Random Access Files in .NET
Random Access Files in .NET
Random Access Files in .NET
Random Access Files in .NET
Random Access Files in .NET
Random Access Files in .NET Random Access Files in .NET Random Access Files in .NET Random Access Files in .NET Random Access Files in .NET Random Access Files in .NET Random Access Files in .NET Random Access Files in .NET
Random Access Files in .NET Random Access Files in .NET
Random Access Files in .NET
Go Back  Xtreme Visual Basic Talk > > > Random Access Files in .NET


Reply
 
Thread Tools Display Modes
  #1  
Old 11-02-2009, 11:53 PM
Roger_Wgnr's Avatar
Roger_Wgnr Roger_Wgnr is offline
CodeASaurus Hex

Forum Leader
* Expert *
 
Join Date: Jul 2006
Location: San Antonio TX
Posts: 2,427
Default Random Access Files in .NET


Ok guys and gals.

I am looking for the answer on how to read and write random access records in .NET.
Now I know random access files are dead and the response on every query I found with google/bing/yahoo searches was don't use random access use Binary serialization but I can not. I have a situation where I need to read and modify 256 byte records in a file that is used by another application.

Now for a bit of background the file is made up of 27 fixed length records of 256 bytes. Each of the records has a separate structure consisting of 30-120 fields.
The proplem I am having is I can create the record structures but when using the FileGet method I am receiving an error. Now I have found samples that showed the same format I am using that claim to work.

The error I am getting is
Option Strict On disallows narrowing from type 'System.ValueType' to type 'ProgramName.ModuleName.Record' in copying the value of 'ByRef' paramater 'Value' back to the matching argument.

The basic layout (simplified) is as follows:
Code:
Structure Record
    <VBFixedString(6)> Public SomeParam as String
    Public SomeNumber as Short
    Public AnotherNumber as Short
    .... and so on
End Structure

Public Function ReadFile() as Record
    Dim ARecord as New Record
    Dim Fhnd as Integer
    Fhnd = FreeFile()
    FileOpen(Fhnd, "TheFile", OpenMode.Random, , , 256)
    FileGet(Fhnd, ARecord, 1)
    FileClose(Fhnd)
    Return ARecord
End Function
Now the overloads for FileGet show all of the data types including Object. But I have tried various methods to retreive the 256 byte record but without success. Do I have to actually read each of the several hundred fields of the structure one by one?
Or am i overlooking something (I have been known for this in the past)

Or (and I'll kick myself if this is the only way) do I just have to set option strict Off in the module and the code as above will actually work.

I would love to be able to just ditch the random access file but I have to keep it for compatibility with several other applications at this time.

Looking forward to finding out about this.
__________________
Code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. ~Martin Golding
The user is a peripheral that types when you issue a read request. ~Peter Williams
MSDN Visual Basic .NET General FAQ
Reply With Quote
  #2  
Old 11-03-2009, 08:20 AM
AtmaWeapon's Avatar
AtmaWeaponRandom Access Files in .NET AtmaWeapon is offline
Fabulous Florist

Forum Leader
* Guru *
 
Join Date: Feb 2004
Location: Austin, TX
Posts: 9,500
Default

Ack! Don't use FileGet() or any other VB6 File I/O methods. You've got a pretty rich System.IO namespace that can handle all of your needs.

Random Access I/O doesn't have the easiest API to work with in .NET, but it's because it's an advanced use case and most of the design in .NET APIs is spent on the "90%" cases. Anyway, that's off-topic.

Like any other I/O, you start with Stream. If you are reading from a file, it's probably better to start with FileStream. There's a few members available for random access here. Position indicates where the "cursor" is in the file. You can set it to a particular location, or use the Seek() method if you just want to offset without having to do the math yourself. Read() and ReadByte() are available to get raw data from the stream.

By itself, Stream has all you need to do random access I/O. However, it's not the most convenient interface. For example, here's how you read an Integer safely:
  • Check to ensure that there are at least 4 bytes left in the file.
  • Read 4 bytes into a Byte array.
  • Convert the bytes into an Integer

That's tedious, and you'll have to write something similar for each type. So, to make things more convenient, you wrap this stream with a BinaryReader. BinaryReader has a lot of convenience methods like ReadInt32() that handle the details of reading different data types from a stream for you. Unfortunately, BinaryReader's interface doesn't provide random access ability directly; again it's considered the advanced case. However, the BaseStream property provides access to the Stream that is being wrapped, so you can use that in conjunction.

Rockoon and I once had a loooong private discussion about whether this API is good or not. I see his point, but the .NET designers felt like random access I/O is an advanced case and thus didn't polish the API for it. You take what you can get. Keep in mind that methods like BinaryReader.ReadInt32() will throw exceptions if the end of the stream is reached. There's no EndOfStream like on StreamReader() because there's no way for BinaryReader to know how many bytes you might want to read next. In the end, if you don't want to use exceptions for normal control flow (and you don't!) then you'll have to write code to verify there's enough bytes left in the stream before reading.

I'm preparing an example in case you need it, but wanted to get the information to you in case you could connect the dots without it.

*edit*
Looking back, it looks like you want something as easy as "read this struct from the file using hand-wavy code". Unfortunately, .NET's answer to that is serialization. I'm going to fool around with FileGet() a little bit and see if I can get it working; this seems like a scenario where it might actually add value. The example will have both if I get FileGet() to work and I'll let you decide which works.

*edit 2*
You might want to have a look at this post. I still don't like the FileGet() method, and structs have some weird design implications in .NET. In particular, it's advised to avoid mutable structs, which is what you have.

*edit 3*
On my machine, the VB compiler warns that FilePut() and FileGet() are deprecated in favor of FilePutObject() and FileGetObject(). Have you tried them?
__________________
.NET Resources
My FAQ threads | Tutor's Corner | Code Library
I would bet money 2/3 of .NET questions are already answered in one of these three places.

Last edited by AtmaWeapon; 11-03-2009 at 09:14 AM.
Reply With Quote
  #3  
Old 11-03-2009, 09:53 AM
Rockoon's Avatar
Rockoon Rockoon is offline
Joseph Koss

* Guru *
 
Join Date: Aug 2003
Location: Unfashionable End
Posts: 3,615
Default

The thing to keep in mind about attributes such as VBFixedString and VBFixedArray is that they are metadata sugar that doesnt actually enforce a reality upon the data that is in memory.

A virgin Record (as defined in the first post) isnt going to have a 6-character SomeParam just because VBFixedString(6) is present. In fact, you will throw an object reference not set if you access SomeParam without assigning a string to it first. Also if you set SomeParam = "x" then its length in vb.net land will be 1, not 6.

This metadata is consumed by, among other things, the file i/o functions. So in theory you should be able to do more or less trivial "works like Random" i/o, although I have not actually tried. Option Strict would definitely cause issues, but I suspect casting to ValueType (as atma's link recommends) works just fine. Not as simple as it once was, but not as verbose as reinventing the wheel for every structure in your legacy data.

Edit: On second thought, I'm not so sure that casting to ValueType actually works "trivially" for wouldn't the metadata then be unfindable with reflection? The metadata isnt actually a payload carried by the in-memory structure, and certainly isnt attached to ValueType...

Last edited by Rockoon; 11-03-2009 at 10:05 AM.
Reply With Quote
  #4  
Old 11-03-2009, 09:53 AM
AtmaWeapon's Avatar
AtmaWeaponRandom Access Files in .NET AtmaWeapon is offline
Fabulous Florist

Forum Leader
* Guru *
 
Join Date: Feb 2004
Location: Austin, TX
Posts: 9,500
Default

*edit* This post is somewhat mistaken; FileGet() has some extra magic that makes the post we talked about work. However, since I can't create a file that can be read by it I can't demonstrate it.

Now I'm going to *strongly* recommend that you don't use FilePut() or FileGet(). Here's my justification.

I started writing an example and noticed that the documentation recommends FilePutObject() and FileGetObject(). When I finished my implementation, FilePutObject() kept throwing an ArgumentException, complaining "Unhandled Exception: System.ArgumentException: 'FilePutObject' of structure 'DataStructure' is not valid." I figured I got my record length wrong, and tried a few modifications based on some documentation I hadn't seen, but nothing seemed to be working. So I fired up Reflector to see what it took to get the exception I was getting.

Here's FilePutObject():
Code:
Public Shared Sub FilePutObject(ByVal FileNumber As Integer, ByVal Value As Object, ByVal Optional RecordNumber As Long = -1)
    Try 
        FileSystem.ValidateGetPutRecordNumber(RecordNumber)
        FileSystem.GetStream(Assembly.GetCallingAssembly, FileNumber, (OpenModeTypes.Binary Or OpenModeTypes.Random)).PutObject(Value, RecordNumber, True)
    Catch exception As Exception
        Throw exception
    End Try
End Sub
ValidateGetPutRecordNumber() isn't throwing an exception, because my number is valid the exception message I get doesn't match the one it builds. So what about GetStream()? It only throws exceptions if the file number is invalid or if we picked a bad file mode. That leaves PutObject:
Code:
Friend Overridable Sub PutObject(ByVal Value As Object, ByVal Optional RecordNumber As Long = 0, ByVal Optional ContainedInVariant As Boolean = True)
    Throw ExceptionUtils.VbMakeException(&H36)
End Sub
Oh dear. Now I can speculate on what's happening. FilePutObject() is a convenience member for when you may not know the type of value. Internally, it chooses the internal PutObject() overload that will be called (it looks like maybe there's some compiler magic involved.) When it can't find an overload that makes sense, it calls this overload, which always throws an exception. Strangely enough, this is still not the exception message I get, so this is pure speculation, but I think it's clear how I came to the conclusion.

Basically, it looks like FilePut() and FileGet() are not supported for custom types in .NET, even if you use the FilePutObject() and FileGetObject() methods. It's a shame because it looked fairly elegant, but now you're left with serialization or writing it yourself.
Reply With Quote
  #5  
Old 11-03-2009, 10:24 AM
Rockoon's Avatar
Rockoon Rockoon is offline
Joseph Koss

* Guru *
 
Join Date: Aug 2003
Location: Unfashionable End
Posts: 3,615
Default

What if you dont use Random mode? Binary should be fine as long as the programmer knows the length of the records.

Edit:

This seems to create the expected file, with Option Strict On:

Code:
    Public Structure foo
        <VBFixedString(6)> Public SomeParam As String
        Public x As Integer
    End Structure

        Dim bar As New foo
        bar.SomeParam = "woot" ' this is 4 characters, but 6 are written to disk.
        bar.x = &H30405060       ' in ascii this is "`P@0"

        Dim outfile As Integer = FreeFile()
        FileOpen(outfile, "TheFile", OpenMode.Binary)
        FilePut(outfile, bar, 1)
        FilePut(outfile, bar, 11)
        FilePut(outfile, bar, 21)
        FileClose(outfile)
and reading seems fine too:

Code:
        Dim infile As Integer = FreeFile()

        Dim vbar As ValueType = CType(New foo, ValueType)
        FileOpen(infile, "TheFile", OpenMode.Binary)
        FileGet(infile, vbar, 1)
        FileClose(infile)
        bar = CType(vbar, foo)
The file loaded in my text editor is:
Code:
woot  `P@0woot  `P@0woot  `P@0
I'm not sure if the two chars after the woot's are ascii-0's or something else that isnt printable tho (I believe VB6 would have made them ascii-0's)

More edited to add:

FileGet and FilePut use 1-based byte indexes... sadly
Also, after reading, SomeParam is in fact a 6-character string as you would have expected. I'm not sure how this is accomplished considering that ValueType doesnt have any attributes on it.

Last edited by Rockoon; 11-03-2009 at 10:57 AM.
Reply With Quote
  #6  
Old 11-03-2009, 11:04 AM
Rockoon's Avatar
Rockoon Rockoon is offline
Joseph Koss

* Guru *
 
Join Date: Aug 2003
Location: Unfashionable End
Posts: 3,615
Default

I suppose, given that the above works, that its possible to make a generic record reader/writer object for structures ..

Len(New Foo) returns the expected length (10) even tho that string isnt even initialized yet, so some magic is happening there as well.
Reply With Quote
  #7  
Old 11-03-2009, 11:55 AM
AtmaWeapon's Avatar
AtmaWeaponRandom Access Files in .NET AtmaWeapon is offline
Fabulous Florist

Forum Leader
* Guru *
 
Join Date: Feb 2004
Location: Austin, TX
Posts: 9,500
Default

Interesting, I'll do the compatibility method then. It's definitely less code, but I'm going to berate it and generally not like it when I'm done
Reply With Quote
  #8  
Old 11-03-2009, 12:12 PM
Rockoon's Avatar
Rockoon Rockoon is offline
Joseph Koss

* Guru *
 
Join Date: Aug 2003
Location: Unfashionable End
Posts: 3,615
Default

Quote:
Originally Posted by AtmaWeapon View Post
Interesting, I'll do the compatibility method then. It's definitely less code, but I'm going to berate it and generally not like it when I'm done
One mans compatibility method is another mans standard library function
Reply With Quote
  #9  
Old 11-03-2009, 12:23 PM
AtmaWeapon's Avatar
AtmaWeaponRandom Access Files in .NET AtmaWeapon is offline
Fabulous Florist

Forum Leader
* Guru *
 
Join Date: Feb 2004
Location: Austin, TX
Posts: 9,500
Default

I'm still having problems with it to be honest. I can read the first record, but attempting to read record #2 reports that I tried to read past the end of the stream. I presume it's because I'm not mucking about with trying to calculate the size of the structure and letting VB try to figure it out?
Reply With Quote
  #10  
Old 11-03-2009, 12:43 PM
Rockoon's Avatar
Rockoon Rockoon is offline
Joseph Koss

* Guru *
 
Join Date: Aug 2003
Location: Unfashionable End
Posts: 3,615
Default

Got some code to work off of? (is this in Random mode, or Binary?)

here is what I had last cooked up, and its output:

Code:
Option Strict On

Module Module1

    Public Structure myStruct
        <VBFixedString(6)> Public s As String
        Public x As Integer

        Public Sub New(ByVal s As String, ByVal x As Integer)
            Me.s = s
            Me.x = x
        End Sub

        Public Overrides Function ToString() As String
            Return "s = <" & s & "> Length = " & s.Length & " x = " & x
        End Function

    End Structure

    Sub Main()

        ' create a file of 3 myStruct's

        Dim alpha As New myStruct("alpha", 0)
        Dim beta As New myStruct("beta", 1)
        Dim gamma As New myStruct("gamma", 2)

        Dim outfile As Integer = FreeFile()
        FileOpen(outfile, "Test.Txt", OpenMode.Binary)
        FilePut(outfile, alpha)
        FilePut(outfile, beta)
        FilePut(outfile, gamma)
        FileClose(outfile)

        ' read file in backwards, displaying it

        Dim infile As Integer = FreeFile()

        For i As Integer = 2 To 0 Step -1
            Dim temp As ValueType = CType(New myStruct, ValueType)
            FileOpen(infile, "Test.Txt", OpenMode.Binary)
            FileGet(infile, temp, 1 + i * Len(temp))
            FileClose(infile)
            Console.WriteLine(CType(temp, myStruct).ToString)
        Next

        Console.ReadKey()

    End Sub

End Module

and the output:

Code:
s = <gamma > Length = 6 x = 2
s = <beta  > Length = 6 x = 1
s = <alpha > Length = 6 x = 0
So I was successfully able to index around the file without precognitive knowledge of the structures size.
Reply With Quote
  #11  
Old 11-03-2009, 02:23 PM
AtmaWeapon's Avatar
AtmaWeaponRandom Access Files in .NET AtmaWeapon is offline
Fabulous Florist

Forum Leader
* Guru *
 
Join Date: Feb 2004
Location: Austin, TX
Posts: 9,500
Default

I got it working; I was still using OpenMode.Random in the reader. *** is that even for if you have to use OpenMode.Binary for reading/writing structs? Expect an angry treatise on why I hate "kitchen sink" methods. Good frameworks are made of small, focused methods.
Reply With Quote
  #12  
Old 11-03-2009, 03:45 PM
Rockoon's Avatar
Rockoon Rockoon is offline
Joseph Koss

* Guru *
 
Join Date: Aug 2003
Location: Unfashionable End
Posts: 3,615
Default

I don't know if you have to use Binary, but Binary is more flexible. The only advantage to Random seems to be that you dont have to calculate byte offsets yourself, which also means that you *cant* calculate byte offsets yourself. Not the sort of tradeoff I like to make.

Some more food for thought:

I was thinking that VB.NET Structure's probably have some padding rules, so the attributes LayoutKind.Explicit and FieldOffset are probably necessary in a surprising number of cases to read arbitrary structures written out by other languages (may even be the source of some of your Random woes?)
Reply With Quote
  #13  
Old 11-04-2009, 04:59 AM
Rockoon's Avatar
Rockoon Rockoon is offline
Joseph Koss

* Guru *
 
Join Date: Aug 2003
Location: Unfashionable End
Posts: 3,615
Default

Trying to make a generic getter function, I ran into a stumbling block and learned about type constraints on generics...


Code:
    Public Function GetRecord(Of T As Structure)(ByVal filename As String, ByVal recordnum As Integer) As T
        Dim infile As Integer = FreeFile()
        Dim temp As ValueType = CType(New T, ValueType)

        FileOpen(infile, filename, OpenMode.Binary)
        FileGet(infile, temp, 1 + recordnum * Len(temp))
        FileClose(infile)

        Return CType(temp, T)
    End Function

    Public Sub PutRecord(Of T As Structure)(ByVal filename As String, ByVal recordnum As Integer, ByVal record As T)
        Dim outfile As Integer = FreeFile()
        FileOpen(outfile, filename, OpenMode.Binary)
        FilePut(outfile, record, 1 + recordnum * Len(record))
        FileClose(outfile)
    End Sub
Record numbers are 0-based in my implementation, and of course this is all entirely inefficient (open, read/write, close methodology) .. but with a little more love (error handling, input validation) these will surely become part-and-parcel of my standard toolbox.
Reply With Quote
  #14  
Old 11-04-2009, 06:05 PM
Roger_Wgnr's Avatar
Roger_Wgnr Roger_Wgnr is offline
CodeASaurus Hex

Forum Leader
* Expert *
 
Join Date: Jul 2006
Location: San Antonio TX
Posts: 2,427
Default

Thanks For all the input on this issue. I did not get a chance to work on this yesterday as they pulled me for a Priority Fix.
Your Direction allowed me to know what I was doing wrong and actually OpenMode.Random does work.
Without actually listing the Structure here is the basic method I used. It only required a small change and I verified it was working by reading all the structures and writing to a new file.
I then did a Binary Diff of the two files and they were identical.

Code:
Public Function ReadRecord(recordNo as Integer) As DataStructure
        Dim _FileData As ValueType = DirectCast(New DataStructure, ValueType)
        Dim _filehndl As Integer = FreeFile()

        FileOpen(_filehndl, theDataFile, OpenMode.Random, , , recordSize)
        FileGet(_filehndl, _FileData, recordNo)
        FileClose(_filehndl)

        Return DirectCast(_FileData, DataStructure)
End Function

Public Sub WriteRecord(fileData as DataStructure, recordNo as integer)
        Dim _filehndl As Integer = FreeFile()

        FileOpen(_filehndl, theDataFile, OpenMode.Random, , , recordSize)
        FilePut(_filehndl, fileData, recordNo)
        FileClose(_filehndl)
End Sub
I knew I was overlooking something because with Option Strict Off (shudder)
This code worked
Code:
Public Function ReadRecord(recordNo as Integer) As DataStructure
        Dim _FileData As New DataStructure
        Dim _filehndl As Integer = FreeFile()
        FileOpen(_filehndl, theDataFile, OpenMode.Random, , , recordSize)
        FileGet(_filehndl, _FileData, recordNo)
        FileClose(_filehndl)
        Return _FileData
End Function
Again Thanks for the Help and great input (I have added the examples to my code bank as well).

Roger
__________________
Code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. ~Martin Golding
The user is a peripheral that types when you issue a read request. ~Peter Williams
MSDN Visual Basic .NET General FAQ
Reply With Quote
  #15  
Old 11-04-2009, 09:05 PM
Rockoon's Avatar
Rockoon Rockoon is offline
Joseph Koss

* Guru *
 
Join Date: Aug 2003
Location: Unfashionable End
Posts: 3,615
Default

Indeed. VB.NET with Strict Off is much worse than VB6's implicit type conversions.

At least in VB6 there is never more than a single conversion step, but in VB.NET a single innocuous statement such as 'a = b' might trigger a chain reaction of many type conversions, and there might be more than one string of conversions which 'make sense' so good luck predicting which the compiler actually picks for you.
Reply With Quote
Reply

Tags
file i/o, random access


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Advertisement:





Free Publications
The ASP.NET 2.0 Anthology
101 Essential Tips, Tricks & Hacks - Free 156 Page Preview. Learn the most practical features and best approaches for ASP.NET.
subscribe
Programmers Heaven C# School Book -Free 338 Page eBook
The Programmers Heaven C# School book covers the .NET framework and the C# language.
subscribe
Build Your Own ASP.NET 3.5 Web Site Using C# & VB, 3rd Edition - Free 219 Page Preview!
This comprehensive step-by-step guide will help get your database-driven ASP.NET web site up and running in no time..
subscribe
Random Access Files in .NET
Random Access Files in .NET
Random Access Files in .NET Random Access Files in .NET
Random Access Files in .NET
Random Access Files in .NET
Random Access Files in .NET Random Access Files in .NET Random Access Files in .NET Random Access Files in .NET Random Access Files in .NET Random Access Files in .NET Random Access Files in .NET
Random Access Files in .NET
Random Access Files in .NET
 
Random Access Files in .NET
Random Access Files in .NET
 
-->