www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Reading a structured binary file?

reply "Gary Willoughby" <dev nomad.so> writes:
What library commands do i use to read from a structured binary 
file? I want to read the byte stream 1, 2 maybe 4 bytes at a time 
and cast these to bytes, shorts and ints respectively. I can't 
seem to find anything like readByte().
Aug 02 2013
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 2 August 2013 at 17:49:55 UTC, Gary Willoughby wrote:
 What library commands do i use to read from a structured binary 
 file? I want to read the byte stream 1, 2 maybe 4 bytes at a 
 time and cast these to bytes, shorts and ints respectively. I 
 can't seem to find anything like readByte().
http://dlang.org/phobos/std_file.html#.read http://dlang.org/phobos/std_stdio.html#.File.rawRead ?
Aug 02 2013
prev sibling next sibling parent Justin Whear <justin economicmodeling.com> writes:
On Fri, 02 Aug 2013 19:49:54 +0200, Gary Willoughby wrote:

 What library commands do i use to read from a structured binary file? I
 want to read the byte stream 1, 2 maybe 4 bytes at a time and cast these
 to bytes, shorts and ints respectively. I can't seem to find anything
 like readByte().
You can use File.rawRead: ushort[1] myShort; file.rawRead(myShort); Or if you have structures in the file: struct Foo { align(1): int bar; short k; char[7] str; } Foo[1] foo; file.rawRead(foo);
Aug 02 2013
prev sibling next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Friday, 2 August 2013 at 17:49:55 UTC, Gary Willoughby wrote:
 What library commands do i use to read from a structured binary 
 file? I want to read the byte stream 1, 2 maybe 4 bytes at a 
 time and cast these to bytes, shorts and ints respectively. I 
 can't seem to find anything like readByte().
How big is the file? If it's not too huge i'd just read it in with std.file.read and then sort out splitting it up from there.
Aug 02 2013
parent "Gary Willoughby" <dev nomad.so> writes:
 How big is the file?

 If it's not too huge i'd just read it in with std.file.read and 
 then sort out splitting it up from there.
Quite large so i'll probably stream it. Thanks guys.
Aug 02 2013
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Friday, 2 August 2013 at 17:49:55 UTC, Gary Willoughby wrote:
 What library commands do i use to read from a structured binary 
 file? I want to read the byte stream 1, 2 maybe 4 bytes at a 
 time and cast these to bytes, shorts and ints respectively. I 
 can't seem to find anything like readByte().
You've gotten some help already around functions D provides. But I thought I would mention I'd recently tried to do some large file parsing for binary data, and decided to try and blog about it. http://he-the-great.livejournal.com/47550.html I can't say this is the best solution, but it worked. I was parsing a 20 gig OpenStreetMap planet file.
Aug 02 2013
prev sibling next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, August 02, 2013 19:49:54 Gary Willoughby wrote:
 What library commands do i use to read from a structured binary
 file? I want to read the byte stream 1, 2 maybe 4 bytes at a time
 and cast these to bytes, shorts and ints respectively. I can't
 seem to find anything like readByte().
I'd probably use std.mmfile and std.bitmanip to do it. MmFile will allow you to efficiently operate on the file as a ubyte[] in memory thanks to mmap, and std.bitmanip's peek and read functions make it easy to convert multiple bytes into integral values. - Jonathan M Davis
Aug 02 2013
next sibling parent reply captaindet <2krnk gmx.net> writes:
On 2013-08-02 17:13, Jonathan M Davis wrote:
 On Friday, August 02, 2013 19:49:54 Gary Willoughby wrote:
 What library commands do i use to read from a structured binary
 file? I want to read the byte stream 1, 2 maybe 4 bytes at a time
 and cast these to bytes, shorts and ints respectively. I can't
 seem to find anything like readByte().
I'd probably use std.mmfile and std.bitmanip to do it. MmFile will allow you to efficiently operate on the file as a ubyte[] in memory thanks to mmap, and std.bitmanip's peek and read functions make it easy to convert multiple bytes into integral values. - Jonathan M Davis
FWIW i have to deal with big data files that can be a few GB. for some data analysis software i wrote in C a while back i did some testing with caching and such. turns out that for Win7-64 the automatic caching done by the OS is really good and any attempt to speed things up actually slowed it down. no kidding, i have seen more than 2GB of data being automatically cached. of course the system RAM must be larger than the file size (if i remember my tests correctly by a factor of ~2, but this is maybe not a linear relationship, i did not actually change the RAM just the size of the data file) and it will hold it in the cache only as long as there are no concurrent applications requiring RAM or caching. i guess my point is, if your target is Win7 and your files are >5x smaller than the installed RAM i would not bother at all trying to optimize file access. i suppose -nix machine will do a similar good job these days. /det
Aug 02 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Aug 02, 2013 at 06:38:20PM -0500, captaindet wrote:
[...]
 FWIW
 i have to deal with big data files that can be a few GB. for some data
 analysis software i wrote in C a while back i did some testing with
 caching and such. turns out that for Win7-64 the automatic caching
 done by the OS is really good and any attempt to speed things up
 actually slowed it down. no kidding, i have seen more than 2GB of data
 being automatically cached. of course the system RAM must be larger
 than the file size (if i remember my tests correctly by a factor of
 ~2, but this is maybe not a linear relationship, i did not actually
 change the RAM just the size of the data file) and it will hold it in
 the cache only as long as there are no concurrent applications
 requiring RAM or caching. i guess my point is, if your target is Win7
 and your files are >5x smaller than the installed RAM i would not
 bother at all trying to optimize file access. i suppose -nix machine
 will do a similar good job these days.
[...] IIRC, Linux has been caching files (or disk blocks, rather) in memory since the days of Win95. Of course, memory in those days was much scarcer, but file sizes were smaller too. :) There's still a cost to copy the kernel buffers into userspace, though, which should not be disregarded. But if you use mmap, then you're essentially accessing that memory cache directly, which is as good as it gets. I don't know how well mmap works on windows, though, IIRC it doesn't have the same semantics as Posix, so you could accidentally run into performance issues by using it the wrong way on windows. T -- There is no gravity. The earth sucks.
Aug 02 2013
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 2 August 2013 at 23:51:27 UTC, H. S. Teoh wrote:
 On Fri, Aug 02, 2013 at 06:38:20PM -0500, captaindet wrote:
 [...]
 FWIW
 i have to deal with big data files that can be a few GB. for 
 some data
 analysis software i wrote in C a while back i did some testing 
 with
 caching and such. turns out that for Win7-64 the automatic 
 caching
 done by the OS is really good and any attempt to speed things 
 up
 actually slowed it down. no kidding, i have seen more than 2GB 
 of data
 being automatically cached. of course the system RAM must be 
 larger
 than the file size (if i remember my tests correctly by a 
 factor of
 ~2, but this is maybe not a linear relationship, i did not 
 actually
 change the RAM just the size of the data file) and it will 
 hold it in
 the cache only as long as there are no concurrent applications
 requiring RAM or caching. i guess my point is, if your target 
 is Win7
 and your files are >5x smaller than the installed RAM i would 
 not
 bother at all trying to optimize file access. i suppose -nix 
 machine
 will do a similar good job these days.
[...] IIRC, Linux has been caching files (or disk blocks, rather) in memory since the days of Win95. Of course, memory in those days was much scarcer, but file sizes were smaller too. :) There's still a cost to copy the kernel buffers into userspace, though, which should not be disregarded. But if you use mmap, then you're essentially accessing that memory cache directly, which is as good as it gets. I don't know how well mmap works on windows, though, IIRC it doesn't have the same semantics as Posix, so you could accidentally run into performance issues by using it the wrong way on windows. T
I did some benching a while back with user bioinfornatics. He had to do some pretty large file reads, preferably in very little time. Observations showed my algo was *much* faster under windows then linux. What we observed is that under windows, as soon as you open a file for reading, windows starts buffering the file in a parallel thread. What we did was create two threads. The first did nothing but read the file, store it into chunks of memory, and then pass it to a worker thread. The worker thread did the parsing proper. Doing this *halved* the linux runtime, tying it with the "monothreaded" windows run time. Windows saw no change. FYI, the full thread is here: forum.dlang.org/thread/gmfqwzgtjfnqiajghmsx forum.dlang.org
Aug 03 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Aug 03, 2013 at 11:29:01PM +0200, monarch_dodra wrote:
 On Friday, 2 August 2013 at 23:51:27 UTC, H. S. Teoh wrote:
On Fri, Aug 02, 2013 at 06:38:20PM -0500, captaindet wrote:
[...]
FWIW
i have to deal with big data files that can be a few GB. for some
data analysis software i wrote in C a while back i did some testing
with caching and such. turns out that for Win7-64 the automatic
caching done by the OS is really good and any attempt to speed
things up actually slowed it down. no kidding, i have seen more than
2GB of data being automatically cached. of course the system RAM
must be larger than the file size (if i remember my tests correctly
by a factor of ~2, but this is maybe not a linear relationship, i
did not actually change the RAM just the size of the data file) and
it will hold it in the cache only as long as there are no concurrent
applications requiring RAM or caching. i guess my point is, if your
target is Win7 and your files are >5x smaller than the installed RAM
i would not bother at all trying to optimize file access. i suppose
-nix machine will do a similar good job these days.
[...] IIRC, Linux has been caching files (or disk blocks, rather) in memory since the days of Win95. Of course, memory in those days was much scarcer, but file sizes were smaller too. :) There's still a cost to copy the kernel buffers into userspace, though, which should not be disregarded. But if you use mmap, then you're essentially accessing that memory cache directly, which is as good as it gets. I don't know how well mmap works on windows, though, IIRC it doesn't have the same semantics as Posix, so you could accidentally run into performance issues by using it the wrong way on windows.
[...]
 I did some benching a while back with user bioinfornatics. He had to
 do some pretty large file reads, preferably in very little time.
 Observations showed my algo was *much* faster under windows then
 linux.
Sorry, I lost the context of this discussion, what algo are you referring to?
 What we observed is that under windows, as soon as you open a file
 for reading, windows starts buffering the file in a parallel thread.
 
 What we did was create two threads. The first did nothing but read
 the file, store it into chunks of memory, and then pass it to a
 worker thread. The worker thread did the parsing proper.
 
 Doing this *halved* the linux runtime, tying it with the
 "monothreaded" windows run time. Windows saw no change.
Interesting. I wonder if you could, under Linux, mmap a file then have one thread access the first byte of each file block while another thread does the real work with the data.
 FYI, the full thread is here:
 forum.dlang.org/thread/gmfqwzgtjfnqiajghmsx forum.dlang.org
I'll take a look, thanks. T -- The diminished 7th chord is the most flexible and fear-instilling chord. Use it often, use it unsparingly, to subdue your listeners into submission!
Aug 03 2013
prev sibling parent reply "Gary Willoughby" <dev nomad.so> writes:
On Friday, 2 August 2013 at 22:13:28 UTC, Jonathan M Davis wrote:
 I'd probably use std.mmfile and std.bitmanip to do it. MmFile 
 will allow you to
 efficiently operate on the file as a ubyte[] in memory thanks 
 to mmap, and
 std.bitmanip's peek and read functions make it easy to convert 
 multiple bytes
 into integral values.

 - Jonathan M Davis
This sounds a great idea but once the file has been opened as a MmFile how to i convert this to a ubyte[] so the std.bitmanip functions work with it?
Aug 03 2013
parent reply "Gary Willoughby" <dev nomad.so> writes:
On Saturday, 3 August 2013 at 18:14:47 UTC, Gary Willoughby wrote:
 This sounds a great idea but once the file has been opened as a 
 MmFile how to i convert this to a ubyte[] so the std.bitmanip 
 functions work with it?
I'm currently doing this: auto file = new MmFile("file.dat"); ubyte[] buffer = cast(ubyte[])file[]; buffer.read!uint(); //etc. Is this how you would recommend?
Aug 03 2013
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Saturday, 3 August 2013 at 18:23:58 UTC, Gary Willoughby wrote:
 On Saturday, 3 August 2013 at 18:14:47 UTC, Gary Willoughby 
 wrote:
 This sounds a great idea but once the file has been opened as 
 a MmFile how to i convert this to a ubyte[] so the 
 std.bitmanip functions work with it?
I'm currently doing this: auto file = new MmFile("file.dat"); ubyte[] buffer = cast(ubyte[])file[]; buffer.read!uint(); //etc. Is this how you would recommend?
That defeats the object of memory mapping, as the [] at the end of cast(ubyte[])file[] implies copying the whole file in to memory. 3 options I can think of: 1) copy read from std.bitmanip and modify it to work nicely with MmFile 2) write a wrapper for MmFile to let it work nicely with read 3) rewrite/modify MmFile I would love to do 3) at some point, but I'm too busy at the moment.
Aug 03 2013
next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, August 03, 2013 23:10:12 John Colvin wrote:
 On Saturday, 3 August 2013 at 18:23:58 UTC, Gary Willoughby wrote:
 On Saturday, 3 August 2013 at 18:14:47 UTC, Gary Willoughby
 
 wrote:
 This sounds a great idea but once the file has been opened as
 a MmFile how to i convert this to a ubyte[] so the
 std.bitmanip functions work with it?
I'm currently doing this: auto file = new MmFile("file.dat"); ubyte[] buffer = cast(ubyte[])file[]; buffer.read!uint(); //etc. Is this how you would recommend?
That defeats the object of memory mapping, as the [] at the end of cast(ubyte[])file[] implies copying the whole file in to memory.
Are you sure about that? Maybe I'm just not familiar enough with mmap, but I don't see anything in MmFile which would result in it copying the whole file into memory. I guess that I'll have to do some more reading up on mmap. Certainly, if slicing it like that copies it all into memory, that's a big problem. - Jonathan M Davis
Aug 03 2013
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Aug 03, 2013 at 02:25:23PM -0700, Jonathan M Davis wrote:
 On Saturday, August 03, 2013 23:10:12 John Colvin wrote:
 On Saturday, 3 August 2013 at 18:23:58 UTC, Gary Willoughby wrote:
 On Saturday, 3 August 2013 at 18:14:47 UTC, Gary Willoughby
 
 wrote:
 This sounds a great idea but once the file has been opened as
 a MmFile how to i convert this to a ubyte[] so the
 std.bitmanip functions work with it?
I'm currently doing this: auto file = new MmFile("file.dat"); ubyte[] buffer = cast(ubyte[])file[]; buffer.read!uint(); //etc. Is this how you would recommend?
That defeats the object of memory mapping, as the [] at the end of cast(ubyte[])file[] implies copying the whole file in to memory.
Are you sure about that? Maybe I'm just not familiar enough with mmap, but I don't see anything in MmFile which would result in it copying the whole file into memory. I guess that I'll have to do some more reading up on mmap. Certainly, if slicing it like that copies it all into memory, that's a big problem.
[...] I think he meant that the OS will have to load the entire file into memory if you sliced the mmap'ed file, not that you'll copy all the data. I'm not certain this is true, though, because slicing as I understand it only returns the address of the start of the mmap'ed addresses coupled with its length. I don't think the OS will actually load anything into memory until you reference an address within that mmap'ed range. And even then, only those disk blocks that correspond with the referenced addresses will actually be loaded -- this is the point of virtual memory, after all. T -- The computer is only a tool. Unfortunately, so is the user. -- Armaphine, K5
Aug 03 2013
prev sibling next sibling parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Saturday, 3 August 2013 at 18:23:58 UTC, Gary Willoughby wrote:
 On Saturday, 3 August 2013 at 18:14:47 UTC, Gary Willoughby 
 wrote:
 This sounds a great idea but once the file has been opened as 
 a MmFile how to i convert this to a ubyte[] so the 
 std.bitmanip functions work with it?
I'm currently doing this: auto file = new MmFile("file.dat"); ubyte[] buffer = cast(ubyte[])file[]; buffer.read!uint(); //etc. Is this how you would recommend?
You will need to slice the size of the data you want, otherwise you're effectively doing std.file.read(). It doesn't need to be for a single value (as in the example), it could be a block of data which is then individual parsed for the pieces. auto file = new MmFile("file.dat"); ubyte[] buffer = cast(ubyte[])file[indexInFile..uint.sizeof]; indexInFile += uint.sizeof; buffer.read!uint(); //etc. The only way I'm seeing to advance through the file is to keep an index on where you're currently reading from. This actually works perfect for the FileRange I mentioned in the previous post. Though I'm not familiar with how mmfile manages its memory, but hopefully there isn't buffer reuse or storing the slice could be overridden (not an issue for value data, but string data).
Aug 05 2013
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Aug 06, 2013 at 06:48:12AM +0200, Jesse Phillips wrote:
[...]
 The only way I'm seeing to advance through the file is to keep an
 index on where you're currently reading from. This actually works
 perfect for the FileRange I mentioned in the previous post. Though
 I'm not familiar with how mmfile manages its memory, but hopefully
 there isn't buffer reuse or storing the slice could be overridden
 (not an issue for value data, but string data).
I don't know about D's Mmfile, but AFAIK, it maps directly to the OS mmap(), which basically maps a portion of your program's address space to the data on the disk. Meaning that the memory is managed by the OS, and addresses will not change from under you. In the underlying physical memory, pages may get swapped out and reused, but this is invisible to your program, since referencing them will cause the OS to swap the pages back in, so you'll never end up with invalid pointers. The worst that could happen is the I/O performance hit associated with swapping. Such is the utility of virtual memory. T -- Error: Keyboard not attached. Press F1 to continue. -- Yoon Ha Lee, CONLANG
Aug 05 2013
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, August 05, 2013 23:04:58 H. S. Teoh wrote:
 On Tue, Aug 06, 2013 at 06:48:12AM +0200, Jesse Phillips wrote:
 [...]
 
 The only way I'm seeing to advance through the file is to keep an
 index on where you're currently reading from. This actually works
 perfect for the FileRange I mentioned in the previous post. Though
 I'm not familiar with how mmfile manages its memory, but hopefully
 there isn't buffer reuse or storing the slice could be overridden
 (not an issue for value data, but string data).
I don't know about D's Mmfile, but AFAIK, it maps directly to the OS mmap(), which basically maps a portion of your program's address space to the data on the disk. Meaning that the memory is managed by the OS, and addresses will not change from under you. In the underlying physical memory, pages may get swapped out and reused, but this is invisible to your program, since referencing them will cause the OS to swap the pages back in, so you'll never end up with invalid pointers. The worst that could happen is the I/O performance hit associated with swapping. Such is the utility of virtual memory.
mmap is awesome. It makes handling large files _way_ easier, especially when you have to worry about performance. It was a huge performance boost for one of our video recorder programs where I work when we switched to using mmap on it (this device is recording multiple video streams from cameras 24/7, and performance is critical). Trying to do what mmap does on your own is incredibly bug-prone and bound to be worse for performance (since you're doing it instead of the kernel). One of our older products tries to do it on its own (probably because the developers didn't know about mmap), and it's a royal mess. - Jonathan M Davis
Aug 05 2013
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, August 03, 2013 20:23:55 Gary Willoughby wrote:
 On Saturday, 3 August 2013 at 18:14:47 UTC, Gary Willoughby wrote:
 This sounds a great idea but once the file has been opened as a
 MmFile how to i convert this to a ubyte[] so the std.bitmanip
 functions work with it?
I'm currently doing this: auto file = new MmFile("file.dat"); ubyte[] buffer = cast(ubyte[])file[]; buffer.read!uint(); //etc. Is this how you would recommend?
Yeah. That's how you do it. - Jonathan M Davis
Aug 03 2013
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, August 03, 2013 14:31:16 H. S. Teoh wrote:
 On Sat, Aug 03, 2013 at 02:25:23PM -0700, Jonathan M Davis wrote:
 On Saturday, August 03, 2013 23:10:12 John Colvin wrote:
 On Saturday, 3 August 2013 at 18:23:58 UTC, Gary Willoughby wrote:
 On Saturday, 3 August 2013 at 18:14:47 UTC, Gary Willoughby
 
 wrote:
 This sounds a great idea but once the file has been opened as
 a MmFile how to i convert this to a ubyte[] so the
 std.bitmanip functions work with it?
I'm currently doing this: auto file = new MmFile("file.dat"); ubyte[] buffer = cast(ubyte[])file[]; buffer.read!uint(); //etc. Is this how you would recommend?
That defeats the object of memory mapping, as the [] at the end of cast(ubyte[])file[] implies copying the whole file in to memory.
Are you sure about that? Maybe I'm just not familiar enough with mmap, but I don't see anything in MmFile which would result in it copying the whole file into memory. I guess that I'll have to do some more reading up on mmap. Certainly, if slicing it like that copies it all into memory, that's a big problem.
[...] I think he meant that the OS will have to load the entire file into memory if you sliced the mmap'ed file, not that you'll copy all the data. I'm not certain this is true, though, because slicing as I understand it only returns the address of the start of the mmap'ed addresses coupled with its length. I don't think the OS will actually load anything into memory until you reference an address within that mmap'ed range. And even then, only those disk blocks that correspond with the referenced addresses will actually be loaded -- this is the point of virtual memory, after all.
That's what I thought that mmap did, but it's not something that I've studied in detail. Aside from that though, my main complaint about MmFile is the fact that it's a class when it really should be a reference-counted struct. At some point, we should probably create MMFile or somesuch which _is_ a reference counted struct and then deprecate MmFile. But if we do that, then we should be sure of whatever other changes the implementation needs and do those with it. - Jonathan M Davis
Aug 03 2013