www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - What is best way to read and interpret binary files?

reply welkam <wwwelkam gmail.com> writes:
So my question is in subject/title. I want to parse binary file 
into D structs and cant really find any good way of doing it. 
What I try to do now is something like this

byte[4] fake_integer;
auto fd = File("binary.data", "r");
fd.rawRead(fake_integer);
int real_integer = *(cast(int*)  fake_integer.ptr);

What I ideally want is to have some kind of c style array and 
just cast it into struct or take existing struct and populate 
fields one by one with data from file. Is there a D way of doing 
it or should I call core.stdc.stdio functions instead?
Nov 19 2018
parent reply Neia Neutuladh <neia ikeran.org> writes:
On Mon, 19 Nov 2018 21:30:36 +0000, welkam wrote:
 So my question is in subject/title. I want to parse binary file into D
 structs and cant really find any good way of doing it. What I try to do
 now is something like this
 
 byte[4] fake_integer;
 auto fd = File("binary.data", "r");
 fd.rawRead(fake_integer);
 int real_integer = *(cast(int*)  fake_integer.ptr);
 
 What I ideally want is to have some kind of c style array and just cast
 it into struct or take existing struct and populate fields one by one
 with data from file. Is there a D way of doing it or should I call
 core.stdc.stdio functions instead?
Nothing stops you from writing: SomeStruct myStruct; fd.rawRead((cast(ubyte*)&myStruct)[0..SomeStruct.sizeof]); Standard caveats about byte order and alignment.
Nov 19 2018
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Nov 19, 2018 at 10:14:25PM +0000, Neia Neutuladh via
Digitalmars-d-learn wrote:
 On Mon, 19 Nov 2018 21:30:36 +0000, welkam wrote:
 So my question is in subject/title. I want to parse binary file into D
 structs and cant really find any good way of doing it. What I try to do
 now is something like this
 
 byte[4] fake_integer;
 auto fd = File("binary.data", "r");
 fd.rawRead(fake_integer);
 int real_integer = *(cast(int*)  fake_integer.ptr);
 
 What I ideally want is to have some kind of c style array and just cast
 it into struct or take existing struct and populate fields one by one
 with data from file. Is there a D way of doing it or should I call
 core.stdc.stdio functions instead?
Nothing stops you from writing: SomeStruct myStruct; fd.rawRead((cast(ubyte*)&myStruct)[0..SomeStruct.sizeof]);
Actually, the case is unnecessary, because arrays implicitly convert to void[], and pointers are sliceable. So all you need is: SomeStruct myStruct; fd.rawRead((&myStruct)[0 .. 1]); This works for all POD types. Writing the struct out to file is the same thing: SomeStruct myStruct; fd.rawWrite((&myStruct)[0 .. 1]); with the nice symmetry that you just have to rename rawRead to rawWrite. For arrays: SomeStruct[] arr; fd.rawWrite(arr); ... arr.length = ... /* expected length */ fd.rawRead(arr); To correctly store length information, you'll have to manually write out array lengths as well, and read it before reading the array. Should be straightforward to figure out.
 Standard caveats about byte order and alignment.
Alignment shouldn't be a problem, since local variables should already be properly aligned. Endianness, however, will be a problem if you intend to transport this data to/from a different platform / hardware. You'll need to manually fix the endianness yourself. T -- This is not a sentence.
Nov 19 2018
next sibling parent Neia Neutuladh <neia ikeran.org> writes:
On Mon, 19 Nov 2018 14:32:55 -0800, H. S. Teoh wrote:
 Standard caveats about byte order and alignment.
Alignment shouldn't be a problem, since local variables should already be properly aligned.
Right, and the IO layer probably doesn't need to read to aligned memory anyway. Struct fields, however, need to have the same relative alignment as the file.
Nov 19 2018
prev sibling parent reply mw <mingwu gmail.com> writes:
On Monday, 19 November 2018 at 22:32:55 UTC, H. S. Teoh wrote:
 Actually, the case is unnecessary, because arrays implicitly 
 convert to void[], and pointers are sliceable.  So all you need 
 is:

 	SomeStruct myStruct;
 	fd.rawRead((&myStruct)[0 .. 1]);

 This works for all POD types.

 Writing the struct out to file is the same thing:

 	SomeStruct myStruct;
 	fd.rawWrite((&myStruct)[0 .. 1]);
This works, but I'm just wondering why we do not just add more functions to the library: rawRead(ref T t), and rawWrite(ref T t) to read & write single value.
 For arrays:

 	SomeStruct[] arr;
 	fd.rawWrite(arr);
 	...

 	arr.length = ... /* expected length */
 	fd.rawRead(arr);
Currently, the library only have this two functions for arrays.
Mar 29
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Mar 30, 2021 at 12:32:36AM +0000, mw via Digitalmars-d-learn wrote:
 On Monday, 19 November 2018 at 22:32:55 UTC, H. S. Teoh wrote:
 Actually, the case is unnecessary, because arrays implicitly convert
 to void[], and pointers are sliceable.  So all you need is:
 
 	SomeStruct myStruct;
 	fd.rawRead((&myStruct)[0 .. 1]);
 
 This works for all POD types.
 
 Writing the struct out to file is the same thing:
 
 	SomeStruct myStruct;
 	fd.rawWrite((&myStruct)[0 .. 1]);
This works, but I'm just wondering why we do not just add more functions to the library: rawRead(ref T t), and rawWrite(ref T t) to read & write single value.
If you wish, submit a PR for this. It's not hard to write your own overloads for it, though: void rawWrite(File f, ref T t) trusted { f.rawWrite((cast(ubyte*) &t)[0 .. T.sizeof]); } // ditto for rawRead T -- A linguistics professor was lecturing to his class one day. "In English," he said, "A double negative forms a positive. In some languages, though, such as Russian, a double negative is still a negative. However, there is no language wherein a double positive can form a negative." A voice from the back of the room piped up, "Yeah, yeah."
Mar 29
prev sibling parent reply welkam <wwwelkam gmail.com> writes:
On Monday, 19 November 2018 at 22:14:25 UTC, Neia Neutuladh wrote:
 Nothing stops you from writing:

     SomeStruct myStruct;
     fd.rawRead((cast(ubyte*)&myStruct)[0..SomeStruct.sizeof]);

 Standard caveats about byte order and alignment.
Never would I thought about casting struct to static array. If I understood correctly you cast myStruct pointer to ubyte pointer and then construct static array on stack with tmpArray.ptr = (ubyte pointer) and tmpArray.sizeof = SomeStruct.sizeof What I figured out when I woke up is that I never needed c style arrays. What I could do is to allocate enough data for all file in ubyte array and just use slices to read data by chunks and cast them into necessary structs. Thanks Neia Neutuladh and H. S. Teoh for giving me some pointers https://www.explainxkcd.com/wiki/index.php/138:_Pointers
Nov 20 2018
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 20 November 2018 at 11:54:59 UTC, welkam wrote:
 On Monday, 19 November 2018 at 22:14:25 UTC, Neia Neutuladh 
 wrote:
 Nothing stops you from writing:

     SomeStruct myStruct;
     fd.rawRead((cast(ubyte*)&myStruct)[0..SomeStruct.sizeof]);

 Standard caveats about byte order and alignment.
Never would I thought about casting struct to static array. If I understood correctly you cast myStruct pointer to ubyte pointer and then construct static array on stack with tmpArray.ptr = (ubyte pointer) and tmpArray.sizeof = SomeStruct.sizeof
Almost correct, except it's not a static array, it's just a slice, i.e. ubyte[].
Nov 20 2018
parent welkam <wwwelkam gmail.com> writes:
On Tuesday, 20 November 2018 at 12:01:49 UTC, Stanislav Blinov 
wrote:
 On Tuesday, 20 November 2018 at 11:54:59 UTC, welkam wrote:
 On Monday, 19 November 2018 at 22:14:25 UTC, Neia Neutuladh 
 wrote:
 Nothing stops you from writing:

     SomeStruct myStruct;
     fd.rawRead((cast(ubyte*)&myStruct)[0..SomeStruct.sizeof]);

 Standard caveats about byte order and alignment.
Never would I thought about casting struct to static array. If I understood correctly you cast myStruct pointer to ubyte pointer and then construct static array on stack with tmpArray.ptr = (ubyte pointer) and tmpArray.sizeof = SomeStruct.sizeof
Almost correct, except it's not a static array, it's just a slice, i.e. ubyte[].
I guess it came from inseparability with C where you want to slice C arrays? Thats useful to know
Nov 20 2018