www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Converting a ubyte[] to a struct with respect to endianness?

reply Felix <sdfsdfs dfads.com> writes:
I'm trying to read in just the first part of a .png file to peek 
at it's width and height without loading in the whole file. I'm 
using FreeImage for reading the whole file but since it doesn't 
have a function to let me peek at the image size before loading 
it all in I'm rolling my own.

I've gotten as a far as reading in the first 16 bytes which 
includes an 8 byte signature, then the length and type of the 
first chunk:

struct Header {
     ubyte[8] signature;
     uint length;
     char[4] type;
}
union Un {
     Header h;
     ubyte[16] u;
}

auto f = File(path, "r");
foreach (ubyte[] buffer; f.byChunk(16)){
     Un fff;
     fff.u = buffer[0..16];
     writeln(fff.h);
     break;
}

It prints out:
Header([137, 80, 78, 71, 13, 10, 26, 10], 218103808, "IHDR")

The signature is correct, the type is correct, but the value I 
get for length should be 13, not 218103808. So I'm guessing my 
ubytes are in the wrong order in the uint... how should I put 
them around the correct way so that my code won't break on 
another machine with different endianness?
Jun 23
next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 06/23/2017 07:52 PM, Felix wrote:

 So I'm guessing my ubytes are in the
 wrong order in the uint... how should I put them around the correct way
 so that my code won't break on another machine with different endianness?
Yes, that would happen when your system is little-endian. (According to spec, the length field is big-endian.) You can detect what endianness the system has and swap the bytes of the length field: http://ddili.org/ders/d.en/union.html#ix_union.endian,%20std.system import std.system; import core.bitop; // ... if (endian == Endian.littleEndian) { address.value = bswap(address.value); } std.bitmanip may be useful as well: https://dlang.org/phobos/std_bitmanip.html Ali
Jun 23
parent reply Felix <sdfsdfs dfads.com> writes:
That works, thanks!
Jun 23
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 06/23/2017 09:26 PM, Felix wrote:
 That works, thanks!
I've just tried this, which seems cleaner: import std.stdio; import std.system; import std.bitmanip; void ensureBigEndian(T)(ref T value) { if (endian == Endian.littleEndian) { value = *cast(T*)nativeToBigEndian(value).ptr; } } void main() { ubyte[] bytes = [ 0, 0, 0, 13 ]; uint u = *cast(uint*)bytes.ptr; writefln("Just read: %s", u); u.ensureBigEndian; writefln("Converted: %s", u); } Just read: 218103808 Converted: 13 Ali
Jun 23
parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Fri, Jun 23, 2017 at 10:10:22PM -0700, Ali Çehreli via Digitalmars-d-learn
wrote:
 On 06/23/2017 09:26 PM, Felix wrote:
 That works, thanks!
I've just tried this, which seems cleaner: import std.stdio; import std.system; import std.bitmanip; void ensureBigEndian(T)(ref T value) { if (endian == Endian.littleEndian) { value = *cast(T*)nativeToBigEndian(value).ptr; }
This is wrong, you should be detecting the endianness of the input and use {big,little}EndianToNative instead. For example, if the input is big endian, you should use bigEndianToNative. Internally, if native is already big endian, it will do nothing; otherwise it will swap the endianness. This way you don't have to check the current machine's endianness yourself; you can just recompile on a machine of different endianness and it will Just Work. T -- People who are more than casually interested in computers should have at least some idea of what the underlying hardware is like. Otherwise the programs they write will be pretty weird. -- D. Knuth
Jun 23
next sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 06/23/2017 10:18 PM, H. S. Teoh via Digitalmars-d-learn wrote:
 On Fri, Jun 23, 2017 at 10:10:22PM -0700, Ali Çehreli via Digitalmars-d-learn
wrote:
 On 06/23/2017 09:26 PM, Felix wrote:
 That works, thanks!
I've just tried this, which seems cleaner: import std.stdio; import std.system; import std.bitmanip; void ensureBigEndian(T)(ref T value) { if (endian == Endian.littleEndian) { value = *cast(T*)nativeToBigEndian(value).ptr; }
This is wrong, you should be detecting the endianness of the input and use {big,little}EndianToNative instead. For example, if the input is big endian, you should use bigEndianToNative. Internally, if native is already big endian, it will do nothing; otherwise it will swap the endianness. This way you don't have to check the current machine's endianness yourself; you can just recompile on a machine of different endianness and it will Just Work. T
Thanks. Something like this: import std.stdio; import std.system; import std.bitmanip; void ensureCorrectFromBigEndian(T)(ref T value) { value = bigEndianToNative!(T, T.sizeof)(*cast(ubyte[T.sizeof]*)&value); } struct S { uint u; } void main() { // Bytes on the wire ubyte[] bytes = [ 0, 0, 0, 13 ]; // Overlaying an object on those bytes S s = *cast(S*)bytes.ptr; void checkValue(uint expectedOnLE, uint expectedOnBE) { if (endian == Endian.littleEndian) { assert(s.u == expectedOnLE); } else if (endian == Endian.bigEndian) { assert(s.u == expectedOnBE); } else { assert(false, "What is this?"); } } // The value should be wrong no a little-endian system checkValue(218103808, 13); s.u.ensureCorrectFromBigEndian; // No matter what, now the result will be correct on any system checkValue(13, 13); } Ali
Jun 23
prev sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/24/17 1:18 AM, H. S. Teoh via Digitalmars-d-learn wrote:
 On Fri, Jun 23, 2017 at 10:10:22PM -0700, Ali Çehreli via Digitalmars-d-learn
wrote:
 On 06/23/2017 09:26 PM, Felix wrote:
 That works, thanks!
I've just tried this, which seems cleaner: import std.stdio; import std.system; import std.bitmanip; void ensureBigEndian(T)(ref T value) { if (endian == Endian.littleEndian) { value = *cast(T*)nativeToBigEndian(value).ptr; }
This is wrong, you should be detecting the endianness of the input and use {big,little}EndianToNative instead. For example, if the input is big endian, you should use bigEndianToNative. Internally, if native is already big endian, it will do nothing; otherwise it will swap the endianness. This way you don't have to check the current machine's endianness yourself; you can just recompile on a machine of different endianness and it will Just Work.
I would also point out that there are pre-defined versions for endianness: version(BigEndian) and version(LittleEndian). This makes static checking a lot easier when you are writing your own code that deals with endiannness. For byte-swapping one field, I would use the standard tools as H.S. has advised. -Steve
Jun 24
prev sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 24 June 2017 at 02:52:23 UTC, Felix wrote:
 I'm trying to read in just the first part of a .png file to 
 peek at it's width and height without loading in the whole 
 file. I'm using FreeImage for reading the whole file but since 
 it doesn't have a function to let me peek at the image size 
 before loading it all in I'm rolling my own.

 I've gotten as a far as reading in the first 16 bytes which 
 includes an 8 byte signature, then the length and type of the 
 first chunk:

 struct Header {
     ubyte[8] signature;
     uint length;
     char[4] type;
 }
 union Un {
     Header h;
     ubyte[16] u;
 }

 auto f = File(path, "r");
 foreach (ubyte[] buffer; f.byChunk(16)){
     Un fff;
     fff.u = buffer[0..16];
     writeln(fff.h);
     break;
 }

 It prints out:
 Header([137, 80, 78, 71, 13, 10, 26, 10], 218103808, "IHDR")

 The signature is correct, the type is correct, but the value I 
 get for length should be 13, not 218103808. So I'm guessing my 
 ubytes are in the wrong order in the uint... how should I put 
 them around the correct way so that my code won't break on 
 another machine with different endianness?
I would advise a look at sqlite.d https://github.com/UplinkCoder/sqlite-d which solves the same problem albeit for sqlite database files and for png. But it provides the means of simply putting a BigEndian!uint into your struct and have conversion on stuff be automatically handled.
Jun 24