www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Converting a ubyte[] to a struct with respect to endianness?

reply Felix <sdfsdfs dfads.com> writes:
I'm trying to read in just the first part of a .png file to peek 
at it's width and height without loading in the whole file. I'm 
using FreeImage for reading the whole file but since it doesn't 
have a function to let me peek at the image size before loading 
it all in I'm rolling my own.

I've gotten as a far as reading in the first 16 bytes which 
includes an 8 byte signature, then the length and type of the 
first chunk:

struct Header {
     ubyte[8] signature;
     uint length;
     char[4] type;
}
union Un {
     Header h;
     ubyte[16] u;
}

auto f = File(path, "r");
foreach (ubyte[] buffer; f.byChunk(16)){
     Un fff;
     fff.u = buffer[0..16];
     writeln(fff.h);
     break;
}

It prints out:
Header([137, 80, 78, 71, 13, 10, 26, 10], 218103808, "IHDR")

The signature is correct, the type is correct, but the value I 
get for length should be 13, not 218103808. So I'm guessing my 
ubytes are in the wrong order in the uint... how should I put 
them around the correct way so that my code won't break on 
another machine with different endianness?
Jun 23 2017
next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 06/23/2017 07:52 PM, Felix wrote:


 So I'm guessing my ubytes are in the
 wrong order in the uint... how should I put them around the correct way
 so that my code won't break on another machine with different endianness?


Yes, that would happen when your system is little-endian. (According to 
spec, the length field is big-endian.)

You can detect what endianness the system has and swap the bytes of the 
length field:

   http://ddili.org/ders/d.en/union.html#ix_union.endian,%20std.system

import std.system;
import core.bitop;

// ...

     if (endian == Endian.littleEndian) {
         address.value = bswap(address.value);
     }

std.bitmanip may be useful as well:

   https://dlang.org/phobos/std_bitmanip.html

Ali
Jun 23 2017
parent reply Felix <sdfsdfs dfads.com> writes:
That works, thanks!
Jun 23 2017
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 06/23/2017 09:26 PM, Felix wrote:

 That works, thanks!


I've just tried this, which seems cleaner:

import std.stdio;
import std.system;
import std.bitmanip;

void ensureBigEndian(T)(ref T value) {
     if (endian == Endian.littleEndian) {
         value = *cast(T*)nativeToBigEndian(value).ptr;
     }
}

void main() {
     ubyte[] bytes = [ 0, 0, 0, 13 ];
     uint u = *cast(uint*)bytes.ptr;

     writefln("Just read: %s", u);
     u.ensureBigEndian;
     writefln("Converted: %s", u);
}

Just read: 218103808
Converted: 13

Ali
Jun 23 2017
parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Fri, Jun 23, 2017 at 10:10:22PM -0700, Ali Çehreli via Digitalmars-d-learn
wrote:

 On 06/23/2017 09:26 PM, Felix wrote:

 That works, thanks!


 
 I've just tried this, which seems cleaner:
 
 import std.stdio;
 import std.system;
 import std.bitmanip;
 
 void ensureBigEndian(T)(ref T value) {
     if (endian == Endian.littleEndian) {
         value = *cast(T*)nativeToBigEndian(value).ptr;
     }


This is wrong, you should be detecting the endianness of the input and
use {big,little}EndianToNative instead.  For example, if the input is
big endian, you should use bigEndianToNative.  Internally, if native is
already big endian, it will do nothing; otherwise it will swap the
endianness. This way you don't have to check the current machine's
endianness yourself; you can just recompile on a machine of different
endianness and it will Just Work.


T

-- 
People who are more than casually interested in computers should have at least
some idea of what the underlying hardware is like. Otherwise the programs they
write will be pretty weird. -- D. Knuth
Jun 23 2017
next sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 06/23/2017 10:18 PM, H. S. Teoh via Digitalmars-d-learn wrote:

 On Fri, Jun 23, 2017 at 10:10:22PM -0700, Ali Çehreli via Digitalmars-d-learn
wrote:

 On 06/23/2017 09:26 PM, Felix wrote:

 That works, thanks!


 I've just tried this, which seems cleaner:

 import std.stdio;
 import std.system;
 import std.bitmanip;

 void ensureBigEndian(T)(ref T value) {
     if (endian == Endian.littleEndian) {
         value = *cast(T*)nativeToBigEndian(value).ptr;
     }


 This is wrong, you should be detecting the endianness of the input and
 use {big,little}EndianToNative instead.  For example, if the input is
 big endian, you should use bigEndianToNative.  Internally, if native is
 already big endian, it will do nothing; otherwise it will swap the
 endianness. This way you don't have to check the current machine's
 endianness yourself; you can just recompile on a machine of different
 endianness and it will Just Work.


 T


Thanks. Something like this:

import std.stdio;
import std.system;
import std.bitmanip;

void ensureCorrectFromBigEndian(T)(ref T value) {
     value = bigEndianToNative!(T, T.sizeof)(*cast(ubyte[T.sizeof]*)&value);
}

struct S {
     uint u;
}

void main() {
     // Bytes on the wire
     ubyte[] bytes = [ 0, 0, 0, 13 ];

     // Overlaying an object on those bytes
     S s = *cast(S*)bytes.ptr;

     void checkValue(uint expectedOnLE, uint expectedOnBE) {
         if (endian == Endian.littleEndian) {
             assert(s.u == expectedOnLE);
         } else if (endian == Endian.bigEndian) {
             assert(s.u == expectedOnBE);
         } else {
             assert(false, "What is this?");
         }
     }

     // The value should be wrong no a little-endian system
     checkValue(218103808, 13);

     s.u.ensureCorrectFromBigEndian;

     // No matter what, now the result will be correct on any system
     checkValue(13, 13);
}

Ali
Jun 23 2017
prev sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/24/17 1:18 AM, H. S. Teoh via Digitalmars-d-learn wrote:

 On Fri, Jun 23, 2017 at 10:10:22PM -0700, Ali Çehreli via Digitalmars-d-learn
wrote:

 On 06/23/2017 09:26 PM, Felix wrote:

 That works, thanks!


 I've just tried this, which seems cleaner:

 import std.stdio;
 import std.system;
 import std.bitmanip;

 void ensureBigEndian(T)(ref T value) {
      if (endian == Endian.littleEndian) {
          value = *cast(T*)nativeToBigEndian(value).ptr;
      }


 
 This is wrong, you should be detecting the endianness of the input and
 use {big,little}EndianToNative instead.  For example, if the input is
 big endian, you should use bigEndianToNative.  Internally, if native is
 already big endian, it will do nothing; otherwise it will swap the
 endianness. This way you don't have to check the current machine's
 endianness yourself; you can just recompile on a machine of different
 endianness and it will Just Work.


I would also point out that there are pre-defined versions for 
endianness: version(BigEndian) and version(LittleEndian). This makes 
static checking a lot easier when you are writing your own code that 
deals with endiannness.

For byte-swapping one field, I would use the standard tools as H.S. has 
advised.

-Steve
Jun 24 2017
prev sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 24 June 2017 at 02:52:23 UTC, Felix wrote:

 I'm trying to read in just the first part of a .png file to 
 peek at it's width and height without loading in the whole 
 file. I'm using FreeImage for reading the whole file but since 
 it doesn't have a function to let me peek at the image size 
 before loading it all in I'm rolling my own.

 I've gotten as a far as reading in the first 16 bytes which 
 includes an 8 byte signature, then the length and type of the 
 first chunk:

 struct Header {
     ubyte[8] signature;
     uint length;
     char[4] type;
 }
 union Un {
     Header h;
     ubyte[16] u;
 }

 auto f = File(path, "r");
 foreach (ubyte[] buffer; f.byChunk(16)){
     Un fff;
     fff.u = buffer[0..16];
     writeln(fff.h);
     break;
 }

 It prints out:
 Header([137, 80, 78, 71, 13, 10, 26, 10], 218103808, "IHDR")

 The signature is correct, the type is correct, but the value I 
 get for length should be 13, not 218103808. So I'm guessing my 
 ubytes are in the wrong order in the uint... how should I put 
 them around the correct way so that my code won't break on 
 another machine with different endianness?


I would advise a look at sqlite.d 
https://github.com/UplinkCoder/sqlite-d
which solves the same problem albeit for sqlite database files 
and for png.
But it provides the means of simply putting a BigEndian!uint into 
your struct and have conversion on stuff be automatically handled.
Jun 24 2017