digitalmars.D.learn - buffer to struct type conversion...TArrayStream?
- Charles Hixson via Digitalmars-d-learn (38/38) Mar 19 2015 I've read a chunk of data into a buffer and want to convert it into a
- ketmar (6/6) Mar 19 2015 On Thu, 19 Mar 2015 10:47:05 -0700, Charles Hixson via Digitalmars-d-lea...
- Charles Hixson via Digitalmars-d-learn (5/10) Mar 19 2015 Whee! Thanks, I don't think I *ever* would have thought of that. I got
- "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> (34/60) Mar 19 2015 Please use D-style array declarations, the syntax you're using is
- =?UTF-8?B?QWxpIMOHZWhyZWxp?= (7/9) Mar 19 2015 I fin the following compiler switches useful:
- "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> (5/23) Mar 19 2015 Urgh... it seems `peek()` and `read()` only work with numerical
- Charles Hixson via Digitalmars-d-learn (26/51) Mar 19 2015 Note: Thanks for the prior comment about where to place the array
- Kagamin (2/2) Mar 20 2015 If you don't want to run into alignment issues, you shouldn't
I've read a chunk of data into a buffer and want to convert it into a struct. The reading routine is in a class that doesn't know about the struct, but the size should be exactly the same. (I.e., I want to use the converse procedure to write it.) Is there a better way to do this than using TArrayStream? The idea here is to have a file of fixed length records (rather like Fortran binary records, except that there's a header record which is a different size and which specifies various things about the file). The class (well, struct) that handles the fixed length records only knows what the record size is, and a couple of other quite general things. The class which uses it holds each record in a fixed length struct with no indirections. So I thought I could just cast the buffer to the struct...but this doesn't work. Every straightforward way I've tried of doing it yields: Error: cannot cast from Node_ to ubyte[] or something reasonably analogous. The current version of the (non-working) code is: ubyte buf[]; auto len = btFile.read(nodeId, buf); assert (len == n.self.sizeof); n.self = to!(Node.Node_)(buf); // TODO write the code which yields the error: Error: template instance std.conv.to!(Node_).to!(ubyte[]) error instantiating Node_ is (approximately, I've renamed aliased values to their base value): struct Node_ { ulong idvalue; ulong keyvalue; int eLen; Entry e[23]; } and Entry is (approximately): struct Entry { ulong key; ulong d; ulong d2; }
Mar 19 2015
On Thu, 19 Mar 2015 10:47:05 -0700, Charles Hixson via Digitalmars-d-learn wrote: turn it 90 degrees. ;-) auto cvt =3D cast(Node_*)buf.ptr; n =3D cvt[0]; =
Mar 19 2015
On 03/19/2015 11:18 AM, ketmar via Digitalmars-d-learn wrote:On Thu, 19 Mar 2015 10:47:05 -0700, Charles Hixson via Digitalmars-d-learn wrote: turn it 90 degrees. ;-) auto cvt = cast(Node_*)buf.ptr; n = cvt[0];Whee! Thanks, I don't think I *ever* would have thought of that. I got as far as getting a pointer to buf, but then couldn't figure out how to turn the Node_* into a Node. (Obvious now that you showed me, and I even think I've seen that trick done before.)
Mar 19 2015
On Thursday, 19 March 2015 at 17:48:00 UTC, Charles Hixson wrote:I've read a chunk of data into a buffer and want to convert it into a struct. The reading routine is in a class that doesn't know about the struct, but the size should be exactly the same. (I.e., I want to use the converse procedure to write it.) Is there a better way to do this than using TArrayStream? The idea here is to have a file of fixed length records (rather like Fortran binary records, except that there's a header record which is a different size and which specifies various things about the file). The class (well, struct) that handles the fixed length records only knows what the record size is, and a couple of other quite general things. The class which uses it holds each record in a fixed length struct with no indirections. So I thought I could just cast the buffer to the struct...but this doesn't work. Every straightforward way I've tried of doing it yields: Error: cannot cast from Node_ to ubyte[] or something reasonably analogous. The current version of the (non-working) code is: ubyte buf[];Please use D-style array declarations, the syntax you're using is deprecated: ubyte[] buf;auto len = btFile.read(nodeId, buf); assert (len == n.self.sizeof); n.self = to!(Node.Node_)(buf); // TODO write the code which yields the error: Error: template instance std.conv.to!(Node_).to!(ubyte[]) error instantiatingIf you want to reinterpret a byte array as another type, there are three ways: 1) Using a cast (unsafe): n.self = *cast(Node.Node_*)buf.ptr; 2) With a union: union { ubyte[Node.Node_.sizeof] asBytes; Node.Node_ asNode; } buf; auto len = btFile.read(nodeId, buf.asBytes[]); assert(len == n.self.sizeof); n.self = buf.asNode 3) Using std.bitmap.peek(), which also supports conversion between big- and little-endian: import std.bitmap; n.self = buf.peek!(Node.Node_, Endian.bigEndian); (The examples are untested, it's possible you'll need to make some adjustments to make them work.) You should probably use option 3). It is safe, because it checks that the buffer has the right size, and it also allows you to specify the endian-ness (many file formats have a standardized endian-ness). While you're at it, you can also try std.bitmanip.read(). It can be applied to any range, so you can probably do something like this (also untested): auto input = btFile.byChunk(4096).joiner; while(!input.empty) { auto node = input.read!(Node.Node_, Endian.bigEndian); // use `node` }
Mar 19 2015
On 03/19/2015 11:42 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:Please use D-style array declarations, the syntax you're using is deprecated:I fin the following compiler switches useful: -de show use of deprecated features as errors (halt compilation) -w warnings as errors (compilation will halt) Ali
Mar 19 2015
On Thursday, 19 March 2015 at 18:42:03 UTC, Marc Schütz wrote:3) Using std.bitmap.peek(), which also supports conversion between big- and little-endian: import std.bitmap; n.self = buf.peek!(Node.Node_, Endian.bigEndian); (The examples are untested, it's possible you'll need to make some adjustments to make them work.) You should probably use option 3). It is safe, because it checks that the buffer has the right size, and it also allows you to specify the endian-ness (many file formats have a standardized endian-ness). While you're at it, you can also try std.bitmanip.read(). It can be applied to any range, so you can probably do something like this (also untested): auto input = btFile.byChunk(4096).joiner; while(!input.empty) { auto node = input.read!(Node.Node_, Endian.bigEndian); // use `node` }Urgh... it seems `peek()` and `read()` only work with numerical types :-( Is this intentional? It would be quite useful for arbitrary types, IMO, even if care must be taken with regard to pointers.
Mar 19 2015
On 03/19/2015 12:05 PM, via Digitalmars-d-learn wrote:On Thursday, 19 March 2015 at 18:42:03 UTC, Marc Schütz wrote:Note: Thanks for the prior comment about where to place the array markers. I keep forgetting. Umnf...I don't plan on needing conversion between Endian versions, but I guess that's important. But more significant if it only handles numeric types I wonder about handling arrays of structs, even if those structs *are* composed entirely of numerical types. Being secure against a future need to handle Endian variations would be good, but not enough to both increase the complexity that way and deal with possible unexpected errors. P.S.: Looking at the std.bitmanip documentation causes me to feel that the restriction to integral types was intentional, though admittedly it doesn't seem to be stated explicitly, but all of the examples seem to be simple arrays. I'll grant that this could be just to keep things simple. And the wording of the docs causes me to think it would probably only work for integral values, as in not even floats. This is reasonable if you are thinking of it as a set of routines for bit manipulation. At all events, I think that it involves excessive overhead (in code verbosity) and that I'ld likely need to use a loop to read the array within the struct. A simple bit copy is much more straightforwards (I was already checking that the read was the correct length, though I haven't decided what to do if that ever fails. Currently it's an assert statement, but this clearly needs to be changed to either an enforce statement or to a thrown exception...but I haven't yet thought of a valid case where the block would be an incorrect length.3) Using std.bitmap.peek(), which also supports conversion between big- and little-endian: import std.bitmap; n.self = buf.peek!(Node.Node_, Endian.bigEndian); (The examples are untested, it's possible you'll need to make some adjustments to make them work.) You should probably use option 3). It is safe, because it checks that the buffer has the right size, and it also allows you to specify the endian-ness (many file formats have a standardized endian-ness). While you're at it, you can also try std.bitmanip.read(). It can be applied to any range, so you can probably do something like this (also untested): auto input = btFile.byChunk(4096).joiner; while(!input.empty) { auto node = input.read!(Node.Node_, Endian.bigEndian); // use `node` }Urgh... it seems `peek()` and `read()` only work with numerical types :-( Is this intentional? It would be quite useful for arbitrary types, IMO, even if care must be taken with regard to pointers.
Mar 19 2015
If you don't want to run into alignment issues, you shouldn't cast the buffer directly: http://dpaste.dzfl.pl/b522f911871a
Mar 20 2015