• Charles Hixson via Digitalmars-d-learn (34/34) Mar 15 2015 I've got, say, a file header in a routine that looks like:
• ketmar (18/18) Mar 15 2015 On Sun, 15 Mar 2015 16:34:14 -0700, Charles Hixson via Digitalmars-d-lea...
• Charles Hixson via Digitalmars-d-learn (11/29) Mar 16 2015 The original method had no knowledge of the layout that the using module...
• Charles Hixson via Digitalmars-d-learn (14/50) Mar 16 2015 My current best answer is to turn the "unused" into a vector of bytes,
• ketmar (16/29) Mar 16 2015 if you passing the array, not a slice, it can't be resized. i.e.:
• Charles Hixson via Digitalmars-d-learn (61/88) Mar 16 2015 Yes, but if I pass an array, the using program can't change it's own
• "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> (27/27) Mar 16 2015 The problem in your example is that your making a copy of the
• Charles Hixson via Digitalmars-d-learn (2/28) Mar 16 2015 Nice! That looks like exactly what I was hoping for.
• ketmar (30/30) Mar 16 2015 On Mon, 16 Mar 2015 12:49:40 -0700, Charles Hixson via Digitalmars-d-lea...

Charles Hixson via Digitalmars-d-learn writes:

I've got, say, a file header in a routine that looks like:
struct    BlockHead
{  uint    magic    =    20150312;        //block magic
     uint    magic2;                                //magic for use by 
the using routine
     uint    blockSize;
     uint    unused1;
     ulong    unused2;
     ulong    spare[61];
}

The "unused" vars are intended to be reserved for future use.  The 
"spare" vars are intended for use by another routine that uses the 
routines of the owner of BlockHead to deal with basic things.  So it 
needs to define the data in spare.

How?
The only almost nice way I've thought of is with a struct pointer to 
spare, and calling that nice is a statement about how bad the other ways 
I've thought of are.  (E.g., redefining spare as a ubyte vector and then 
casting the struct of the using program as a ubyte vector and copying it 
over.)

I'm sure that there must be a better way, but I can't think of what it 
is.  BlockHead needs to be a struct to allow the data to be written with 
a writeBlock.  I don't want to use a buffered file because this needs 
random access.  Perhaps I just shouldn't define spare, but then I 
wouldn't retrieve the data with the readBlock.  So I'd double the number 
of disk accesses (to the BlockHead).

One alternative is to have each using routine define it's own header 
struct, and to cast each BlockHead to
the appropriate routine's own header type, but that begs for errors to 
creep in if any of the structs isn't the right length, of ever becomes 
modified to not be the correct length.

Every way I've thought of looks like something that's just begging for 
errors to creep into the code, if not now then later...so *is* there a 
good way?

ketmar <ketmar ketmar.no-ip.org> writes:

On Sun, 15 Mar 2015 16:34:14 -0700, Charles Hixson via Digitalmars-d-learn
wrote:

if you know the exact layouts of `spare`, you can use union for that:

struct S {
  // ...
  union {
    ulong[61] spare;
    struct { int vala; ubyte valb; }
    // etc.
  }
}

and then you can use it like this:

  auto s =3D S();
  s.vala =3D 42;
  assert(s.spare[0] =3D=3D 42);

you can also write a CTFE function that builds struct to cast where=20
`spare` is changed to something else, using `S` as a source, but this=20
solution will not be pretty. ;-)=

Charles Hixson via Digitalmars-d-learn writes:

On 03/15/2015 04:51 PM, ketmar via Digitalmars-d-learn wrote:
 On Sun, 15 Mar 2015 16:34:14 -0700, Charles Hixson via Digitalmars-d-learn
 wrote:

 if you know the exact layouts of `spare`, you can use union for that:

 struct S {
    // ...
    union {
      ulong[61] spare;
      struct { int vala; ubyte valb; }
      // etc.
    }
 }

 and then you can use it like this:

    auto s = S();
    s.vala = 42;
    assert(s.spare[0] == 42);

 you can also write a CTFE function that builds struct to cast where
 `spare` is changed to something else, using `S` as a source, but this
 solution will not be pretty. ;-)

The original method had no knowledge of the layout that the using module 
will need, merely that it will need to store some information.  (In this 
particular case I *could* just modify the header in the original file, 
but that isn't solving the design problem, and means that working code 
needs to be modified to accommodate new code in a different file.)

IOW, if I could write the union, I wouldn't have needed to label the 
unused header memory "spare".  That solution *would* allow for multiple 
secondary users, with different union values, but it would again mean 
that new code would require the modifying of the file containing 
existing code.  This is normally considered bad practice.

Charles Hixson via Digitalmars-d-learn writes:

On 03/16/2015 09:16 AM, Charles Hixson via Digitalmars-d-learn wrote:
 On 03/15/2015 04:51 PM, ketmar via Digitalmars-d-learn wrote:
 On Sun, 15 Mar 2015 16:34:14 -0700, Charles Hixson via 
 Digitalmars-d-learn
 wrote:

 if you know the exact layouts of `spare`, you can use union for that:

 struct S {
    // ...
    union {
      ulong[61] spare;
      struct { int vala; ubyte valb; }
      // etc.
    }
 }

 and then you can use it like this:

    auto s = S();
    s.vala = 42;
    assert(s.spare[0] == 42);

 you can also write a CTFE function that builds struct to cast where
 `spare` is changed to something else, using `S` as a source, but this
 solution will not be pretty. ;-)

 The original method had no knowledge of the layout that the using 
 module will need, merely that it will need to store some information.  
 (In this particular case I *could* just modify the header in the 
 original file, but that isn't solving the design problem, and means 
 that working code needs to be modified to accommodate new code in a 
 different file.)

 IOW, if I could write the union, I wouldn't have needed to label the 
 unused header memory "spare".  That solution *would* allow for 
 multiple secondary users, with different union values, but it would 
 again mean that new code would require the modifying of the file 
 containing existing code.  This is normally considered bad practice.

My current best answer is to turn the "unused" into a vector of bytes, 
and then pass that to the using routines as a ref.  This still is wide 
open to errors at the calling end, but the BlockHead end can ensure that 
the length of the vector remains constant whenever it accesses it (well, 
whenever the head is being written to disk.  The byte vector would be a 
static array at the BlockHead end.  This would seem to allow the using 
end to cast the byte vector into an appropriate struct for its own use, 
and that writes to it would be seen by BlockHead as writes to the 
header...but only to restricted parts of the header.  The problem here 
is that the ref array might allow itself to be resized at the user end.  
That's not a problem as long as the resizing is to something smaller, 
but I'm not sure what happens if it gets resized to something larger.  
It looks like allowing a buffer overflow.

ketmar <ketmar ketmar.no-ip.org> writes:

On Mon, 16 Mar 2015 11:18:16 -0700, Charles Hixson via Digitalmars-d-learn
wrote:

 My current best answer is to turn the "unused" into a vector of bytes,
 and then pass that to the using routines as a ref.  This still is wide
 open to errors at the calling end, but the BlockHead end can ensure that
 the length of the vector remains constant whenever it accesses it (well,
 whenever the head is being written to disk.  The byte vector would be a
 static array at the BlockHead end.  This would seem to allow the using
 end to cast the byte vector into an appropriate struct for its own use,
 and that writes to it would be seen by BlockHead as writes to the
 header...but only to restricted parts of the header.  The problem here
 is that the ref array might allow itself to be resized at the user end.
 That's not a problem as long as the resizing is to something smaller,
 but I'm not sure what happens if it gets resized to something larger. It
 looks like allowing a buffer overflow.

if you passing the array, not a slice, it can't be resized. i.e.:

  void foo (ref ubyte[8] a) {
    a[2] =3D 42;
    //a.length =3D 6; // this will not compile: constant a.length is not an=
=20
lvalue
  }

  void main () {
    ubyte[8] aa;
    assert(aa[2] =3D=3D 0);
    foo(aa);
    assert(aa[2] =3D=3D 42);
  }
=

Charles Hixson via Digitalmars-d-learn writes:

On 03/16/2015 11:55 AM, ketmar via Digitalmars-d-learn wrote:
 On Mon, 16 Mar 2015 11:18:16 -0700, Charles Hixson via Digitalmars-d-learn
 wrote:

 My current best answer is to turn the "unused" into a vector of bytes,
 and then pass that to the using routines as a ref.  This still is wide
 open to errors at the calling end, but the BlockHead end can ensure that
 the length of the vector remains constant whenever it accesses it (well,
 whenever the head is being written to disk.  The byte vector would be a
 static array at the BlockHead end.  This would seem to allow the using
 end to cast the byte vector into an appropriate struct for its own use,
 and that writes to it would be seen by BlockHead as writes to the
 header...but only to restricted parts of the header.  The problem here
 is that the ref array might allow itself to be resized at the user end.
 That's not a problem as long as the resizing is to something smaller,
 but I'm not sure what happens if it gets resized to something larger. It
 looks like allowing a buffer overflow.

 if you passing the array, not a slice, it can't be resized. i.e.:

    void foo (ref ubyte[8] a) {
      a[2] = 42;
      //a.length = 6; // this will not compile: constant a.length is not an
 lvalue
    }

    void main () {
      ubyte[8] aa;
      assert(aa[2] == 0);
      foo(aa);
      assert(aa[2] == 42);
    }

Yes, but if I pass an array, the using program can't change it's own 
values (which need to be stored in the header).  Additionally, if I pass 
an array I'm passing over 400 bytes rather than around 16, but that's 
very secondary.
So, attempting to rewrite your example into more what I am talking about:
import    std.stdio;

struct    tstHead
{
     ubyte    data[400];

     ref    ubyte[400]    value    ()
     {    return    data;    }

     void    write(int i)
     {    writefln ("%d", data[i]);    }
}

void    main()
{    tstHead    tst;
     auto    val    =    tst.value;
     val[23]    =    23;
     writefln("%d", val[23]);
     tst.write(23);
     tst.write(0);
}
Yields:
23
0
0
instead of:
23
23
0

And:import    std.stdio;

struct    tstHead
{
     ubyte    data[400];

     void    value    (ref byte[400] val)
     {    val.ptr    =    data;    }

     void    write(int i)
     {    writefln ("%d", data[i]);    }
}

void    main()
{    tstHead    tst;
     ubyte    val[400];
     tst.value (val);
     val[23]    =    23;
     writefln("%d", val[23]);
     tst.write(23);
     tst.write(0);
}
Yields:
test.d(8): Error: val.ptr is not an lvalue
test.d(17): Error: function test.tstHead.value (ref byte[400] val) is 
not callable using argument types (ubyte[400])
Failed: ["dmd", "-unittest", "-Dddocs", "-v", "-o-", "test.d", "-I."]

It is *necessary* that the using routine be able to store its data into 
the parts of the header reserved for it.  Perhaps multiple copying will 
be necessary.  I could write getter and setter routines, but they will 
inevitably be doing a lot of excess copying as the base routine doesn't 
know what data the calling routine will need.  Every solution I've 
thought of either requires a lot of excess copying, or requires a very 
tight coupling between the "library" routine and the using routine.

"Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:

The problem in your example is that your making a copy of the 
returned data. Of course any changes to that copy won't affect 
the original. You need to return a pointer to it (`ref` won't do 
if you want to store it in a local variable, because these can't 
be `ref`).

struct BlockHead
{
     uint magic = 20150312;
     uint magic2;
     uint blockSize;
     uint unused1;
     ulong unused2;
     ulong spare[61];

     T* embedded(T)() {
         static assert(T.sizeof < spare.length);
         return cast(T*) spare.ptr;
     }
}

How to use it:

void useIt(ref BlockHead bh) {
     static struct InternalData {
         int foo;
         ubyte[20] bar;
     }

     auto data = bh.embedded!InternalData;
     data.foo = 42;
}

Charles Hixson via Digitalmars-d-learn writes:

On 03/16/2015 01:24 PM, via Digitalmars-d-learn wrote:
 The problem in your example is that your making a copy of the returned 
 data. Of course any changes to that copy won't affect the original. 
 You need to return a pointer to it (`ref` won't do if you want to 
 store it in a local variable, because these can't be `ref`).

 struct BlockHead
 {
     uint magic = 20150312;
     uint magic2;
     uint blockSize;
     uint unused1;
     ulong unused2;
     ulong spare[61];

     T* embedded(T)() {
         static assert(T.sizeof < spare.length);
         return cast(T*) spare.ptr;
     }
 }

 How to use it:

 void useIt(ref BlockHead bh) {
     static struct InternalData {
         int foo;
         ubyte[20] bar;
     }

     auto data = bh.embedded!InternalData;
     data.foo = 42;
 }

Nice!  That looks like exactly what I was hoping for.

ketmar <ketmar ketmar.no-ip.org> writes:

On Mon, 16 Mar 2015 12:49:40 -0700, Charles Hixson via Digitalmars-d-learn
wrote:

yep, you're doing it wrong, as Marc already wrote. ;-)

it's not very obvious, but this line makes a copy:

  auto val =3D tst.value;

there are no `ref` type in D. `ref` is modifier for function arguments,=20
but not part of the type, and you can't declare `ref int`, for example.=20
so that line transforms to this:

  ubyte[400] val =3D tst.value;

which, obviously, does copying. yet this will work as expected:

  import iv.writer;

  struct tstHead {
    ubyte[400] data;

    ubyte[] value () { return data; }
    void write (int i) { writeln(data[i]); }
  }


  void test (ref ubyte[400] a) {
    a[2] =3D 42;
  }


  void main () {
    tstHead tst;
    auto val =3D tst.value;
    val[23] =3D 23;
    writeln(val[23]);
    tst.write(23);
    tst.write(0);
    test(val[0..400]);
    tst.write(2);
  }

=