www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Why must butfields sum to a multiple of a byte?

reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
It says for std.bitmanip.bitfields:

"The sum of all bit lengths in one bitfield instantiation must be
exactly 8, 16, 32, or 64."
It has a static assert to verify this. But why does this limitation exist?

I was thinking of cases where I'm wrapping a C++ POD struct which has
bitfields and is packed on a 1-byte boundary. GCC example:

#include <stdio.h>
#pragma pack(1)
struct Foo
{
    unsigned int bits1 : 1;
    unsigned int bits2 : 2;
};

int main()
{
    printf("%d\n", sizeof(Foo));
    return 0;
}

I can't translate this to D using the bitfields mixin since it
requires a multiple of a byte, 3 bits isn't a byte:

import std.stdio;
import std.bitmanip;

align(1) struct Foo
{
    mixin(bitfields!(
        uint, "bits1", 1,
        uint, "bits2", 2
    ));
}

void main() { writeln(Foo.sizeof); }

std\bitmanip.d(151): Error: static assert  "Field widths must sum to
8, 16, 32, or 64"

Is this a language limitation? I guess because 'byte' is the smallest
legal type in D this is what the bitfields mixin uses internally?
Jul 30 2012
next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 30 July 2012 at 14:19:56 UTC, Andrej Mitrovic wrote:
 It says for std.bitmanip.bitfields:

 "The sum of all bit lengths in one bitfield instantiation must 
 be
 exactly 8, 16, 32, or 64."
 It has a static assert to verify this. But why does this 
 limitation exist?

 I was thinking of cases where I'm wrapping a C++ POD struct 
 which has
 bitfields and is packed on a 1-byte boundary. GCC example:

 #include <stdio.h>
 #pragma pack(1)
 struct Foo
 {
     unsigned int bits1 : 1;
     unsigned int bits2 : 2;
 };

 int main()
 {
     printf("%d\n", sizeof(Foo));
     return 0;
 }

 I can't translate this to D using the bitfields mixin since it
 requires a multiple of a byte, 3 bits isn't a byte:

 import std.stdio;
 import std.bitmanip;

 align(1) struct Foo
 {
     mixin(bitfields!(
         uint, "bits1", 1,
         uint, "bits2", 2
     ));
 }

 void main() { writeln(Foo.sizeof); }

 std\bitmanip.d(151): Error: static assert  "Field widths must 
 sum to
 8, 16, 32, or 64"

 Is this a language limitation? I guess because 'byte' is the 
 smallest
 legal type in D this is what the bitfields mixin uses 
 internally?

---- align(1) struct Foo { mixin(bitfields!( uint, "bits1", 1, uint, "bits2", 2, uint, "", 5 )); } ---- The rationale (I *think*) Is that bitfields wants to know how big it should be: While YOU know you want to put it into a "Foo" object that is 1 byte big, bitfields has no idea. Either that, or having the developer provide this by hand made the implementation easier enough to justify the burden? Even then, I think it would have been more convenient to specify the total size as a first argument, rather than manually padding at the end. Either way, nothing world shattering for me :D
Jul 30 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/30/12, monarch_dodra <monarchdodra gmail.com> wrote:
 I think you are supposed to explicitly pad the field yourself:

You're right, I missed that part of the documentation. :)
Jul 30 2012
prev sibling next sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 07/30/2012 07:19 AM, Andrej Mitrovic wrote:

 I was thinking of cases where I'm wrapping a C++ POD struct which has
 bitfields and is packed on a 1-byte boundary.

It is inherently unportable because unlike D, bitfields in C and C++ give too much freedom to the compiler. The compiler can move the fields around at will. Ali
Jul 30 2012
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 07/30/2012 09:44 AM, Andrej Mitrovic wrote:
 On 7/30/12, Ali Çehreli<acehreli yahoo.com>  wrote:
 It is inherently unportable because unlike D, bitfields in C and C++
 give too much freedom to the compiler. The compiler can move the fields
 around at will.

I really need to figure out a reliable way to extract this information, at least for the target platform a library is compiled on.

The program should be able to detect the placements of the fields. You can expect a certain bit pattern for a given set of values of the bit fields, set those values, read them on the D side, detect potential inconsistencies, and fail the execution (or hopefully compilation). Ali
Jul 30 2012
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 07/30/2012 10:15 AM, Andrej Mitrovic wrote:

 import std.bitmanip;
 struct Foo
 {
      mixin(bitfields!(
          uint, "bits1", 32,
      ));
 }

 D:\DMD\dmd2\windows\bin\..\..\src\phobos\std\bitmanip.d(76): Error:
 shift by 32 is outside the range 0..31

 Should I file this?

Yes, it's a bug. Ali
Jul 30 2012
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 07/31/2012 06:57 PM, Era Scarecrow wrote:
 On Tuesday, 31 July 2012 at 16:48:37 UTC, Andrej Mitrovic wrote:
 On 7/31/12, Era Scarecrow <rtcvb32 yahoo.com> wrote:
 I wonder, is it really a bug? If you are going to have it fill a
 whole size it would fit anyways, why even put it in as a bitfield?
 You could just declare it separately.

I don't really know, I'm looking at this from a point of wrapping C++. I haven't used bitfields myself in my own code.

I'd say it's not a bug since C/C++ is free to reorder the fields you'd need to tinker with it anyways; HOWEVER if you still need to be able to have it then who's to stop you from doing it? I think more likely a flag/version or some indicator that you didn't make a mistake, such as making them depreciated so it complains to you. Kinda like how you can't make assignments in if statements or do useless compares, it's an error and helps prevent issues that are quite obviously mistakes.

This is obviously a mistake in the bitfield implementation. What else could be concluded from the error message: std\bitmanip.d(76): Error: shift by 32 is outside the range 0..31 Requesting a 32 bit or 64 bit member on the other hand is not a mistake, and it is not useless, therefore the analogy breaks down. (Also IMO, the once-in-a-year wtf that is caused by accidentally assigning in an if condition does not justify special casing assignment expressions inside if conditions. Also, what is an useless compare?)
Jul 31 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 31-Jul-12 22:21, Era Scarecrow wrote:
 On Tuesday, 31 July 2012 at 17:17:43 UTC, Timon Gehr wrote:

 This is obviously a mistake in the bitfield implementation. What else
 could be concluded from the error message:

 std\bitmanip.d(76): Error: shift by 32 is outside the range 0..31

 Requesting a 32 bit or 64 bit member on the other hand is not a
 mistake, and it is not useless, therefore the analogy breaks down.

Well curiously it was easier to fix than I thought (a line for a static if, and a modification of the masking)... Was there any other bugs that come to mind? Anything of consequence?

Great to see things moving. Could you please do a separate pull for bitfields it should get merged easier and it seems like a small but important bugfix. -- Dmitry Olshansky
Jul 31 2012
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/2/12 5:14 AM, Era Scarecrow wrote:
 On Thursday, 2 August 2012 at 09:03:54 UTC, monarch_dodra wrote:
 I had an (implementation) question for you: Does the implementation
 actually require knowing what the size of the padding is?

 eg:
 struct A
 {
 int a;
 mixin(bitfields!(
 uint, "x", 2,
 int, "y", 3,
 ulong, "", 3 // <- This line right there
 ));
 }

 It that highlighted line really mandatory?
 I'm fine with having it optional, in case I'd want to have, say, a 59
 bit padding, but can't the implementation figure it out on it's own?

The original code has it set that way, why? Perhaps so you are aware and actually have in place where all the bits are assigned (even if you aren't using them); Be horrible if you used accidently 33 bits and it extended to 64 without telling you (Wouldn't it?).

Yes, that's the intent. The user must define exactly how an entire ubyte/ushort/uint/ulong is filled, otherwise ambiguities and bugs are soon to arrive.
 However, having it fill the size in and ignore the last x bits wouldn't
 be too hard to do, I've been wondering if I should remove it.

Please don't. The effort on the programmer side is virtually nil, and keeps things in check. In no case would the use of bitfields() be so intensive that the bloat of one line gets any significance. Andrei
Aug 02 2012
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/2/12 5:26 AM, monarch_dodra wrote:
 One of the *big* reasons I'm against having a hand chosen padding, is
 that the implementation *should* be able to find out what the most
 efficient padding is on the current machine (could be 32 on some, could
 be 64 on some)

In my neck of the woods they call that "non-portability". If your code is dependent on the machine's characteristics you use version() and whatnot. Andrei
Aug 02 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/2/12 9:48 AM, monarch_dodra wrote:
 On Thursday, 2 August 2012 at 12:38:10 UTC, Andrei Alexandrescu wrote:
 On 8/2/12 5:26 AM, monarch_dodra wrote:
 One of the *big* reasons I'm against having a hand chosen padding, is
 that the implementation *should* be able to find out what the most
 efficient padding is on the current machine (could be 32 on some, could
 be 64 on some)

In my neck of the woods they call that "non-portability". If your code is dependent on the machine's characteristics you use version() and whatnot.

Well, isn't that the entire point: Making your code NOT dependent on the machine's characteristics? By forcing the developer to chose the bitfield size (32 or 64), you ARE forcing him to make a choice dependent on the machine's characteristics.

I think that's backwards. Andrei
Aug 02 2012
prev sibling parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 07/31/2012 09:15 AM, Era Scarecrow wrote:
 On Tuesday, 31 July 2012 at 15:25:55 UTC, Andrej Mitrovic wrote:
 On 7/31/12, monarch_dodra <monarchdodra gmail.com> wrote:
 The bug is only when the field is EXACTLY 32 bits BTW. bitfields
 works quite nice with 33 or whatever. More details in the report.

Yeah 32 or 64 bits, thanks for changing the title.

I wonder, is it really a bug? If you are going to have it fill a whole size it would fit anyways, why even put it in as a bitfield? You could just declare it separately.

It can happen in templated code where the width of the first field may be a template parameter. I wouldn't want to 'static if (width == 32)'. But thanks for fixing the bug already! :) Ali
Jul 31 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/30/12, Ali =C7ehreli <acehreli yahoo.com> wrote:
 It is inherently unportable because unlike D, bitfields in C and C++
 give too much freedom to the compiler. The compiler can move the fields
 around at will.

I really need to figure out a reliable way to extract this information, at least for the target platform a library is compiled on. gccxml gives some information which seems to be somewhat reliable but according to the authors it's more of a guess than accurate info (2nd paragraph): http://www.gccxml.org/pipermail/gccxml/2012-February/001483.html Still this isn't too shabby, there could be configuration options where the user could set pack/bitsize/compiler options and similar stuff which would make a codegenerator more reliable.
Jul 30 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/30/12, Ali =C7ehreli <acehreli yahoo.com> wrote:
 The program should be able to detect the placements of the fields. You
 can expect a certain bit pattern for a given set of values of the bit
 fields, set those values, read them on the D side, detect potential
 inconsistencies, and fail the execution (or hopefully compilation).

Yeah that was exactly my thinking, these things can be automated although they might slow down the whole codegeneration process a little bit (not that it matters much since you would rarely have to regenerate). Btw, I get an internal std error with this mixin: import std.bitmanip; struct Foo { mixin(bitfields!( uint, "bits1", 32, )); } D:\DMD\dmd2\windows\bin\..\..\src\phobos\std\bitmanip.d(76): Error: shift by 32 is outside the range 0..31 Should I file this?
Jul 30 2012
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 30 July 2012 at 17:15:54 UTC, Andrej Mitrovic wrote:
 On 7/30/12, Ali Çehreli <acehreli yahoo.com> wrote:
 The program should be able to detect the placements of the 
 fields. You
 can expect a certain bit pattern for a given set of values of 
 the bit
 fields, set those values, read them on the D side, detect 
 potential
 inconsistencies, and fail the execution (or hopefully 
 compilation).

Yeah that was exactly my thinking, these things can be automated although they might slow down the whole codegeneration process a little bit (not that it matters much since you would rarely have to regenerate). Btw, I get an internal std error with this mixin: import std.bitmanip; struct Foo { mixin(bitfields!( uint, "bits1", 32, )); } D:\DMD\dmd2\windows\bin\..\..\src\phobos\std\bitmanip.d(76): Error: shift by 32 is outside the range 0..31 Should I file this?

Yes, that should work. By the way, what's a "butfield"?
Jul 30 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/30/12, monarch_dodra <monarchdodra gmail.com> wrote:
 By the way, what's a "butfield"?

LOL I haven't even noticed.
Jul 30 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Monday, 30 July 2012 at 17:43:28 UTC, Ali Çehreli wrote:
 On 07/30/2012 10:15 AM, Andrej Mitrovic wrote:

 import std.bitmanip;
 struct Foo
 {
      mixin(bitfields!(
          uint, "bits1", 32,
      ));
 }

 D:\DMD\dmd2\windows\bin\..\..\src\phobos\std\bitmanip.d(76):

 shift by 32 is outside the range 0..31

 Should I file this?

Yes, it's a bug.

And likely one I'll be working on fairly soon. I've been concentrating on the BitArray, but I'll get more of the bitfields very very soon.
Jul 30 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/31/12, Era Scarecrow <rtcvb32 yahoo.com> wrote:
   And likely one I'll be working on fairly soon. I've been
 concentrating on the BitArray, but I'll get more of the bitfields
 very very soon.

Cool. What about the max limit of 64bits per bitfield instantiation? I don't suppose this is common in C++ but I wouldn't know..
Jul 30 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Monday, 30 July 2012 at 23:41:39 UTC, Andrej Mitrovic wrote:
 On 7/31/12, Era Scarecrow <rtcvb32 yahoo.com> wrote:
 And likely one I'll be working on fairly soon. I've been 
 concentrating on the BitArray, but I'll get more of the 
 bitfields very very soon.

Cool. What about the Max limit of 64bits per bitfield instantiation? I don't suppose this is common in C++ but I wouldn't know..

The limitation is based on what kind of types you can use. If you want a field 1000 bits long, i don't see why not; But obviously you can't treat it like an int or bool. (No cheap way to make a BigInt here :P), and packing say a struct, it would be slow since it has to copy each bit individually, or if it's byte aligned, by bytes. It assumes the largest type we can currently use which is ulong, if cent ever gets properly added, then 128bits will become available. If they want to go higher than likely type 'dime' or 'nickel' or bicent (256bits? :P) can be used once the template is modified. Maybe tricent can be 384 and quadcent can be 512.... Mmmm :) or it could just be large256, ularge256. As for the 'why', is that all of the bitfields work by assuming bit-shifting and low level binary operators to do the job. So, 2 for 4 ints would be int value; int a() property { return value & 0x7; } void a(int v) property { value &= ~0x7; value |= v & 0x7; } int b() property { return (value>>4) & 0x7; } void a(int v) property { value &= ~(0x7 << 4); value |= (v & 0x7) << 4; } That may be flawed but it gives you a basic idea.
Jul 30 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 00:00:24 UTC, Era Scarecrow wrote:
Corrections:
  So, 2 variables using 4 bit ints would be

 void a(int v)  property {
   value &= ~(0x7 << 4);
   value |= (v & 0x7) << 4;
 }

the second setter should be void b(int v) property {
Jul 30 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/31/12, Era Scarecrow <rtcvb32 yahoo.com> wrote:
   It assumes the largest type we can currently use which is ulong

Ah yes, it makes sense now. Thanks for the brain cereal. :p
Jul 30 2012
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Tuesday, 31 July 2012 at 00:44:16 UTC, Andrej Mitrovic wrote:
 On 7/31/12, Era Scarecrow <rtcvb32 yahoo.com> wrote:
   It assumes the largest type we can currently use which is 
 ulong

Ah yes, it makes sense now. Thanks for the brain cereal. :p

I saw your bug report: http://d.puremagic.com/issues/show_bug.cgi?id=8474 The bug is only when the field is EXACTLY 32 bits BTW. bitfields works quite nice with 33 or whatever. More details in the report.
Jul 31 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/31/12, monarch_dodra <monarchdodra gmail.com> wrote:
 The bug is only when the field is EXACTLY 32 bits BTW. bitfields
 works quite nice with 33 or whatever. More details in the report.

Yeah 32 or 64 bits, thanks for changing the title.
Jul 31 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 15:25:55 UTC, Andrej Mitrovic wrote:
 On 7/31/12, monarch_dodra <monarchdodra gmail.com> wrote:
 The bug is only when the field is EXACTLY 32 bits BTW. 
 bitfields works quite nice with 33 or whatever. More details 
 in the report.

Yeah 32 or 64 bits, thanks for changing the title.

I wonder, is it really a bug? If you are going to have it fill a whole size it would fit anyways, why even put it in as a bitfield? You could just declare it separately. I get the feeling it's not so much a bug as a design feature. If you really needed a full size not aligned with whole bytes (or padded appropriately) then I could understand, but still... And I'm the one that changed the title name. Suddenly I'm reminded of south park (movie) and the buttfor.
Jul 31 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/31/12, Era Scarecrow <rtcvb32 yahoo.com> wrote:
   I wonder, is it really a bug? If you are going to have it fill a
 whole size it would fit anyways, why even put it in as a
 bitfield? You could just declare it separately.

I don't really know, I'm looking at this from a point of wrapping C++. I haven't used bitfields myself in my own code.
Jul 31 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 16:48:37 UTC, Andrej Mitrovic wrote:
 On 7/31/12, Era Scarecrow <rtcvb32 yahoo.com> wrote:
 I wonder, is it really a bug? If you are going to have it fill 
 a whole size it would fit anyways, why even put it in as a 
 bitfield? You could just declare it separately.

I don't really know, I'm looking at this from a point of wrapping C++. I haven't used bitfields myself in my own code.

I'd say it's not a bug since C/C++ is free to reorder the fields you'd need to tinker with it anyways; HOWEVER if you still need to be able to have it then who's to stop you from doing it? I think more likely a flag/version or some indicator that you didn't make a mistake, such as making them depreciated so it complains to you. Kinda like how you can't make assignments in if statements or do useless compares, it's an error and helps prevent issues that are quite obviously mistakes.
Jul 31 2012
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Tuesday, 31 July 2012 at 16:16:00 UTC, Era Scarecrow wrote:
 On Tuesday, 31 July 2012 at 15:25:55 UTC, Andrej Mitrovic wrote:
 On 7/31/12, monarch_dodra <monarchdodra gmail.com> wrote:
 The bug is only when the field is EXACTLY 32 bits BTW. 
 bitfields works quite nice with 33 or whatever. More details 
 in the report.

Yeah 32 or 64 bits, thanks for changing the title.

I wonder, is it really a bug? If you are going to have it fill a whole size it would fit anyways, why even put it in as a bitfield? You could just declare it separately. I get the feeling it's not so much a bug as a design feature. If you really needed a full size not aligned with whole bytes (or padded appropriately) then I could understand, but still... And I'm the one that changed the title name. Suddenly I'm reminded of south park (movie) and the buttfor.

No, it's a bug. There is no reason for it to fail (and it certainly isn't a feature). Maybe the user wants to pack an "uint, ushort, ubyte, ubyte" together in a struct, but doesn't want the rest of that struct's members 1-aligned? Maybe the user needs a 32 bit ulong? This way the ulong only takes 32 bits, but can still be implicitly passed to functions expecting ulongs. Maybe the user generated the mixin using another template? For example, testing integers from 10 bits wide to 40 bits wide? Should he write a special case for 32? ... Now I'm not saying it is big bug or anything, but it is something that should be supported... Or at least explicitly asserted until fixed...
Jul 31 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 16:59:11 UTC, monarch_dodra wrote:
 No, it's a bug. There is no reason for it to fail (and it 
 certainly isn't a feature).

If I made two fields in a 64bit bitfield, each 32bits int's I'd like it to complain; If it's calculated from something else then finding the problem may be a little more difficult. But that's how my mind works, give you the tools you need to do whatever you want including shooting yourself in the foot (although you need to work harder to do that than C++).
 Maybe the user wants to pack an "uint, ushort, ubyte, ubyte" 
 together in a struct, but doesn't want the rest of that 
 struct's members 1-aligned?

That would be the one reason that makes sense; Course can't you align sections at 1 byte and then by the default afterwards? Course now that I think about it, some systems don't have options to do byte alignments and require to access memory at 4/8 byte alignments. This makes sense needing to support it.
 Maybe the user needs a 32 bit ulong? This way the ulong only 
 takes 32 bits, but can still be implicitly passed to functions 
 expecting ulongs.

I would think the bug only showed itself if you did int at 32bits and ulong at 64 bits, not ulong at 32bits.
 Maybe the user generated the mixin using another template? For 
 example, testing integers from 10 bits wide to 40 bits wide? 
 Should he write a special case for 32?

Ummm..... Yeah I think I see what your saying; And you're probably right a special case would be less easily documented (and more a pain) when it shouldn't need a special case.
 Now I'm not saying it is big bug or anything, but it is 
 something that should be supported... Or at least explicitly 
 asserted until fixed...

Jul 31 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 17:17:43 UTC, Timon Gehr wrote:

 (Also IMO, the once-in-a-year wtf that is caused by 
 accidentally assigning in an if condition does not justify 
 special casing assignment expressions inside if conditions. 
 Also, what is an useless compare?)

I've noticed in my experience, DMD gives you an error if you do a statement that has no effect; IE: 1 + 2; //statement has no effect a == b;//ditto
Jul 31 2012
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Tuesday, 31 July 2012 at 17:17:25 UTC, Era Scarecrow wrote:
 On Tuesday, 31 July 2012 at 16:59:11 UTC, monarch_dodra wrote:
 Maybe the user needs a 32 bit ulong? This way the ulong only 
 takes 32 bits, but can still be implicitly passed to functions 
 expecting ulongs.

I would think the bug only showed itself if you did int at 32bits and ulong at 64 bits, not ulong at 32bits.

No, the bug shows itself if the first field is 32 bits, regardless of (ulong included). I would add though that requesting a field in bits that is bigger than the type of the field should not work (IMO). EG: -------- struct A { mixin(bitfields!( ushort, "a", 24, uint, "", 8 ) ); } -------- I don't see any way how that could make sense... But it *is* legal in C and C++... But it does generates warnings... I think it should static assert in D. On Tuesday, 31 July 2012 at 17:21:00 UTC, Era Scarecrow wrote:
 On Tuesday, 31 July 2012 at 17:17:43 UTC, Timon Gehr wrote:

 (Also IMO, the once-in-a-year wtf that is caused by 
 accidentally assigning in an if condition does not justify 
 special casing assignment expressions inside if conditions. 
 Also, what is an useless compare?)

I've noticed in my experience, DMD gives you an error if you do a statement that has no effect; IE: 1 + 2; //statement has no effect a == b;//ditto

That's part of the standard: Statements that have no effect are illegal. This is a good think, IMO. I've seen MANY C++ bugs that could have been saved by that. Regarding the assignment in if. I think it is a good thing. D sides with safety. If you *really* want to test assign, you can always comma operator it. -------- void main() { int a = 0, b = 5; while(a = --b, a) //YES, I *DO* want to assign! { write(a); } }; --------I'm sure the compiler will optimize away what it needs.
Jul 31 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 17:34:33 UTC, monarch_dodra wrote:
 No, the bug shows itself if the first field is 32 bits, 
 regardless of (ulong included).

 I would add though that requesting a field in bits that is 
 bigger than the type of the field should not work (IMO). EG:
 --------
 struct A
 {
     mixin(bitfields!(
           ushort, "a", 24,
           uint,    "",  8
         )
     );
 }
 --------
 I don't see any way how that could make sense...
 But it *is* legal in C and C++...
 But it does generates warnings...

Maybe so ushort has extra padding for expansion at some later date when they change it to uint?? could put an assert in, but if it doesn't break code...
 I think it should static assert in D.

Glancing over the issue, the [0..31] is a compiler error based on bit shifting (not bitfields itself); if the storage type is ulong then it shouldn't matter if the first one is a 32bit size or not. Unless... Nah, couldn't be... I'll look it over later to be sure.
 That's part of the standard: Statements that have no effect are 
 illegal. This is a good think, IMO. I've seen MANY C++ bugs 
 that could have been saved by that.

 Regarding the assignment in if. I think it is a good thing. D 
 sides with safety. If you *really* want to test assign, you can 
 always comma operator it.

Jul 31 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 17:17:43 UTC, Timon Gehr wrote:

 This is obviously a mistake in the bitfield implementation. 
 What else could be concluded from the error message:

 std\bitmanip.d(76): Error: shift by 32 is outside the range 
 0..31

 Requesting a 32 bit or 64 bit member on the other hand is not a 
 mistake, and it is not useless, therefore the analogy breaks 
 down.

Well curiously it was easier to fix than I thought (a line for a static if, and a modification of the masking)... Was there any other bugs that come to mind? Anything of consequence?
Jul 31 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 17:17:43 UTC, Timon Gehr wrote:

 This is obviously a mistake in the bitfield implementation. 
 What else could be concluded from the error message:

 std\bitmanip.d(76): Error: shift by 32 is outside the range 
 0..31

 Requesting a 32 bit or 64 bit member on the other hand is not a 
 mistake, and it is not useless, therefore the analogy breaks 
 down.

Well curiously it was easier to fix than I thought (a line for a static if, and a modification of the masking)... Was there any other bugs that come to mind? Anything of consequence?
Jul 31 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 20:41:55 UTC, Dmitry Olshansky wrote:
 On 31-Jul-12 22:21, Era Scarecrow wrote:
 Well curiously it was easier to fix than I thought (a line for 
 a static if, and a modification of the masking)... Was there 
 any other bugs that come to mind? Anything of consequence?

Great to see things moving. Could you please do a separate pull for bitfields it should get merged easier and it seems like a small but important bugfix.

Guess this means I'll be working on BitArrays a bit later and work instead on the bitfields code. What fun... :) I thought I had bitfields as separate already, but it's kinda thrown both sets of changes in. Once I figure it out I'll get them separated and finish work on the bitfields.
Jul 31 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 20:41:55 UTC, Dmitry Olshansky wrote:

 Great to see things moving. Could you please do a separate pull 
 for bitfields it should get merged easier and it seems like a 
 small but important bugfix.

https://github.com/rtcvb32/phobos/commit/620ba57cc0a860245a2bf03f7b7f5d6a1bb58312 I've updated the next update in my bitfields branch. All unittests pass for me.
Aug 01 2012
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Wednesday, 1 August 2012 at 07:24:09 UTC, Era Scarecrow wrote:
 On Tuesday, 31 July 2012 at 20:41:55 UTC, Dmitry Olshansky 
 wrote:

 Great to see things moving. Could you please do a separate 
 pull for bitfields it should get merged easier and it seems 
 like a small but important bugfix.

https://github.com/rtcvb32/phobos/commit/620ba57cc0a860245a2bf03f7b7f5d6a1bb58312 I've updated the next update in my bitfields branch. All unittests pass for me.

I had an (implementation) question for you: Does the implementation actually require knowing what the size of the padding is? eg: struct A { int a; mixin(bitfields!( uint, "x", 2, int, "y", 3, ulong, "", 3 // <- This line right there )); } It that highlighted line really mandatory? I'm fine with having it optional, in case I'd want to have, say, a 59 bit padding, but can't the implementation figure it out on it's own?
Aug 02 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Thursday, 2 August 2012 at 09:03:54 UTC, monarch_dodra wrote:
 I had an (implementation) question for you: Does the 
 implementation actually require knowing what the size of the 
 padding is?

 eg:
 struct A
 {
     int a;
     mixin(bitfields!(
         uint,  "x",    2,
         int,   "y",    3,
         ulong,  "",    3 // <- This line right there
     ));
 }

 It that highlighted line really mandatory?
 I'm fine with having it optional, in case I'd want to have, 
 say, a 59 bit padding, but can't the implementation figure it 
 out on it's own?

The original code has it set that way, why? Perhaps so you are aware and actually have in place where all the bits are assigned (even if you aren't using them); Be horrible if you used accidently 33 bits and it extended to 64 without telling you (Wouldn't it?). However, having it fill the size in and ignore the last x bits wouldn't be too hard to do, I've been wondering if I should remove it.
Aug 02 2012
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 2 August 2012 at 09:14:15 UTC, Era Scarecrow wrote:
 On Thursday, 2 August 2012 at 09:03:54 UTC, monarch_dodra wrote:
 I had an (implementation) question for you: Does the 
 implementation actually require knowing what the size of the 
 padding is?

 eg:
 struct A
 {
    int a;
    mixin(bitfields!(
        uint,  "x",    2,
        int,   "y",    3,
        ulong,  "",    3 // <- This line right there
    ));
 }

 It that highlighted line really mandatory?
 I'm fine with having it optional, in case I'd want to have, 
 say, a 59 bit padding, but can't the implementation figure it 
 out on it's own?

The original code has it set that way, why? Perhaps so you are aware and actually have in place where all the bits are assigned (even if you aren't using them); Be horrible if you used accidently 33 bits and it extended to 64 without telling you (Wouldn't it?). However, having it fill the size in and ignore the last x bits wouldn't be too hard to do, I've been wondering if I should remove it.

Well, I was just trying to figure out the rationale: The most obvious one for me being "it is much easier on the implementation". One of the *big* reasons I'm against having a hand chosen padding, is that the implementation *should* be able to find out what the most efficient padding is on the current machine (could be 32 on some, could be 64 on some) That said, something that could fix the above "problem" could be: *Bitfields are automatically padded if the final field is not a "padding field". **Padding size is implementation chosen. *If the final field is a "padding field", then the total size must be 8/16/32/64. EG: //Case 1 bitfields!( bool, "x", 1, uint, "", 3, //Interfield padding bool, "y", 1 ) //Fine, implementation chosen bitfield size //Case 2 bitfields!( bool, "x", 1, uint, "", 3, //Interfield padding bool, "y", 1 ulong, "", 59, //Pad to 64 ) //Fine, imposed 64 bit //Case 3 bitfields!( bool, "x", 1, uint, "", 3, //Interfield padding bool, "y", 1 ulong, "", 32, //Pad to 37 ) //ERROR: Padding requests the bitfield to be 37 bits longs But I'd say that's another development anyways, if we ever decide to go this way.
Aug 02 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Thursday, 2 August 2012 at 09:26:04 UTC, monarch_dodra wrote:
 Well, I was just trying to figure out the rationale: The most 
 obvious one for me being "it is much easier on the 
 implementation".

Since the template is recursive and at the end after bit counting would know how much it needed, that doesn't seem right; far more likely just to be explicit.
 One of the *big* reasons I'm against having a hand chosen 
 padding, is that the implementation *should* be able to find 
 out what the most efficient padding is on the current machine 
 (could be 32 on some, could be 64 on some)

If your using bitfields, then you are going for space, and to be as small as reasonably possible. Especially important for packets of information like headers for compression, and making it compatible with C/C++'s bitpacking.
 That said, something that could fix the above "problem" could 
 be:
 *Bitfields are automatically padded if the final field is not a 
 "padding field".

Workable
 **Padding size is implementation chosen.

I assume you mean by word size (size_t), meaning always 32/64bit. In that case many applicable cases would go away and be useless.
 *If the final field is a "padding field", then the total size 
 must be 8/16/32/64.

 EG:
 //Case 1
 bitfields!(
     bool, "x",    1,
     uint,  "",    3, //Interfield padding
     bool, "y",    1
 )
 //Fine, implementation chosen bitfield size

It would go with a ubyte as is likely obvious, although any of the types would work.
 //Case 2
 bitfields!(
     bool, "x",    1,
     uint,  "",    3, //Interfield padding
     bool, "y",    1
     ulong, "",   59, //Pad to 64
 )
 //Fine, imposed 64 bit

 //Case 3
 bitfields!(
     bool, "x",    1,
     uint,  "",    3, //Interfield padding
     bool, "y",    1
     ulong, "",   32, //Pad to 37
 )
 //ERROR: Padding requests the bitfield to be 37 bits longs

 But I'd say that's another development anyways, if we ever 
 decide to go this way.

In the end, either explicit in size, or let it round up to the size it can accept. If you let it decide, then padding would be treated as though it's a variable (but isn't), so... bitfields!( bool, "x", 1, uint, "", 3, //Interfield padding bool, "y", 1 ulong, "", 5, ) the total size is 9bits, the padding forces it to 16bit afterwards. In cases like this it could easily be abused or leave it in a confusing state; So likely the padding would have to be missing (and assumed) or explicit in size. So previous one would err, while: bitfields!( bool, "x", 1, uint, "", 3, //Interfield padding bool, "y", 1, // void, "", 4 //implied by missing, if padded and final size not ^2 would statically assert. )
Aug 02 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Thursday, 2 August 2012 at 12:35:20 UTC, Andrei Alexandrescu 
wrote:

 Please don't. The effort on the programmer side is virtually 
 nil, and keeps things in check. In no case would the use of 
 bitfields() be so intensive that the bloat of one line gets any 
 significance.>

If you're using a template or something to fill in the sizes, then having to calculate the remainder could be an annoyance; but those cases would be small in number. I'll agree, and it's best leaving it as it is. BTW, Wasn't there a new/reserved type of cent/ucent (128bit)?
Aug 02 2012
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 2 August 2012 at 12:38:10 UTC, Andrei Alexandrescu 
wrote:
 On 8/2/12 5:26 AM, monarch_dodra wrote:
 One of the *big* reasons I'm against having a hand chosen 
 padding, is
 that the implementation *should* be able to find out what the 
 most
 efficient padding is on the current machine (could be 32 on 
 some, could
 be 64 on some)

In my neck of the woods they call that "non-portability". If your code is dependent on the machine's characteristics you use version() and whatnot.

Well, isn't that the entire point: Making your code NOT dependent on the machine's characteristics? By forcing the developer to chose the bitfield size (32 or 64), you ARE forcing him to make a choice dependent on the machine's characteristics. The developer just knows how he wants to pack his bits, not how he wants to pad them. Why should the developer be burdened with figuring out what the optimal size of his bitfield should be? By leaving the field blank, *that* guarantees portability.
Aug 02 2012
prev sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Thursday, 2 August 2012 at 14:52:58 UTC, Andrei Alexandrescu 
wrote:
 On 8/2/12 9:48 AM, monarch_dodra wrote:
 By forcing the developer to chose the bitfield size (32 or 
 64), you ARE forcing him to make a choice dependent on the 
 machine's characteristics.

I think that's backwards.

I think specifying bitfields, you're already going really low level; meaning you need explicit and full control. As for portability, if the size is specified, it comes down to endinness or order of the fields. Zlib header: http://tools.ietf.org/html/rfc1950 A zlib stream has the following structure: CMF, FLaG CMF: bits 0 to 3 CM Compression method bits 4 to 7 CINFO Compression info FLaG: bits 0 to 4 FCHECK (check bits for CMF and FLG) bit 5 FDICT (preset dictionary) bits 6 to 7 FLEVEL (compression level) This easily becomes: //assumes small-endian mixin(bitfields( ubyte, "CM", 4, ubyte, "CINFO", 4, ubyte, "FCHECK", 5, ubyte, "FDICT", 1, ubyte, "FLEVEL", 2)); //should be exactly 16 bits Now if a typo was done (wrong number of bits) or endianness is questionable, this becomes completely different as the fields could be as much as 3-7 bytes off (24 bits, or worse if it assumes 64bits, 56 bits off), or cover a much larger area than it should for the bits. This is unacceptable, especially if you need to get low level with the code.
Aug 02 2012