www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - bit fields

reply Walter Bright <newshound2 digitalmars.com> writes:
I have been working on implementing bit fields for #ImportC, with invaluable 
help from Iain.

And I've learned an unexpected lesson. Bit fields with DMC and Microsoft VC are 
simple. Bit fields on clang and gcc are complicated and messy. I've had a hard 
time duplicating that behavior. Hence,

1. https://dlang.org/phobos/std_bitmanip.html#bitfields does not create bit 
fields that match gcc/clang behavior. Not at all.

2. It's unreasonable to expect users to figure out how to lay out bit fields. 
Heck, how gcc/clang do it is essentially undocumented. They should be able to 
use bit fields that "just work" with the associated C compiler. We should be 
doing the hard work of compatibility, not dumping it on users.

3. DasBetterC is not in a great place since it can't do matching bit fields.

4. D is supposed to have relatively seamless compatibility with C. But we've 
left users twisting in the wind with C bit fields.

But there is good news. The result of figuring out bit fields for ImportC is 
that the logic is already in the D compiler, and we merely have to "turn it on" 
and it will work for D. There'll be a bit of extra work to deny trying to take 
the address of a bit field, pass it by `ref`, etc., but not too bad.

I propose adding bit fields to D to resolve these problems.

There are other advantages, too. I often use bit flags:

     enum Flags { A = 1, B = 2, C = 4, D = 8 }

     Flags f;
     f |= Flags.A;
     f &= ~Flags.B;

but this is tedious, and so this often just winds up as:

     bool a, b, c, d;

which is expensive in memory. With bit fields it is:

     bool a:1, b:1, c:1, d:1;

If we do it right, the code will be the same as the Flags version, and we can 
just do away with the Flags declaration.

P.S. Since the DMD backend already does bit fields, it was pretty simple to
hook 
that up with ImportC.

Thoughts welcome.
Aug 28
next sibling parent Daniel N <no public.email> writes:
On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:
 I have been working on implementing bit fields for #ImportC, 
 with invaluable help from Iain.


 P.S. Since the DMD backend already does bit fields, it was 
 pretty simple to hook that up with ImportC.

 Thoughts welcome.
As someone who uses bitfields alot, YES Please! My current workaround was to keep some C files in my projects just to be able to use bitfields, this would allow me to go pure D! I didn't even bring it up as I thought it was a lost cause. One very happy user, Daniel N
Aug 28
prev sibling next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:
 They should be able to use bit fields that "just work" with the 
 associated C compiler.
How does dmd know which C compiler you're interfacing with?
Aug 28
next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On Saturday, 28 August 2021 at 22:46:14 UTC, Dennis wrote:
 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:
 They should be able to use bit fields that "just work" with 
 the associated C compiler.
How does dmd know which C compiler you're interfacing with?
Windows targets default to MS bit-fields, Posix targets default to GCC bit-fields. It's a run-time configurable option, so the likes of gdc will re-use the -mms-bitfields option.
Aug 28
prev sibling next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/28/21 6:46 PM, Dennis wrote:
 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:
 They should be able to use bit fields that "just work" with the 
 associated C compiler.
How does dmd know which C compiler you're interfacing with?
We were discussing this on beerconf. https://issues.dlang.org/show_bug.cgi?id=22248 -Steve
Aug 28
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/28/2021 3:46 PM, Dennis wrote:
 How does dmd know which C compiler you're interfacing with?
Trade secret.
Aug 28
prev sibling next sibling parent reply Johan <j j.nl> writes:
On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:
 I have been working on implementing bit fields for #ImportC, 
 with invaluable help from Iain.

 And I've learned an unexpected lesson. Bit fields with DMC and 
 Microsoft VC are simple. Bit fields on clang and gcc are 
 complicated and messy. I've had a hard time duplicating that 
 behavior.
A link to an article that demonstrates what kind of complexities are involved: http://jkz.wtf/bit-field-packing-in-gcc-and-clang -Johan
Aug 29
parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On Sunday, 29 August 2021 at 07:57:11 UTC, Johan wrote:
 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:
 I have been working on implementing bit fields for #ImportC, 
 with invaluable help from Iain.

 And I've learned an unexpected lesson. Bit fields with DMC and 
 Microsoft VC are simple. Bit fields on clang and gcc are 
 complicated and messy. I've had a hard time duplicating that 
 behavior.
A link to an article that demonstrates what kind of complexities are involved: http://jkz.wtf/bit-field-packing-in-gcc-and-clang -Johan
I see more tests that can be added to the testsuite. ;-)
Aug 29
parent reply Johan <j j.nl> writes:
On Sunday, 29 August 2021 at 15:09:14 UTC, Iain Buclaw wrote:
 On Sunday, 29 August 2021 at 07:57:11 UTC, Johan wrote:
 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:
 I have been working on implementing bit fields for #ImportC, 
 with invaluable help from Iain.

 And I've learned an unexpected lesson. Bit fields with DMC 
 and Microsoft VC are simple. Bit fields on clang and gcc are 
 complicated and messy. I've had a hard time duplicating that 
 behavior.
A link to an article that demonstrates what kind of complexities are involved: http://jkz.wtf/bit-field-packing-in-gcc-and-clang -Johan
I see more tests that can be added to the testsuite. ;-)
I think it's worthwhile to create a random testcase generator and let it run (+ execute the test) for a day or two... -Johan
Aug 30
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/30/21 6:40 AM, Johan wrote:
 
 I think it's worthwhile to create a random testcase generator and let it 
 run (+ execute the test) for a day or two...
 
Max had said he was going to try and cobble something like this together during beerconf. -Steve
Aug 30
parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On Monday, 30 August 2021 at 10:43:33 UTC, Steven Schveighoffer 
wrote:
 On 8/30/21 6:40 AM, Johan wrote:
 
 I think it's worthwhile to create a random testcase generator 
 and let it run (+ execute the test) for a day or two...
 
Max had said he was going to try and cobble something like this together during beerconf.
I had raised a bug report about doing such a thing a couple months back too. https://issues.dlang.org/show_bug.cgi?id=22074
Aug 30
parent Johan <j j.nl> writes:
On Monday, 30 August 2021 at 11:07:31 UTC, Iain Buclaw wrote:
 On Monday, 30 August 2021 at 10:43:33 UTC, Steven Schveighoffer 
 wrote:
 On 8/30/21 6:40 AM, Johan wrote:
 
 I think it's worthwhile to create a random testcase generator 
 and let it run (+ execute the test) for a day or two...
 
Max had said he was going to try and cobble something like this together during beerconf.
I had raised a bug report about doing such a thing a couple months back too. https://issues.dlang.org/show_bug.cgi?id=22074
Something like that, but then specific for bit fields that includes execution of C/D code, something like this: 1. Randomly generate C file with a struct+bitfields type definition: ```c struct Test { {<empty>|some __alignment...} {<empty>|signed|unsigned} {int|char|long|size_t|float|...} member1 {<empty>|:<some number 0..bitwidth of the storage type}; ... repeat for member2, member3, random number of members } void resetMember1_C(Test* t) { t->member1 = 0; } ...repeat for how many members there are size_t sizeofTest_C() { return sizeof(Test); } ``` 2. While generating the C file, you also generate a D file: ```d extern(C) struct Test { // ofcourse corresponding to the generated C type! {<empty>|align(...)} {int|uint|bool|byte|char|long|size_t|float|...} member1 {<empty>|:<some number 0..bitwidth of the storage type}; ... repeat for member2, member3, random number of members } extern(C) void resetMember1_C(Test* t); void resetMember1_D(Test* t) { t.member1 = 0; } ...repeat for how many members there are extern(C) size_t sizeofTest_C(); size_t sizeofTest_D() { return Test.sizeof(); } void main() { assert(sizeofTest_C() == sizeofTest_D()); foreach(i; number of members in Test) { Test s_C; Test s_D; memset(&s_C, 0xFF, Test.sizeof); memset(&s_D, 0xFF, Test.sizeof); resetMemberi_C(&s_C); resetMemberi_D(&s_D); assert(memcmp(&s_C, &s_D, Test.sizeof)); } } ``` 3. Compile and run at -O0 and -O3 on every arch supported. Make sure that asserts are not discarded at high optimization level. And then loop&repeat these steps for a long time. cheers, Johan
Aug 30
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Another reason for D bit fields are the various C to D and C++ to D
translators. 
It's an unreasonable burden on those translation developers to account for all 
the different undocumented bit field layout algorithms.
Aug 29
prev sibling next sibling parent reply Elronnd <elronnd elronnd.net> writes:
On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:
 how gcc/clang do it is essentially undocumented
x86-64-psABI, sec 3.1.2 (data representation)
Aug 29
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/29/2021 11:32 PM, Elronnd wrote:
 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:
 how gcc/clang do it is essentially undocumented
x86-64-psABI, sec 3.1.2 (data representation)
https://github.com/hjl-tools/x86-psABI/wiki/x86-64-psABI-1.0.pdf That only partially documents it. It doesn't cover things like alignment and 0 length fields.
Aug 30
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 30 August 2021 at 18:39:20 UTC, Walter Bright wrote:
 On 8/29/2021 11:32 PM, Elronnd wrote:
 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:
 how gcc/clang do it is essentially undocumented
x86-64-psABI, sec 3.1.2 (data representation)
https://github.com/hjl-tools/x86-psABI/wiki/x86-64-psABI-1.0.pdf That only partially documents it. It doesn't cover things like alignment and 0 length fields.
Zero-length bitfields are covered by the C standard:
 As a special case, a bit-field structure member with a width of 
 0 indicates that no further bit-field is to be packed into the 
 unit in which the previous bit-field, if any, was placed.
Aug 30
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/30/2021 12:40 PM, Paul Backus wrote:
 On Monday, 30 August 2021 at 18:39:20 UTC, Walter Bright wrote:
 On 8/29/2021 11:32 PM, Elronnd wrote:
 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:
 how gcc/clang do it is essentially undocumented
x86-64-psABI, sec 3.1.2 (data representation)
https://github.com/hjl-tools/x86-psABI/wiki/x86-64-psABI-1.0.pdf That only partially documents it. It doesn't cover things like alignment and 0 length fields.
Zero-length bitfields are covered by the C standard:
 As a special case, a bit-field structure member with a width of 0 indicates 
 that no further bit-field is to be packed into the unit in which the previous 
 bit-field, if any, was placed.
Doesn't cover alignment, either. Doesn't cover what happens with a struct where the only member is a 0 bit field. Doesn't cover what a sequence of 2 or more 0 bit fields does.
Aug 31
prev sibling next sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:
 I have been working on implementing bit fields for #ImportC, 
 with invaluable help from Iain.

 [...]
Splendid ☀
Aug 30
prev sibling next sibling parent reply TheGag96 <thegag96 gmail.com> writes:
On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:
 3. DasBetterC is not in a great place since it can't do 
 matching bit fields.
I think I may be experiencing a bug in a betterC project of mine as a result of me assuming that `bitfields` is doing the same thing as gcc would...! Is there a way I can get things synced up on at least one platform in the meantime? I feel pretty in favor of having bit fields as a language feature, warts and all!
Aug 30
next sibling parent max haughton <maxhaton gmail.com> writes:
On Monday, 30 August 2021 at 22:45:36 UTC, TheGag96 wrote:
 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:
 3. DasBetterC is not in a great place since it can't do 
 matching bit fields.
I think I may be experiencing a bug in a betterC project of mine as a result of me assuming that `bitfields` is doing the same thing as gcc would...! Is there a way I can get things synced up on at least one platform in the meantime? I feel pretty in favor of having bit fields as a language feature, warts and all!
Read the assembly code/print bit patterns and make sure your code is doing the right shifts. The write unittests.
Aug 30
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 30 August 2021 at 22:45:36 UTC, TheGag96 wrote:
 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:
 3. DasBetterC is not in a great place since it can't do 
 matching bit fields.
I think I may be experiencing a bug in a betterC project of mine as a result of me assuming that `bitfields` is doing the same thing as gcc would...! Is there a way I can get things synced up on at least one platform in the meantime?
Don't use bitfields for anything that needs to have a stable ABI. They're not even guaranteed to have the same layout when using the same compiler on different platforms [1], much less between different languages. [1] https://yarchive.net/comp/linux/bitfields.html
Aug 30
next sibling parent TheGag96 <thegag96 gmail.com> writes:
On Tuesday, 31 August 2021 at 01:04:57 UTC, Paul Backus wrote:
 On Monday, 30 August 2021 at 22:45:36 UTC, TheGag96 wrote:
 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:
 3. DasBetterC is not in a great place since it can't do 
 matching bit fields.
I think I may be experiencing a bug in a betterC project of mine as a result of me assuming that `bitfields` is doing the same thing as gcc would...! Is there a way I can get things synced up on at least one platform in the meantime?
Don't use bitfields for anything that needs to have a stable ABI. They're not even guaranteed to have the same layout when using the same compiler on different platforms [1], much less between different languages. [1] https://yarchive.net/comp/linux/bitfields.html
I'm working on a single, very specific platform, and the library I'm interfacing with was the one who chose to make data structures that had bitfields, not me.
Aug 30
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Tuesday, 31 August 2021 at 01:04:57 UTC, Paul Backus wrote:
 On Monday, 30 August 2021 at 22:45:36 UTC, TheGag96 wrote:
 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:
 3. DasBetterC is not in a great place since it can't do 
 matching bit fields.
I think I may be experiencing a bug in a betterC project of mine as a result of me assuming that `bitfields` is doing the same thing as gcc would...! Is there a way I can get things synced up on at least one platform in the meantime?
Don't use bitfields for anything that needs to have a stable ABI. They're not even guaranteed to have the same layout when using the same compiler on different platforms [1], much less between different languages. [1] https://yarchive.net/comp/linux/bitfields.html
Then maybe introduce a bitfields with a keyword indicating that it's not cross platform? __bitfields bool a:1, b:1, c:1, d:1;
Aug 31
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/30/2021 6:04 PM, Paul Backus wrote:
 Don't use bitfields for anything that needs to have a stable ABI. They're not 
 even guaranteed to have the same layout when using the same compiler on 
 different platforms [1], much less between different languages.
Things are pretty consistent if sticking with simple int/unsigned bit fields.
Aug 31
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/30/2021 3:45 PM, TheGag96 wrote:
 I think I may be experiencing a bug in a betterC project of mine as a result
of 
 me assuming that `bitfields` is doing the same thing as gcc would...! Is there
a 
 way I can get things synced up on at least one platform in the meantime?
The ImportC bit fields will be in the next dmd beta. Until D itself gets bit fields, the thing to do will be to declare the bit field struct in a separate .c file, and import it. Otherwise, the only way is to implement your own using get/set member functions.
Aug 31
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 28.08.21 10:20, Walter Bright wrote:
 
 There are other advantages, too. I often use bit flags:
 
      enum Flags { A = 1, B = 2, C = 4, D = 8 }
 
      Flags f;
      f |= Flags.A;
      f &= ~Flags.B;
 
 but this is tedious, and so this often just winds up as:
 
      bool a, b, c, d;
 
 which is expensive in memory. With bit fields it is:
 
      bool a:1, b:1, c:1, d:1;
+1. This is one of those things you leave in to fix later because the syntax is more convenient, then it does not get fixed later.
Aug 30