www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - bit fields

reply Walter Bright <newshound2 digitalmars.com> writes:
I have been working on implementing bit fields for #ImportC, with invaluable 
help from Iain.

And I've learned an unexpected lesson. Bit fields with DMC and Microsoft VC are 
simple. Bit fields on clang and gcc are complicated and messy. I've had a hard 
time duplicating that behavior. Hence,

1. https://dlang.org/phobos/std_bitmanip.html#bitfields does not create bit 
fields that match gcc/clang behavior. Not at all.

2. It's unreasonable to expect users to figure out how to lay out bit fields. 
Heck, how gcc/clang do it is essentially undocumented. They should be able to 
use bit fields that "just work" with the associated C compiler. We should be 
doing the hard work of compatibility, not dumping it on users.

3. DasBetterC is not in a great place since it can't do matching bit fields.

4. D is supposed to have relatively seamless compatibility with C. But we've 
left users twisting in the wind with C bit fields.

But there is good news. The result of figuring out bit fields for ImportC is 
that the logic is already in the D compiler, and we merely have to "turn it on" 
and it will work for D. There'll be a bit of extra work to deny trying to take 
the address of a bit field, pass it by `ref`, etc., but not too bad.

I propose adding bit fields to D to resolve these problems.

There are other advantages, too. I often use bit flags:

     enum Flags { A = 1, B = 2, C = 4, D = 8 }

     Flags f;
     f |= Flags.A;
     f &= ~Flags.B;

but this is tedious, and so this often just winds up as:

     bool a, b, c, d;

which is expensive in memory. With bit fields it is:

     bool a:1, b:1, c:1, d:1;

If we do it right, the code will be the same as the Flags version, and we can 
just do away with the Flags declaration.

P.S. Since the DMD backend already does bit fields, it was pretty simple to
hook 
that up with ImportC.

Thoughts welcome.
Aug 28 2021
next sibling parent Daniel N <no public.email> writes:
On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:

 I have been working on implementing bit fields for #ImportC, 
 with invaluable help from Iain.


 P.S. Since the DMD backend already does bit fields, it was 
 pretty simple to hook that up with ImportC.

 Thoughts welcome.


As someone who uses bitfields alot, YES Please!

My current workaround was to keep some C files in my projects 
just to be able to use bitfields, this would allow me to go pure 
D! I didn't even bring it up as I thought it was a lost cause.

One very happy user,
Daniel N
Aug 28 2021
prev sibling next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:

 They should be able to use bit fields that "just work" with the 
 associated C compiler.


How does dmd know which C compiler you're interfacing with?
Aug 28 2021
next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On Saturday, 28 August 2021 at 22:46:14 UTC, Dennis wrote:

 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:

 They should be able to use bit fields that "just work" with 
 the associated C compiler.


 How does dmd know which C compiler you're interfacing with?


Windows targets default to MS bit-fields, Posix targets default 
to GCC bit-fields.  It's a run-time configurable option, so the 
likes of gdc will re-use the -mms-bitfields option.
Aug 28 2021
prev sibling next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/28/21 6:46 PM, Dennis wrote:

 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:

 They should be able to use bit fields that "just work" with the 
 associated C compiler.


 
 How does dmd know which C compiler you're interfacing with?
 


We were discussing this on beerconf.

https://issues.dlang.org/show_bug.cgi?id=22248

-Steve
Aug 28 2021
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/28/2021 3:46 PM, Dennis wrote:

 How does dmd know which C compiler you're interfacing with?


Trade secret.
Aug 28 2021
prev sibling next sibling parent reply Johan <j j.nl> writes:
On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:

 I have been working on implementing bit fields for #ImportC, 
 with invaluable help from Iain.

 And I've learned an unexpected lesson. Bit fields with DMC and 
 Microsoft VC are simple. Bit fields on clang and gcc are 
 complicated and messy. I've had a hard time duplicating that 
 behavior.


A link to an article that demonstrates what kind of complexities 
are involved: http://jkz.wtf/bit-field-packing-in-gcc-and-clang

-Johan
Aug 29 2021
parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On Sunday, 29 August 2021 at 07:57:11 UTC, Johan wrote:

 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:

 I have been working on implementing bit fields for #ImportC, 
 with invaluable help from Iain.

 And I've learned an unexpected lesson. Bit fields with DMC and 
 Microsoft VC are simple. Bit fields on clang and gcc are 
 complicated and messy. I've had a hard time duplicating that 
 behavior.


 A link to an article that demonstrates what kind of 
 complexities are involved: 
 http://jkz.wtf/bit-field-packing-in-gcc-and-clang

 -Johan


I see more tests that can be added to the testsuite. ;-)
Aug 29 2021
parent reply Johan <j j.nl> writes:
On Sunday, 29 August 2021 at 15:09:14 UTC, Iain Buclaw wrote:

 On Sunday, 29 August 2021 at 07:57:11 UTC, Johan wrote:

 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:

 I have been working on implementing bit fields for #ImportC, 
 with invaluable help from Iain.

 And I've learned an unexpected lesson. Bit fields with DMC 
 and Microsoft VC are simple. Bit fields on clang and gcc are 
 complicated and messy. I've had a hard time duplicating that 
 behavior.


 A link to an article that demonstrates what kind of 
 complexities are involved: 
 http://jkz.wtf/bit-field-packing-in-gcc-and-clang

 -Johan


 I see more tests that can be added to the testsuite. ;-)


I think it's worthwhile to create a random testcase generator and 
let it run (+ execute the test) for a day or two...

-Johan
Aug 30 2021
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/30/21 6:40 AM, Johan wrote:

 
 I think it's worthwhile to create a random testcase generator and let it 
 run (+ execute the test) for a day or two...
 


Max had said he was going to try and cobble something like this together 
during beerconf.

-Steve
Aug 30 2021
parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On Monday, 30 August 2021 at 10:43:33 UTC, Steven Schveighoffer 
wrote:

 On 8/30/21 6:40 AM, Johan wrote:

 
 I think it's worthwhile to create a random testcase generator 
 and let it run (+ execute the test) for a day or two...
 


 Max had said he was going to try and cobble something like this 
 together during beerconf.


I had raised a bug report about doing such a thing a couple 
months back too.

https://issues.dlang.org/show_bug.cgi?id=22074
Aug 30 2021
parent Johan <j j.nl> writes:
On Monday, 30 August 2021 at 11:07:31 UTC, Iain Buclaw wrote:

 On Monday, 30 August 2021 at 10:43:33 UTC, Steven Schveighoffer 
 wrote:

 On 8/30/21 6:40 AM, Johan wrote:

 
 I think it's worthwhile to create a random testcase generator 
 and let it run (+ execute the test) for a day or two...
 


 Max had said he was going to try and cobble something like 
 this together during beerconf.


 I had raised a bug report about doing such a thing a couple 
 months back too.

 https://issues.dlang.org/show_bug.cgi?id=22074


Something like that, but then specific for bit fields that 
includes execution of C/D code, something like this:

1. Randomly generate C file with a struct+bitfields type 
definition:
```c
struct Test {
   {<empty>|some __alignment...} {<empty>|signed|unsigned} 
{int|char|long|size_t|float|...} member1 {<empty>|:<some number 
0..bitwidth of the storage type};
   ... repeat for member2, member3, random number of members
}

void resetMember1_C(Test* t)
{
   t->member1 = 0;
}
...repeat for how many members there are

size_t sizeofTest_C()
{
    return sizeof(Test);
}
```

2. While generating the C file, you also generate a D file:
```d
extern(C)
struct Test {
// ofcourse corresponding to the generated C type!
   {<empty>|align(...)} 
{int|uint|bool|byte|char|long|size_t|float|...} member1 
{<empty>|:<some number 0..bitwidth of the storage type};
   ... repeat for member2, member3, random number of members
}

extern(C) void resetMember1_C(Test* t);
void resetMember1_D(Test* t)
{
   t.member1 = 0;
}
...repeat for how many members there are

extern(C) size_t sizeofTest_C();
size_t sizeofTest_D()
{
    return Test.sizeof();
}

void main()
{
   assert(sizeofTest_C() == sizeofTest_D());

   foreach(i; number of members in Test) {
      Test s_C;
      Test s_D;
      memset(&s_C, 0xFF, Test.sizeof);
      memset(&s_D, 0xFF, Test.sizeof);
      resetMemberi_C(&s_C);
      resetMemberi_D(&s_D);
      assert(memcmp(&s_C, &s_D, Test.sizeof));
   }
}
```

3. Compile and run at -O0 and -O3 on every arch supported. Make 
sure that asserts are not discarded at high optimization level.

And then loop&repeat these steps for a long time.

cheers,
   Johan
Aug 30 2021
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Another reason for D bit fields are the various C to D and C++ to D
translators. 
It's an unreasonable burden on those translation developers to account for all 
the different undocumented bit field layout algorithms.
Aug 29 2021
prev sibling next sibling parent reply Elronnd <elronnd elronnd.net> writes:
On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:

 how gcc/clang do it is essentially undocumented


x86-64-psABI, sec 3.1.2 (data representation)
Aug 29 2021
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/29/2021 11:32 PM, Elronnd wrote:

 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:

 how gcc/clang do it is essentially undocumented


 
 x86-64-psABI, sec 3.1.2 (data representation)


https://github.com/hjl-tools/x86-psABI/wiki/x86-64-psABI-1.0.pdf

That only partially documents it. It doesn't cover things like alignment and 0 
length fields.
Aug 30 2021
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 30 August 2021 at 18:39:20 UTC, Walter Bright wrote:

 On 8/29/2021 11:32 PM, Elronnd wrote:

 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:

 how gcc/clang do it is essentially undocumented


 
 x86-64-psABI, sec 3.1.2 (data representation)


 https://github.com/hjl-tools/x86-psABI/wiki/x86-64-psABI-1.0.pdf

 That only partially documents it. It doesn't cover things like 
 alignment and 0 length fields.


Zero-length bitfields are covered by the C standard:


 As a special case, a bit-field structure member with a width of 
 0 indicates that no further bit-field is to be packed into the 
 unit in which the previous bit-field, if any, was placed.
Aug 30 2021
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/30/2021 12:40 PM, Paul Backus wrote:

 On Monday, 30 August 2021 at 18:39:20 UTC, Walter Bright wrote:

 On 8/29/2021 11:32 PM, Elronnd wrote:

 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:

 how gcc/clang do it is essentially undocumented


 x86-64-psABI, sec 3.1.2 (data representation)


 https://github.com/hjl-tools/x86-psABI/wiki/x86-64-psABI-1.0.pdf

 That only partially documents it. It doesn't cover things like alignment and 0 
 length fields.


 
 Zero-length bitfields are covered by the C standard:
 

 As a special case, a bit-field structure member with a width of 0 indicates 
 that no further bit-field is to be packed into the unit in which the previous 
 bit-field, if any, was placed.


 



Doesn't cover alignment, either. Doesn't cover what happens with a struct where 
the only member is a 0 bit field. Doesn't cover what a sequence of 2 or more 0 
bit fields does.
Aug 31 2021
prev sibling next sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:

 I have been working on implementing bit fields for #ImportC, 
 with invaluable help from Iain.

 [...]


Splendid ☀
Aug 30 2021
prev sibling next sibling parent reply TheGag96 <thegag96 gmail.com> writes:
On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright wrote:

 3. DasBetterC is not in a great place since it can't do 
 matching bit fields.


I think I may be experiencing a bug in a betterC project of mine 
as a result of me assuming that `bitfields` is doing the same 
thing as gcc would...! Is there a way I can get things synced up 
on at least one platform in the meantime?

I feel pretty in favor of having bit fields as a language 
feature, warts and all!
Aug 30 2021
next sibling parent max haughton <maxhaton gmail.com> writes:
On Monday, 30 August 2021 at 22:45:36 UTC, TheGag96 wrote:

 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:

 3. DasBetterC is not in a great place since it can't do 
 matching bit fields.


 I think I may be experiencing a bug in a betterC project of 
 mine as a result of me assuming that `bitfields` is doing the 
 same thing as gcc would...! Is there a way I can get things 
 synced up on at least one platform in the meantime?

 I feel pretty in favor of having bit fields as a language 
 feature, warts and all!


Read the assembly code/print bit patterns and make sure your code 
is doing the right shifts.

The write unittests.
Aug 30 2021
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 30 August 2021 at 22:45:36 UTC, TheGag96 wrote:

 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:

 3. DasBetterC is not in a great place since it can't do 
 matching bit fields.


 I think I may be experiencing a bug in a betterC project of 
 mine as a result of me assuming that `bitfields` is doing the 
 same thing as gcc would...! Is there a way I can get things 
 synced up on at least one platform in the meantime?


Don't use bitfields for anything that needs to have a stable ABI. 
They're not even guaranteed to have the same layout when using 
the same compiler on different platforms [1], much less between 
different languages.

[1] https://yarchive.net/comp/linux/bitfields.html
Aug 30 2021
next sibling parent TheGag96 <thegag96 gmail.com> writes:
On Tuesday, 31 August 2021 at 01:04:57 UTC, Paul Backus wrote:

 On Monday, 30 August 2021 at 22:45:36 UTC, TheGag96 wrote:

 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:

 3. DasBetterC is not in a great place since it can't do 
 matching bit fields.


 I think I may be experiencing a bug in a betterC project of 
 mine as a result of me assuming that `bitfields` is doing the 
 same thing as gcc would...! Is there a way I can get things 
 synced up on at least one platform in the meantime?


 Don't use bitfields for anything that needs to have a stable 
 ABI. They're not even guaranteed to have the same layout when 
 using the same compiler on different platforms [1], much less 
 between different languages.

 [1] https://yarchive.net/comp/linux/bitfields.html


I'm working on a single, very specific platform, and the library 
I'm interfacing with was the one who chose to make data 
structures that had bitfields, not me.
Aug 30 2021
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Tuesday, 31 August 2021 at 01:04:57 UTC, Paul Backus wrote:

 On Monday, 30 August 2021 at 22:45:36 UTC, TheGag96 wrote:

 On Saturday, 28 August 2021 at 08:20:25 UTC, Walter Bright 
 wrote:

 3. DasBetterC is not in a great place since it can't do 
 matching bit fields.


 I think I may be experiencing a bug in a betterC project of 
 mine as a result of me assuming that `bitfields` is doing the 
 same thing as gcc would...! Is there a way I can get things 
 synced up on at least one platform in the meantime?


 Don't use bitfields for anything that needs to have a stable 
 ABI. They're not even guaranteed to have the same layout when 
 using the same compiler on different platforms [1], much less 
 between different languages.

 [1] https://yarchive.net/comp/linux/bitfields.html


Then maybe introduce a bitfields with a keyword indicating that 
it's not cross platform?

__bitfields bool a:1, b:1, c:1, d:1;
Aug 31 2021
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/30/2021 6:04 PM, Paul Backus wrote:

 Don't use bitfields for anything that needs to have a stable ABI. They're not 
 even guaranteed to have the same layout when using the same compiler on 
 different platforms [1], much less between different languages.


Things are pretty consistent if sticking with simple int/unsigned bit fields.
Aug 31 2021
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/30/2021 3:45 PM, TheGag96 wrote:

 I think I may be experiencing a bug in a betterC project of mine as a result
of 
 me assuming that `bitfields` is doing the same thing as gcc would...! Is there
a 
 way I can get things synced up on at least one platform in the meantime?


The ImportC bit fields will be in the next dmd beta.

Until D itself gets bit fields, the thing to do will be to declare the bit
field 
struct in a separate .c file, and import it.

Otherwise, the only way is to implement your own using get/set member functions.
Aug 31 2021
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 28.08.21 10:20, Walter Bright wrote:

 
 There are other advantages, too. I often use bit flags:
 
      enum Flags { A = 1, B = 2, C = 4, D = 8 }
 
      Flags f;
      f |= Flags.A;
      f &= ~Flags.B;
 
 but this is tedious, and so this often just winds up as:
 
      bool a, b, c, d;
 
 which is expensive in memory. With bit fields it is:
 
      bool a:1, b:1, c:1, d:1;



+1. This is one of those things you leave in to fix later because the 
syntax is more convenient, then it does not get fixed later.
Aug 30 2021