www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Creating array of structs being used in C interface

reply Timoses <timosesu gmail.com> writes:
Hi there,

I've got a problem interfacing to a C library.
The following structs are used by the library's .d file that I've 
written.

--------------------
struct neo4j_map_entry_t
{
     neo4j_value_t key;
     neo4j_value_t value;
};

struct neo4j_value_t
{
     uint8_t _vt_off;
     uint8_t _type; /*TODO: combine with _vt_off? (both always 
have same value)*/
     uint16_t _pad1;
     uint32_t _pad2;
     _neo4j_value_data _vdata;
};

union _neo4j_value_data
{
     uint64_t _int;
     uintptr_t _ptr;
     double _dbl;
};
--------------------

Now I'd also like to use them in my own code. However, I fail at 
generating an array of map entries:

--------------------
void test()
{
     // These work
     neo4j_map_entry_t[] mapa2; // yes
     neo4j_map_entry_t entry1 = { key: neo4j_string("prop1"), 
value: neo4j_string("testprop1")}; // yes
     neo4j_map_entry_t entry2 = { key: neo4j_string("prop2"), 
value: neo4j_string("testprop2")}; // yes
     neo4j_map_entry_t* mapp; // yes

     // These don't
     neo4j_map_entry_t[2] mapa1; // no
     mapa2.length = 2; // no
     mapa2 ~= entry1; // no
     neo4j_map_entry_t[] mapa3 = [{ key: neo4j_null, value: 
neo4j_null}]; // no
}
--------------------

The output is:
______________________
myprogram ~master: building configuration "unittest"...
Linking...
Undefined symbols for architecture x86_64:
   "_D10neo4jTypes17neo4j_map_entry_t6__initZ", referenced from:
       _D6myprogram6myprogram5test2MFZv in myprogram.o
ld: symbol(s) not found for architecture x86_64
______________________

Why is it a linker problem? I'm not linking to the c interface 
but merely using D structs...
Nov 27 2016
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 27 November 2016 at 12:59:32 UTC, Timoses wrote:
 Hi there,

 I've got a problem interfacing to a C library.
 The following structs are used by the library's .d file that 
 I've written.

 --------------------
 struct neo4j_map_entry_t
 {
     neo4j_value_t key;
     neo4j_value_t value;
 };

 struct neo4j_value_t
 {
     uint8_t _vt_off;
     uint8_t _type; /*TODO: combine with _vt_off? (both always 
 have same value)*/
     uint16_t _pad1;
     uint32_t _pad2;
     _neo4j_value_data _vdata;
 };

 union _neo4j_value_data
 {
     uint64_t _int;
     uintptr_t _ptr;
     double _dbl;
 };
 --------------------

 Now I'd also like to use them in my own code. However, I fail 
 at generating an array of map entries:

 --------------------
 void test()
 {
     // These work
     neo4j_map_entry_t[] mapa2; // yes
     neo4j_map_entry_t entry1 = { key: neo4j_string("prop1"), 
 value: neo4j_string("testprop1")}; // yes
     neo4j_map_entry_t entry2 = { key: neo4j_string("prop2"), 
 value: neo4j_string("testprop2")}; // yes
     neo4j_map_entry_t* mapp; // yes

     // These don't
     neo4j_map_entry_t[2] mapa1; // no
     mapa2.length = 2; // no
     mapa2 ~= entry1; // no
     neo4j_map_entry_t[] mapa3 = [{ key: neo4j_null, value: 
 neo4j_null}]; // no
 }
 --------------------

 The output is:
 ______________________
 myprogram ~master: building configuration "unittest"...
 Linking...
 Undefined symbols for architecture x86_64:
   "_D10neo4jTypes17neo4j_map_entry_t6__initZ", referenced from:
       _D6myprogram6myprogram5test2MFZv in myprogram.o
 ld: symbol(s) not found for architecture x86_64
 ______________________

 Why is it a linker problem? I'm not linking to the c interface 
 but merely using D structs...
The missing symbol is the struct initialiser for neo4j_map_entry_t. Not sure why is not being generated (it should), possibly because of the union. That seems like a bug please report it. http://issues.dlang.org/
Nov 27 2016
parent reply Timoses <timosesu gmail.com> writes:
On Sunday, 27 November 2016 at 13:22:36 UTC, Nicholas Wilson 
wrote:
 The missing symbol is the struct initialiser for 
 neo4j_map_entry_t. Not sure why is not being generated (it 
 should), possibly because of the union.

 That seems like a bug please report it. http://issues.dlang.org/
Thanks for the answer, Nicholas! I've created a small example: source/app.d -------------------- import mytypes; void main() { myunion[2] t; } -------------------- include/mytypes.d: -------------------- module mytypes; union myunion { double b; }; -------------------- dub.json: -------------------- "importPaths": ["include"] -------------------- Output: _____________________ dmd -c -of.dub/build/application-debug-posix.osx-x86_64-dmd_2072-99D5F2E2DCAF9FF19FE5AE4 3C120D52/testomat.o -debug -g -w -version=Have_testomat -Iinclude source/app.d -vcolumns Linking... dmd -of.dub/build/application-debug-posix.osx-x86_64-dmd_2072-99D5F2E2DCAF9FF19FE5A 403C120D52/testomat .dub/build/application-debug-posix.osx-x86_64-dmd_2072-99D5F2E2DCAF9FF19FE5AE4 3C120D52/testomat.o -g Undefined symbols for architecture x86_64: "_D7mytypes7myunion6__initZ", referenced from: __Dmain in testomat.o ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) Error: linker exited with status 1 FAIL .dub/build/application-debug-posix.osx-x86_64-dmd_2072-99D5F2E2DCAF9 F19FE5AE403C120D52/ testomat executable dmd failed with exit code 1. _____________________ If I change the union's variable type to "int" (or any other) it compiles just fine. So the problem seems to be the "double" value. It also compiles just fine when I move "mytypes.d" into the source/ folder (even with "double"!!!). Should I still report it as a bug here: https://issues.dlang.org/ ? Or is it dub related?
Nov 27 2016
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 27 November 2016 at 14:12:31 UTC, Timoses wrote:
 If I change the union's variable type to "int" (or any other) 
 it compiles just fine. So the problem seems to be the "double" 
 value.
That's because int is zero initialized by default and thus doesn't need anything more than a call to zero memory function, and double isn't (it is NaN), so it gets an initializer data blob. If you make it = 0 it might work, but even then, the compiler may want to reference the initializer just because there's a non-default init value!
 Should I still report it as a bug here:
It isn't a bug, this is working as designed [1]. Making it skip the initializer with =void or =0 explicit init values is a valid enhancement request though. 1: search for "default" on this page http://dlang.org/spec/struct.html#static_struct_init
Nov 27 2016
parent reply Timoses <timosesu gmail.com> writes:
On Sunday, 27 November 2016 at 14:27:54 UTC, Adam D. Ruppe wrote:
 That's because int is zero initialized by default and thus 
 doesn't need anything more than a call to zero memory function, 
 and double isn't (it is NaN), so it gets an initializer data 
 blob. If you make it = 0 it might work, but even then, the 
 compiler may want to reference the initializer just because 
 there's a non-default init value!
Does your answer also explain why it works when I move the mytypes.d into the source/ folder?
Nov 27 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 27 November 2016 at 15:23:33 UTC, Timoses wrote:
 On Sunday, 27 November 2016 at 14:27:54 UTC, Adam D. Ruppe 
 wrote:
 That's because int is zero initialized by default and thus 
 doesn't need anything more than a call to zero memory 
 function, and double isn't (it is NaN), so it gets an 
 initializer data blob. If you make it = 0 it might work, but 
 even then, the compiler may want to reference the initializer 
 just because there's a non-default init value!
Does your answer also explain why it works when I move the mytypes.d into the source/ folder?
dub will link it in when it is in the source folder.
Nov 27 2016
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 27 November 2016 at 15:56:13 UTC, Stefan Koch wrote:
 Does your answer also explain why it works when I move the 
 mytypes.d into the source/ folder?
dub will link it in when it is in the source folder.
exactly.
Nov 27 2016
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 27 November 2016 at 12:59:32 UTC, Timoses wrote:
 Why is it a linker problem? I'm not linking to the c interface 
 but merely using D structs...
It is a linker problem because you didn't link to it... D structs have an initializer, even if they are used for interfacing with C. This is a small blob of data called something like `_D10neo4jTypes17neo4j_map_entry_t6__initZ`, or can be all zeroes, and is generated with the interface module. So if you use the initializer from the interface module, but do not link it in, you get an undefined symbol. Easiest fix: just link in that .d file and use the automatic features. Why does your struct have an initializer data symbol instead of being all zeroes? I'm not entirely sure, but I think it is because the contents are structs and it didn't look inside those structs to see if they too are zeroes, it just assumed struct = fancy initialized. That's arguably an enhancement request, but I wouldn't call it a bug. You can often bypass the initializer by using `=0` or `=void` in definitions, but not here... and that is arguably a bug, if you `=void` all fields, it should be free to zero initialize the whole thing, but it doesn't. Why do you use the initializer? Look at each of these lines: neo4j_map_entry_t[] mapa2; // yes neo4j_map_entry_t entry1 = { key: neo4j_string("prop1"), value: neo4j_string("testprop1")}; // yes neo4j_map_entry_t entry2 = { key: neo4j_string("prop2"), value: neo4j_string("testprop2")}; // yes neo4j_map_entry_t* mapp; // yes Those work because nothing is default initialized: pointers and arrays start as null (and thus point to no actual struct data). The middle lines have you explicitly initializing it, so again, no default needed. // These don't neo4j_map_entry_t[2] mapa1; // no This is basically the same as `neo4j_map_entry_y a;` done twice - you create a struct, which is default initialized, which references that data. You can bypass this with `=void` at the end of that line. mapa2.length = 2; // no This also default initializes the new items you add. Just don't use that length function if you don't want to link in the other .d file. mapa2 ~= entry1; // no I think the implementation does length += 1, then copy arr[$-1] = entry1, so there's a temporary default in there. There's arguably a better implementation possible so this one *could* work. But if it doesn't now, just avoid that function (try something like std.array.uninitializedArray perhaps). neo4j_map_entry_t[] mapa3 = [{ key: neo4j_null, value: neo4j_null}]; // no And I don't know with this one, since I don't know what neo4j_null is. Perhaps you just didn't do the extern symbol right, or perhaps it too is a little data blob that should be linked in. But while this can be annoying, it isn't really a bug - it is D's auto initialize feature creating some data that you didn't link. Avoiding D-specific features and bypassing initialization with =void can skip it, but I recommend just linking in the module.
Nov 27 2016
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 27 November 2016 at 13:54:54 UTC, Adam D. Ruppe wrote:
 On Sunday, 27 November 2016 at 12:59:32 UTC, Timoses wrote:
 [...]
It is a linker problem because you didn't link to it... D structs have an initializer, even if they are used for interfacing with C. This is a small blob of data called something like `_D10neo4jTypes17neo4j_map_entry_t6__initZ`, or can be all zeroes, and is generated with the interface module. [...]
Unions are supposed to be default initialised with the first member, a ulong and thus should be zero initialised.
 You can often bypass the initializer by using `=0` or `=void` 
 in definitions, but not here... and that is arguably a bug, if 
 you `=void` all fields, it should be free to zero initialize 
 the whole thing, but it doesn't.

 [...]
Nov 27 2016
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 27 November 2016 at 23:25:57 UTC, Nicholas Wilson 
wrote:
 `_D10neo4jTypes17neo4j_map_entry_t6__initZ`, or
Unions are supposed to be default initialised with the first member, a ulong and thus should be zero initialised.
Right, but neo4j_map_entry_t is a struct and that's the one it is complaining about.
Nov 27 2016