www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How to compile Phobos with other D code to create a shared library?

reply data pulverizer <data.pulverizer gmail.com> writes:
Hi,

I am trying to compile D code to a shared library to be called by 
another language (in this case Julia). I can get a basic call to 
work but I can't call functions that use Phobos (I have no idea 
how to compile against it). I have found some documentation 
[here](https://github.com/mesonbuild/meson/issues/6786) and 
[here](https://dlang.org/articles/dll-linux.html) that lists 
compilation against specific flags so for instance I have tried:

```
ldc2 jbasic.d -O3 -link-defaultlib-shared --betterC 
--boundscheck=off -nogc -shared -of=jbasic.so
```

But in Julia (and the equivalent thing happens in  R), when I 
attempt to load the function I get an error:

```
ERROR: could not load library "./jbasic.so"
./jbasic.so: undefined symbol: 
_D3std6random__T21MersenneTwisterEngineTmVmi64Vmi312Vmi156Vmi31VmN5403634167711393303Vmi29Vmi6148914691236517205Vmi17Vmi8202884508482404352Vmi37VmN2270628950310912Vmi43Vmi6364136223846793005ZQGt6__initZ
```

I tried the equivalent with `dmd`:

```
dmd jbasic.d -O -defaultlib=libphobos2.so -betterC -fPIC 
-boundscheck=off  -shared -of=jbasic.so
```

with the same result.


Thank you.
May 31 2021
parent reply kinke <noone nowhere.com> writes:
On Monday, 31 May 2021 at 19:21:52 UTC, data pulverizer wrote:

 ldc2 jbasic.d -O3 -link-defaultlib-shared --betterC 
 --boundscheck=off -nogc -shared -of=jbasic.so


The problem is almost certainly `-betterC`, which disables 
linking against Phobos and druntime.
May 31 2021
parent reply data pulverizer <data.pulverizer gmail.com> writes:
On Monday, 31 May 2021 at 20:32:11 UTC, kinke wrote:

 On Monday, 31 May 2021 at 19:21:52 UTC, data pulverizer wrote:

 ldc2 jbasic.d -O3 -link-defaultlib-shared --betterC 
 --boundscheck=off -nogc -shared -of=jbasic.so


 The problem is almost certainly `-betterC`, which disables 
 linking against Phobos and druntime.


Thanks - I should have know that one! I've removed the flag but 
now I'm getting the following error when I try to call the 
function:

```
Aborting from src/core/time.d(2131) 
MonoTimeImpl!(ClockType.normal) failed to get the frequency of 
the system's monotonic clock.
signal (6): Aborted
in expression starting at REPL[2]:1
gsignal at x86_64-linux-gnu/libc.so.6 (unknown line)
abort at x86_64-linux-gnu/libc.so.6 (unknown line)
_D4core8internal5abortQgFNbNiNfMAyaMQemZv at 
/lib64/libphobos2.so.0.90 (unknown line)
_D4core4time__T12MonoTimeImplVEQBdQBb9ClockTypei0ZQBj8currTimeFNbNdNiNeZSQC
QCr__TQCpVQCei0ZQCz at dmd/current/lib64/libphobos2.so.0.90 (unknown line)
_D3std6random13bootstrapSeedFNbNiZm at 
dmd/current/lib64/libphobos2.so.0.90 (unknown line)
_D3std6random__T17unpredictableSeedTmZQwFNbNdNiNeZm at 
dmd/current/lib64/libphobos2.so.0.90 (unknown line)
_D3std6random17unpredictableSeedFNbNdNiNeZk at 
dmd/current/lib64/libphobos2.so.0.90 (unknown line)

```
May 31 2021
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/31/21 4:41 PM, data pulverizer wrote:

 On Monday, 31 May 2021 at 20:32:11 UTC, kinke wrote:

 On Monday, 31 May 2021 at 19:21:52 UTC, data pulverizer wrote:

 ldc2 jbasic.d -O3 -link-defaultlib-shared --betterC --boundscheck=off 
 -nogc -shared -of=jbasic.so


 The problem is almost certainly `-betterC`, which disables linking 
 against Phobos and druntime.


 
 Thanks - I should have know that one! I've removed the flag but now I'm 
 getting the following error when I try to call the function:
 
 ```
 Aborting from src/core/time.d(2131) MonoTimeImpl!(ClockType.normal) 
 failed to get the frequency of the system's monotonic clock.
 signal (6): Aborted
 in expression starting at REPL[2]:1
 gsignal at x86_64-linux-gnu/libc.so.6 (unknown line)
 abort at x86_64-linux-gnu/libc.so.6 (unknown line)
 _D4core8internal5abortQgFNbNiNfMAyaMQemZv at /lib64/libphobos2.so.0.90 
 (unknown line)
 _D4core4time__T12MonoTimeImplVEQBdQBb9ClockTypei0ZQBj8currTimeFNbNdNiNeZSQC
QCr__TQCpVQCei0ZQCz 
 at dmd/current/lib64/libphobos2.so.0.90 (unknown line)
 _D3std6random13bootstrapSeedFNbNiZm at 
 dmd/current/lib64/libphobos2.so.0.90 (unknown line)
 _D3std6random__T17unpredictableSeedTmZQwFNbNdNiNeZm at 
 dmd/current/lib64/libphobos2.so.0.90 (unknown line)
 _D3std6random17unpredictableSeedFNbNdNiNeZk at 
 dmd/current/lib64/libphobos2.so.0.90 (unknown line)
 
 ```


ticksPerSecond is initialized in the runtime just before static 
constructors are run. See 
https://github.com/dlang/druntime/blob/2d8b28da39e8bc3bc3172c69bb96c35d77f40d2a/src/rt/dmain2.d#L130

Are you calling Runtime.initialize()?

-Steve
May 31 2021
parent reply data pulverizer <data.pulverizer gmail.com> writes:
On Monday, 31 May 2021 at 21:01:19 UTC, Steven Schveighoffer 
wrote:

 ticksPerSecond is initialized in the runtime just before static 
 constructors are run. See 
 https://github.com/dlang/druntime/blob/2d8b28da39e8bc3bc3172c69bb96c35d77f40d2a/src/rt/dmain2.d#L130

 Are you calling Runtime.initialize()?



Nope, I guess I'm supposed to be? if so where do I place the 
call(s) for initialize() and terminate()?
May 31 2021
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/31/21 5:20 PM, data pulverizer wrote:

 On Monday, 31 May 2021 at 21:01:19 UTC, Steven Schveighoffer wrote:

 ticksPerSecond is initialized in the runtime just before static 
 constructors are run. See 
 https://github.com/dlang/druntime/blob/2d8b28da39e8bc3bc3172c69bb96c35d77f40d2a/
rc/rt/dmain2.d#L130 


 Are you calling Runtime.initialize()?


 
 
 Nope, I guess I'm supposed to be? if so where do I place the call(s) for 
 initialize() and terminate()?
 
 


You need to call it wherever you think it might not have been called yet.

It's reentrant, so if you call it more than once, it will only 
initialize once, and count how many times you have to call 
`Runtime.terminate`.

Best to use a `scope(exit)` to call `Runtime.terminate` if you are 
calling it periodically.

-Steve
May 31 2021
parent reply data pulverizer <data.pulverizer gmail.com> writes:
On Monday, 31 May 2021 at 21:26:15 UTC, Steven Schveighoffer 
wrote:

 You need to call it wherever you think it might not have been 
 called yet.

 It's reentrant, so if you call it more than once, it will only 
 initialize once, and count how many times you have to call 
 `Runtime.terminate`.

 Best to use a `scope(exit)` to call `Runtime.terminate` if you 
 are calling it periodically.


Many thanks!

Something interesting is using arrays. I can see that if you 
instantiate an array within the D function using `new`, for 
instance

```
auto result = new double[n];
```

you can't return that array to Julia (or whatever) if your 
function does

```
scope(exit) Runtime.terminate();
```

and it segfaults. But while you can do:

```
auto result = cast(double*)malloc(double.sizeof * n);
```

you're left with the issue of not being able to free the memory 
since you've terminated the runtime.

So I guess you have to appropriately pair up `initialize` and 
`terminate` as appropriate to kind of *startup* and and 
*shutdown* the connection with druntime with your allocations if 
they stay within D?
May 31 2021
next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 31 May 2021 at 21:46:09 UTC, data pulverizer wrote:

 Something interesting is using arrays. I can see that if you 
 instantiate an array within the D function using `new`, for 
 instance


Passing one of those to a C function is iffy anyway because the C 
function can hide it from the garbage collector.

What happens here is Runtime.terminate does a final GC sweep to 
gracefully clean up as much as it can before it close... and it 
thinks that array is unused, so it frees it.

GC.addRoot(array) can tell it not to free normally... but i'm not 
sure if runtime.terminate cares about that since terminate makes 
it think the whole thing is going down anyway.


 auto result = cast(double*)malloc(double.sizeof * n);
 ```

 you're left with the issue of not being able to free the memory 
 since you've terminated the runtime.


You can free still, malloc and free are not controlled by the D 
runtime.

This is sometimes a better idea anyway when passing it to C 
function. I don't know how Julia does it though.


 So I guess you have to appropriately pair up `initialize` and 
 `terminate` as appropriate to kind of *startup* and and 
 *shutdown* the connection with druntime with your allocations 
 if they stay within D?


But yeah the best way to do it is to initialize once on library 
load then terminate once when the whole thing is finished.

Typically C plugin systems have some hook for this you can use. 
Like some onload function they define you can use or something.
May 31 2021
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/31/21 5:46 PM, data pulverizer wrote:

 On Monday, 31 May 2021 at 21:26:15 UTC, Steven Schveighoffer wrote:

 You need to call it wherever you think it might not have been called yet.

 It's reentrant, so if you call it more than once, it will only 
 initialize once, and count how many times you have to call 
 `Runtime.terminate`.

 Best to use a `scope(exit)` to call `Runtime.terminate` if you are 
 calling it periodically.


 
 Many thanks!
 
 Something interesting is using arrays. I can see that if you instantiate 
 an array within the D function using `new`, for instance
 
 ```
 auto result = new double[n];
 ```
 
 you can't return that array to Julia (or whatever) if your function does
 
 ```
 scope(exit) Runtime.terminate();
 ```
 
 and it segfaults. But while you can do:
 
 ```
 auto result = cast(double*)malloc(double.sizeof * n);
 ```
 
 you're left with the issue of not being able to free the memory since 
 you've terminated the runtime.
 
 So I guess you have to appropriately pair up `initialize` and 
 `terminate` as appropriate to kind of *startup* and and *shutdown* the 
 connection with druntime with your allocations if they stay within D?
 


You should use whatever Julia is expecting for memory management. I 
don't know what that is, but typically you want to use your host 
language's memory management system to create blocks instead of the D GC.

-Steve
Jun 01 2021
parent reply data pulverizer <data.pulverizer gmail.com> writes:
Doing `Runtime.initialize` is working with Julia but not yet R, 
I'm getting a clock/GLIBC error

```
Error in dyn.load("rbasic.so") :
   unable to load shared object 'code/rbasic.so':
   lib/x86_64-linux-gnu/librt.so.1: undefined symbol: 
__clock_nanosleep, version GLIBC_PRIVATE
```

do I need to import anything else?
Jun 01 2021
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/1/21 2:18 PM, data pulverizer wrote:

 Doing `Runtime.initialize` is working with Julia but not yet R, I'm 
 getting a clock/GLIBC error
 
 ```
 Error in dyn.load("rbasic.so") :
    unable to load shared object 'code/rbasic.so':
    lib/x86_64-linux-gnu/librt.so.1: undefined symbol: __clock_nanosleep, 
 version GLIBC_PRIVATE
 ```
 
 do I need to import anything else?


What's happening is that the dynamic linker is trying to resolve that 
symbol, but cannot find it in the given library.

Try `ldd code/rbasic.so` and see if it tells you the things it is 
looking for. Many times, you will see "Not found" or something and you 
know what library to install/look for.

Otherwise, it might be that librt.so.1 seemed to have __clock_nanosleep 
when it was built, but the librt.so.1 it is loading during runtime does 
not have it. Are you building on one machine and then running on another?

-Steve
Jun 02 2021
parent data pulverizer <data.pulverizer gmail.com> writes:
On Wednesday, 2 June 2021 at 17:49:49 UTC, Steven Schveighoffer 
wrote:

 What's happening is that the dynamic linker is trying to 
 resolve that symbol, but cannot find it in the given library.

 Try `ldd code/rbasic.so` and see if it tells you the things it 
 is looking for. Many times, you will see "Not found" or 
 something and you know what library to install/look for.


I ran `ldd`, it looks as if all the symbols are resolved

```
$ ldd rbasic.so
	linux-vdso.so.1 (0x00007ffe7af56000)
	libR.so => /lib/libR.so (0x00007f269adb8000)
	libdruntime-ldc-shared.so.94 => 
/snap/ldc2/170/bin/../lib/libdruntime-ldc-shared.so.94 
(0x00007f269ac0c000)
	libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 
(0x00007f269920d000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f26990be000)
	libreadline.so.8 => /lib/x86_64-linux-gnu/libreadline.so.8 
(0x00007f269906e000)
	libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 
(0x00007f2698fdc000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 
(0x00007f2698fb3000)
	libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 
(0x00007f2698fa0000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f2698f84000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
(0x00007f2698f7e000)
	libicuuc.so.66 => /lib/x86_64-linux-gnu/libicuuc.so.66 
(0x00007f2698d98000)
	libicui18n.so.66 => /lib/x86_64-linux-gnu/libicui18n.so.66 
(0x00007f2698a97000)
	libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 
(0x00007f2698a55000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x00007f2698a32000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2698840000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f269b29e000)
	librt.so.1 => /snap/core/current/lib/x86_64-linux-gnu/librt.so.1 
(0x00007f2698638000)
	libgcc_s.so.1 => 
/snap/core/current/lib/x86_64-linux-gnu/libgcc_s.so.1 
(0x00007f2698420000)
	libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 
(0x00007f26983f0000)
	libicudata.so.66 => /lib/x86_64-linux-gnu/libicudata.so.66 
(0x00007f269692f000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 
(0x00007f269674e000)
```


 Otherwise, it might be that librt.so.1 seemed to have 
 __clock_nanosleep when it was built, but the librt.so.1 it is 
 loading during runtime does not have it. Are you building on 
 one machine and then running on another?


I am building and running on the same machine.
Jun 02 2021
prev sibling parent reply bachmeier <no spam.net> writes:
On Tuesday, 1 June 2021 at 18:18:35 UTC, data pulverizer wrote:

 Doing `Runtime.initialize` is working with Julia but not yet R, 
 I'm getting a clock/GLIBC error

 ```
 Error in dyn.load("rbasic.so") :
   unable to load shared object 'code/rbasic.so':
   lib/x86_64-linux-gnu/librt.so.1: undefined symbol: 
 __clock_nanosleep, version GLIBC_PRIVATE
 ```

 do I need to import anything else?


Are you aware of my embedr project, which handles all that for 
you, and does a lot of other stuff?

https://embedr.netlify.app

If you want to do it yourself, you can see the boilerplate you 
need to add to call a shared library from R here: 
https://bitbucket.org/bachmeil/embedr/src/bebf67e5b30cd4163214c7f44f9907e7d7490dc0/R/compile.R#lines-99

If you want to read the R manual, the relevant section is here:
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#dyn_002eload-and-dyn_002eunload
If you don't do that, it's unlikely to work unless you get lucky. 
If you want to load foo.so, you need a corresponding R_init_foo 
function where you call Runtime.initialize.
Jun 02 2021
parent reply data pulverizer <data.pulverizer gmail.com> writes:
On Wednesday, 2 June 2021 at 23:23:58 UTC, bachmeier wrote:

 Are you aware of my embedr project, which handles all that for 
 you, and does a lot of other stuff?

 https://embedr.netlify.app

 If you want to do it yourself, you can see the boilerplate you 
 need to add to call a shared library from R here: 
 https://bitbucket.org/bachmeil/embedr/src/bebf67e5b30cd4163214c7f44f9907e7d7490dc0/R/compile.R#lines-99

 If you want to read the R manual, the relevant section is here:
 https://cran.r-project.org/doc/manuals/r-release/R-exts.html#dyn_002eload-and-dyn_002eunload
 If you don't do that, it's unlikely to work unless you get 
 lucky. If you want to load foo.so, you need a corresponding 
 R_init_foo function where you call Runtime.initialize.


Thanks. Looks like I have some more reading to do. I did know 
about embedr, but I saw it had dependencies and I wanted a 
standalone and fully transparent D solution. I'm already calling 
R routines from D using standalone Rmath and being able to call D 
from R and D have independent full access to R's internals would 
be awesome. But your package certainly provides very useful 
information.

The reference to R's dyn.load internals looks useful and relevant 
and I'll be trying those once I get the time. Probably during the 
weekend.
Jun 04 2021
parent reply bachmeier <no spam.net> writes:
On Friday, 4 June 2021 at 07:26:53 UTC, data pulverizer wrote:


 Thanks. Looks like I have some more reading to do. I did know 
 about embedr, but I saw it had dependencies and I wanted a 
 standalone and fully transparent D solution.


It requires an R package if you want to call D functions from R. 
You need to link to R itself if you want to do something like 
pass a vector from R to D and then access that data from D. Since 
R is aware of the location of all the underlying libraries, it's 
easiest to let it generate the dub.sdl for you.

I guess you're using the older .C interface [as done 
here](http://users.stat.umn.edu/~geyer/rc/). Good enough if 
passing double* and int* pointers to D functions will suffice. 
Only thing you need to do to initialize the D runtime is add this 
to your D code

```
struct DllInfo;

extern(C) void R_init_libfoo(DllInfo * info) {
     Runtime.initialize();
}
```

where foo is the name of the library you're loading (foo.so).
Jun 04 2021
next sibling parent reply Alain De Vos <devosalain ymail.com> writes:
Dub is probably not much of a help :)
Jun 04 2021
parent bachmeier <no spam.net> writes:
On Friday, 4 June 2021 at 15:33:32 UTC, Alain De Vos wrote:

 Dub is probably not much of a help :)


That's right. I typically don't use Dub when I'm calling D 
functions from R. It's the only way you can use a Dub package 
like Mir, though, so that's why you might want it to generate a 
dub.sdl for you.
Jun 04 2021
prev sibling parent data pulverizer <data.pulverizer gmail.com> writes:
On Friday, 4 June 2021 at 15:19:17 UTC, bachmeier wrote:

 It requires an R package if you want to call D functions from 
 R. You need to link to R itself if you want to do something 
 like pass a vector from R to D and then access that data from 
 D. Since R is aware of the location of all the underlying 
 libraries, it's easiest to let it generate the dub.sdl for you.

 I guess you're using the older .C interface [as done 
 here](http://users.stat.umn.edu/~geyer/rc/). Good enough if 
 passing double* and int* pointers to D functions will suffice. 
 Only thing you need to do to initialize the D runtime is add 
 this to your D code

 ```
 struct DllInfo;

 extern(C) void R_init_libfoo(DllInfo * info) {
     Runtime.initialize();
 }
 ```

 where foo is the name of the library you're loading (foo.so).


I've just read the documentation and had a go, **compiling with 
GDC worked for both `.C` and `.Call`**.

I tried with all three compilers DMD, LDC, and GDC, and looked at 
the outputs with the nm tool. I don't think this matters much but 
DMD and LDC are more likely to list functions (DMD) or symbols in 
referenced modules (LDC) as weak symbols. I think the more likely 
the reason GDC worked is because it is a GNU compiler this has 
full GLIBC compatibility with reference to the `librt.so.1` 
`GLIBC_PRIVATE` error.

The R ext documentation you kindly referenced says:

*"By default, R uses the operating-system-specific dynamic loader 
to lookup the symbol in all loaded DLLs and the R executable or 
libraries it is linked to."*

and that:

*"Registering routines has two main advantages: it provides a 
faster way to find the address of the entry point via tables 
stored in the DLL at compilation time, and it provides a run-time 
check that the entry point is called with the right number of 
arguments and, optionally, the right argument types."*

So registering symbols is not the issue. The error was specific 
to GLIBC which is probably why the GNU compiler worked.

Thank you.
Jun 05 2021