www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Compiling to shared library with static dependencies

reply Tanel L <liivtanel gmail.com> writes:
Hello,
I am very new to the D world - but serious in moving over to it 
after I have seen what a cool language it really is.
I am trying to incorporate D into our main product as an 
externally loadable module.

Basically I have a minimal project:

./app.d
./helpers/image.d (mir and dcv libraries imported)
./helpers/utils.d

I am using the latest (statically compiled) ldc2

In my app file(besides the void main) I have a test function 
which I am trying to call from python using ctypes:
extern (C) int doit()

Currently I have gotten as far as getting a loading error in 
python:
library.so: undefined symbol: _d_eh_personality

Googling disclosed that this is satisfied by libdruntime.so, but 
I don't have a shared libdruntime anywhere. I guess I have to 
compile ldc2 in dynamic mode and use the libdruntime.so from 
there?

My build steps until now:

dub.sdl (pseudocode):
   project_description "nudesc"
   dependency "mir"
   dependency "dcv"
   targetPath "lib"
   dflags "-relocation-mode=pic"

dub build --compiler ldc2
cp .dub/obj/*.o lib/
cd lib
ldc2 -of library.so helpers.image.o helpers.utils.o nudesc.o 
-shared -defaultlib=phobos2

strace shows that ldc2 loads "phobos2" from DMD: 
"/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libphobos2.so"

The LDC2 folder only seems to contain libphobos2-ldc.a, no .so's.
Compilation succeeds, but trying to load the so from python:

python  -c "from ctypes import *; CDLL('library.so', 
mode=RTLD_GLOBAL)"
OSError: library.so: undefined symbol: _d_eh_personality

I figured that I don't have druntime.so available because 
druntime is incorporated into LDC2, but mine is statically 
compiled (downloaded precompiled from github i think).

So I tried to compile LDC2 dynamically:
cmake .. -DBUILD_SHARED=ON

But I get a bunch of these errors before the LDC2 build fails:
/home/tanel/thirdparty/ldc-dynamic/build/bin/ldc2: error while 
loading shared libraries: libldc.so: cannot open shared object 
file: No such file or directory
runtime/CMakeFiles/druntime-ldc.dir/build.make:93: recipe for 
target 'runtime/src/core/bitop.o' failed
make[2]: *** [runtime/src/core/bitop.o] Error 127


Basically I am cornered... I really need to be able to compile 
dcv and mir into my dynamic libraries. Probably LDC2 is a must 
because DMD and GDC have given me compilation errors on the 
project even in regular mode. Also LDC2 seems to be the best.

Any ideas what to do or how to proceed?
Thanks a bunch!
Dec 14 2016
next sibling parent reply Relja Ljubobratovic <ljubobratovic.relja gmail.com> writes:
On Wednesday, 14 December 2016 at 12:33:04 UTC, Tanel L wrote:

 Hello,


Hi Tanel,

I have just tried replicating the task you describe, and it 
worked flawlessly for me.


 I am very new to the D world - but serious in moving over to it 
 after I have seen what a cool language it really is.
 I am trying to incorporate D into our main product as an 
 externally loadable module.

 Basically I have a minimal project:

 ./app.d
 ./helpers/image.d (mir and dcv libraries imported)
 ./helpers/utils.d

 I am using the latest (statically compiled) ldc2

 In my app file(besides the void main) I have a test function 
 which I am trying to call from python using ctypes:
 extern (C) int doit()


Probably a dumb question - have you tried compiling your library 
without dependencies, and using it through ctypes? Does this work?


 Currently I have gotten as far as getting a loading error in 
 python:
 library.so: undefined symbol: _d_eh_personality

 Googling disclosed that this is satisfied by libdruntime.so, 
 but I don't have a shared libdruntime anywhere. I guess I have 
 to compile ldc2 in dynamic mode and use the libdruntime.so from 
 there?

 My build steps until now:

 dub.sdl (pseudocode):
   project_description "nudesc"
   dependency "mir"
   dependency "dcv"
   targetPath "lib"
   dflags "-relocation-mode=pic"

 dub build --compiler ldc2
 cp .dub/obj/*.o lib/
 cd lib
 ldc2 -of library.so helpers.image.o helpers.utils.o nudesc.o 
 -shared -defaultlib=phobos2


Why are you linking manually? Why not let dub do the job? Also, I 
believe if you're adding dcv as dependency, you don't need to add 
mir, since mir is the dependency of dcv. (it has worked for me 
this way so far) but this has nothing to do with errors you 
describe.


 strace shows that ldc2 loads "phobos2" from DMD: 
 "/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libphobos2.so"

 The LDC2 folder only seems to contain libphobos2-ldc.a, no 
 .so's.
 Compilation succeeds, but trying to load the so from python:

 python  -c "from ctypes import *; CDLL('library.so', 
 mode=RTLD_GLOBAL)"
 OSError: library.so: undefined symbol: _d_eh_personality

 I figured that I don't have druntime.so available because 
 druntime is incorporated into LDC2, but mine is statically 
 compiled (downloaded precompiled from github i think).


Again, dumb one (and I'm quite sure this is not the issue) - have 
you initialized druntime in the doit function?


 So I tried to compile LDC2 dynamically:
 cmake .. -DBUILD_SHARED=ON

 But I get a bunch of these errors before the LDC2 build fails:
 /home/tanel/thirdparty/ldc-dynamic/build/bin/ldc2: error while 
 loading shared libraries: libldc.so: cannot open shared object 
 file: No such file or directory
 runtime/CMakeFiles/druntime-ldc.dir/build.make:93: recipe for 
 target 'runtime/src/core/bitop.o' failed
 make[2]: *** [runtime/src/core/bitop.o] Error 127


 Basically I am cornered... I really need to be able to compile 
 dcv and mir into my dynamic libraries. Probably LDC2 is a must 
 because DMD and GDC have given me compilation errors on the 
 project even in regular mode. Also LDC2 seems to be the best.

 Any ideas what to do or how to proceed?
 Thanks a bunch!


What is your configuration - OS, ldc and gcc version etc.? 
Anyhow, I'm surely not the best person to help here - I hope some 
of more experienced guys will drop a word to help you out...

And for the end, have you tried creating the python module with 
pyd[1]?

Cheers,
Relja

[1] https://github.com/ariovistus/pyd
Dec 14 2016
parent reply Tanel L <liivtanel gmail.com> writes:
Hi, thanks for the answer.

I had tried disabling all imports, but now I created a clean new 
project to test this - it worked.

After that I moved the compiling and linking over to DUB, with 
dependencies:

dependency "dcv" version="0.1.7"
dependency "mir" version="0.22.0"
targetPath "output"
targetType "dynamicLibrary"
dflags "-defaultlib=phobos2"

The library source:
import std.stdio;
import dcv;
import core.runtime:Runtime;
extern (C) int doit(int a, int b) {
	return a*b + dcv.imread("1.jpg").width.to!int;
}

This created an .so with huge amount of all kinds of external so 
deps, but it still failed with the same error:
OSError: libdynlibtest.so: undefined symbol: _d_eh_personality

Otherwise:
Linux Mint 18 (basically Ubuntu 16.04)
LDC - the LLVM D compiler (1.1.0-beta6):
   based on DMD v2.071.2 and LLVM 3.9.0
   built with LDC - the LLVM D compiler (1.0.0)
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
DUB version 1.1.1, built on Nov 30 2016

Thank you for helping! :)
Dec 14 2016
parent reply Relja Ljubobratovic <ljubobratovic.relja gmail.com> writes:
On Wednesday, 14 December 2016 at 20:46:15 UTC, Tanel L wrote:

 Hi, thanks for the answer.

 I had tried disabling all imports, but now I created a clean 
 new project to test this - it worked.


Awesome. So LDC compile-link is ok, and also python runtime 
linking is good.


 After that I moved the compiling and linking over to DUB, with 
 dependencies:

 dependency "dcv" version="0.1.7"
 dependency "mir" version="0.22.0"
 targetPath "output"
 targetType "dynamicLibrary"
 dflags "-defaultlib=phobos2"


Why are you defining defaultlib explicitly here? - wouldn't this 
work fine without it? Also, like I said before, try removing mir 
from dependencies since its included with dcv.


 The library source:
 import std.stdio;
 import dcv;
 import core.runtime:Runtime;
 extern (C) int doit(int a, int b) {
 	return a*b + dcv.imread("1.jpg").width.to!int;
 }


Did you actually call Runtime.initialize before calling doit? 
Also, did you try the same code in ordinary D app? Anyways, this 
works for me:

D source
========
import std.stdio;
import core.runtime;

extern (C) void initd()
{
     Runtime.initialize;
}

extern (C) void terminated()
{
     Runtime.terminate();
}

extern (C) void showim()
{
     import dcv;
     auto im = imread("img.jpg");
     imshow(im);
     waitKey();
}
========

Python source
========
from ctypes import *

d = CDLL("libctypestest.so")
d.initd()
d.showim()
d.terminated()
========


 This created an .so with huge amount of all kinds of external 
 so deps, but it still failed with the same error:
 OSError: libdynlibtest.so: undefined symbol: _d_eh_personality


All kinds of external deps are expected since dcv is linking to 
ggplotd and ffmpeg, which also link to whole lotta libraries. I'm 
not sure about the binary size though. If you can, use only 
dcv:core which has no other dependencies other than Mir. Also, 
dcv:io links to ffmpeg at the moment, but if you need only image 
i/o and would like to avoid linking to ffmpeg, you could use just 
dcv:core with imageformats[1] (dcv wraps it's io methods in 
imread). If you need help with this let me know.


 Otherwise:
 Linux Mint 18 (basically Ubuntu 16.04)
 LDC - the LLVM D compiler (1.1.0-beta6):
   based on DMD v2.071.2 and LLVM 3.9.0
   built with LDC - the LLVM D compiler (1.0.0)
 gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
 DUB version 1.1.1, built on Nov 30 2016

 Thank you for helping! :)


Hey, no sweat - its a pleasure! I'm really glad you'd like to use 
D (and Mir and DCV for that matter). I'd really like to help you 
out as much as I can, so this can work out for you! :)

Cheers,
Relja
Dec 14 2016
next sibling parent Relja Ljubobratovic <ljubobratovic.relja gmail.com> writes:
On Wednesday, 14 December 2016 at 21:41:53 UTC, Relja 
Ljubobratovic wrote:

 use just dcv:core with imageformats[1]


http://code.dlang.org/packages/imageformats
Dec 14 2016
prev sibling parent reply Tanel L <liivtanel gmail.com> writes:
First of all thanks Johan for urging me to compile it with only 
with that flag. It worked! Previously I tried to compile it with 
only BUILD_SHARED or both - and that failed. But I got it 
working. Thanks! More details below.

On Wednesday, 14 December 2016 at 21:41:53 UTC, Relja 
Ljubobratovic wrote:

 On Wednesday, 14 December 2016 at 20:46:15 UTC, Tanel L wrote:

 Hi, thanks for the answer.

 I had tried disabling all imports, but now I created a clean 
 new project to test this - it worked.


 Awesome. So LDC compile-link is ok, and also python runtime 
 linking is good.


 After that I moved the compiling and linking over to DUB, with 
 dependencies:

 dependency "dcv" version="0.1.7"
 dependency "mir" version="0.22.0"
 targetPath "output"
 targetType "dynamicLibrary"
 dflags "-defaultlib=phobos2"


 Why are you defining defaultlib explicitly here? - wouldn't 
 this work fine without it? Also, like I said before, try 
 removing mir from dependencies since its included with dcv.


Yes it probably would, call it inexperience :) I'm now much 
smarter b/c I got it working :D

 The library source:
 import std.stdio;
 import dcv;
 import core.runtime:Runtime;
 extern (C) int doit(int a, int b) {
 	return a*b + dcv.imread("1.jpg").width.to!int;
 }


 Did you actually call Runtime.initialize before calling doit? 
 Also, did you try the same code in ordinary D app? Anyways, 
 this works for me:


No, I will use your example as an example for the future, but It 
seems it is not required. Although I really did it with 
Initialize, I just forgot it from the sample I pasted here - it 
didn't make a difference.

 D source
 ========
 import std.stdio;
 import core.runtime;

 extern (C) void initd()
 {
     Runtime.initialize;
 }

 extern (C) void terminated()
 {
     Runtime.terminate();
 }

 extern (C) void showim()
 {
     import dcv;
     auto im = imread("img.jpg");
     imshow(im);
     waitKey();
 }
 ========


Thanks, I will use this as a template in the future.


 Python source
 ========
 from ctypes import *

 d = CDLL("libctypestest.so")
 d.initd()
 d.showim()
 d.terminated()
 ========


 This created an .so with huge amount of all kinds of external 
 so deps, but it still failed with the same error:
 OSError: libdynlibtest.so: undefined symbol: _d_eh_personality


 All kinds of external deps are expected since dcv is linking to 
 ggplotd and ffmpeg, which also link to whole lotta libraries. 
 I'm not sure about the binary size though. If you can, use only 
 dcv:core which has no other dependencies other than Mir. Also, 
 dcv:io links to ffmpeg at the moment, but if you need only 
 image i/o and would like to avoid linking to ffmpeg, you could 
 use just dcv:core with imageformats[1] (dcv wraps it's io 
 methods in imread). If you need help with this let me know.


Thanks, I don't need ffmpeg and I will surely use your advice. I 
am not worried about the size much. Is there any way to include 
many/most of them into the binary? Or should I just ship the deps 
alongside the shared lib?

 Otherwise:
 Linux Mint 18 (basically Ubuntu 16.04)
 LDC - the LLVM D compiler (1.1.0-beta6):
   based on DMD v2.071.2 and LLVM 3.9.0
   built with LDC - the LLVM D compiler (1.0.0)
 gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
 DUB version 1.1.1, built on Nov 30 2016

 Thank you for helping! :)


 Hey, no sweat - its a pleasure! I'm really glad you'd like to 
 use D (and Mir and DCV for that matter). I'd really like to 
 help you out as much as I can, so this can work out for you! :)

 Cheers,
 Relja


So I got it working.
Going to leave a mark behind about what I did incase someone else 
runs into this.

First of all, I successfully compiled ldc2 with:
cmake .. -DBUILD_SHARED_LIBS=ON
make -j 7

I then removed the installed "dmd" DEB package because it would 
always link with its libphobos2.so. I installed only "dub" from 
the apt repo instead.

Added the "lflags" param to dub, to have a final dub.sdl:
targetPath "output"
targetType "dynamicLibrary"
--dflags "-defaultlib=phobos2-ldc"
lflags "-L/home/tanel/thirdparty/ldc-dynamic-working/build/lib/"

Mucked around figuring out why dub enforces a 
"-defaultlib=phobos2" argument to ldc2 - didn't figure it out. 
LDC won't compile because it doesn't exists and if I add my own 
to dub.sdl LDC whines that the flag is declared twice.
So i created a symlink in the dynamic ldc lib dir and removed the 
"dflags" from my .sdl:
ln -s libphobos2-ldc.so libphobos2.so

Then it still couldn't find it, so I discovered that I had used 
LD_LIBRARY_PATH, instead of LIBRARY_PATH - doh! Fixed it.
In .bashrc:
export 
LIBRARY_PATH=/home/tanel/thirdparty/ldc-dynamic-working/build/lib/

$ dub run --config=library --compiler=ldc2
$ python -c "from ctypes import *; print 
CDLL('libdynlibtest.so').doit(10,10)"

 2020


Success!
Thank you both so much. With this I can successfully move forward 
with using D in our company :)
Dec 14 2016
parent Relja Ljubobratovic <ljubobratovic.relja gmail.com> writes:
On Wednesday, 14 December 2016 at 22:17:25 UTC, Tanel L wrote:

 First of all thanks Johan for urging me to compile it with only 
 with that flag. It worked! Previously I tried to compile it 
 with only BUILD_SHARED or both - and that failed. But I got it 
 working. Thanks! More details below.


Awesome!


 Thanks, I will use this as a template in the future.


Please don't :) - consider other options (D is full of them, I'm 
sure you'll find much more elegant solution), I pulled this just 
as a quick test!


 Thanks, I don't need ffmpeg and I will surely use your advice. 
 I am not worried about the size much. Is there any way to 
 include many/most of them into the binary? Or should I just 
 ship the deps alongside the shared lib?


You could build ffmpeg for static linking, but I'm not sure about 
ggplotd's deps - I suppose there's a way I just haven't done it 
before.


 Success!
 Thank you both so much. With this I can successfully move 
 forward with using D in our company :)


Congrats! - if it's possible keep us posted of your endeavors for 
which you're using Mir and DCV! And of course, feel free to make 
a question, bug report, feature request etc at our repositories:
https://github.com/libmir/mir
https://github.com/libmir/dcv

All of the best,
Relja
Dec 14 2016
prev sibling next sibling parent Johan Engelen <j j.nl> writes:
On Wednesday, 14 December 2016 at 12:33:04 UTC, Tanel L wrote:

 So I tried to compile LDC2 dynamically:
 cmake .. -DBUILD_SHARED=ON


Have you tried `cmake .. -DBUILD_SHARED_LIBS=ON` ?

-Johan
Dec 14 2016
prev sibling parent Dominikus Dittes Scherkl <Dominikus.Scherkl continental-corporation.com> writes:
On Wednesday, 14 December 2016 at 12:33:04 UTC, Tanel L wrote:

 Any ideas what to do or how to proceed?
 Thanks a bunch!


This kind of questions is better adressed to the "Learn" forum.
Dec 14 2016