www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dynamic library building and loading

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Haven't done any dynamic linking with D and I need to. I'm using dmd 
2.058/Linux at work to build and use dynamic libraries. Here's my attempt:

*** Makefile
all: main lib.so

main: main.d
	dmd main

lib.so: lib.d
	dmd -fPIC -shared lib.d -of./lib.so

*** lib.d
extern(C) int fun(string s)
{
     return 42;
}

*** main.d
import std.stdio;
void main()
{
}

Running make prints:

dmd -fPIC -shared lib.d -of./lib.so
/usr/bin/ld: 
/mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/li
phobos2.a(minfo.o): 
relocation R_X86_64_32 against `a local symbol' can not be used when 
making a shared object; recompile with -fPIC
/mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/rele
se/64/libphobos2.a: 
could not read symbols: Bad value
collect2: ld returned 1 exit status

What steps do I need to take to get off the ground?


Thanks,

Andrei
Sep 26 2012
next sibling parent reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
26.09.2012 21:58, Andrei Alexandrescu пишет:
 Haven't done any dynamic linking with D and I need to. I'm using dmd
 2.058/Linux at work to build and use dynamic libraries.

Sorry for the OT (as you are a Linux user), but if you will ever do it on Windows, be aware of this nasty trap: http://d.puremagic.com/issues/show_bug.cgi?id=8130 -- Денис В. Шеломовский Denis V. Shelomovskij
Sep 26 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/12 2:25 PM, Denis Shelomovskij wrote:
 26.09.2012 21:58, Andrei Alexandrescu пишет:
 Haven't done any dynamic linking with D and I need to. I'm using dmd
 2.058/Linux at work to build and use dynamic libraries.

Sorry for the OT (as you are a Linux user), but if you will ever do it on Windows, be aware of this nasty trap: http://d.puremagic.com/issues/show_bug.cgi?id=8130

Thanks! Andrei
Sep 26 2012
prev sibling next sibling parent reply "Michael" <pr m1xa.com> writes:
On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei
Alexandrescu wrote:
 Haven't done any dynamic linking with D and I need to. I'm 
 using dmd 2.058/Linux at work to build and use dynamic 
 libraries. Here's my attempt:

Maybe it will help you D: https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/symload.d Fortran: https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/simple.f95 Dmd 2.060 used. DLL on OpenSUSE 64bit compiled with command: gfortran -shared -fPIC -O3 -o simple.so simple.f95
Sep 26 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/12 2:59 PM, Michael wrote:
 On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei
 Alexandrescu wrote:
 Haven't done any dynamic linking with D and I need to. I'm using dmd
 2.058/Linux at work to build and use dynamic libraries. Here's my
 attempt:

Maybe it will help you D: https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/symload.d Fortran: https://bitbucket.org/AnyCPU/codewithd/raw/de0a2f0ea1af/simple.f95 Dmd 2.060 used. DLL on OpenSUSE 64bit compiled with command: gfortran -shared -fPIC -O3 -o simple.so simple.f95

Thanks. The loading part is very useful, but I'm still lost when it comes to build the shared library itself. Andrei
Sep 26 2012
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-09-26 19:58, Andrei Alexandrescu wrote:
 Haven't done any dynamic linking with D and I need to. I'm using dmd
 2.058/Linux at work to build and use dynamic libraries. Here's my attempt:

 Running make prints:

 dmd -fPIC -shared lib.d -of./lib.so
 /usr/bin/ld:
 /mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/libphobos2.a(minfo.o):
 relocation R_X86_64_32 against `a local symbol' can not be used when
 making a shared object; recompile with -fPIC
 /mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/libphobos2.a:
 could not read symbols: Bad value
 collect2: ld returned 1 exit status

 What steps do I need to take to get off the ground?

You need to properly implement support for dynamic libraries in druntime and DMD. I'm not entirely sure if there are some problems left in DMD, but druntime is definitely not ready yet. In general what needs to be done is to properly handle: * Module infos * TLS * Exception handling tables Martin Nowak has a branch for this: https://github.com/dawgfoto/druntime/tree/SharedRuntime -- /Jacob Carlborg
Sep 26 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/12 3:18 PM, Jacob Carlborg wrote:
 Martin Nowak has a branch for this:

 https://github.com/dawgfoto/druntime/tree/SharedRuntime

Thanks, I'll follow up with him. Andrei
Sep 26 2012
prev sibling next sibling parent "Michael" <pr m1xa.com> writes:
 Thanks. The loading part is very useful, but I'm still lost 
 when it comes to build the shared library itself.

 Andrei

Program loads dll at runtime using loader which is configured to load concrete dll file(s). Like in gtkD http://www.dsource.org/projects/gtkd/browser/trunk/src/gtkc/Loader.d
Sep 26 2012
prev sibling next sibling parent "nazriel" <spam dzfl.pl> writes:
On Wednesday, 26 September 2012 at 20:10:47 UTC, Michael wrote:
 Thanks. The loading part is very useful, but I'm still lost 
 when it comes to build the shared library itself.

 Andrei

Program loads dll at runtime using loader which is configured to load concrete dll file(s). Like in gtkD http://www.dsource.org/projects/gtkd/browser/trunk/src/gtkc/Loader.d

Loading Shared lib isn't big issues here. The bigger one is building Shared library (written in D) and running it in host application without issues (EH, shared GC etc). Andrei, if you find out how to make those things work, please share your findings. I'm also in need of using shared libraries. And yeah, probably Martin Nowak will be the best bet to get information from.
Sep 26 2012
prev sibling next sibling parent "Michael" <pr m1xa.com> writes:
 Loading Shared lib isn't big issues here.

 The bigger one is building Shared library (written in D) and 
 running it in host application without issues (EH, shared GC 
 etc).

 Andrei, if you find out how to make those things work, please 
 share your findings. I'm also in need of using shared libraries.

 And yeah, probably Martin Nowak will be the best bet to get 
 information from.

I understand. As I remember on forum were topic with same approach for dll written in D. But I can't find a link.
Sep 26 2012
prev sibling next sibling parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
You can build shared libraries on linux by manually compiling 
object files and linking them. On windows last time I tries it 
was not possible.
Sep 26 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-09-27 10:04, Maxim Fomin wrote:
 On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller
 wrote:
 Maxim Fomin wrote:
 You can build shared libraries on linux by manually compiling object
 files and linking them. On windows last time I tries it was not
 possible.

Can you give detailed steps for doing this on Linux? Because nobody as far as I know has made this work yet? Jens

Dpaste seems not working, so, sorry for code ----lib.d--- import std.stdio; static this() { writeln("module ctor"); } static ~this() { writeln("module dtor"); } class A { private string text;; this(string text) { writeln("class ctor"); this.text = text; } void tell() { writeln(this.text); } ~this() { writeln(this.text); writeln("dtor"); } static this() { writeln("static ctor"); } static ~this() { writeln("static dtor"); } } --------------- -----main.d---- import lib; void main() { auto a = new A("some text"); a.tell(); } --------------- dmd -c -fPIC lib.d gcc -shared lib.o -o liblib.so dmd -c main.d gcc main.o -llib -lphobos2 -lrt -lpthread -L. -Wl,-rpath=. ./a.out ldd a.out linux-vdso.so.1 (0x00007fff703ff000) liblib.so => ./liblib.so (0x00007f48158f1000) librt.so.1 => /lib64/librt.so.1 (0x00007f48156cd000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f48154b1000) libc.so.6 => /lib64/libc.so.6 (0x00007f481510c000) /lib64/ld-linux-x86-64.so.2 (0x00007f4815af4000)

1. Does this actually run? 2. This is statically linked with druntime and Phobos. What happens when you create an executable that links with the D dynamic library? Last time I tried this (on Mac OS X) I got several symbols missing. This was all symbols that are usually pointing to the executable, inserted by the compiler. One of them would be "main" and symbols like these: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27 -- /Jacob Carlborg
Sep 27 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-09-27 10:55, Maxim Fomin wrote:

 If it were non-runnable, I wouldn't posted it.

Ok, I see.
 Solution depends on a problem. I understood Andrei's post that he wanted
 a .so file or DLL. I told originally that it is possible to make shared
 libraries on linux. Now I see there is some misunderstanding. Is the
 problem in diving D application on executables and shared libraries or
 making druntime+phobos a shared library or loading a library at runtime?
 A was speaking about the first.

Obviously he wants to use the shared library, otherwise it would be rather useless and pointless. Last I checked there was a problem with: * Using a shared library written in D with a D application * Making druntime and Phobos shared libraries * Possibly also compiling shared libraries -- /Jacob Carlborg
Sep 27 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-09-27 12:02, Daniel Kozak wrote:

 With DMD 2.060 last two points are possible, only first point (Using a
 shared ...) I dont try

I really need to try this when I get home. -- /Jacob Carlborg
Sep 27 2012
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-09-27 10:55, Maxim Fomin wrote:
 On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg wrote:
 1. Does this actually run?

If it were non-runnable, I wouldn't posted it.
 2. This is statically linked with druntime and Phobos. What happens
 when you create an executable that links with the D dynamic library?

Solution depends on a problem. I understood Andrei's post that he wanted a .so file or DLL. I told originally that it is possible to make shared libraries on linux. Now I see there is some misunderstanding. Is the problem in diving D application on executables and shared libraries or making druntime+phobos a shared library or loading a library at runtime? A was speaking about the first.

Actually, I seriously doubt everything is working as expected. For example, what happens when an application loads (via dlopen or similar) a D dynamic library: * Are exception handlers registered * Are module infos properly registered * What happens if I call Object.factory, will that find a class in the dynamic library * Are GC sections registered * What happens when the library is unloaded, will all of the above be unregistered and cleaned up A quick look in rt.minfo in druntime shows the it uses a single array or linked list for all module infos. It expect the module infos to be in just one place, i.e. in the application. There's no handling when a dynamic library is loaded. Have a look at this: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/minfo.d#L184 That symbol points to the start of the linked list containing module infos. That's only a single linked list, no handling of module infos from multiple places. rt.minfo needs to be changed to use an associative array with the keys pointing to images (executables and loaded dynamic libraries) and the values mapped to module infos of the given image. It also needs to intercept when a library is dynamically loaded, extract the module infos and register it with the runtime. I would think that the same is true for exception handling tables, TLS and GC sections. -- /Jacob Carlborg
Sep 27 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/27/12 3:37 PM, Maxim Fomin wrote:
 Posted code doesn't load libraries at runtime, it is just linked to
 shared libraries.

Exactly! (I can't believe I'm starting to get the hang of this...) But what we ultimately need is true dynamic loading of never-seen modules. After the initial test I tried to load symbols with dlopen(). Indeed this works: auto p = dlopen("liblib.so", RTLD_LAZY); And it does work, all fine. To my dismay, as soon as I removed the flag "-llib" from the linker command line in the makefile, dlopen() didn't want to work anymore. So I think in order to enable "true" dynamic loading, I'll need to generate PIC for druntime and phobos, and then link liblib.so like this: dmd -fPIC -c lib.d gcc -shared lib.o -o liblib.so -L/path/to/phobos -lphobos2 Is that correct? At that point, of course, a variety of issues will need to be resolved as people pointed out. Andrei
Sep 27 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-09-28 01:21, Rob T wrote:
 For me to get C or C++ to run a D function, I had to do the following:

 // ====================
 // C/C++ library source
 // ====================

 // sample D function from library
 void foo(int i, int j, int k);
 // D runtime initialization & shutdown
 void init();
 void done();

 void bar()
 {
      foo(6, 7, 8);
 }

 int main(int argc, char **argv)
 {
 init();
      bar();
      done();
      return 0;
 }


 // ================
 // D library source
 // ================
 import std.stdio;
 static import core.runtime;

 // sample D function for test
 extern (C++) //    int foo(int i, int j, int k)
      void foo(int i, int j, int k)
      {
          writefln("i = %s", i);
          writefln("j = %s", j);
          writefln("k = %s", k);
          int t = i + j + k;
          writefln("Total = %s", t);
      }

 // Had to initialize and shutdown D system from C/C++.
 extern (C++) export void init() { // to be called once after loading
 shared lib
      core.runtime.Runtime.initialize();
 }

 extern (C++) export void done() { // to be called before unloading
 shared lib
      core.runtime.Runtime.terminate();
 }


 // I had to include main even though this is a library.
 int main()
 {
     return 0;
 }

That is not sufficient to have everything work. It might be ok if you limit yourself to a C subset of D. On some platforms, calling Runtime.initialize, will miss initializing some parts. The implementation of the runtime initialization contains a lot of code duplication and some functions/branches are missing functionality. Example: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/dmain2.d#L346 The C main function handles the runtime initialization for a regular statically linked executable. If you initialize the runtime via "rt_init", which is called by "Runtime.initialize", it will miss to initialize some parts on some platforms. The C main function should really call "rt_init" to remove the code duplication. Also see this post why everything will not properly work: http://forum.dlang.org/thread/k3vfm9$1tq$1 digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com -- /Jacob Carlborg
Sep 27 2012
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-09-28 20:25, Maxim Fomin wrote:

 I tried to check how TLS, EX, etc. (mostly exposed to dll issue) are
 working and here is some kind of test:
 https://github.com/mxfm/sharedtest. Unfortunately scope(exit) isn't
 executed when it is situated in a shared library and which calls some
 throwing function from another shared library. Unittests aren't working
 either. Regarding other parts - they seem to work.

That's a fairly uninteresting test. You are linking to the dynamic library. What's interesting is loading a dynamic library using dlopen, or similar. What's the point of using dynamic libraries if you're linking with them? -- /Jacob Carlborg
Sep 29 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-09-29 17:40, Maxim Fomin wrote:

 I was not taking about dynamic loading, but about dynamic linking. If I
 understand topic right, the issue is the former, not the latter.

The title of the thread says "... and loading".
 BTW, in majority cases dynamic loading gives no advantages over dynamic
 linking (just the opposite: doing unnecessary job which can be done by
 linker and loader). In most cases, when an app is being written, it is
 known which functional from which third-party libraries would be used.

I agree, in these cases their no point in using dynamic loading.
 The only exceptions I can name are pluggin support and hacker's binary
 ELF/PE tools. I can count few apps in windows and linux which are used
 often, which dynamically load something, and if they do, they likely
 load pluggins. This was made for cases when app developers by definition
 cannot know beforehand full list of used shared libraries.

Exactly, pluggins are one of the main reasons for using dynamic loading. An other reason is license issues. There's a D library called Derelict which uses dynamic loading to load C libraries (OpenGL, SDL, OpenAL and others). One of the many reason for Derelict using dynamic loading is due to license issues/complications with these libraries. http://dsource.org/projects/derelict/
 So, the question is just opposite: "What's the point of using dynamic loading
if
 you know beforehand which libraries you use, which happens almost in all
 cases?".

There's no point in doing that. But what I'm saying is if you know beforehand the libraries you will use you can get quite far with static libraries. -- /Jacob Carlborg
Sep 29 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-09-30 08:41, Rob T wrote:

 There are plenty of cases where you have to use a dynamically loaded lib
 even if you know before hand what will be loaded. I think you understand
 this if I read your posts correctly.

 In my case I have a pre-existing C++ app that is designed to load user
 defined C++ plugins. I wanted to interface D to one of my own C++
 plugins, but I apparently cannot do it reliably because of
 initialization issues with the GC and possibly some other obscure items.

But now we're back at plugins. I think this part of the discussion is starting to run in circles and become quite pointless. I think most of us know that we need to support all types of libraries. Static and dynamic, both for link time and runtime.
 If I can figure out what needs to be done to resolve the problem in
 enough detail, then maybe I can hack the runtime source and roll out a
 solution.

You can have a look at the work done by Martin Nowak: https://github.com/dawgfoto/druntime/commits/SharedRuntime He has a couple of other useful branches as well, for example: https://github.com/dawgfoto/druntime/commits/dmain2Refactoring On Mac OS X there's also the problem with TLS. There is no native support for TLS in Mac OS X prior to Lion (10.7). DMD has rolled its own implementation that needs to be adapted to support dynamic libraries.
 The GC always seems to pop up as a source of problems. For long term
 solution, the GC should be made 100% optional (in practice as opposed to
 in theory), the standard library ought to be made to work with or
 wothout a GC (or simply without), and the GC itself should be easily
 replaceable with alternate versions. I think this idea has already been
 discussed elsewhere, and is on the TODO list (I hope!).

You can replace the GC at link time. Here's an example of a stubbed implemented of the GC: http://dsource.org/projects/tango/browser/trunk/tango/core/rt/gc/stub This is for Tango but the druntime is based on the Tango runtime so I would guess most of this would be the same. -- /Jacob Carlborg
Sep 30 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-10-01 01:42, Rob T wrote:

 It seems that an attempt to make the runtime shared is well under way.
 Did anything get into the main dmd branch or has the effort been stalled
 or ...?

Seems pretty stalled.
 I will look at this too. Thanks for the pointers.

No problem. -- /Jacob Carlborg
Sep 30 2012
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-01 12:42, Johannes Pfau wrote:

 There are some reasons for dynamic libraries linked at compile time,
 one is that we have to start somewhere and they are required for
 plugins / dynamically loaded libraries as well ;-)

 So I started a small test suite for GDC (could be adapted to other
 compilers). It currently only tests compile time linking of dynamic
 libraries, but adjusting the test to use runtime loading should be
 easy. But it's pointless as long as we have no runtime support.
 https://github.com/jpf91/dso-test

 It should also be enhanced to test multiple shared libraries.

 The good news:
 * Exception handling is working
 * ModuleInfos are working
 * unit tests are working
 * Static variables, gshared variables, tls variables are working
 * Object.factory is working
 * Calling functions, passing function pointers, passing classes between
    dso/app is working

That's good. Have you tested this with DMD?
 The bad news:
 * The GC doesn't scan TLS/__gshared/static data in dynamic libraries,
    it only scans the main app.

That should be fairly trivial on Mac OS X. But I'm suspecting it won't be that easy on Linux. BTW, is the runtime and phobos statically linked both with the dynamic library and the executable? -- /Jacob Carlborg
Oct 01 2012
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-10-01 14:40, Iain Buclaw wrote:

 On Linux, there has already been an runtime implementation written
 that scans /proc/self/maps and adds all data sections to the GC that
 way.  Whether or not DMD wishes to go down that route is their own
 decision.  I am looking into a solution that doesn't have any bearing
 on what platform it's running on...

Well, /proc isn't available on Mac OS X so I think you have to continue looking. -- /Jacob Carlborg
Oct 01 2012
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-10-01 14:40, Iain Buclaw wrote:

 On Linux, there has already been an runtime implementation written
 that scans /proc/self/maps and adds all data sections to the GC that
 way.  Whether or not DMD wishes to go down that route is their own
 decision.  I am looking into a solution that doesn't have any bearing
 on what platform it's running on...

I think this is already working on Mac OS X. It's handled by: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memory_osx.d#L82 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memory_osx.d#L133 -- /Jacob Carlborg
Oct 01 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-10-01 17:03, Johannes Pfau wrote:

 I tested two different configurations:

 druntime and phobos are shared libraries as well (this is the correct
 solution, it's mostly working except for the GC issues)

I agree.
 druntime and phobos static linking: phobos and druntime are statically
 linked into the app, libdso.so is not linked against druntime/phobos at
 all. (this is a hack to get better test results: With a shared
 druntime, you can't call GC.collect twice, because the first call
 frees important objects in druntime and then the second call segfaults)

Ok. -- /Jacob Carlborg
Oct 01 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-01 12:42, Johannes Pfau wrote:

 There are some reasons for dynamic libraries linked at compile time,
 one is that we have to start somewhere and they are required for
 plugins / dynamically loaded libraries as well ;-)

 So I started a small test suite for GDC (could be adapted to other
 compilers). It currently only tests compile time linking of dynamic
 libraries, but adjusting the test to use runtime loading should be
 easy. But it's pointless as long as we have no runtime support.
 https://github.com/jpf91/dso-test

Isn't "dmain2" used when building shared libraries using GDC? That's where the implementation of "rt_init" is located. -- /Jacob Carlborg
Oct 01 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-10-01 17:06, Johannes Pfau wrote:

 the problem is that we don't want the C main function in a shared
 libgdruntime.so, because you might want to use libgdruntime.so in a
 C/C++ app which has it's own main function.

 So we currently don't link in dmain2.o into the shared library and it
 must be included manually when linking an application.
 (But dmain2 also contains some stuff that really should be in
 libdruntime.so, so this source file should probably be split up at some
 time.)

I'm not sure if I follow this correctly or not, but why is this needed to be handled manually? If you pass "-shared" to the compiler just skip linking dmain2.o, otherwise link with it. Would that work? -- /Jacob Carlborg
Oct 01 2012
prev sibling parent 1100110 <0b1100110 gmail.com> writes:
On 06/18/2013 10:14 AM, Bottled Gin wrote:
 Actually, I seriously doubt everything is working as expected. For
 example, what happens when an application loads (via dlopen or
 similar) a D dynamic library:

 * Are exception handlers registered
 * Are module infos properly registered
 * What happens if I call Object.factory, will that find a class in the
 dynamic library
 * Are GC sections registered
 * What happens when the library is unloaded, will all of the above be
 unregistered and cleaned up

Hello D Experts I am replying to an old thread since I want to know if and how the situation has improved over the past few months. In particular I want to link and call fairly complex D functions from a C++ application. I am exclusively using Linux. Can somebody please guide me if I should expect things to work with current DMD master from github? Regards - Puneet

There's a DConf talk about shared libraries in D. Perhaps there's useful information there? http://www.youtube.com/watch?v=i63VeudjZM4 (I'm watching it now so I don't know how detailed it is.)
Jun 18 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/27/12 6:26 AM, Maxim Fomin wrote:
[snip]

Thanks! I adapted your code as follows and it works with 2.058 on Centos.

*** lib.d
import std.stdio;

extern(C) int fun()
{
     writeln(", world!");
     return 42;
}

*** main.d
import std.stdio;

extern(C) int fun();

void main()
{
     write("Hello");
     fun();
}

*** Makefile

PHOBOS_PATH=/path/to/phobos/

main: main.o liblib.so
	gcc main.o -llib -lphobos2 -lpthread -lrt -L. -L${PHOBOS_PATH} 
-Wl,-rpath=. -o main

liblib.so: lib.d
	dmd -fPIC -c lib.d
	gcc -shared lib.o -o liblib.so

%.o: %.d
	dmd -c $<

clean:
	rm -f main main.o lib.o liblib.so

Building the makefile and running ./main prints the classic message. 
Thanks all for answering, this will get us going.


Andrei
Sep 27 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-09-27 20:25, Andrei Alexandrescu wrote:
 On 9/27/12 6:26 AM, Maxim Fomin wrote:
 [snip]

 Thanks! I adapted your code as follows and it works with 2.058 on Centos.

I seriously doubt that everything is working properly, have a look at my reply to Maxim Fomin: http://forum.dlang.org/thread/k3vfm9$1tq$1 digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com -- /Jacob Carlborg
Sep 27 2012
prev sibling next sibling parent Jens Mueller <jens.k.mueller gmx.de> writes:
Maxim Fomin wrote:
 You can build shared libraries on linux by manually compiling object
 files and linking them. On windows last time I tries it was not
 possible.

Can you give detailed steps for doing this on Linux? Because nobody as far as I know has made this work yet? Jens
Sep 26 2012
prev sibling next sibling parent reply "Daniel Kozak" <kozzi11 gmail.com> writes:
Hi,

I have same issue, but it is possible make shared library,
first of all you have to make shared variant of druntime and 
phobos library, than it should work ok.

Now I am at work, when I come back home I will post some more 
details about this.

Daniel Kozak

On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei 
Alexandrescu wrote:
 Haven't done any dynamic linking with D and I need to. I'm 
 using dmd 2.058/Linux at work to build and use dynamic 
 libraries. Here's my attempt:

 *** Makefile
 all: main lib.so

 main: main.d
 	dmd main

 lib.so: lib.d
 	dmd -fPIC -shared lib.d -of./lib.so

 *** lib.d
 extern(C) int fun(string s)
 {
     return 42;
 }

 *** main.d
 import std.stdio;
 void main()
 {
 }

 Running make prints:

 dmd -fPIC -shared lib.d -of./lib.so
 /usr/bin/ld: 
 /mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/li
phobos2.a(minfo.o): 
 relocation R_X86_64_32 against `a local symbol' can not be used 
 when making a shared object; recompile with -fPIC
 /mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/rele
se/64/libphobos2.a: 
 could not read symbols: Bad value
 collect2: ld returned 1 exit status

 What steps do I need to take to get off the ground?


 Thanks,

 Andrei

Sep 26 2012
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/27/12 2:26 AM, Daniel Kozak wrote:
 Now I try it, and it is not required to build shared variant of druntime
 and phobos, only rebuild it with -fPIC

Could you please send a troika composed of one dynlib, one loader using it, and a makefile that puts the all together? Thanks much! Andrei
Sep 27 2012
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-09-28 02:04, Rob T wrote:

 I'm using C++ main app that dynamically loads shared libs at runtime,
 but I cannot load D shared libs because they cannot be linked to the
 non-fPIC built phobos2/druntime lib. It seems that the only way to get
 things working is to re-build the C++ code so that it is linked with
 non-fPIC phobos2, but this is not something I want to be doing.

 So if I read this right, I can build phobos/druntime with -fPIC from the
 source code and it will work OK? If so, then why was this not done with
 the latest distrubution package, or is this only possible after a
 certain version number?

No, it will not work ok. Depending on what you do in the D code it might work, but everything will not work. Perhaps if you limit yourself to a C subset. See one of my other posts: http://forum.dlang.org/thread/k3vfm9$1tq$1 digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com -- /Jacob Carlborg
Sep 27 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-09-28 19:54, Rob T wrote:

 I suppose the answer is very complicated, but why can't the runtime
 execute as a shared lib? Is it a design limitation of the runtime model?
 This sort of problem should have been dealt with from the ground up, not
 now. What can be done to fix it, or is it a major redesign + re-write
 effort?

See one of my other posts: http://forum.dlang.org/thread/k3vfm9$1tq$1 digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com -- /Jacob Carlborg
Sep 29 2012
prev sibling next sibling parent "Daniel Kozak" <kozzi11 gmail.com> writes:
Now I try it, and it is not required to build shared variant of 
druntime and phobos, only rebuild it with -fPIC



On Thursday, 27 September 2012 at 06:12:38 UTC, Daniel Kozak 
wrote:
 Hi,

 I have same issue, but it is possible make shared library,
 first of all you have to make shared variant of druntime and 
 phobos library, than it should work ok.

 Now I am at work, when I come back home I will post some more 
 details about this.

 Daniel Kozak

 On Wednesday, 26 September 2012 at 17:57:29 UTC, Andrei 
 Alexandrescu wrote:
 Haven't done any dynamic linking with D and I need to. I'm 
 using dmd 2.058/Linux at work to build and use dynamic 
 libraries. Here's my attempt:

 *** Makefile
 all: main lib.so

 main: main.d
 	dmd main

 lib.so: lib.d
 	dmd -fPIC -shared lib.d -of./lib.so

 *** lib.d
 extern(C) int fun(string s)
 {
    return 42;
 }

 *** main.d
 import std.stdio;
 void main()
 {
 }

 Running make prints:

 dmd -fPIC -shared lib.d -of./lib.so
 /usr/bin/ld: 
 /mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/release/64/li
phobos2.a(minfo.o): 
 relocation R_X86_64_32 against `a local symbol' can not be 
 used when making a shared object; recompile with -fPIC
 /mnt/vol/engshare/third-party/centos5.2-native/dmd/dmd-2.058-centos5.2-native/bin/../../../../centos5.2-native/phobos/phobos-2.058/ffad884/generated/linux/rele
se/64/libphobos2.a: 
 could not read symbols: Bad value
 collect2: ld returned 1 exit status

 What steps do I need to take to get off the ground?


 Thanks,

 Andrei


Sep 26 2012
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Wednesday, 26 September 2012 at 20:15:35 UTC, nazriel wrote:
 On Wednesday, 26 September 2012 at 20:10:47 UTC, Michael wrote:
 Thanks. The loading part is very useful, but I'm still lost 
 when it comes to build the shared library itself.

 Andrei

Program loads dll at runtime using loader which is configured to load concrete dll file(s). Like in gtkD http://www.dsource.org/projects/gtkd/browser/trunk/src/gtkc/Loader.d

Loading Shared lib isn't big issues here. The bigger one is building Shared library (written in D) and running it in host application without issues (EH, shared GC etc). Andrei, if you find out how to make those things work, please share your findings. I'm also in need of using shared libraries. And yeah, probably Martin Nowak will be the best bet to get information from.

Native Oberon/BlueBottle OS really a lot in dynamic modules, as everything is dynamically loaded. As far as I know both operating systems have a global GC. Not sure how Spin does it. Singularity, uses local GC per process and does not support dynamic loading. Bringing this into D's context, I imagine a solution for the GC would be to have it shared across all modules. The biggest problem I see, is that all shared libraries have to use the same D compiler, or a standard GC ABI needs to be defined. -- Paulo
Sep 26 2012
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 27 Sep 2012 08:26:36 +0200
schrieb "Daniel Kozak" <kozzi11 gmail.com>:

 Now I try it, and it is not required to build shared variant of 
 druntime and phobos, only rebuild it with -fPIC
 

In the end you'll probably need a shared druntime & phobos: Let's say your main app doesn't use std.datetime and you statically link libphobos. Then the linker might strip std.datetime from your executable. If your shared library now needs std.datetime it won't work. The funny part is that on ARM+GDC we can at least build druntime & phobos as a shared library, but on x86/64 last time I checked it didn't work because of non-PIC assembly. https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support
Sep 27 2012
prev sibling next sibling parent "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller
wrote:
 Maxim Fomin wrote:
 You can build shared libraries on linux by manually compiling 
 object
 files and linking them. On windows last time I tries it was not
 possible.

Can you give detailed steps for doing this on Linux? Because nobody as far as I know has made this work yet? Jens

Dpaste seems not working, so, sorry for code ----lib.d--- import std.stdio; static this() { writeln("module ctor"); } static ~this() { writeln("module dtor"); } class A { private string text;; this(string text) { writeln("class ctor"); this.text = text; } void tell() { writeln(this.text); } ~this() { writeln(this.text); writeln("dtor"); } static this() { writeln("static ctor"); } static ~this() { writeln("static dtor"); } } --------------- -----main.d---- import lib; void main() { auto a = new A("some text"); a.tell(); } --------------- dmd -c -fPIC lib.d gcc -shared lib.o -o liblib.so dmd -c main.d gcc main.o -llib -lphobos2 -lrt -lpthread -L. -Wl,-rpath=. ./a.out ldd a.out linux-vdso.so.1 (0x00007fff703ff000) liblib.so => ./liblib.so (0x00007f48158f1000) librt.so.1 => /lib64/librt.so.1 (0x00007f48156cd000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f48154b1000) libc.so.6 => /lib64/libc.so.6 (0x00007f481510c000) /lib64/ld-linux-x86-64.so.2 (0x00007f4815af4000)
Sep 27 2012
prev sibling next sibling parent "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg 
wrote:
 1. Does this actually run?

If it were non-runnable, I wouldn't posted it.
 2. This is statically linked with druntime and Phobos. What 
 happens when you create an executable that links with the D 
 dynamic library?

Solution depends on a problem. I understood Andrei's post that he wanted a .so file or DLL. I told originally that it is possible to make shared libraries on linux. Now I see there is some misunderstanding. Is the problem in diving D application on executables and shared libraries or making druntime+phobos a shared library or loading a library at runtime? A was speaking about the first.
 Last time I tried this (on Mac OS X) I got several symbols 
 missing. This was all symbols that are usually pointing to the 
 executable, inserted by the compiler. One of them would be 
 "main" and symbols like these:

 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27

I have no idea about D support of shared libraries on Mac OS.
Sep 27 2012
prev sibling next sibling parent Jens Mueller <jens.k.mueller gmx.de> writes:
Jacob Carlborg wrote:
 On 2012-09-27 10:04, Maxim Fomin wrote:
On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller
wrote:
Maxim Fomin wrote:
You can build shared libraries on linux by manually compiling object
files and linking them. On windows last time I tries it was not
possible.

Can you give detailed steps for doing this on Linux? Because nobody as far as I know has made this work yet? Jens

Dpaste seems not working, so, sorry for code ----lib.d--- import std.stdio; static this() { writeln("module ctor"); } static ~this() { writeln("module dtor"); } class A { private string text;; this(string text) { writeln("class ctor"); this.text = text; } void tell() { writeln(this.text); } ~this() { writeln(this.text); writeln("dtor"); } static this() { writeln("static ctor"); } static ~this() { writeln("static dtor"); } } --------------- -----main.d---- import lib; void main() { auto a = new A("some text"); a.tell(); } --------------- dmd -c -fPIC lib.d gcc -shared lib.o -o liblib.so dmd -c main.d gcc main.o -llib -lphobos2 -lrt -lpthread -L. -Wl,-rpath=. ./a.out ldd a.out linux-vdso.so.1 (0x00007fff703ff000) liblib.so => ./liblib.so (0x00007f48158f1000) librt.so.1 => /lib64/librt.so.1 (0x00007f48156cd000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f48154b1000) libc.so.6 => /lib64/libc.so.6 (0x00007f481510c000) /lib64/ld-linux-x86-64.so.2 (0x00007f4815af4000)

1. Does this actually run?

I just tried. $ ./a.out module ctor static ctor class ctor some text static dtor module dtor some text dtor
 2. This is statically linked with druntime and Phobos. What happens
 when you create an executable that links with the D dynamic library?

a.out is linked dynamically against liblib.so.
 Last time I tried this (on Mac OS X) I got several symbols missing.
 This was all symbols that are usually pointing to the executable,
 inserted by the compiler. One of them would be "main" and symbols
 like these:
 
 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27

I'm running Linux. You can test on Mac OS X. I'm also astonished that this works. This is great. Jens
Sep 27 2012
prev sibling next sibling parent "Daniel Kozak" <kozzi11 gmail.com> writes:
On Thursday, 27 September 2012 at 09:16:48 UTC, Jacob Carlborg 
wrote:
 On 2012-09-27 10:55, Maxim Fomin wrote:

 If it were non-runnable, I wouldn't posted it.

Ok, I see.
 Solution depends on a problem. I understood Andrei's post that 
 he wanted
 a .so file or DLL. I told originally that it is possible to 
 make shared
 libraries on linux. Now I see there is some misunderstanding. 
 Is the
 problem in diving D application on executables and shared 
 libraries or
 making druntime+phobos a shared library or loading a library 
 at runtime?
 A was speaking about the first.

Obviously he wants to use the shared library, otherwise it would be rather useless and pointless. Last I checked there was a problem with: * Using a shared library written in D with a D application * Making druntime and Phobos shared libraries * Possibly also compiling shared libraries

With DMD 2.060 last two points are possible, only first point (Using a shared ...) I dont try
Sep 27 2012
prev sibling next sibling parent "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg
wrote:
 Last time I tried this (on Mac OS X) I got several symbols 
 missing. This was all symbols that are usually pointing to the 
 executable, inserted by the compiler. One of them would be 
 "main" and symbols like these:

 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/deh2.d#L27

"_deh_" problem AFAIK arise when non-D and D code is merged and can be avoided. For example, assume C/C++ application using D library. This can be done by writing D main function which forwards call to C/C++ main function which uses D shared library code. Again, sorry for code post, dpaste isn't working: ---main.cpp--- extern "C" void foo(void); #include <stdio.h> class CPP { public: CPP() { printf("CPP ctor\n"); } }; CPP cpp; extern "C" int c_main_func() { printf("C main function reached\n"); foo(); // C/C++ application return 0; } ---dmain.d--- import std.stdio; extern(C) int c_main_func(); static this() { writeln("main module ctor"); } static ~this() { writeln("main module dtor"); } int main() { return c_main_func(); } ---test.d--- import std.stdio; static this() { writeln("lib module ctor"); } static ~this() { writeln("lib module dtor"); } extern(C) void foo() { auto a = new A("data"); writeln("in foo"); } class A { string x; static this() { writeln("static ctor"); } static ~this() { writeln("static dtor"); } this(string x) { writeln("ctor"); this.x = x; } ~this() { writeln("dtor"); writeln(x); } } ------ # g++ main.cpp -c # dmd -c dmain.d # dmd -c test.d -fPIC # gcc -shared test.o -o libtest.so # gcc dmain.o main.o -ltest -lphobos2 -lpthread -lrt -L. -Wl,-rpath=. # ./a.out CPP ctor main module ctor lib module ctor static ctor C main function reached ctor in foo static dtor lib module dtor main module dtor dtor data # ldd ./a.out linux-vdso.so.1 (0x00007fff4cd55000) libtest.so => ./libtest.so (0x00007f970520b000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9704fd3000) librt.so.1 => /lib64/librt.so.1 (0x00007f9704dcb000) libc.so.6 => /lib64/libc.so.6 (0x00007f9704a26000) /lib64/ld-linux-x86-64.so.2 (0x00007f970540e000) #
Sep 27 2012
prev sibling next sibling parent "nazriel" <spam dzfl.pl> writes:
On Thursday, 27 September 2012 at 08:03:34 UTC, Maxim Fomin wrote:
 On Thursday, 27 September 2012 at 05:52:44 UTC, Jens Mueller
 wrote:
 Maxim Fomin wrote:
 You can build shared libraries on linux by manually compiling 
 object
 files and linking them. On windows last time I tries it was 
 not
 possible.

Can you give detailed steps for doing this on Linux? Because nobody as far as I know has made this work yet? Jens

Dpaste seems not working, so, sorry for code [snip]

Hmm, that looks nice, seems that stuff improved from last time I checked. Gonna test this out today. I have project that would gain from "Hot swapping" of Shared libraries. Temporarily solution I made was just writing plugins in C++. Btw, sorry for OT. What exactly doesn't work in Dpaste? It seems to work fine for me(TM). If those are UI glitches, try pressing F5 2-3 times ;D
Sep 27 2012
prev sibling next sibling parent "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 27 September 2012 at 13:00:24 UTC, nazriel wrote:
 Btw, sorry for OT.
 What exactly doesn't work in Dpaste? It seems to work fine for 
 me(TM).
 If those are UI glitches, try pressing F5 2-3 times ;D

Dpaste was completely down when I tried to post code. It was in the middle of today.
Sep 27 2012
prev sibling next sibling parent "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 27 September 2012 at 17:10:07 UTC, Jacob Carlborg 
wrote:
 On 2012-09-27 10:55, Maxim Fomin wrote:
 On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg 
 wrote:
 1. Does this actually run?

If it were non-runnable, I wouldn't posted it.
 2. This is statically linked with druntime and Phobos. What 
 happens
 when you create an executable that links with the D dynamic 
 library?

Solution depends on a problem. I understood Andrei's post that he wanted a .so file or DLL. I told originally that it is possible to make shared libraries on linux. Now I see there is some misunderstanding. Is the problem in diving D application on executables and shared libraries or making druntime+phobos a shared library or loading a library at runtime? A was speaking about the first.

Actually, I seriously doubt everything is working as expected. For example, what happens when an application loads (via dlopen or similar) a D dynamic library: * Are exception handlers registered * Are module infos properly registered * What happens if I call Object.factory, will that find a class in the dynamic library * Are GC sections registered * What happens when the library is unloaded, will all of the above be unregistered and cleaned up A quick look in rt.minfo in druntime shows the it uses a single array or linked list for all module infos. It expect the module infos to be in just one place, i.e. in the application. There's no handling when a dynamic library is loaded. Have a look at this: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/minfo.d#L184 That symbol points to the start of the linked list containing module infos. That's only a single linked list, no handling of module infos from multiple places. rt.minfo needs to be changed to use an associative array with the keys pointing to images (executables and loaded dynamic libraries) and the values mapped to module infos of the given image. It also needs to intercept when a library is dynamically loaded, extract the module infos and register it with the runtime. I would think that the same is true for exception handling tables, TLS and GC sections.

Posted code doesn't load libraries at runtime, it is just linked to shared libraries.
Sep 27 2012
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
For me to get C or C++ to run a D function, I had to do the 
following:

// ====================
// C/C++ library source
// ====================

// sample D function from library
void foo(int i, int j, int k);
// D runtime initialization & shutdown
void init();
void done();

void bar()
{
     foo(6, 7, 8);
}

int main(int argc, char **argv)
{
init();
	bar();
	done();
	return 0;
}


// ================
// D library source
// ================
import std.stdio;
static import core.runtime;

// sample D function for test
extern (C++) //	int foo(int i, int j, int k)
	void foo(int i, int j, int k)
	{
		writefln("i = %s", i);
		writefln("j = %s", j);
		writefln("k = %s", k);
		int t = i + j + k;
		writefln("Total = %s", t);
	}

// Had to initialize and shutdown D system from C/C++.
extern (C++) export void init() { // to be called once after 
loading shared lib
     core.runtime.Runtime.initialize();
}

extern (C++) export void done() { // to be called before 
unloading shared lib
     core.runtime.Runtime.terminate();
}


// I had to include main even though this is a library.
int main()
{
    return 0;
}
Sep 27 2012
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
On Thursday, 27 September 2012 at 07:54:29 UTC, Johannes Pfau 
wrote:
 Am Thu, 27 Sep 2012 08:26:36 +0200
 schrieb "Daniel Kozak" <kozzi11 gmail.com>:

 Now I try it, and it is not required to build shared variant 
 of druntime and phobos, only rebuild it with -fPIC
 

In the end you'll probably need a shared druntime & phobos: Let's say your main app doesn't use std.datetime and you statically link libphobos. Then the linker might strip std.datetime from your executable. If your shared library now needs std.datetime it won't work.

I'm using C++ main app that dynamically loads shared libs at runtime, but I cannot load D shared libs because they cannot be linked to the non-fPIC built phobos2/druntime lib. It seems that the only way to get things working is to re-build the C++ code so that it is linked with non-fPIC phobos2, but this is not something I want to be doing. So if I read this right, I can build phobos/druntime with -fPIC from the source code and it will work OK? If so, then why was this not done with the latest distrubution package, or is this only possible after a certain version number? --rt
Sep 27 2012
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
On Thursday, 27 September 2012 at 19:57:13 UTC, Andrei 
Alexandrescu wrote:
 So I think in order to enable "true" dynamic loading, I'll need 
 to generate PIC for druntime and phobos, and then link 
 liblib.so like this:

 dmd -fPIC -c lib.d
 gcc -shared lib.o -o liblib.so -L/path/to/phobos -lphobos2

 Is that correct?

Yes, that is correct. The other thing missing is a phobos2.so for dynamic runtime linking. -rt
Sep 27 2012
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 28 Sep 2012 02:04:11 +0200
schrieb "Rob T" <rob ucora.com>:

 
 So if I read this right, I can build phobos/druntime with -fPIC 
 from the source code and it will work OK? If so, then why was 
 this not done with the latest distrubution package, or is this 
 only possible after a certain version number?
 
 --rt

It shouldn't compile on x86 (32bit) with -fPIC because there's some incompatible asm code in druntime and phobos. If dmd doesn't warn in these cases, this code will fail as soon as it's called. GDC used to warn about these errors, but as we dropped the D inline assembler that code doesn't affect us anymore and there are no more warnings. The fix for this issue is simple: Do not modify the EBX register, or at least save & restore it. Old patch, which shows what needs to be done: https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329 Example asm code which probably doesn't work: https://github.com/D-Programming-Language/druntime/blob/master/src/core/cpuid.d#L432 I just tried building a shared druntime & phobos with gdc: Druntime seems to be working, Phobos doesn't link here because something's wrong with the fstat64 symbol. I haven't done any real testing and we probably need some additional support in the runtime, but the compiler part seems to be OK with gdc. http://gdcproject.org/wiki/SharedRuntime https://github.com/D-Programming-GDC/gdc/pull/32
Sep 28 2012
prev sibling next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
On 28 September 2012 12:14, Johannes Pfau <nospam example.com> wrote:
 Am Fri, 28 Sep 2012 02:04:11 +0200
 schrieb "Rob T" <rob ucora.com>:

 So if I read this right, I can build phobos/druntime with -fPIC
 from the source code and it will work OK? If so, then why was
 this not done with the latest distrubution package, or is this
 only possible after a certain version number?

 --rt

It shouldn't compile on x86 (32bit) with -fPIC because there's some incompatible asm code in druntime and phobos. If dmd doesn't warn in these cases, this code will fail as soon as it's called. GDC used to warn about these errors, but as we dropped the D inline assembler that code doesn't affect us anymore and there are no more warnings. The fix for this issue is simple: Do not modify the EBX register, or at least save & restore it.

In GDC D1 phobos, one hack around it was to use db to manually write out the bytecode instruction (thus bypassing GCC 'ebx' clobbered checks). Which was a fiendish workaround to say the least. :~) -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0';
Sep 28 2012
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
I am trying out gdc 4.7 branch with -fPIC, but the info I'm 
getting is that even with a successulf PIC build I will still not 
be able to reliably execute D functions directly from C/C++ code.

 Old patch, which shows what needs to be done:
 https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329

Does anyone know why patches like this are taking so [bleeping] long to get recognized and implemented? -rt
Sep 28 2012
prev sibling next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
On 28 September 2012 17:43, Rob T <rob ucora.com> wrote:
 I am trying out gdc 4.7 branch with -fPIC, but the info I'm getting is that
 even with a successulf PIC build I will still not be able to reliably
 execute D functions directly from C/C++ code.


 Old patch, which shows what needs to be done:

 https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329

Does anyone know why patches like this are taking so [bleeping] long to get recognized and implemented? -rt

Because the [bleeping] runtime does not support running from a shared library. -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0';
Sep 28 2012
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
On Friday, 28 September 2012 at 17:07:16 UTC, Iain Buclaw wrote:
 Because the [bleeping] runtime does not support running from a 
 shared library.

I suppose the answer is very complicated, but why can't the runtime execute as a shared lib? Is it a design limitation of the runtime model? This sort of problem should have been dealt with from the ground up, not now. What can be done to fix it, or is it a major redesign + re-write effort? -rt
Sep 28 2012
prev sibling next sibling parent "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 27 September 2012 at 17:10:07 UTC, Jacob Carlborg 
wrote:
 Actually, I seriously doubt everything is working as expected.

I tried to check how TLS, EX, etc. (mostly exposed to dll issue) are working and here is some kind of test: https://github.com/mxfm/sharedtest. Unfortunately scope(exit) isn't executed when it is situated in a shared library and which calls some throwing function from another shared library. Unittests aren't working either. Regarding other parts - they seem to work.
Sep 28 2012
prev sibling next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
On 28 September 2012 18:54, Rob T <rob ucora.com> wrote:
 On Friday, 28 September 2012 at 17:07:16 UTC, Iain Buclaw wrote:
 Because the [bleeping] runtime does not support running from a shared
 library.

I suppose the answer is very complicated, but why can't the runtime execute as a shared lib? Is it a design limitation of the runtime model? This sort of problem should have been dealt with from the ground up, not now. What can be done to fix it, or is it a major redesign + re-write effort? -rt

The big implementation issue is that the runtime only passes two data ranges to the GC to scan. Global data segment and TLS data segment of the resultant binary/executable. So any static data stored in shared libraries aren't scanned / recognised as data we want to keep. So the use of them may cause sudden unwanted collections. The way I intend to address it is to have each module handle it's own gshared/thread data and pass the information to the D runtime during the module construction stage (.ctor) - there is already something done this way for _Dmodule_ref - so it may end up being that two new fields will be tacked onto it; void[] tlsdata, void[] gsharedata; Regards -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0';
Sep 28 2012
prev sibling next sibling parent "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Saturday, 29 September 2012 at 13:19:01 UTC, Jacob Carlborg 
wrote:
 That's a fairly uninteresting test.

I am not a D developer which means I have no incentive in blindly portraying D as a language having shared libraries support when in fact it has some issues. I am a D user which has incentive to gladly report which part of D does work (or seems to work) and which doesn't when making shared libraries.
 You are linking to the dynamic library. What's interesting is 
 loading a dynamic library using dlopen, or similar. What's the 
 point of using dynamic libraries if you're linking with them?

I was not taking about dynamic loading, but about dynamic linking. If I understand topic right, the issue is the former, not the latter. BTW, in majority cases dynamic loading gives no advantages over dynamic linking (just the opposite: doing unnecessary job which can be done by linker and loader). In most cases, when an app is being written, it is known which functional from which third-party libraries would be used. The only exceptions I can name are pluggin support and hacker's binary ELF/PE tools. I can count few apps in windows and linux which are used often, which dynamically load something, and if they do, they likely load pluggins. This was made for cases when app developers by definition cannot know beforehand full list of used shared libraries. So, the question is just opposite: "What's the point of using dynamic loading if you know beforehand which libraries you use, which happens almost in all cases?". But I certainly agree that dynamic loading is essential for some applications, if not irreplaceable, and support of it also needed to be inspected and improved.
Sep 29 2012
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
On Saturday, 29 September 2012 at 16:09:14 UTC, Jacob Carlborg 
wrote:
 There's no point in doing that. But what I'm saying is if you 
 know beforehand the libraries you will use you can get quite 
 far with static libraries.

There are plenty of cases where you have to use a dynamically loaded lib even if you know before hand what will be loaded. I think you understand this if I read your posts correctly. In my case I have a pre-existing C++ app that is designed to load user defined C++ plugins. I wanted to interface D to one of my own C++ plugins, but I apparently cannot do it reliably because of initialization issues with the GC and possibly some other obscure items. If I can figure out what needs to be done to resolve the problem in enough detail, then maybe I can hack the runtime source and roll out a solution. The GC always seems to pop up as a source of problems. For long term solution, the GC should be made 100% optional (in practice as opposed to in theory), the standard library ought to be made to work with or wothout a GC (or simply without), and the GC itself should be easily replaceable with alternate versions. I think this idea has already been discussed elsewhere, and is on the TODO list (I hope!). --rt
Sep 29 2012
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
On Friday, 28 September 2012 at 21:26:47 UTC, Iain Buclaw wrote:
 The way I intend to address it is to have each module handle 
 it's own
 gshared/thread data and pass the information to the D runtime 
 during
 the module construction stage (.ctor) - there is already 
 something
 done this way for _Dmodule_ref - so it may end up being that 
 two new
 fields will be tacked onto it; void[] tlsdata, void[] 
 gsharedata;

I re-built libgphobos and libgdruntime with -fPIC and I can now successfully create dynamically loaded D libs. I have successfully linked a dynamic D lib to a C++ app, executing some test code successfully. I have not yet tried to dlopen a dynamic D lib from C++, but I will try maybe tomorrow. My simple dynamic lib test seems to run fine, but I understand that there may be problems, such as the GC failing to work properly, and perhaps more can/will go wrong as per Jacob's post http://forum.dlang.org/post/k4219f$uft$1 digitalmars.com Any idea when/if you will get around to implementing a fix? Wish I could help but I've only just started looking at the source code, so whatever I try to fix will probably cause more harm than good for a while (but it's a start). --rt
Sep 30 2012
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
On Sunday, 30 September 2012 at 10:57:24 UTC, Jacob Carlborg 
wrote:
 On 2012-09-30 08:41, Rob T wrote:

 I think most of us know that we need to support all types of 
 libraries. Static and dynamic, both for link time and runtime.

OK we're all in agreement on this point.
 You can have a look at the work done by Martin Nowak:

 https://github.com/dawgfoto/druntime/commits/SharedRuntime

It seems that an attempt to make the runtime shared is well under way. Did anything get into the main dmd branch or has the effort been stalled or ...? I've only recently been learning what's going on with D and why it is what it is, so a breif history lesson may be needed once in a while.
 You can replace the GC at link time. Here's an example of a 
 stubbed implemented of the GC:
 http://dsource.org/projects/tango/browser/trunk/tango/core/rt/gc/stub

I will look at this too. Thanks for the pointers. --rt
Sep 30 2012
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Sat, 29 Sep 2012 15:19:30 +0200
schrieb Jacob Carlborg <doob me.com>:

 On 2012-09-28 20:25, Maxim Fomin wrote:
 
 I tried to check how TLS, EX, etc. (mostly exposed to dll issue) are
 working and here is some kind of test:
 https://github.com/mxfm/sharedtest. Unfortunately scope(exit) isn't
 executed when it is situated in a shared library and which calls
 some throwing function from another shared library. Unittests
 aren't working either. Regarding other parts - they seem to work.

That's a fairly uninteresting test. You are linking to the dynamic library. What's interesting is loading a dynamic library using dlopen, or similar. What's the point of using dynamic libraries if you're linking with them?

There are some reasons for dynamic libraries linked at compile time, one is that we have to start somewhere and they are required for plugins / dynamically loaded libraries as well ;-) So I started a small test suite for GDC (could be adapted to other compilers). It currently only tests compile time linking of dynamic libraries, but adjusting the test to use runtime loading should be easy. But it's pointless as long as we have no runtime support. https://github.com/jpf91/dso-test It should also be enhanced to test multiple shared libraries. The good news: * Exception handling is working * ModuleInfos are working * unit tests are working * Static variables, gshared variables, tls variables are working * Object.factory is working * Calling functions, passing function pointers, passing classes between dso/app is working The bad news: * The GC doesn't scan TLS/__gshared/static data in dynamic libraries, it only scans the main app.
Oct 01 2012
prev sibling next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
On 1 October 2012 13:34, Jacob Carlborg <doob me.com> wrote:
 On 2012-10-01 12:42, Johannes Pfau wrote:

 There are some reasons for dynamic libraries linked at compile time,
 one is that we have to start somewhere and they are required for
 plugins / dynamically loaded libraries as well ;-)

 So I started a small test suite for GDC (could be adapted to other
 compilers). It currently only tests compile time linking of dynamic
 libraries, but adjusting the test to use runtime loading should be
 easy. But it's pointless as long as we have no runtime support.
 https://github.com/jpf91/dso-test

 It should also be enhanced to test multiple shared libraries.

 The good news:
 * Exception handling is working
 * ModuleInfos are working
 * unit tests are working
 * Static variables, gshared variables, tls variables are working
 * Object.factory is working
 * Calling functions, passing function pointers, passing classes between
    dso/app is working

That's good. Have you tested this with DMD?
 The bad news:
 * The GC doesn't scan TLS/__gshared/static data in dynamic libraries,
    it only scans the main app.

That should be fairly trivial on Mac OS X. But I'm suspecting it won't be that easy on Linux. BTW, is the runtime and phobos statically linked both with the dynamic library and the executable? -- /Jacob Carlborg

On Linux, there has already been an runtime implementation written that scans /proc/self/maps and adds all data sections to the GC that way. Whether or not DMD wishes to go down that route is their own decision. I am looking into a solution that doesn't have any bearing on what platform it's running on... Regards -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0';
Oct 01 2012
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Mon, 01 Oct 2012 14:34:21 +0200
schrieb Jacob Carlborg <doob me.com>:

 On 2012-10-01 12:42, Johannes Pfau wrote:
 
 There are some reasons for dynamic libraries linked at compile time,
 one is that we have to start somewhere and they are required for
 plugins / dynamically loaded libraries as well ;-)

 So I started a small test suite for GDC (could be adapted to other
 compilers). It currently only tests compile time linking of dynamic
 libraries, but adjusting the test to use runtime loading should be
 easy. But it's pointless as long as we have no runtime support.
 https://github.com/jpf91/dso-test

 It should also be enhanced to test multiple shared libraries.

 The good news:
 * Exception handling is working
 * ModuleInfos are working
 * unit tests are working
 * Static variables, gshared variables, tls variables are working
 * Object.factory is working
 * Calling functions, passing function pointers, passing classes
 between dso/app is working

That's good. Have you tested this with DMD?

Not yet.
 
 The bad news:
 * The GC doesn't scan TLS/__gshared/static data in dynamic
 libraries, it only scans the main app.

That should be fairly trivial on Mac OS X. But I'm suspecting it won't be that easy on Linux. BTW, is the runtime and phobos statically linked both with the dynamic library and the executable?

I tested two different configurations: druntime and phobos are shared libraries as well (this is the correct solution, it's mostly working except for the GC issues) druntime and phobos static linking: phobos and druntime are statically linked into the app, libdso.so is not linked against druntime/phobos at all. (this is a hack to get better test results: With a shared druntime, you can't call GC.collect twice, because the first call frees important objects in druntime and then the second call segfaults)
Oct 01 2012
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Mon, 01 Oct 2012 15:11:49 +0200
schrieb Jacob Carlborg <doob me.com>:

 On 2012-10-01 12:42, Johannes Pfau wrote:
 
 There are some reasons for dynamic libraries linked at compile time,
 one is that we have to start somewhere and they are required for
 plugins / dynamically loaded libraries as well ;-)

 So I started a small test suite for GDC (could be adapted to other
 compilers). It currently only tests compile time linking of dynamic
 libraries, but adjusting the test to use runtime loading should be
 easy. But it's pointless as long as we have no runtime support.
 https://github.com/jpf91/dso-test

Isn't "dmain2" used when building shared libraries using GDC? That's where the implementation of "rt_init" is located.

the problem is that we don't want the C main function in a shared libgdruntime.so, because you might want to use libgdruntime.so in a C/C++ app which has it's own main function. So we currently don't link in dmain2.o into the shared library and it must be included manually when linking an application. (But dmain2 also contains some stuff that really should be in libdruntime.so, so this source file should probably be split up at some time.)
Oct 01 2012
prev sibling next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
On 1 October 2012 16:06, Johannes Pfau <nospam example.com> wrote:
 Am Mon, 01 Oct 2012 15:11:49 +0200
 schrieb Jacob Carlborg <doob me.com>:

 On 2012-10-01 12:42, Johannes Pfau wrote:

 There are some reasons for dynamic libraries linked at compile time,
 one is that we have to start somewhere and they are required for
 plugins / dynamically loaded libraries as well ;-)

 So I started a small test suite for GDC (could be adapted to other
 compilers). It currently only tests compile time linking of dynamic
 libraries, but adjusting the test to use runtime loading should be
 easy. But it's pointless as long as we have no runtime support.
 https://github.com/jpf91/dso-test

Isn't "dmain2" used when building shared libraries using GDC? That's where the implementation of "rt_init" is located.

the problem is that we don't want the C main function in a shared libgdruntime.so, because you might want to use libgdruntime.so in a C/C++ app which has it's own main function.

That is at least one of the theories behind it anyway. :-) The more I think about it, the less I think I would want a C++ app to link against a D shared library though. With C, there will need to be defined a common interface header - similar to what used to be mars.h in rt/ but also contain some useful runtime functions. Regards -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0';
Oct 01 2012
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Mon, 01 Oct 2012 19:18:46 +0200
schrieb Jacob Carlborg <doob me.com>:

 On 2012-10-01 17:06, Johannes Pfau wrote:
 
 the problem is that we don't want the C main function in a shared
 libgdruntime.so, because you might want to use libgdruntime.so in a
 C/C++ app which has it's own main function.

 So we currently don't link in dmain2.o into the shared library and
 it must be included manually when linking an application.
 (But dmain2 also contains some stuff that really should be in
 libdruntime.so, so this source file should probably be split up at
 some time.)

I'm not sure if I follow this correctly or not, but why is this needed to be handled manually? If you pass "-shared" to the compiler just skip linking dmain2.o, otherwise link with it. Would that work?

Yes something similar would work, it's just not yet implemented. GDC should detect if we're linking against a shared druntime and then it should automatically add dmain2.o to the linker command.
Oct 02 2012
prev sibling next sibling parent "Bottled Gin" <gin bottled.com> writes:
 Actually, I seriously doubt everything is working as expected. 
 For example, what happens when an application loads (via dlopen 
 or similar) a D dynamic library:

 * Are exception handlers registered
 * Are module infos properly registered
 * What happens if I call Object.factory, will that find a class 
 in the dynamic library
 * Are GC sections registered
 * What happens when the library is unloaded, will all of the 
 above be unregistered and cleaned up

Hello D Experts I am replying to an old thread since I want to know if and how the situation has improved over the past few months. In particular I want to link and call fairly complex D functions from a C++ application. I am exclusively using Linux. Can somebody please guide me if I should expect things to work with current DMD master from github? Regards - Puneet
Jun 18 2013
prev sibling parent "Etienne" <etcimon gmail.com> writes:
On Tuesday, 18 June 2013 at 20:23:42 UTC, 1100110 wrote:
 On 06/18/2013 10:14 AM, Bottled Gin wrote:
 Actually, I seriously doubt everything is working as 
 expected. For
 example, what happens when an application loads (via dlopen or
 similar) a D dynamic library:

 * Are exception handlers registered
 * Are module infos properly registered
 * What happens if I call Object.factory, will that find a 
 class in the
 dynamic library
 * Are GC sections registered
 * What happens when the library is unloaded, will all of the 
 above be
 unregistered and cleaned up

Hello D Experts I am replying to an old thread since I want to know if and how the situation has improved over the past few months. In particular I want to link and call fairly complex D functions from a C++ application. I am exclusively using Linux. Can somebody please guide me if I should expect things to work with current DMD master from github? Regards - Puneet

There's a DConf talk about shared libraries in D. Perhaps there's useful information there? http://www.youtube.com/watch?v=i63VeudjZM4 (I'm watching it now so I don't know how detailed it is.)

Hello all, I'm wondering if this has been resolved to date. I get this error on master: /usr/bin/ld: generated/linux/release/64/libphobos2.so.0.66.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC generated/linux/release/64/libphobos2.so.0.66.o: could not read symbols: Bad value collect2: error: ld returned 1 exit status even though I've built with fPIC: cc -c -m64 -fPIC -O3 etc/c/zlib/trees.c -ogenerated/linux/release/64/etc/c/zlib/trees.o cc -c -m64 -fPIC -O3 etc/c/zlib/uncompr.c -ogenerated/linux/release/64/etc/c/zlib/uncompr.o cc -c -m64 -fPIC -O3 etc/c/zlib/zutil.c -ogenerated/linux/release/64/etc/c/zlib/zutil.o etc...
Mar 24 2014