www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - shared libraries in D

reply Iain Buclaw <ibuclaw ubuntu.com> writes:
So I've been prototyping, and have built a fully working shared D2
druntime/phobos library on Linux (will come to caveats in a moment). Just for
sake of visual proof.

[iain natty gdc]$ cat hello.d
import std.stdio;
void main()
{
    writeln("Hello World");
}
[iain natty gdc]$ gdc hello.d -lrt -ldl # Never needed these before...
[iain natty gdc]$ du -hs a.out
8.0K	a.out
[iain natty gdc]$ ldd a.out
	linux-gate.so.1 =>  (0x00a32000)
	librt.so.1 => /lib/librt.so.1 (0x00c32000)
	libdl.so.2 => /lib/libdl.so.2 (0x00914000)
	libgphobos2 => /usr/local/lib/libgphobos2 (0x0050d000)
	libgdruntime => /usr/local/lib/libgdruntime (0x00a90000)
	libm.so.6 => /lib/libm.so.6 (0x00921000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00e8f000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x007b9000)
	libc.so.6 => /lib/libc.so.6 (0x00110000)
	/lib/ld-linux.so.2 (0x00f07000)



Now, what's the catch?

The curent catch is that *none* of the external static ctors/dtors are called,
because the linked module list (_Dmodule_ref) is generated before the program
enters C main, thus before the shared druntime and phobos libraries have been
loaded into memory (don't know if this is the same for DMD, but I'll assume
that will likely be the case too).

Breakpoint 1, hello.__modinit() () at hello.d:1
(gdb) bt
#0  hello.__modinit() () at hello.d:1
#1  0x0804889d in __do_global_ctors_aux ()
#2  0x080485d0 in _init ()
#3  0x08048829 in __libc_csu_init ()
#4  0x003a8c84 in __libc_start_main () from /lib/libc.so.6
#5  0x08048661 in _start ()


Each modules own __modinit() is written out as such:

  struct _modref_t {
      _modref_t  * next;
      ModuleInfo m;
  }
  extern (C) _modref_t * _Dmodule_ref;
  private _modref_t our_mod_ref =
    { next: null, m: _ModuleInfo_xxx };
  void ___modinit() { // a static constructor
     our_mod_ref.next = _Dmodule_ref;
     _Dmodule_ref = & our_mod_ref;
  }

These functions are created by the compiler, and inserted into the .ctor list.

Before I progress, posting to ask if anyone has any good implementation ideas
to get this fully functional?

Regards
Iain
Feb 14 2011
next sibling parent reply Johannes Pfau <spam example.com> writes:
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

Iain Buclaw wrote:
So I've been prototyping, and have built a fully working shared D2
druntime/phobos library on Linux (will come to caveats in a moment).
Just for sake of visual proof.

Before I progress, posting to ask if anyone has any good
implementation ideas to get this fully functional?

Regards
Iain

I think dmd/druntime support shared libraries on MacOS. I'm not sure though whether that also includes using a shared druntime/phobos. Probably that code could help (or maybe the MacOS code is too different, then just ignore this comment ;-)) --=20 Johannes Pfau
Feb 14 2011
parent reply Iain Buclaw <ibuclaw ubuntu.com> writes:
== Quote from Johannes Pfau (spam example.com)'s article
 --Sig_/a6ST_/ke_QlFF5lFi4BGNvw
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: quoted-printable
 Iain Buclaw wrote:
So I've been prototyping, and have built a fully working shared D2
druntime/phobos library on Linux (will come to caveats in a moment).
Just for sake of visual proof.

Before I progress, posting to ask if anyone has any good
implementation ideas to get this fully functional?

Regards
Iain

though whether that also includes using a shared druntime/phobos. Probably that code could help (or maybe the MacOS code is too different, then just ignore this comment ;-)) --=20 Johannes Pfau

The support for OSX uses an entirely different, and questionable implementation. Regards
Feb 14 2011
next sibling parent reply Iain Buclaw <ibuclaw ubuntu.com> writes:
== Quote from Iain Buclaw (ibuclaw ubuntu.com)'s article
 == Quote from Johannes Pfau (spam example.com)'s article
 --Sig_/a6ST_/ke_QlFF5lFi4BGNvw
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: quoted-printable
 Iain Buclaw wrote:
So I've been prototyping, and have built a fully working shared D2
druntime/phobos library on Linux (will come to caveats in a moment).
Just for sake of visual proof.

Before I progress, posting to ask if anyone has any good
implementation ideas to get this fully functional?

Regards
Iain

though whether that also includes using a shared druntime/phobos. Probably that code could help (or maybe the MacOS code is too different, then just ignore this comment ;-)) --=20 Johannes Pfau

Regards

Came across this obscure documentation in the tldp. "Libraries should export initialization and cleanup routines using the gcc __attribute__((constructor)) and __attribute__((destructor)) function attributes." This is what gdc was doing anyway. "Constructor routines are executed before dlopen returns (or before main() is started if the library is loaded at load time). Destructor routines are executed before dlclose returns (or after exit() or completion of main() if the library is loaded at load time)." This is what should have been happening, but wasn't earlier because: "Shared libraries must not be compiled with the gcc arguments `-nostartfiles' or `-nostdlib'. If those arguments are used, the constructor/destructor routines will not be executed (unless special measures are taken)." Whoops! Fixed and it now works. :~) Will be making shared libraries default in GDC pretty soon now...
Feb 14 2011
next sibling parent reply Christian Kamm <kamm-incasoftware removethis.de> writes:
Iain Buclaw wrote:
 Will be making shared libraries default in GDC pretty soon now...

Did you adjust the GC to check the shared libraries' data sections for references? When we looked at this for LDC that turned out to slow down GC runs significantly. I'm pretty sure bearophile benchmarked it at the time. As far as I remember the main problem was that we were scanning all data sections - even of plain C libraries. Regards, Christian
Feb 14 2011
parent reply Iain Buclaw <ibuclaw ubuntu.com> writes:
== Quote from Christian Kamm (kamm-incasoftware removethis.de)'s article
 Iain Buclaw wrote:
 Will be making shared libraries default in GDC pretty soon now...

references? When we looked at this for LDC that turned out to slow down GC runs significantly. I'm pretty sure bearophile benchmarked it at the time. As far as I remember the main problem was that we were scanning all data sections - even of plain C libraries. Regards, Christian

When adding more ranges in the GC to check, slowdowns are inevitable. So far in my personal testing, the slowdowns seem pretty negligible in comparison (never more than 0.300 seconds, though that could be a sign that I'm not pushing it in the right way). To prevent the GC from scanning C libraries, I've added an extra check to only add ranges that have a D module inside them. Think: ModuleReference *mr; for ( mr = _Dmodule_ref; mr; mr = mr.next ) { if ( mr >= start && mr <= end ) { addrange = 1; break; } } This is viable for GDC, as all platforms share the same implementation. Note, this also adds a variable overhead at startup, but I think the unskewed runtime justifies it to remain. Regards
Feb 15 2011
parent reply Christian Kamm <kamm-incasoftware removethis.de> writes:
Iain Buclaw wrote:

 == Quote from Christian Kamm (kamm-incasoftware removethis.de)'s article
 Iain Buclaw wrote:
 Will be making shared libraries default in GDC pretty soon now...

references? When we looked at this for LDC that turned out to slow down GC runs significantly. I'm pretty sure bearophile benchmarked it at the time. As far as I remember the main problem was that we were scanning all data sections - even of plain C libraries. Regards, Christian

When adding more ranges in the GC to check, slowdowns are inevitable. So far in my personal testing, the slowdowns seem pretty negligible in comparison (never more than 0.300 seconds, though that could be a sign that I'm not pushing it in the right way). To prevent the GC from scanning C libraries, I've added an extra check to only add ranges that have a D module inside them.

That sounds good. I was asking because I didn't see any code like this in http://bitbucket.org/goshawk/gdc - which repository is this in? Christian
Feb 15 2011
parent reply Iain Buclaw <ibuclaw ubuntu.com> writes:
== Quote from Christian Kamm (kamm-incasoftware removethis.de)'s article
 Iain Buclaw wrote:
 == Quote from Christian Kamm (kamm-incasoftware removethis.de)'s article
 Iain Buclaw wrote:
 Will be making shared libraries default in GDC pretty soon now...

references? When we looked at this for LDC that turned out to slow down GC runs significantly. I'm pretty sure bearophile benchmarked it at the time. As far as I remember the main problem was that we were scanning all data sections - even of plain C libraries. Regards, Christian

When adding more ranges in the GC to check, slowdowns are inevitable. So far in my personal testing, the slowdowns seem pretty negligible in comparison (never more than 0.300 seconds, though that could be a sign that I'm not pushing it in the right way). To prevent the GC from scanning C libraries, I've added an extra check to only add ranges that have a D module inside them.

http://bitbucket.org/goshawk/gdc - which repository is this in? Christian

It's currently on my local workstation, all mashed together with the D2 beta merge. I have faith that my workstation won't crash and die in the meantime <touchwood> and will sift through it all (and break it down into sizeable commits) once I'm satisfied that there's nothing more I can do. Regards Iain
Feb 16 2011
parent Iain Buclaw <ibuclaw ubuntu.com> writes:
== Quote from Iain Buclaw (ibuclaw ubuntu.com)'s article
 == Quote from Christian Kamm (kamm-incasoftware removethis.de)'s article
 Iain Buclaw wrote:
 == Quote from Christian Kamm (kamm-incasoftware removethis.de)'s article
 Iain Buclaw wrote:
 Will be making shared libraries default in GDC pretty soon now...

references? When we looked at this for LDC that turned out to slow down GC runs significantly. I'm pretty sure bearophile benchmarked it at the time. As far as I remember the main problem was that we were scanning all data sections - even of plain C libraries. Regards, Christian

When adding more ranges in the GC to check, slowdowns are inevitable. So far in my personal testing, the slowdowns seem pretty negligible in comparison (never more than 0.300 seconds, though that could be a sign that I'm not pushing it in the right way). To prevent the GC from scanning C libraries, I've added an extra check to only add ranges that have a D module inside them.

http://bitbucket.org/goshawk/gdc - which repository is this in? Christian

merge. I have faith that my workstation won't crash and die in the meantime <touchwood> and will sift through it all (and break it down into sizeable commits) once I'm satisfied that there's nothing more I can do. Regards Iain

Module/Routine is now up here: https://bitbucket.org/goshawk/gdc/src/a402fef51a6a/d/druntime/rt/gccmemory.d
Feb 19 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-02-15 16:02, Johannes Pfau wrote:
 Iain Buclaw wrote:
 Came across this obscure documentation in the tldp.

 "Libraries should export initialization and cleanup routines using the
 gcc __attribute__((constructor)) and __attribute__((destructor))
 function attributes."

 This is what gdc was doing anyway.


 "Constructor routines are executed before dlopen returns (or before
 main() is started if the library is loaded at load time). Destructor
 routines are executed before dlclose returns (or after exit() or
 completion of main() if the library is loaded at load time)."

 This is what should have been happening, but wasn't earlier because:


 "Shared libraries must not be compiled with the gcc arguments
 `-nostartfiles' or `-nostdlib'. If those arguments are used, the
 constructor/destructor routines will not be executed (unless special
 measures are taken)."

 Whoops! Fixed and it now works. :~)

 Will be making shared libraries default in GDC pretty soon now...

Great, now I've got 2 questions: 1.) Are shared libraries compiled with gdc abi compitible with dmd? (if dmd did support shared libraries)

I'm not completely sure but the runtimes are different.
 2.) Is it now possible to load a shared d library into a C program? Will
 jut loading the library initialize the runtime or is it necessary to
 call some runtime setup functions manually? If the C program loads 2 D
 libraries, will those share the gc/runtime?

I don't know how Iain has implemented this but it should be easy to do. Just add, to the runtime, a C function with __attribute__((constructor)) that initializes the runtime. -- /Jacob Carlborg
Feb 15 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-02-14 16:29, Iain Buclaw wrote:
 == Quote from Johannes Pfau (spam example.com)'s article
 --Sig_/a6ST_/ke_QlFF5lFi4BGNvw
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: quoted-printable
 Iain Buclaw wrote:
 So I've been prototyping, and have built a fully working shared D2
 druntime/phobos library on Linux (will come to caveats in a moment).
 Just for sake of visual proof.

 Before I progress, posting to ask if anyone has any good
 implementation ideas to get this fully functional?

 Regards
 Iain

though whether that also includes using a shared druntime/phobos. Probably that code could help (or maybe the MacOS code is too different, then just ignore this comment ;-)) --=20 Johannes Pfau

The support for OSX uses an entirely different, and questionable implementation. Regards

If you know any other way of implementing it I very open for suggestions. -- /Jacob Carlborg
Feb 14 2011
prev sibling parent Johannes Pfau <spam example.com> writes:
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

Iain Buclaw wrote:
Came across this obscure documentation in the tldp.

"Libraries should export initialization and cleanup routines using the
gcc __attribute__((constructor)) and __attribute__((destructor))
function attributes."

This is what gdc was doing anyway.


"Constructor routines are executed before dlopen returns (or before
main() is started if the library is loaded at load time). Destructor
routines are executed before dlclose returns (or after exit() or
completion of main() if the library is loaded at load time)."

This is what should have been happening, but wasn't earlier because:


"Shared libraries must not be compiled with the gcc arguments
`-nostartfiles' or `-nostdlib'. If those arguments are used, the
constructor/destructor routines will not be executed (unless special
measures are taken)."

Whoops! Fixed and it now works. :~)

Will be making shared libraries default in GDC pretty soon now...

Great, now I've got 2 questions: 1.) Are shared libraries compiled with gdc abi compitible with dmd? (if dmd did support shared libraries) 2.) Is it now possible to load a shared d library into a C program? Will jut loading the library initialize the runtime or is it necessary to call some runtime setup functions manually? If the C program loads 2 D libraries, will those share the gc/runtime? --=20 Johannes Pfau
Feb 15 2011