www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Dynamic loading,

reply Carl Sturtivant <sturtivant gmail.com> writes:
--f46d0444ec19b9bead04c7aa4476
Content-Type: multipart/alternative; boundary=f46d0444ec19b9beab04c7aa4474

--f46d0444ec19b9beab04c7aa4474
Content-Type: text/plain; charset=UTF-8

(By dynamic loading I mean using something like the C library function
dlopen to explicitly load a shared object at runtime. I do not mean dynamic
linking in the usual passive sense.)

I just successfully got working a toy example of dynamically loading (from
a D program) a D shared object and then finding and calling functions
defined in the shared object, and not just with C linkage.

The main program, main.d, was compiled and linked somewhat normally, except
for passing the linker via gcc the flags necessary to ensure that the whole
of libphobos2.a is present and that all symbols in the resulting executable
are exposed dynamically.

The shared object source, dload.d was compiled to an object file containing
position independent code by dmd. Then I invoked the linker explicitly and
had it make a shared object without the D runtime system or Phobos. This is
the novel step, and it enables the shared object to resolve its linkage to
the D runtime system and Phobos at the point of being loaded, via callbacks
to the main program. Thus the troubles of D in shared objects are largely
circumvented. There is only one instance of phobos and D-runtime, in the
main program. (Once phobos and druntime are shared objects in the future
somewhere this will work with no code bloat.)

The static initialization code in dload.d is automatically executed when
the shared object libdload.so is loaded by the main program, because the
linker is also passed a flag indicating the static initialization block's
mangled name, dynamically determined from dload.o before linkage to
libdload.so occurs.

Finally, the mangled names of the functions to load are determined by a
call of a function with C linkage in dload.d from main.d that looks up
those names in an associative array initialized in the static
initialization block of dload.d where those mangled names are directly
available, so that full D linkage can be emulated, at least for functions.

One thing: the garbage collector needs to be aware of static and 'global' D
variables in the shared object. Can a technical expert verify that I've
done the right thing to achieve that happy state of affairs in this unusual
context?

So, what's overlooked here? I know that the static initialization code
cannot successfully throw an exception. Yet if a function in the shared
object is called from the main program and throws an exception, all is
well. (Try these.) See my comments in dload.d about this. What is it about
the implementation of exceptions that's problematic here?

All files attached, including a Makefile with the exact options passed to
dmd, gcc and ld.

--f46d0444ec19b9beab04c7aa4474
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

(By dynamic loading I mean using something like the C library function dlop=
en to explicitly load a shared object at runtime. I do not mean dynamic lin=
king in the usual passive sense.)<br><br>I just successfully got working a =
toy example of dynamically loading (from a D program) a D shared object and=
 then finding and calling functions defined in the shared object, and not j=
ust with C linkage. <br>
<br>The main program, main.d, was compiled and linked somewhat normally, ex=
cept for passing the linker via gcc the flags necessary to ensure that the =
whole of libphobos2.a is present and that all symbols in the resulting exec=
utable are exposed dynamically.<br>
<br>The shared object source, dload.d was compiled to an object file contai=
ning position independent code by dmd. Then I invoked the linker explicitly=
 and had it make a shared object without the D runtime system or Phobos. Th=
is is the novel step, and it enables the shared object to resolve its linka=
ge to the D runtime system and Phobos at the point of being loaded, via cal=
lbacks to the main program. Thus the troubles of D in shared objects are la=
rgely circumvented. There is only one instance of phobos and D-runtime, in =
the main program. (Once phobos and druntime are shared objects in the futur=
e somewhere this will work with no code bloat.)<br>
<br>The static initialization code in dload.d is automatically executed whe=
n the shared object libdload.so is loaded by the main program, because the =
linker is also passed a flag indicating the static initialization block&#39=
;s mangled name, dynamically determined from dload.o before linkage to libd=
load.so occurs.<br>
<br>Finally, the mangled names of the functions to load are determined by a=
 call of a function with C linkage in dload.d from main.d that looks up tho=
se names in an associative array initialized in the static initialization b=
lock of dload.d where those mangled names are directly available, so that f=
ull D linkage can be emulated, at least for functions.<br>
<br>One thing: the garbage collector needs to be aware of static and &#39;g=
lobal&#39; D variables in the shared object. Can a technical expert verify =
that I&#39;ve done the right thing to achieve that happy state of affairs i=
n this unusual context?<br>
<br>So, what&#39;s overlooked here? I know that the static initialization c=
ode cannot successfully throw an exception. Yet if a function in the shared=
 object is called from the main program and throws an exception, all is wel=
l. (Try these.) See my comments in dload.d about this. What is it about the=
 implementation of exceptions that&#39;s problematic here?<br>
<br>All files attached, including a Makefile with the exact options passed =
to dmd, gcc and ld.<br>
<br>

--f46d0444ec19b9beab04c7aa4474--
--f46d0444ec19b9bead04c7aa4476
Content-Type: application/octet-stream; name="dload.d"
Content-Disposition: attachment; filename="dload.d"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_h630h08w0

CmltcG9ydCBzdGQuc3RkaW8sIGNvcmUubWVtb3J5LCBzdGQuZXhjZXB0aW9uOwoKc3RhdGljIHN0
cmluZ1tzdHJpbmddIGxvb2t1cDsgLy9tYXAgZnJvbSBuYW1lcyB0byBtYW5nbGVkIG5hbWVzCgpl
eHRlcm4oQykgc3RyaW5nIGRsb2FkKHN0cmluZyBuYW1lKSB7Ci8vZ2V0IG1hbmdsZWQgbmFtZTsg
dG95IGV4YW1wbGU6IGlnbm9yZSBwb3NzaWJpbGl5IG9mIG92ZXJsb2FkaW5nCiAgcmV0dXJuIGxv
b2t1cFtuYW1lXTsgCn0KCgpzdGF0aWMgdGhpcygpIHsKICAvL21ha2UgdGhpcyBtb2R1bGUgR0Mg
c2FmZSBoZXJlCiAgc3RhdGljIHN0cmluZ1tzdHJpbmddKiBscHRyOwogIGxwdHIgPSAmbG9va3Vw
OwogIC8vd3Jvbmc/LCBhc2sgZXhwZXJ0LCBuZWVkIGNsZWFuIHNvbHV0aW9uCiAgR0MuYWRkUm9v
dCggJmxwdHIpOyAvL3doeSBkb2VzIHRoaXMgY2FsbGJhY2sgd29yayB3aGVuIG5vdCBsb2FkZWQg
ZnVsbHk/CiAgLy9jYW4gY2FsbCBzdGRpbyBmdW5jdGlvbnMgaGVyZSB3aXRoIG5vIHByb2JsZW0g
dG9vCgogIC8vZXhjZXB0aW9uIGhhbmRsaW5nIHByb2JsZW1hdGljIGZvciBleGVjdXRpb24gaW4g
aGVyZQogIC8vbm90IHlldCBsb2FkZWQgZnVsbHksIHNvIGNhbGxiYWNrcyBmb3IgdGhyb3dpbmcg
JmMgbm90IGxpbmthYmxlIHlldD8KCiAgLy9zZXR1cCBhbGwgY2FsbGJhY2sgbmFtZXMKICBsb29r
dXBbImNvbmNhdCJdID0gY29uY2F0Lm1hbmdsZW9mOwogIGxvb2t1cFsic3VtIl0gPSBzdW0ubWFu
Z2xlb2Y7Cgp9CgovL2V4YW1wbGUgZnVuY3Rpb25zIHRvIGNhbGwgYWZ0ZXIgbG9hZAoKc3RyaW5n
IGNvbmNhdChzdHJpbmdbXSBhcmcuLi4pIHsKICBzdHJpbmcgYW5zOwogIGZvcmVhY2goIHMgOyBh
cmcgKSBhbnMgfj0gczsKICByZXR1cm4gYW5zOwp9CgppbnQgc3VtKGludFtdIG51bS4uLikgewog
IGludCB0b3RhbCA9IDA7CiAgZm9yZWFjaCggeCA7IG51bSApIHRvdGFsICs9IHg7CiAgcmV0dXJu
IHRvdGFsOwp9Cg==
--f46d0444ec19b9bead04c7aa4476
Content-Type: application/octet-stream; name="main.d"
Content-Disposition: attachment; filename="main.d"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_h630joi81

CmltcG9ydCBzdGQuc3RkaW8sIGNvcmUuc3lzLnBvc2l4LmRsZmNuOwoKdm9pZCBtYWluKCkgewoK
ICBhdXRvIGhhbmRsZSA9IGRsb3BlbigiLi9saWJkbG9hZC5zbyIucHRyLCBSVExEX0xBWlkgfCBS
VExEX0dMT0JBTCk7CiAgaWYoIGhhbmRsZT09bnVsbCApIHsgd3JpdGVsbigiITEiKTsgcmV0dXJu
OyB9CgogIGRsb2FkID0gY2FzdCh0eXBlb2YoZGxvYWQpKSBkbHN5bShoYW5kbGUsICJkbG9hZCIu
cHRyKTsKICBpZiggZGxvYWQ9PW51bGwgKSB7IHdyaXRlbG4oIiEyIik7IHJldHVybjsgfQoKICBz
dHJpbmcgY29uY2F0X3N5bWJvbCA9IGRsb2FkKCJjb25jYXQiKTsgLy9nZXQgbWFuZ2xlZCBuYW1l
cwogIHN0cmluZyBzdW1fc3ltYm9sID0gZGxvYWQoInN1bSIpOwoKICBjb25jYXQgPSBjYXN0KHR5
cGVvZihjb25jYXQpKSBkbHN5bShoYW5kbGUsIGNvbmNhdF9zeW1ib2wucHRyICk7CiAgc3VtID0g
Y2FzdCh0eXBlb2Yoc3VtKSkgZGxzeW0oaGFuZGxlLCBzdW1fc3ltYm9sLnB0ciApOwogIGlmKCBj
b25jYXQ9PW51bGwgfHwgc3VtPT1udWxsICkgeyB3cml0ZWxuKCIhMyIpOyByZXR1cm47IH0KCiAg
d3JpdGVsbiggY29uY2F0KCJIZWxsbyIsICIgdGhpcyBpcyBhIGNhbGwgIiwgIm9mIGNvbmNhdCIp
ICk7CiAgd3JpdGVsbiggIjMgKyA0ICsgNSB2aWEgc3VtIGlzICIsIHN1bSgzLDQsNSkgKTsKfQoK
c3RyaW5nIGZ1bmN0aW9uKHN0cmluZykgZGxvYWQ7IC8vbmFtZSBtYW5nbGluZyBmdW5jdGlvbiBp
biBsaWJkbG9hZC5zbwoKLy9EIGZ1bmN0aW9ucyB0byBkeW5hbWljYWxseSBsb2FkCnN0cmluZyBm
dW5jdGlvbihzdHJpbmdbXSBhcmcuLi4pIGNvbmNhdDsKaW50IGZ1bmN0aW9uKGludFtdIG51bS4u
Likgc3VtOwo=
--f46d0444ec19b9bead04c7aa4476
Content-Type: application/octet-stream; name="staticCtor.icn"
Content-Disposition: attachment; filename="staticCtor.icn"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_h630k1mu2

CiMgc3RhdGljQ3RvciBkbG9hZC5vCiMgLS0tIHJ1bnMgbm0gb24gZGxvYWQubyAmIGZpbmRzIHRo
ZSBsaW5lIGNvbnRhaW5pbmcgInN0YXRpY0N0b3IiLAojIC0tLSBhbmFseXplcyB0aGF0IGxpbmUg
YW5kIHByaW50cyBvdXQgdGhlIG1hbmdsZWQgc3ltYm9sIGluIGl0CiMgLS0tIHdoaWNoIGlzIHRo
ZSBuYW1lIG9mIHRoZSBzdGF0aWMgaW5pdGlhbGl6YXRpb24gYmxvY2suCiMgLS0tIFNlZSB0aGUg
TWFrZWZpbGUgdGhhdCBhY2NvbXBhbmllcyB0aGlzLgojIENhcmwgU3R1cnRpdmFudCwgMjAxMi0w
OC0xOQoKI25lZWRzIGFuIGluc3RhbGxhdGlvbiBvZiBJY29uIHNlZSBodHRwOi8vY3MuYXJpem9u
YS5lZHUvaWNvbgojIGljb250ICgidHJhbnNsYXRvciIgaS5lLiBjb21waWxlciB0byBieXRlY29k
ZSkKIyBpY29ueCAoInJ1bnRpbWUiIGkuZS4gaW50ZXJwcmV0ZXIpCiMgSWNvbiB2OS40LjMgZWFz
eSB0byBnZXQgcGFja2FnZSB3aXRoIFVidW50dSBvciBEZWJpYW4uCiMgZW1hY3Mgc3ludGF4IGNv
bG9ycyBJY29uLgoKcHJvY2VkdXJlIG1haW4oYXJnKQogICAgZm5hbWUgOj0gYXJnWzFdIHwgc3Rv
cCgiITEiKQogICAgaW4gOj0gb3Blbigibm0gIiB8fCBmbmFtZSwgInAiKSB8IHN0b3AoIiEyIikK
ICAgIHdoaWxlIGxpbmUgOj0gdHJpbShyZWFkKGluKSkgZG8gbGluZSA/CglpZiBmaW5kKCJzdGF0
aWNDdG9yIikgdGhlbiB7CgkgICAgd3JpdGUoZ2V0X3N5bWJvbChsaW5lKSkKCSAgICByZXR1cm4K
CX0KICAgIHN0b3AoIiEzIikKZW5kCgpwcm9jZWR1cmUgZ2V0X3N5bWJvbChsaW5lKQogICAgbGlu
ZSA/IHsKCXRhYihmaW5kKCJfIikpCglzeW0gOj0gdGFiKDApCglyZXR1cm4gc3ltCiAgICB9CmVu
ZAo=
--f46d0444ec19b9bead04c7aa4476
Content-Type: application/octet-stream; name=Makefile
Content-Disposition: attachment; filename=Makefile
Content-Transfer-Encoding: base64
X-Attachment-Id: f_h630kbd73

CiMgTWFrZWZpbGUgZm9yIGR5bmFtaWMgbG9hZGluZyBleGFtcGxlIGZvciBECiMgU3BlY2lmaWMg
dG8gKGRtZCA2NGJpdCAyLjA2MC9VYnVudHUgMTIuMDQveDg2XzY0KQojIENhcmwgU3R1cnRpdmFu
dCAyMDEyLTA4LTE5CgpkZWZhdWx0IDogbGliZGxvYWQuc28gbWFpbgoKbGliZGxvYWQuc28gOiBk
bG9hZC5vIHN0YXRpY0N0b3IKIwlleGVjdXRlICdzdGF0aWMgdGhpcygpJyBvbiBsb2FkIHNldCB2
aWEgZmxhZyAtaW5pdAoJbGQgZGxvYWQubyAtc2hhcmVkIC1vIGxpYmRsb2FkLnNvIC1tIGVsZl94
ODZfNjQgLUUgLWluaXQ9JChzaGVsbCBzdGF0aWNDdG9yIGRsb2FkLm8pICNiYXJlIGJvbmVzIGxp
bmthZ2UKIwlnY2MgZGxvYWQubyAtc2hhcmVkIC1vIGxpYmRsb2FkLnNvIC1tNjQgLVdsLC1FIC1X
bCwtaW5pdD0kKHNoZWxsIHN0YXRpY0N0b3IgZGxvYWQubykgI2FkZHMgc29tZSBjb2RlCgpkbG9h
ZC5vIDogZGxvYWQuZAoJZG1kIC1jIGRsb2FkLmQgLW02NCAtZlBJQwoKc3RhdGljQ3RvciA6IHN0
YXRpY0N0b3IuaWNuCglpY29udCAtcyBzdGF0aWNDdG9yLmljbiAjY291bGQgdXNlIHVuaXh5IHRo
aW5ncyB0byBkbyB0aGlzLCBidXQgSSdtIHRvbyBsYXp5CgptYWluIDogbWFpbi5vCglnY2MgbWFp
bi5vIC1vIG1haW4gLW02NCAtV2wsLUUgLWxkbCAtV2wsLS13aG9sZS1hcmNoaXZlIC1scGhvYm9z
MiAtV2wsLS1uby13aG9sZS1hcmNoaXZlIC1sY3VybCAtbHB0aHJlYWQgLWxtIC1scnQKCm1haW4u
byA6IG1haW4uZAoJZG1kIC1jIG1haW4uZCAtbTY0CgpjbGVhbiA6CglAcm0gLWYgKi5vICouc28g
Kn4gKlwjIG1haW4gc3RhdGljQ3Rvcgo=
--f46d0444ec19b9bead04c7aa4476--
Aug 19 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-08-20 05:40, Carl Sturtivant wrote:
 (By dynamic loading I mean using something like the C library function
 dlopen to explicitly load a shared object at runtime. I do not mean
 dynamic linking in the usual passive sense.)

 I just successfully got working a toy example of dynamically loading
 (from a D program) a D shared object and then finding and calling
 functions defined in the shared object, and not just with C linkage.

 The main program, main.d, was compiled and linked somewhat normally,
 except for passing the linker via gcc the flags necessary to ensure that
 the whole of libphobos2.a is present and that all symbols in the
 resulting executable are exposed dynamically.

 The shared object source, dload.d was compiled to an object file
 containing position independent code by dmd. Then I invoked the linker
 explicitly and had it make a shared object without the D runtime system
 or Phobos. This is the novel step, and it enables the shared object to
 resolve its linkage to the D runtime system and Phobos at the point of
 being loaded, via callbacks to the main program. Thus the troubles of D
 in shared objects are largely circumvented. There is only one instance
 of phobos and D-runtime, in the main program. (Once phobos and druntime
 are shared objects in the future somewhere this will work with no code
 bloat.)

 The static initialization code in dload.d is automatically executed when
 the shared object libdload.so is loaded by the main program, because the
 linker is also passed a flag indicating the static initialization
 block's mangled name, dynamically determined from dload.o before linkage
 to libdload.so occurs.

 Finally, the mangled names of the functions to load are determined by a
 call of a function with C linkage in dload.d from main.d that looks up
 those names in an associative array initialized in the static
 initialization block of dload.d where those mangled names are directly
 available, so that full D linkage can be emulated, at least for functions.

 One thing: the garbage collector needs to be aware of static and
 'global' D variables in the shared object. Can a technical expert verify
 that I've done the right thing to achieve that happy state of affairs in
 this unusual context?

 So, what's overlooked here? I know that the static initialization code
 cannot successfully throw an exception. Yet if a function in the shared
 object is called from the main program and throws an exception, all is
 well. (Try these.) See my comments in dload.d about this. What is it
 about the implementation of exceptions that's problematic here?

 All files attached, including a Makefile with the exact options passed
 to dmd, gcc and ld.

I'm not sure I'm following what you exactly have done here but in general this is what needs to be done to make dynamic libraries properly work in D : * Initialize module infos (module constructors and similar) * Add TLS variables * Add exception handling tables * Add GC roots The above four things need to be extracted from the loaded dynamic library and it gets loaded and preferably remove them as well when the dynamic library gets unloaded. Currently this is only extracted from the running executable. This is platform dependent but usually it's extracted using bracketed sections via extern C variables. -- /Jacob Carlborg
Aug 20 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-08-20 13:38, Paulo Pinto wrote:

 Should this be made automatically by the compiler?

 This would be my expectation based on my experience with dynamic
 libraries in Turbo Pascal/Delphi.

This should be handled automatically by the runtime. -- /Jacob Carlborg
Aug 20 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-08-20 16:22, Paulo Pinto wrote:

 Ah ok, from your explanation I understood as something we are required
 to do manually.

I just explained what changes need to be done in order for it to work. So we need to modify druntime to do what I listed above. -- /Jacob Carlborg
Aug 20 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-08-20 17:16, Carl Sturtivant wrote:
 I'm not sure I'm following what you exactly have done here but in
 general this is what needs to be done to make dynamic libraries
 properly work in D :

 * Initialize module infos (module constructors and similar)
 * Add TLS variables
 * Add exception handling tables
 * Add GC roots

 The above four things need to be extracted from the loaded dynamic
 library and it gets loaded and preferably remove them as well when the
 dynamic library gets unloaded. Currently this is only extracted from
 the running executable. This is platform dependent but usually it's
 extracted using bracketed sections via extern C variables.

OK, good to know. Any further hints about these, or where I can look?

From druntime: * rt.minfo - module infos. The "_moduleinfo_array" variable contains the module infos. * rt.deh2 - exception handling. On Mac OS X the "_deh_eh_array" contains the exception handling tables. On the other Posix systems the sections bracketed by "_deh_beg" and "_deh_end" contains the tables. * rt.thread - threading, TLS. On Mac OS X the TLS variables are stored in "_tls_data_array". On the other Posix systems they're are again stored in bracketed sections: "_tlsstart" and "_tlsend". * rt.memory - GC roots. I think this module contains code related to adding GC roots. rt.memory_osx for Mac OS X. You can also take a look at work by Martin Nowak: https://github.com/dawgfoto/druntime/commits/SharedRuntime
 What I've done is use the C dynamic loading library (header dlfcn.h) to
 manually load a shared object written in D, dload.d, from a D program
 (main.d), which then successfully calls functions in dload.d that are
 not defined extern(C).

 I am attempting a do-it-yourself dynamic loading in D, where I
 explicitly do all the administration manually to make it work, rather
 than rely upon D to do it automatically. Hence my use of the C dynamic
 loading library, which knows nothing of additional work D must do.

The functions from dlfcn.h is what D would use as well. Possibly wrapped in some D function that does some additional work. -- /Jacob Carlborg
Aug 20 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-08-24 00:26, Philip Daniels wrote:

 But wouldn't that require you to link everything together at, err,
 compile time?

 What I'm getting at is, would it be possible to port a DI/IoC tool such
 as StructureMap (http://docs.structuremap.net/index.html) or Spring to
 D? This can handle tasks such as creating dynamic plug-in architectures.
 For example, given SomeBigApp.exe (not written by me) which looks in
 standard folders for components implementing a particular interface, I
 can just drop my code in that folder and have it loaded at runtime. I
 could even drop it in there after the program starts running. I know how
 to achieve this in the .Net world, just wondered if it was possible in D.

Sure, you just need to come up with a couple of conventions. Say for example that you have your application in a folder structure like this: app |--plugins |---fully.qualified.class.name.so/dll/dylib The host application inspects the "plugins" folder at runtime grabbing all the names of the dynamic libraries. The convention is that in the dynamic library there will be a class with the same fully qualified name as the name of the dynamic library its located in. The next convention is that the class must implement an interface, which the host application provides. The class also needs to have a default constructor (there are ways around that). Then the host application would do something like this: interface Plugin { void initialize (); // ... other needed methods } auto plugin = cast(Plugin) Object.factory("fully.qualified.class.name"); plugin.initialize(); The magic behind this is the "Object.factory" method: http://dlang.org/phobos/object.html#factory -- /Jacob Carlborg
Aug 24 2012
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Monday, 20 August 2012 at 07:26:40 UTC, Jacob Carlborg wrote:
 On 2012-08-20 05:40, Carl Sturtivant wrote:
 (By dynamic loading I mean using something like the C library 
 function
 dlopen to explicitly load a shared object at runtime. I do not 
 mean
 dynamic linking in the usual passive sense.)

 I just successfully got working a toy example of dynamically 
 loading
 (from a D program) a D shared object and then finding and 
 calling
 functions defined in the shared object, and not just with C 
 linkage.

 The main program, main.d, was compiled and linked somewhat 
 normally,
 except for passing the linker via gcc the flags necessary to 
 ensure that
 the whole of libphobos2.a is present and that all symbols in 
 the
 resulting executable are exposed dynamically.

 The shared object source, dload.d was compiled to an object 
 file
 containing position independent code by dmd. Then I invoked 
 the linker
 explicitly and had it make a shared object without the D 
 runtime system
 or Phobos. This is the novel step, and it enables the shared 
 object to
 resolve its linkage to the D runtime system and Phobos at the 
 point of
 being loaded, via callbacks to the main program. Thus the 
 troubles of D
 in shared objects are largely circumvented. There is only one 
 instance
 of phobos and D-runtime, in the main program. (Once phobos and 
 druntime
 are shared objects in the future somewhere this will work with 
 no code
 bloat.)

 The static initialization code in dload.d is automatically 
 executed when
 the shared object libdload.so is loaded by the main program, 
 because the
 linker is also passed a flag indicating the static 
 initialization
 block's mangled name, dynamically determined from dload.o 
 before linkage
 to libdload.so occurs.

 Finally, the mangled names of the functions to load are 
 determined by a
 call of a function with C linkage in dload.d from main.d that 
 looks up
 those names in an associative array initialized in the static
 initialization block of dload.d where those mangled names are 
 directly
 available, so that full D linkage can be emulated, at least 
 for functions.

 One thing: the garbage collector needs to be aware of static 
 and
 'global' D variables in the shared object. Can a technical 
 expert verify
 that I've done the right thing to achieve that happy state of 
 affairs in
 this unusual context?

 So, what's overlooked here? I know that the static 
 initialization code
 cannot successfully throw an exception. Yet if a function in 
 the shared
 object is called from the main program and throws an 
 exception, all is
 well. (Try these.) See my comments in dload.d about this. What 
 is it
 about the implementation of exceptions that's problematic here?

 All files attached, including a Makefile with the exact 
 options passed
 to dmd, gcc and ld.

I'm not sure I'm following what you exactly have done here but in general this is what needs to be done to make dynamic libraries properly work in D : * Initialize module infos (module constructors and similar) * Add TLS variables * Add exception handling tables * Add GC roots The above four things need to be extracted from the loaded dynamic library and it gets loaded and preferably remove them as well when the dynamic library gets unloaded. Currently this is only extracted from the running executable. This is platform dependent but usually it's extracted using bracketed sections via extern C variables.

Should this be made automatically by the compiler? This would be my expectation based on my experience with dynamic libraries in Turbo Pascal/Delphi. -- Paulo
Aug 20 2012
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Monday, 20 August 2012 at 12:15:28 UTC, Jacob Carlborg wrote:
 On 2012-08-20 13:38, Paulo Pinto wrote:

 Should this be made automatically by the compiler?

 This would be my expectation based on my experience with 
 dynamic
 libraries in Turbo Pascal/Delphi.

This should be handled automatically by the runtime.

Ah ok, from your explanation I understood as something we are required to do manually.
Aug 20 2012
prev sibling next sibling parent "Carl Sturtivant" <sturtivant gmail.com> writes:
 I'm not sure I'm following what you exactly have done here but 
 in general this is what needs to be done to make dynamic 
 libraries properly work in D :

 * Initialize module infos (module constructors and similar)
 * Add TLS variables
 * Add exception handling tables
 * Add GC roots

 The above four things need to be extracted from the loaded 
 dynamic library and it gets loaded and preferably remove them 
 as well when the dynamic library gets unloaded. Currently this 
 is only extracted from the running executable. This is platform 
 dependent but usually it's extracted using bracketed sections 
 via extern C variables.

OK, good to know. Any further hints about these, or where I can look? What I've done is use the C dynamic loading library (header dlfcn.h) to manually load a shared object written in D, dload.d, from a D program (main.d), which then successfully calls functions in dload.d that are not defined extern(C). I am attempting a do-it-yourself dynamic loading in D, where I explicitly do all the administration manually to make it work, rather than rely upon D to do it automatically. Hence my use of the C dynamic loading library, which knows nothing of additional work D must do. Reason for this approach: the newsgroups indicate that dynamic loading in D does not work as yet. What's novel is that I explicitly excluded from the shared object anything but code generated directly from dload.d: dmd -c dload.d -m64 -fPIC ld dload.o -shared -o libdload.so -m elf_x86_64 -E And I included the whole of libphobos.a in the build of the main program: dmd -c main.d -m64 gcc main.o -o main -m64 -Wl,-E -ldl -Wl,--whole-archive -lphobos2 \ -Wl,--no-whole-archive -lcurl -lpthread -lm -lrt This tactic is in the hope that all parts of D used in the shared library will find their linkage in the main program when it's dynamically loaded. (And the -E option passed to the linker in both cases is to expose all symbols for exactly this purpose.) Reason for this approach: the newsgroups indicate that D runtime/phobos (all currently in libphobos.a it seems) does not initialize properly in a shared library, so I ensure that it's not present at all, and endeavor to have the shared object implicitly use the properly working D runtime/phobos in the main executable. [And besides, at 64 bits all code in shared libraries apparently must be position independent, so even if I wanted to link parts of libphobos.a into the shared library, I couldn't without recompiling libphobos.a with the -fPIC option!] This bare-bones-in-the-shared-library approach has worked well in my toy example. The only thing that apparently doesn't work is if an exception is thrown from the static initializer 'static this()' in the shared object. If an exception results from a call chain initiated by the main program even if thrown from a function in the shared object, all is well it seems. Incidentally, I fibbed slightly about the ld command used to link the shared object. It also contains a trailing -init=$(shell staticCtor dload.o) which enables the linker to bind in execution of the static initializer 'static this()' so that it runs automatically when the shared library is loaded. staticCtor is a script that analyzes dload.o to find its name, which is mangled of course. If anyone can give me any more specific information about what else I can make happen manually to complete effective linkage at the D level I'd be grateful. I'm not stopping the investigation here!
Aug 20 2012
prev sibling next sibling parent "Philip Daniels" <foo foo.com> writes:
snip<

If we had dynamic loading, would we be able to do dependency injection in D?
Aug 22 2012
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Wednesday, 22 August 2012 at 15:51:05 UTC, Philip Daniels 
wrote:
snip<

If we had dynamic loading, would we be able to do dependency injection in D?

Dependency injection does not require dynamic loading per se. It is all about using interfaces instead of classes, and initializing the corresponding instance members. You just need some piece of code that takes the responsibility of locating such interfaces, by registering the classes somehow, or by compile time reflection, which gets called on program initialization. You can do this in D today. -- Paulo
Aug 22 2012
prev sibling parent "Philip Daniels" <foo foo.com> writes:
On Wednesday, 22 August 2012 at 16:57:26 UTC, Paulo Pinto wrote:
 On Wednesday, 22 August 2012 at 15:51:05 UTC, Philip Daniels 
 wrote:
snip<

If we had dynamic loading, would we be able to do dependency injection in D?

Dependency injection does not require dynamic loading per se. It is all about using interfaces instead of classes, and initializing the corresponding instance members. You just need some piece of code that takes the responsibility of locating such interfaces, by registering the classes somehow, or by compile time reflection, which gets called on program initialization. You can do this in D today. -- Paulo

But wouldn't that require you to link everything together at, err, compile time? What I'm getting at is, would it be possible to port a DI/IoC tool such as StructureMap (http://docs.structuremap.net/index.html) or Spring to D? This can handle tasks such as creating dynamic plug-in architectures. For example, given SomeBigApp.exe (not written by me) which looks in standard folders for components implementing a particular interface, I can just drop my code in that folder and have it loaded at runtime. I could even drop it in there after the program starts running. I know how to achieve this in the .Net world, just wondered if it was possible in D.
Aug 23 2012