www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Desperately looking for a work-around to load and unload D shared

reply ponce <contact gam3sfrommars.fr> writes:
Context: On OSX, a C program can load a D shared library but once 
unloaded the next dlopen will crash, jumping into a callback that 
doesn't exist anymore.

I've filed it here: https://issues.dlang.org/show_bug.cgi?id=15060


It looks like this was known and discussed several times already:
http://forum.dlang.org/post/vixoqmidlbizawbxmsao forum.dlang.org 
(2015)
https://github.com/D-Programming-Language/druntime/pull/228 (2012)


Any idea to work-around this problem would be awesome.

I'm not looking for something correct, future-proof, or pretty. 
Any shit that still stick to the wall will do. Anything!

The only case I need to support is: C host, D shared library, 
with runtime statically linked.

Please help!
Sep 16 2015
next sibling parent reply bitwise <bitwise.pvt gmail.com> writes:
On Wednesday, 16 September 2015 at 22:29:46 UTC, ponce wrote:

 Context: On OSX, a C program can load a D shared library but 
 once unloaded the next dlopen will crash, jumping into a 
 callback that doesn't exist anymore.

 I've filed it here: 
 https://issues.dlang.org/show_bug.cgi?id=15060


 It looks like this was known and discussed several times 
 already:
 http://forum.dlang.org/post/vixoqmidlbizawbxmsao forum.dlang.org (2015)
 https://github.com/D-Programming-Language/druntime/pull/228 
 (2012)


 Any idea to work-around this problem would be awesome.

 I'm not looking for something correct, future-proof, or pretty. 
 Any shit that still stick to the wall will do. Anything!

 The only case I need to support is: C host, D shared library, 
 with runtime statically linked.

 Please help!


I was trying to solve this one myself, but the modifications to 
DMD's backend that are needed are out of reach for me right now.

If you're willing to build your own druntime, you may be able to 
get by.

If I understand correctly, you want to repeatedly load/unload the 
same shared library, correct? I ask because druntime for osx only 
supports loading a single image at a time:

https://github.com/D-Programming-Language/druntime/blob/1e25749cd01ad08dc08319a3853fbe86356c3e62/src/rt/sections_osx.d#L26

Anyways, when main() of a D program runs, it calls rt_init() and 
rt_term(). If you don't have a D entry point in your program, you 
have to retrieve these from your shared lib(which has druntime 
statically linked) using dlsym() and call them yourself.

https://github.com/D-Programming-Language/druntime/blob/478b6c5354470bc70e688c45821eea71b766e70d/src/rt/dmain2.d#L158

Now, initSections() and finiSections() are responsible for 
setting up the image. If you look at initSections(), the function 
"_dyld_register_func_for_add_image" is the one that causes the 
crash, because there is no way to remove the callback, which will 
reside in your shared lib.

https://github.com/D-Programming-Language/druntime/blob/1e25749cd01ad08dc08319a3853fbe86356c3e62/src/rt/sections_osx.d#L76

So what happens is, when you call 
_dyld_register_func_for_add_image, dyld will call your callback 
for every shared-library/image(including the main application's 
image) that is currently loaded. However, you can skip the 
callback and just call "sections_osx_onAddImage" yourself.

You would have to add something like this to sections_osx.d, and 
call it instead of adding the callback:

void callOnAddImage()
{
     // dladdr() should give you information about the
     // shared lib in which the symbol you pass resides.
     // Passing the address of this function should work.
     Dl_info info;
     int ret = dladdr(cast(void*)&callOnAddImage, &info);
     assert(ret);

     // "dli_fbase" is actually a pointer to
     // the mach_header for the shared library.
     // once you have the mach_header, you can
     // also retrieve the image slide, and finally
     // call sections_osx_onAddImage().
     mach_header* header = cast(mach_header*)info.dli_fbase;
     intptr_t slide = _dyld_get_image_slide(header);
     sections_osx_onAddImage(header, slide);
}

Now, if you look at finiSections(), it seems to be incomplete. 
There is nothing like sections_osx_onRemoveImage, so you'll have 
to complete it to make sure the library is unloaded correctly:

https://github.com/D-Programming-Language/druntime/blob/1e25749cd01ad08dc08319a3853fbe86356c3e62/src/rt/sections_osx.d#L83

You'll may have to make other mods here and there to get this 
working correctly, but this is the bulk of it.

      Bit
Sep 16 2015
parent reply ponce <contact gam3sfrommars.fr> writes:
On Wednesday, 16 September 2015 at 23:24:29 UTC, bitwise wrote:

 I was trying to solve this one myself, but the modifications to 
 DMD's backend that are needed are out of reach for me right now.

 If you're willing to build your own druntime, you may be able 
 to get by.


I'd prefer a solution that works with existing compilers, but 
maybe building a custom LDC is possible if I figure it out.


 If I understand correctly, you want to repeatedly load/unload 
 the same shared library, correct? I ask because druntime for 
 osx only supports loading a single image at a time:

 https://github.com/D-Programming-Language/druntime/blob/1e25749cd01ad08dc08319a3853fbe86356c3e62/src/rt/sections_osx.d#L26


In practive I've found that the D shared libraries I produce can 
be dlopen/dlclose any number of times, simultaneous too. Using 
both LDC and DMD, don't know why it works.
The thing that doesn't work is the C host program dlopen'ing the 
shared library, dlclose it, then dlopen another shared library 
written in C.



 Anyways, when main() of a D program runs, it calls rt_init() 
 and rt_term(). If you don't have a D entry point in your 
 program, you have to retrieve these from your shared lib(which 
 has druntime statically linked) using dlsym() and call them 
 yourself.


I don't control the host program. My shared libs do have an 
entrypoint, from which I call Runtime.initialize().

I can also use LDC global constructor/destructor to call 
Runtime.initialize / Runtime.terminate, but it doesn't work any 
better because of the callback.




 https://github.com/D-Programming-Language/druntime/blob/478b6c5354470bc70e688c45821eea71b766e70d/src/rt/dmain2.d#L158

 Now, initSections() and finiSections() are responsible for 
 setting up the image. If you look at initSections(), the 
 function "_dyld_register_func_for_add_image" is the one that 
 causes the crash, because there is no way to remove the 
 callback, which will reside in your shared lib.

 https://github.com/D-Programming-Language/druntime/blob/1e25749cd01ad08dc08319a3853fbe86356c3e62/src/rt/sections_osx.d#L76

 So what happens is, when you call 
 _dyld_register_func_for_add_image, dyld will call your callback 
 for every shared-library/image(including the main application's 
 image) that is currently loaded. However, you can skip the 
 callback and just call "sections_osx_onAddImage" yourself.

 You would have to add something like this to sections_osx.d, 
 and call it instead of adding the callback:

 void callOnAddImage()
 {
     // dladdr() should give you information about the
     // shared lib in which the symbol you pass resides.
     // Passing the address of this function should work.
     Dl_info info;
     int ret = dladdr(cast(void*)&callOnAddImage, &info);
     assert(ret);

     // "dli_fbase" is actually a pointer to
     // the mach_header for the shared library.
     // once you have the mach_header, you can
     // also retrieve the image slide, and finally
     // call sections_osx_onAddImage().
     mach_header* header = cast(mach_header*)info.dli_fbase;
     intptr_t slide = _dyld_get_image_slide(header);
     sections_osx_onAddImage(header, slide);
 }

 Now, if you look at finiSections(), it seems to be incomplete. 
 There is nothing like sections_osx_onRemoveImage, so you'll 
 have to complete it to make sure the library is unloaded 
 correctly:

 https://github.com/D-Programming-Language/druntime/blob/1e25749cd01ad08dc08319a3853fbe86356c3e62/src/rt/sections_osx.d#L83

 You'll may have to make other mods here and there to get this 
 working correctly, but this is the bulk of it.

      Bit


Thanks for your answer. This is really helpful, though I don't 
understand the first thing about what images, headers and 
sections are in this context.
Sep 16 2015
next sibling parent bitwise <bitwise.pvt gmail.com> writes:
On Thursday, 17 September 2015 at 06:40:56 UTC, ponce wrote:

 On Wednesday, 16 September 2015 at 23:24:29 UTC, bitwise wrote:

 I was trying to solve this one myself, but the modifications 
 to DMD's backend that are needed are out of reach for me right 
 now.

 If you're willing to build your own druntime, you may be able 
 to get by.


 I'd prefer a solution that works with existing compilers, but 
 maybe building a custom LDC is possible if I figure it out.


Unfortunately, this is a complex problem and you can't just flip 
a few switches and make it work.


 In practive I've found that the D shared libraries I produce 
 can be dlopen/dlclose any number of times, simultaneous too. 
 Using both LDC and DMD, don't know why it works.
 The thing that doesn't work is the C host program dlopen'ing 
 the shared library, dlclose it, then dlopen another shared 
 library written in C.


Calling dlopen won't initialize the runtime in the shared 
library. Although your program doesn't crash, you may find that 
it does once you start calling into the shared lib... or it may 
leak memory because the GC was never initialized. You can call 
Runtime.initialize() from druntime yourself, but as you know, 
there is the callback issue.

I think LDC may be farther along in terms of shared library 
support. You may want to ask David Nadlinger about this. I think 
the problem with LDC was that it only supported statically loaded 
shared libs for osx. I'm not sure if this is still the case, but 
it looks like it:

https://github.com/ldc-developers/druntime/blob/002d0aed65cd24dfcbbc782d9aed2f9112f27b4e/src/rt/sections_ldc.d#L372


 Anyways, when main() of a D program runs, it calls rt_init() 
 and rt_term(). If you don't have a D entry point in your 
 program, you have to retrieve these from your shared lib(which 
 has druntime statically linked) using dlsym() and call them 
 yourself.


 I don't control the host program. My shared libs do have an 
 entrypoint, from which I call Runtime.initialize().

 I can also use LDC global constructor/destructor to call 
 Runtime.initialize / Runtime.terminate, but it doesn't work any 
 better because of the callback.


I don't think there is any way around this beside building a 
custom runtime and implementing the function I suggested.


 Thanks for your answer. This is really helpful, though I don't 
 understand the first thing about what images, headers and 
 sections are in this context.


As a last resort, you may be able to use OSX Distributed Objects, 
depending on what your shared library does. This is basically a 
wrapper around linux pipes, but with a simple RPC-like interface. 
Depending on how much talk there is between the host program and 
your shared lib, this may or may not be a good solution.

https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/DistrObjects/DistrObjects.html

Basically, you would make an Objective-C shared library with a C 
interface, into which the host program could call. This shared 
library would then spawn a second process as needed, which would 
load the D shared library. You would then use distributed objects 
to call from the Objective-C shared library to the D shared 
library. When you were done with the D shared library, you would 
simply terminate the second process.

    Bit
Sep 17 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/16/2015 11:40 PM, ponce wrote:

 On Wednesday, 16 September 2015 at 23:24:29 UTC, bitwise wrote:

 I was trying to solve this one myself, but the modifications to DMD's backend
 that are needed are out of reach for me right now.

 If you're willing to build your own druntime, you may be able to get by.


 I'd prefer a solution that works with existing compilers, but maybe building a
 custom LDC is possible if I figure it out.


I seriously doubt this issue has anything to do with the compiler's code 
generator back end. It's more likely the problem is in druntime.
Sep 17 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Thursday, 17 September 2015 at 10:12:57 UTC, Walter Bright 
wrote:

 On 9/16/2015 11:40 PM, ponce wrote:

 On Wednesday, 16 September 2015 at 23:24:29 UTC, bitwise wrote:

 I was trying to solve this one myself, but the modifications 
 to DMD's backend
 that are needed are out of reach for me right now.

 If you're willing to build your own druntime, you may be able 
 to get by.


 I'd prefer a solution that works with existing compilers, but 
 maybe building a
 custom LDC is possible if I figure it out.


 I seriously doubt this issue has anything to do with the 
 compiler's code generator back end. It's more likely the 
 problem is in druntime.


Based on previous posts you've made, it seems you already know 
that Martin's solution involves having DMD's back end add linux 
style ctors/dtors to the COMDAT section of every compiled D file, 
so I'm not sure what you mean by this.

If a callback provided to _dyld_register_func_for_add_image does 
not call sections_osx_onAddImage, then it must be called somehow.

    Bit
Sep 17 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Thursday, 17 September 2015 at 16:17:11 UTC, bitwise wrote:

 On Thursday, 17 September 2015 at 10:12:57 UTC, Walter Bright 
 wrote:

 On 9/16/2015 11:40 PM, ponce wrote:

 On Wednesday, 16 September 2015 at 23:24:29 UTC, bitwise 
 wrote:

 I was trying to solve this one myself, but the modifications 
 to DMD's backend
 that are needed are out of reach for me right now.

 If you're willing to build your own druntime, you may be 
 able to get by.


 I'd prefer a solution that works with existing compilers, but 
 maybe building a
 custom LDC is possible if I figure it out.


 I seriously doubt this issue has anything to do with the 
 compiler's code generator back end. It's more likely the 
 problem is in druntime.


 Based on previous posts you've made, it seems you already know 
 that Martin's solution involves having DMD's back end add linux 
 style ctors/dtors to the COMDAT section of every compiled D 
 file, so I'm not sure what you mean by this.

 If a callback provided to _dyld_register_func_for_add_image 
 does not call sections_osx_onAddImage, then it must be called 
 somehow.

    Bit


One solution which could work is to disallow static linking of 
druntime on OSX completely....meaning, either don't even 
distribute a static druntime for OSX, or make shared druntime the 
default. This way, druntime would only ever be initialized once 
per process. Knowing this, linux ctors/dtors could be added to 
druntime to initialize it, eliminating the need to call 
rt_init/rt_term before and after main(). Also, if druntime were 
loaded into a C-hosted binary, druntime would automatically be 
initialized by the ctors/dtors. Also, for the ctors/dtors, they 
could be added to the druntime build, and wouldn't having to be 
compiler generated.

I'm not sure about the difference in performance cost over static 
vs shared druntime, but it seems that this is the way things are 
done on OSX. If you look in /usr/lib/ on a mac, practically 
everything is a shared lib.

     Bit
Sep 17 2015
parent reply ponce <contact gam3sfrommars.fr> writes:
On Thursday, 17 September 2015 at 16:42:52 UTC, bitwise wrote:

 One solution which could work is to disallow static linking of 
 druntime on OSX completely....meaning, either don't even 
 distribute a static druntime for OSX, or make shared druntime 
 the default. This way, druntime would only ever be initialized 
 once per process. Knowing this, linux ctors/dtors could be 
 added to druntime to initialize it, eliminating the need to 
 call rt_init/rt_term before and after main(). Also, if druntime 
 were loaded into a C-hosted binary, druntime would 
 automatically be initialized by the ctors/dtors. Also, for the 
 ctors/dtors, they could be added to the druntime build, and 
 wouldn't having to be compiler generated.

 I'm not sure about the difference in performance cost over 
 static vs shared druntime, but it seems that this is the way 
 things are done on OSX. If you look in /usr/lib/ on a mac, 
 practically everything is a shared lib.

     Bit


I use static linking of druntime already all the time and rely on 
it to be able to do something instead of nothing (where would I 
even found that shared druntime?). Apart from this one horrible 
bug, static runtime seems very much working. Remove possibilities 
to do work would make my situation worse.

I can call rt_init / rt_term at the right place with LDC global 
constructor/destructors no problem. The problem is this callback 
that cannot be removed. Don't know why it's there in the first 
place since by definition a shared library can't control when 
it's unloaded.
Sep 17 2015
parent bitwise <bitwise.pvt gmail.com> writes:
On Thursday, 17 September 2015 at 16:54:09 UTC, ponce wrote:

 On Thursday, 17 September 2015 at 16:42:52 UTC, bitwise wrote:

     [...]


 I use static linking of druntime already all the time and rely 
 on it to be able to do something instead of nothing (where 
 would I even found that shared druntime?). Apart from this one 
 horrible bug, static runtime seems very much working. Remove 
 possibilities to do work would make my situation worse.

 I can call rt_init / rt_term at the right place with LDC global 
 constructor/destructors no problem. The problem is this 
 callback that cannot be removed. Don't know why it's there in 
 the first place since by definition a shared library can't 
 control when it's unloaded.


It seems that you either haven't read, or did not understand my 
previous posts. This is a complicated problem, and unless you're 
willing to dig much deeper into it, then you'll have to deal with 
things the way they are.

    Bit
Sep 17 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-09-17 00:29, ponce wrote:

 Context: On OSX, a C program can load a D shared library but once
 unloaded the next dlopen will crash, jumping into a callback that
 doesn't exist anymore.

 I've filed it here: https://issues.dlang.org/show_bug.cgi?id=15060


 It looks like this was known and discussed several times already:
 http://forum.dlang.org/post/vixoqmidlbizawbxmsao forum.dlang.org (2015)
 https://github.com/D-Programming-Language/druntime/pull/228 (2012)


 Any idea to work-around this problem would be awesome.

 I'm not looking for something correct, future-proof, or pretty. Any shit
 that still stick to the wall will do. Anything!

 The only case I need to support is: C host, D shared library, with
 runtime statically linked.

 Please help!


Easiest would be to not unload the library. If that doesn't work, 
replace "_dyld_register_func_for_add_image" [1] with 
"dyld_register_image_state_change_handler" [2].

[1] 
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L76

[2] 
http://www.opensource.apple.com/source/dyld/dyld-353.2.3/include/mach-o/dyld_priv.h

-- 
/Jacob Carlborg
Sep 17 2015
next sibling parent reply bitwise <bitwise.pvt gmail.com> writes:
On Thursday, 17 September 2015 at 15:12:43 UTC, Jacob Carlborg 
wrote:

 On 2015-09-17 00:29, ponce wrote:

 [...]


 Easiest would be to not unload the library. If that doesn't 
 work, replace "_dyld_register_func_for_add_image" [1] with 
 "dyld_register_image_state_change_handler" [2].

 [1] 
 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L76

 [2] 
 http://www.opensource.apple.com/source/dyld/dyld-353.2.3/include/mach-o/dyld_priv.h


dyld_register_image_state_change_handler does not provide a way 
to unregister the callback either, so I don't see how this helps.
Sep 17 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-09-17 18:20, bitwise wrote:


 dyld_register_image_state_change_handler does not provide a way to
 unregister the callback either, so I don't see how this helps.


The dynamic library holding the callback is pinned. See the 
implementation of registerImageStateSingleChangeHandler, the first few 
lines.

[1] http://www.opensource.apple.com/source/dyld/dyld-353.2.3/src/dyld.cpp

-- 
/Jacob Carlborg
Sep 17 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Thursday, 17 September 2015 at 19:20:34 UTC, Jacob Carlborg 
wrote:

 On 2015-09-17 18:20, bitwise wrote:


 dyld_register_image_state_change_handler does not provide a 
 way to
 unregister the callback either, so I don't see how this helps.


 The dynamic library holding the callback is pinned. See the 
 implementation of registerImageStateSingleChangeHandler, the 
 first few lines.

 [1] 
 http://www.opensource.apple.com/source/dyld/dyld-353.2.3/src/dyld.cpp


Ok, but this kinda defeats the purpose, as the op wants to unload 
the library ;)

     Bit
Sep 17 2015
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-09-17 21:42, bitwise wrote:


 Ok, but this kinda defeats the purpose, as the op wants to unload the
 library ;)


I'm not sure. Apparently he has no control of the host which will, most 
likely, unload regardless if he wants to or not.

-- 
/Jacob Carlborg
Sep 17 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-09-17 21:42, bitwise wrote:


 Ok, but this kinda defeats the purpose, as the op wants to unload the
 library ;)


He said he doesn't want dlopen to crash, if it doesn't unload it fixes 
the problem ;)

-- 
/Jacob Carlborg
Sep 17 2015
next sibling parent reply bitwise <bitwise.pvt gmail.com> writes:
On Thursday, 17 September 2015 at 20:47:49 UTC, Jacob Carlborg 
wrote:

 On 2015-09-17 21:42, bitwise wrote:


 Ok, but this kinda defeats the purpose, as the op wants to 
 unload the
 library ;)


 He said he doesn't want dlopen to crash, if it doesn't unload 
 it fixes the problem ;)


True. Looking at his bug report now, it seems his dylib is a VST 
plugin. Had he just said that to begin with, this conversation 
would have been a lot easier -_-

The op shouldn't have to actually modify druntime in this case. 
He shouldn't have to replace "_dyld_register_func_for_add_image".

He can simply create a second empty callback in VSTPluginMain 
which will pin his library:

static bool didInitRuntime = false;

const char* ignoreImageLoad(
     enum dyld_image_states state,
     uint32_t infoCount,
     const struct dyld_image_info info[])
{
     // ignore.
}

extern(C) void* VSTPluginMain(void* hostCallback)
{
     import core.runtime;

     if(!didInitRuntime)
     {
         Runtime.initialize();
         dyld_register_image_state_change_handler(
             dyld_image_state_initialized,
             &ignoreImageLoad);
         didInitRuntime = true;
     }

     import std.stdio;
     import core.stdc.stdio;
     printf("Hello !\n");

     // Runtime.terminate();
     return null;
}

     Bit
Sep 17 2015
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-09-17 23:13, bitwise wrote:


 True. Looking at his bug report now, it seems his dylib is a VST plugin.
 Had he just said that to begin with, this conversation would have been a
 lot easier -_-

 The op shouldn't have to actually modify druntime in this case. He
 shouldn't have to replace "_dyld_register_func_for_add_image".

 He can simply create a second empty callback in VSTPluginMain which will
 pin his library


That's pretty clever, why didn't I think of that :)

-- 
/Jacob Carlborg
Sep 18 2015
prev sibling next sibling parent reply ponce <contact gam3sfrommars.fr> writes:
On Thursday, 17 September 2015 at 21:13:46 UTC, bitwise wrote:

 On Thursday, 17 September 2015 at 20:47:49 UTC, Jacob Carlborg 
 wrote:

 On 2015-09-17 21:42, bitwise wrote:


 Ok, but this kinda defeats the purpose, as the op wants to 
 unload the
 library ;)


 He said he doesn't want dlopen to crash, if it doesn't unload 
 it fixes the problem ;)


 True. Looking at his bug report now, it seems his dylib is a 
 VST plugin. Had he just said that to begin with, this 
 conversation would have been a lot easier -_-

 The op shouldn't have to actually modify druntime in this case. 
 He shouldn't have to replace 
 "_dyld_register_func_for_add_image".

 He can simply create a second empty callback in VSTPluginMain 
 which will pin his library:

 static bool didInitRuntime = false;

 const char* ignoreImageLoad(
     enum dyld_image_states state,
     uint32_t infoCount,
     const struct dyld_image_info info[])
 {
     // ignore.
 }

 extern(C) void* VSTPluginMain(void* hostCallback)
 {
     import core.runtime;

     if(!didInitRuntime)
     {
         Runtime.initialize();
         dyld_register_image_state_change_handler(
             dyld_image_state_initialized,
             &ignoreImageLoad);
         didInitRuntime = true;
     }

     import std.stdio;
     import core.stdc.stdio;
     printf("Hello !\n");

     // Runtime.terminate();
     return null;
 }

     Bit


Much success.
Not only did this work, it worked (around) right away!

Final patch here:
https://github.com/p0nce/dplug/commit/7dc6385ebb8147cc53cfe69bfd54e41f5341e158

Thanks to you all and I will for sure include your name in the 
release notes.
You removed a huge roadblock on my path to being relevant.
Sep 18 2015
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-09-18 11:43, ponce wrote:


 Much success.
 Not only did this work, it worked (around) right away!

 Final patch here:
 https://github.com/p0nce/dplug/commit/7dc6385ebb8147cc53cfe69bfd54e41f5341e158


The way it behaves is that the runtime uses a function to register the a 
callback which is called by the dynamic linker each time an image 
(dynamic library/exectuable) is loaded.

When the callback points to a function that itself is located in a 
dynamic library it will cause dlopen to crash if the dynamic library has 
been unloaded because the callback is not valid anymore.

For some reason it's not possible to unregister a callback.

When "dyld_register_image_state_change_handler" is used instead, it pins 
the dynamic library which contains the callback and it's actually never 
unloaded, even though dlclose is called.

So when dyld_register_image_state_change_handler is called from you're 
plugin it pins the dynamic library and the callback register in the 
runtime will still be valid.

-- 
/Jacob Carlborg
Sep 18 2015
prev sibling parent reply bitwise <bitwise.pvt gmail.com> writes:
On Friday, 18 September 2015 at 09:43:43 UTC, ponce wrote:

 On Thursday, 17 September 2015 at 21:13:46 UTC, bitwise wrote:

     [...]


 Much success.
 Not only did this work, it worked (around) right away!

 Final patch here:
 https://github.com/p0nce/dplug/commit/7dc6385ebb8147cc53cfe69bfd54e41f5341e158

 Thanks to you all and I will for sure include your name in the 
 release notes.
 You removed a huge roadblock on my path to being relevant.


Woot! Go team! =)
Sep 18 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-09-18 16:53, bitwise wrote:


 Woot! Go team! =)


No, no, no. D team :D

-- 
/Jacob Carlborg
Sep 18 2015
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Thursday, 17 September 2015 at 21:13:46 UTC, bitwise wrote:

 He can simply create a second empty callback in VSTPluginMain 
 which will pin his library:


Yikes, pinning the library is really ugly hack around the actual 
issue.
Anyone has an idea how to use the crappy dyld API w/o crashing on 
unload?

If nothing helps we could try to add init/fini calls to binaries 
like we do w/ ELF.
Sep 19 2015
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-09-19 18:53, Martin Nowak wrote:


 Yikes, pinning the library is really ugly hack around the actual issue.
 Anyone has an idea how to use the crappy dyld API w/o crashing on unload?


I don't think it possible since there's no API to unregister the callback.


 If nothing helps we could try to add init/fini calls to binaries like we
 do w/ ELF.


I think this would be best choice. That's how the libSystem (the C 
standard library and a couple of other libraries) is initialized.

-- 
/Jacob Carlborg
Sep 19 2015
prev sibling parent bitwise <bitwise.pvt gmail.com> writes:
On Saturday, 19 September 2015 at 16:53:28 UTC, Martin Nowak 
wrote:

 On Thursday, 17 September 2015 at 21:13:46 UTC, bitwise wrote:

 He can simply create a second empty callback in VSTPluginMain 
 which will pin his library:


 Yikes, pinning the library is really ugly hack around the 
 actual issue.
 Anyone has an idea how to use the crappy dyld API w/o crashing 
 on unload?

 If nothing helps we could try to add init/fini calls to 
 binaries like we do w/ ELF.


I wanted to attempt this myself, but I think I'm going to throw 
in the towel...

I wunderstand the mach-o format ok, and was able to get druntime 
to build as a shared library, but after spending some time 
looking through DMD's backend... I find myself very discouraged 
by the code quality.

Not trying to be mean, but that code was clearly not written with 
teamwork in mind. With the vast majority of methods and variables 
being named things like "bytes" or "SDshtidx", I find myself all 
but completely helpless.

I understand why people keep complaining about the backend and 
trying to say it should be swapped out for llvm or something.

     Bit
Sep 19 2015
prev sibling parent Martin Nowak <code dawg.eu> writes:
On Thursday, 17 September 2015 at 21:13:46 UTC, bitwise wrote:

 The op shouldn't have to actually modify druntime in this case. 
 He shouldn't have to replace 
 "_dyld_register_func_for_add_image".


Now that you guys digged in the dyld implementation, did anyone 
find a solution to use the callback mechanism and still support 
unloading of libraries?
Pinning libraries is only a hacky workaround.

Let's please continue the discussion in Bugzilla.
https://issues.dlang.org/show_bug.cgi?id=15060#c4
Sep 27 2015
prev sibling parent ponce <contact gam3sfrommars.fr> writes:
On Thursday, 17 September 2015 at 20:47:49 UTC, Jacob Carlborg 
wrote:

 On 2015-09-17 21:42, bitwise wrote:


 Ok, but this kinda defeats the purpose, as the op wants to 
 unload the
 library ;)


 He said he doesn't want dlopen to crash, if it doesn't unload 
 it fixes the problem ;)


Yes, it would help.
Sep 17 2015
prev sibling parent ponce <contact gam3sfrommars.fr> writes:
On Thursday, 17 September 2015 at 15:12:43 UTC, Jacob Carlborg 
wrote:

 Easiest would be to not unload the library.


I don't control what the host program does.



 If that doesn't work, replace 
 "_dyld_register_func_for_add_image" [1] with 
 "dyld_register_image_state_change_handler" [2].

 [1] 
 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L76

 [2] 
 http://www.opensource.apple.com/source/dyld/dyld-353.2.3/include/mach-o/dyld_priv.h


Looks good.
Sep 17 2015