www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - shared libs for OSX

reply "bitwise" <bitwise.pvt gmail.com> writes:
I tried using a shared library for OSX yesterday. I opened it 
with dlopen, retrieved my extern(C) function, and called it. All 
was well, and it seemed to work(wrote to the console with 
writeln).

But, I got a message in the console saying shared libraries were 
not yet implemented for OSX. What exactly is it that's not 
working?

  Thanks,
    Bit
May 20 2015
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-05-20 16:44, bitwise wrote:
 I tried using a shared library for OSX yesterday. I opened it with
 dlopen, retrieved my extern(C) function, and called it. All was well,
 and it seemed to work(wrote to the console with writeln).

 But, I got a message in the console saying shared libraries were not yet
 implemented for OSX. What exactly is it that's not working?
TLS, module constructors, module/class info, perhaps the GC, similar things. -- /Jacob Carlborg
May 20 2015
parent reply "bitwise" <bitwise.pvt gmail.com> writes:
On Wednesday, 20 May 2015 at 18:53:30 UTC, Jacob Carlborg wrote:
 On 2015-05-20 16:44, bitwise wrote:
 I tried using a shared library for OSX yesterday. I opened it 
 with
 dlopen, retrieved my extern(C) function, and called it. All 
 was well,
 and it seemed to work(wrote to the console with writeln).

 But, I got a message in the console saying shared libraries 
 were not yet
 implemented for OSX. What exactly is it that's not working?
TLS, module constructors, module/class info, perhaps the GC, similar things.
Heh.. That's pretty useless. Any idea on the state of things? Like if there are plans to support this in the near future(few months)? I couldn't find many conversations on this. This kinda-totally ruins my plans(on OSX at least).
May 20 2015
next sibling parent reply bitwise <bitwise.pvt gmail.com> writes:
On Wed, 20 May 2015 17:35:37 -0400, bitwise <bitwise.pvt gmail.com> wrote:

 On Wednesday, 20 May 2015 at 18:53:30 UTC, Jacob Carlborg wrote:
 On 2015-05-20 16:44, bitwise wrote:
 I tried using a shared library for OSX yesterday. I opened it with
 dlopen, retrieved my extern(C) function, and called it. All was well,
 and it seemed to work(wrote to the console with writeln).

 But, I got a message in the console saying shared libraries were not  
 yet
 implemented for OSX. What exactly is it that's not working?
TLS, module constructors, module/class info, perhaps the GC, similar things.
Heh.. That's pretty useless. Any idea on the state of things? Like if there are plans to support this in the near future(few months)? I couldn't find many conversations on this. This kinda-totally ruins my plans(on OSX at least).
I've been reading over all the bugs and conversations I can find, but I'm still confused about exactly what's going on here. What I need: -load a dynamic library on OSX using dlopen or Runtime.loadLibrary -use dlsym to retrieve my functions and set up a gc proxy -pull interfaces of classes defined in the dynamic library into my program for use. Not sure how much of this is working properly or not. I'd be willing to throw a couple of bucks at this for a bounty, but I'm not sure how much it's worth though. Any takers for $25? ;) Bit
May 20 2015
parent "Martin Nowak" <code dawg.eu> writes:
On Thursday, 21 May 2015 at 01:31:37 UTC, bitwise wrote:
 I've been reading over all the bugs and conversations I can 
 find, but I'm still confused about exactly what's going on here.

 What I need:
 -load a dynamic library on OSX using dlopen or 
 Runtime.loadLibrary
 -use dlsym to retrieve my functions and set up a gc proxy
Forget about the GC proxy, it's a hack to work around ODR issues. What you really need is a shared phobos library as we have on linux or freebsd.
 -pull interfaces of classes defined in the dynamic library into 
 my program for use.
If you want to exchange data and code across multiple shared libraries and your executable the runtime must be fully aware of all shared libraries to support casting ,exceptions and finalizers.
May 22 2015
prev sibling next sibling parent reply "Joakim" <dlang joakim.fea.st> writes:
On Wednesday, 20 May 2015 at 21:35:38 UTC, bitwise wrote:
 On Wednesday, 20 May 2015 at 18:53:30 UTC, Jacob Carlborg wrote:
 On 2015-05-20 16:44, bitwise wrote:
 I tried using a shared library for OSX yesterday. I opened it 
 with
 dlopen, retrieved my extern(C) function, and called it. All 
 was well,
 and it seemed to work(wrote to the console with writeln).

 But, I got a message in the console saying shared libraries 
 were not yet
 implemented for OSX. What exactly is it that's not working?
TLS, module constructors, module/class info, perhaps the GC, similar things.
Heh.. That's pretty useless. Any idea on the state of things? Like if there are plans to support this in the near future(few months)? I couldn't find many conversations on this. This kinda-totally ruins my plans(on OSX at least).
Actually, TLS works on OS X: dmd emulates it, while ldc has native TLS using undocumented OS X APIs. I'm unsure of how full-fledged the support is on those other issues, but I believe the big issue is that loading multiple shared libraries is not supported, as that will take more work: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L198 If you want to talk to someone about getting more of shared libraries working, I believe Martin is your man.
May 20 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-05-21 07:06, Joakim wrote:

 Actually, TLS works on OS X: dmd emulates it, while ldc has native TLS
 using undocumented OS X APIs.
The emulated TLS won't work with dynamic libraries. The runtime can only fetch the TLS data from one source, which will be the executable itself.
 I'm unsure of how full-fledged the
 support is on those other issues, but I believe the big issue is that
 loading multiple shared libraries is not supported, as that will take
 more work:

 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L198
No, that issue is loading dynamic libraries regardless if it's one or several. As you can see in the comment, it will pick the sections from the executable.
 If you want to talk to someone about getting more of shared libraries
 working, I believe Martin is your man.
Yeah, but I'm not sure how familiar he is with the OS X specific things. And native TLS is a requirement, which requires changes in the compiler and the runtime. Alternatively we need to reimplement the TLS handling from the dynamic linker. -- /Jacob Carlborg
May 21 2015
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-05-20 23:35, bitwise wrote:

 Heh.. That's pretty useless. Any idea on the state of things? Like if
 there are plans to support this in the near future(few months)? I
 couldn't find many conversations on this. This kinda-totally ruins my
 plans(on OSX at least).
I don't think anyone is working on this. Native TLS is a prerequisite which requires changes both to the compiler and the runtime. There's an enhancement request for native TLS [1]. [1] https://issues.dlang.org/show_bug.cgi?id=9476 -- /Jacob Carlborg
May 21 2015
next sibling parent reply Timothee Cour via Digitalmars-d <digitalmars-d puremagic.com> writes:
dlopen() of a D shared library works when called from a C++ file, and that
a C++ file can call multiple D shared libraries simultaneously when using
RTLD_LAZY option.

Before waiting for a full solution with an integrated druntime, is there at
least a way to have a separate runtime in each shared library, so that
dlopen() can be called from a D file?

On Thu, May 21, 2015 at 1:07 AM, Jacob Carlborg via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 On 2015-05-20 23:35, bitwise wrote:

  Heh.. That's pretty useless. Any idea on the state of things? Like if
 there are plans to support this in the near future(few months)? I
 couldn't find many conversations on this. This kinda-totally ruins my
 plans(on OSX at least).
I don't think anyone is working on this. Native TLS is a prerequisite which requires changes both to the compiler and the runtime. There's an enhancement request for native TLS [1]. [1] https://issues.dlang.org/show_bug.cgi?id=9476 -- /Jacob Carlborg
May 21 2015
next sibling parent reply bitwise <bitwise.pvt gmail.com> writes:
On Thu, 21 May 2015 04:12:19 -0400, Timothee Cour via Digitalmars-d  
<digitalmars-d puremagic.com> wrote:

 dlopen() of a D shared library works when called from a C++ file, and  
 that
 a C++ file can call multiple D shared libraries simultaneously when using
 RTLD_LAZY option.

 Before waiting for a full solution with an integrated druntime, is there  
 at
 least a way to have a separate runtime in each shared library, so that
 dlopen() can be called from a D file?
I'm not sure exactly what you mean about "integrated runtime". Looking in /usr/share/dmd/lib, I see phobos as a static library (which I'm assuming includes druntime, which I don't see anywhere). If the runtime is linked as a static library, shouldn't it be able to link to a dynamic library and just work? I think a separate runtime is what I was expecting.
 On Thu, May 21, 2015 at 1:07 AM, Jacob Carlborg via Digitalmars-d <
 digitalmars-d puremagic.com> wrote:
 I don't think anyone is working on this. Native TLS is a prerequisite
 which requires changes both to the compiler and the runtime. There's an
 enhancement request for native TLS [1].

 [1] https://issues.dlang.org/show_bug.cgi?id=9476

 --
 /Jacob Carlborg
(In reply to John Colvin from comment #11)
 I think the time has come that 10.6 support could be dropped, no? It's no
 longer supported by Apple.
+1 Bit
May 21 2015
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-05-21 17:51, bitwise wrote:

 I'm not sure exactly what you mean about "integrated runtime".

 Looking in /usr/share/dmd/lib, I see phobos as a static library (which
 I'm assuming includes druntime, which I don't see anywhere). If the
 runtime is linked as a static library, shouldn't it be able to link to a
 dynamic library and just work?

 I think a separate runtime is what I was expecting.
No, see my answer to Timothee [1]. [1] http://forum.dlang.org/post/mjlc10$2pi5$1 digitalmars.com -- /Jacob Carlborg
May 21 2015
prev sibling parent "Martin Nowak" <code dawg.eu> writes:
On Thursday, 21 May 2015 at 15:51:38 UTC, bitwise wrote:
 I'm not sure exactly what you mean about "integrated runtime".
I think he means Phobos/Druntime as shared library.
 Looking in /usr/share/dmd/lib, I see phobos as a static library 
 (which I'm assuming includes druntime, which I don't see 
 anywhere). If the runtime is linked as a static library, 
 shouldn't it be able to link to a dynamic library and just work?

 I think a separate runtime is what I was expecting.
Yes separate shared libraries (with multiple runtimes) work on every other platform. The problem for OSX is that onAddImage gets called for the executable and every shared library. It would be trivial to only register the image containing the current runtime, by comparing the address of a private symbol with the memory region of the images. https://github.com/D-Programming-Language/druntime/blob/6331ab1ae19f3ff82449a5734b59d81b128685f4/src/rt/sections_osx.d#L186
May 22 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-05-21 10:12, Timothee Cour via Digitalmars-d wrote:
 dlopen() of a D shared library works when called from a C++ file, and
 that a C++ file can call multiple D shared libraries simultaneously when
 using RTLD_LAZY option.

 Before waiting for a full solution with an integrated druntime, is there
 at least a way to have a separate runtime in each shared library, so
 that dlopen() can be called from a D file?
No, not out of the box. I'm not sure if I remember all the details correctly, but this is basically how it works: The runtime uses the "_dyld_register_func_for_add_image" function provided by the dynamic linker. This function is used to register a callback. The callback will be called for each currently loaded image (executable/dynamic library) in the executable. The callback will also be called for every newly loaded image, i.e. using dlopen. You need to keep track of which image is yourself and which are other images you should ignore. Then, the other problem with "_dyld_register_func_for_add_image" is that it's not possible to unregister the callback. Which means, if you register a callback using this function from a dynamic library loaded with dlopen and then unload it. Next time dlopen is called it will crash, because the callback points to an address that doesn't exist anymore. There's an other, undocumented, function that does basically the same but prevents the issue with the callback by not allowing the dynamic library to be unloaded. -- /Jacob Carlborg
May 21 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Thu, 21 May 2015 15:34:56 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2015-05-21 10:12, Timothee Cour via Digitalmars-d wrote:
 dlopen() of a D shared library works when called from a C++ file, and
 that a C++ file can call multiple D shared libraries simultaneously when
 using RTLD_LAZY option.

 Before waiting for a full solution with an integrated druntime, is there
 at least a way to have a separate runtime in each shared library, so
 that dlopen() can be called from a D file?
No, not out of the box. I'm not sure if I remember all the details correctly, but this is basically how it works: The runtime uses the "_dyld_register_func_for_add_image" function provided by the dynamic linker. This function is used to register a callback. The callback will be called for each currently loaded image (executable/dynamic library) in the executable. The callback will also be called for every newly loaded image, i.e. using dlopen. You need to keep track of which image is yourself and which are other images you should ignore. Then, the other problem with "_dyld_register_func_for_add_image" is that it's not possible to unregister the callback. Which means, if you register a callback using this function from a dynamic library loaded with dlopen and then unload it. Next time dlopen is called it will crash, because the callback points to an address that doesn't exist anymore. There's an other, undocumented, function that does basically the same but prevents the issue with the callback by not allowing the dynamic library to be unloaded.
So does that mean that "_dyld_register_func_for_add_image" shouldn't be called at all from shared libs at all then? The fact that there is no "unregister" function seems to suggest that it's meant to be called once from the main executable to add a callbacck that survives the entire life of the program. Can't whatever the callback does be done from the runtime linked to the main executable? Bit
May 21 2015
next sibling parent bitwise <bitwise.pvt gmail.com> writes:
On Thu, 21 May 2015 21:13:56 -0400, bitwise <bitwise.pvt gmail.com> wrote:

 Can't whatever the callback does be done from the runtime linked to the  
 main executable?
Or simply use dlsym from the main executable to call back into the dylib to set things up? Bit
May 21 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-05-22 03:13, bitwise wrote:

 So does that mean that "_dyld_register_func_for_add_image" shouldn't be
 called at all from shared libs at all then? The fact that there is no
 "unregister" function seems to suggest that it's meant to be called once
 from the main executable to add a callbacck that survives the entire
 life of the program.
Yes.
 Can't whatever the callback does be done from the
 runtime linked to the main executable?
The runtime extracts some data from the loaded image and stores it in some global variable. If the executable would do this it needs to store the data in the global variable in the dynamic library. -- /Jacob Carlborg
May 23 2015
prev sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Thursday, 21 May 2015 at 08:07:49 UTC, Jacob Carlborg wrote:
 I don't think anyone is working on this. Native TLS is a 
 prerequisite which requires changes both to the compiler and 
 the runtime. There's an enhancement request for native TLS [1].
One could also make emulated TLS work with shared libraries.
May 22 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-05-22 18:12, Martin Nowak wrote:

 One could also make emulated TLS work with shared libraries.
Yeah, but then you would need to implement something similar to the TLS code already present in the dynamic linker in druntime. Both alternatives would require changes to both the compiler and runtime. Hopefully implementing native TLS in the compiler would not be any more difficult than adding support for shared libraries to the emulated TLS. -- /Jacob Carlborg
May 23 2015
prev sibling parent "Martin Nowak" <code dawg.eu> writes:
On Wednesday, 20 May 2015 at 21:35:38 UTC, bitwise wrote:
 Heh.. That's pretty useless. Any idea on the state of things? 
 Like if there are plans to support this in the near future(few 
 months)? I couldn't find many conversations on this. This 
 kinda-totally ruins my plans(on OSX at least).
Have you tried LDC? They recently got shared library support, maybe also on OSX.
May 22 2015
prev sibling parent reply bitwise <bitwise.pvt gmail.com> writes:
I've read through these now, which I missed the first time around, so  
sorry for making you guys repeat yourselves ;)

<Runtime issue on Mac OS X>
http://comments.gmane.org/gmane.comp.lang.d.runtime/1214

<ideas for runtime loading of shared libraries.>
http://forum.dlang.org/thread/mailman.2052.1325532031.24802.digitalmars-d puremagic.com

So in terms of a shared lib having it's own runtime, we have these  
problems:

[1] problem

On Fri, 22 May 2015 12:04:24 -0400, Martin Nowak <code dawg.eu> wrote:
 Yes separate shared libraries (with multiple runtimes) work on every  
 other platform.
 The problem for OSX is that onAddImage gets called for the executable  
 and every shared library. It would be trivial to only register the image  
 containing the current runtime, by comparing the address of a private  
 symbol with the memory region of the images.

 https://github.com/D-Programming-Language/druntime/blob/6331ab1ae19f3ff82449a5734b59d81b128685f4/src/rt/sections_osx.d#L186
[2] problem On Thu, 21 May 2015 15:34:56 -0400, Jacob Carlborg <doob me.com> wrote:
 The runtime uses the "_dyld_register_func_for_add_image" function  
 provided by the dynamic linker. This function is used to register a  
 callback. The callback will be called for each currently loaded image  
 (executable/dynamic library) in the executable. The callback will also  
 be called for every newly loaded image, i.e. using dlopen. You need to  
 keep track of which image is yourself and which are other images you  
 should ignore.

 Then, the other problem with "_dyld_register_func_for_add_image" is that  
 it's not possible to unregister the callback. Which means, if you  
 register a callback using this function from a dynamic library loaded  
 with dlopen and then unload it. Next time dlopen is called it will  
 crash, because the callback points to an address that doesn't exist  
 anymore.
I think I have found solutions for these problems. [1] solution I've modified sections_osx.d as follows: extern (C) void _sections_osx_onAddImage_STUB(in mach_header* h, intptr_t slide) { // empty } extern (C) void sections_osx_onAddImage(in mach_header* h, intptr_t slide) { // on mac osx, Dl_info.dli_fbase is a pointer to the mach_header for the library. [I] // here we return unless onAddImage is being called for the current library mach_header* myHeader = null; Dl_info info; // this line also makes sure that the stub isn't // removed by the linker(it's needed for [2]) if(dladdr(&_sections_osx_onAddImage_STUB, &info)) { mach_header* mh = cast(mach_header*)info.dli_fbase; if(mh == cast(mach_header*)h) myHeader = mh; } if(!myHeader) return; // initialize sections..... } [2] solution Although the callback passed to "_dyld_register_func_for_add_image" cannot be removed, it can be replaced, so I've replaced it with a pointer to the stub located in the main program. I've modified initSections() in sections_osx.d as follows: void initSections() { pthread_key_create(&_tlsKey, null); // register the callback as usual. This will call the callback // for every library currently loaded before returning, so // once it has returned, it should be safe to set the callback // to something else(the empty stub) _dyld_register_func_for_add_image(&sections_osx_onAddImage); // dlopen(null, ..) will retrieve a handle to the main program [II] // OSX docs says it returns the first symbol found using // "RTLD_DEFAULT" or "the default library search order" which // should(and does as far as I can tell) return the handle // to the main program void *main = dlopen(null, RTLD_NOW); assert(main); alias typeof(&_sections_osx_onAddImage_STUB) addImgFn; addImgFn func = cast(addImgFn)dlsym(main, "_sections_osx_onAddImage_STUB"); assert(func); // set the callback to the empty stub in the main program _dyld_register_func_for_add_image(func); _isRuntimeInitialized = true; } [I] https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dladdr.3.html [II] https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/dlopen.3.html http://linux.die.net/man/3/dlopen So at this point, it seems like these two fixes work as expected, but now, I'm having some new and very strange problems. I have a simple shared library and program I've been using to test this: [main.d] module main; import std.stdio; import std.conv; import std.string; import core.sys.posix.dlfcn; void main(string[] args) { alias void function() fnType; void *handle = dlopen("myShared.dylib", RTLD_NOW); assert(handle); fnType init = cast(fnType)dlsym(handle, "initLib"); assert(init); init(); fnType term = cast(fnType)dlsym(handle, "termLib"); assert(term); term(); dlclose(handle); writeln("done"); } [myShared.d] module myShared; import core.runtime; import std.stdio; extern(C) void initLib() { writeln("Initializing Runtime"); Runtime.initialize(); } extern(C) void termLib() { writeln("Terminating Runtime"); Runtime.terminate(); } So, when I run the above program, rt_init() should be called once for the main program, and once for the shared library. However, when I run the above program, rt_init() from the main program seems to get called twice. To clarify, I mean that when I retrieve "initLib()" with dlsym() and call it, rt_init() from the main module gets called. This seems to prove the above: In dmain2.d, I have modified rt_init() as follows: extern (C) int rt_init() { import core.sys.posix.dlfcn; Dl_info info; if(dladdr(&rt_init, &info)) fprintf(stdout, "RT INIT: %s\n", info.dli_fname); // this prints "main" for both calls if (atomicOp!"+="(_initCount, 1) > 1) { fprintf(stdout, "RT ALREADY INITIALIZED\n"); return 1; } // ... fprintf(stdout, "RT INIT COMPLETE\n"); } When the main program calls rt_init(), the output correctly reads "RT INIT COMPLETE". When I load the dynamic library however, I get the output "RT ALREADY INITIALIZED" How is this possible? I am not using a shared druntime afaik.. Bit
May 24 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Sun, 24 May 2015 15:40:04 -0400, bitwise <bitwise.pvt gmail.com> wrote:

 [snip]

 So at this point, it seems like these two fixes work as expected, but  
 now, I'm having some new and very strange problems.

 I have a simple shared library and program I've been using to test this:

 [main.d]
 module main;
 import std.stdio;
 import std.conv;
 import std.string;
 import core.sys.posix.dlfcn;

 void main(string[] args)
 {
      alias void function() fnType;

      void *handle = dlopen("myShared.dylib", RTLD_NOW);
      assert(handle);

      fnType init = cast(fnType)dlsym(handle, "initLib");
      assert(init);
      init();

      fnType term = cast(fnType)dlsym(handle, "termLib");
      assert(term);
      term();

      dlclose(handle);
      writeln("done");
 }

 [myShared.d]
 module myShared;
 import core.runtime;
 import std.stdio;

 extern(C) void initLib() {
      writeln("Initializing Runtime");
      Runtime.initialize();
 }

 extern(C) void termLib() {
      writeln("Terminating Runtime");
      Runtime.terminate();
 }


 So, when I run the above program, rt_init() should be called once for  
 the main program, and once for the shared library. However, when I run  
 the above program, rt_init() from the main program seems to get called  
 twice. To clarify, I mean that when I retrieve "initLib()" with dlsym()  
 and call it, rt_init() from the main module gets called.

 This seems to prove the above:

 In dmain2.d, I have modified rt_init() as follows:

 extern (C) int rt_init()
 {
      import core.sys.posix.dlfcn;
      Dl_info info;
      if(dladdr(&rt_init, &info))
          fprintf(stdout, "RT INIT: %s\n", info.dli_fname);   // this  
 prints "main" for both calls

      if (atomicOp!"+="(_initCount, 1) > 1)
      {
          fprintf(stdout, "RT ALREADY INITIALIZED\n");
          return 1;
      }

 // ...

      fprintf(stdout, "RT INIT COMPLETE\n");
 }

 When the main program calls rt_init(), the output correctly reads "RT  
 INIT COMPLETE".
 When I load the dynamic library however, I get the output "RT ALREADY  
 INITIALIZED"

 How is this possible? I am not using a shared druntime afaik..
I just found that gcc has "-fvisibility=hidden" and "-fvisibility-ms-compat" which are equivalent. The lack of this functionality for dmd seems to explain the above problem. Is this correct? Any ideas? Bit
May 24 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
I've been reading through the Mach-O docs[1], and it seems that dynamic  
libs are treated the same as static libs in that exported symbols can only  
be defined once, even across dynamically loaded libraries. This is why  
calling rt_init from my dylib ended up calling the one that was already  
defined in the application image. I was able to call the rt_init from the  
dylib by specifically requesting it through dlsym, but then, all the  
global variables it used(_isRuntimeInitialized, etc) still ended up  
resolving to the ones in the application image.

At this point, my impression is that it would be very impractical, if not  
impossible to have separate druntimes for each shared library. Even when  
you do link separate runtimes, dyld still treats all the exported symbols  
as shared.


 Martin
So correct me if I'm wrong, but it seems the _only_ choice is a shared  
druntime for osx.
Could you elaborate a bit on how you've managed to do this for linux?


[1]  
https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/MachOTopics/0-Introduction/introduction.html

Thanks,

   Bit
May 25 2015
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-05-25 16:33, bitwise wrote:
 I've been reading through the Mach-O docs[1], and it seems that dynamic
 libs are treated the same as static libs in that exported symbols can
 only be defined once, even across dynamically loaded libraries. This is
 why calling rt_init from my dylib ended up calling the one that was
 already defined in the application image. I was able to call the rt_init
 from the dylib by specifically requesting it through dlsym, but then,
 all the global variables it used(_isRuntimeInitialized, etc) still ended
 up resolving to the ones in the application image.

 At this point, my impression is that it would be very impractical, if
 not impossible to have separate druntimes for each shared library. Even
 when you do link separate runtimes, dyld still treats all the exported
 symbols as shared.


  Martin
 So correct me if I'm wrong, but it seems the _only_ choice is a shared
 druntime for osx.
 Could you elaborate a bit on how you've managed to do this for linux?
If I recall correctly, this is how it works: On Linux the compiler will generate a function which is placed in a special section in the binary, same as annotating a C function with __attribute__((constructor)). This function calls a function in the druntime. This will give similar properties as _dyld_register_func_for_add_image on OS X but without the issues with registered callbacks Each image then becomes responsible to initialize itself. The image updates the shared data structure containing the necessary data (TLS, exception handling tables, ...). As you noticed yourself, all global symbols are shared across all images. -- /Jacob Carlborg
May 25 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Mon, 25 May 2015 14:39:37 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2015-05-25 16:33, bitwise wrote:
 I've been reading through the Mach-O docs[1], and it seems that dynamic
 libs are treated the same as static libs in that exported symbols can
 only be defined once, even across dynamically loaded libraries. This is
 why calling rt_init from my dylib ended up calling the one that was
 already defined in the application image. I was able to call the rt_init
 from the dylib by specifically requesting it through dlsym, but then,
 all the global variables it used(_isRuntimeInitialized, etc) still ended
 up resolving to the ones in the application image.

 At this point, my impression is that it would be very impractical, if
 not impossible to have separate druntimes for each shared library. Even
 when you do link separate runtimes, dyld still treats all the exported
 symbols as shared.


  Martin
 So correct me if I'm wrong, but it seems the _only_ choice is a shared
 druntime for osx.
 Could you elaborate a bit on how you've managed to do this for linux?
If I recall correctly, this is how it works: On Linux the compiler will generate a function which is placed in a special section in the binary, same as annotating a C function with __attribute__((constructor)). This function calls a function in the druntime. This will give similar properties as _dyld_register_func_for_add_image on OS X but without the issues with registered callbacks Each image then becomes responsible to initialize itself. The image updates the shared data structure containing the necessary data (TLS, exception handling tables, ...). As you noticed yourself, all global symbols are shared across all images.
So then I think I have a full solution: 1) _dyld_register_func_for_add_image should be taken care of with the above two fixes 2) __attribute__((constructor/destructor)) can be added to druntime when building for osx like in the file dylib_fixes.c [1] 3) copy paste rt_init/rt_term, rename them to dylib_init/dylib_term and remove everything except whats needed to initialize a shared lib's image. Does this make sense? Thanks, Bit [1] https://github.com/D-Programming-Language/druntime/blob/61ba4b8d3c0052065c17ffc8eef4f11496f3db3e/src/rt/dylib_fixes.c
May 25 2015
next sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Monday, 25 May 2015 at 19:40:52 UTC, bitwise wrote:
 1) _dyld_register_func_for_add_image should be taken care of 
 with the above two fixes
You still cannot unregister the callback, so it can't be used for dynamically loading druntime. Last time we talked about this problem, we found some undocumented function that could be deregistered.
 2) __attribute__((constructor/destructor)) can be added to 
 druntime when building for osx like in the file dylib_fixes.c 
 [1]
For linux we let the compiler emit comdat constructors into every D object, so you'll end up with exactly a single function for any binary containing D code. I don't think you need ctors/dtors on OSX if you already have the dylib callback.
 3) copy paste rt_init/rt_term, rename them to 
 dylib_init/dylib_term and remove everything except whats needed 
 to initialize a shared lib's image.
Not sure what you want to copy, but for sure you need to extend sections_osx to handle multiple dylibs. It should be very similar to _d_dso_registry for ELF, except that you iterate over the sections of a dylib image to get EH tables and ModuleInfo.
May 25 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-05-25 22:58, Martin Nowak wrote:
 On Monday, 25 May 2015 at 19:40:52 UTC, bitwise wrote:
 1) _dyld_register_func_for_add_image should be taken care of with the
 above two fixes
You still cannot unregister the callback, so it can't be used for dynamically loading druntime. Last time we talked about this problem, we found some undocumented function that could be deregistered.
 2) __attribute__((constructor/destructor)) can be added to druntime
 when building for osx like in the file dylib_fixes.c [1]
For linux we let the compiler emit comdat constructors into every D object, so you'll end up with exactly a single function for any binary containing D code. I don't think you need ctors/dtors on OSX if you already have the dylib callback.
I'm not sure if there is any other solution. There is one private function, "dyld_register_image_state_change_handler", that should work. I think it works because the image is never unloaded. I have not seen any function for deregistering a callback, private or public. Isn't it better to avoid private undocumented functions? -- /Jacob Carlborg
May 25 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Tue, 26 May 2015 02:28:14 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2015-05-25 22:58, Martin Nowak wrote:
 On Monday, 25 May 2015 at 19:40:52 UTC, bitwise wrote:
 1) _dyld_register_func_for_add_image should be taken care of with the
 above two fixes
You still cannot unregister the callback, so it can't be used for dynamically loading druntime. Last time we talked about this problem, we found some undocumented function that could be deregistered.
 2) __attribute__((constructor/destructor)) can be added to druntime
 when building for osx like in the file dylib_fixes.c [1]
For linux we let the compiler emit comdat constructors into every D object, so you'll end up with exactly a single function for any binary containing D code. I don't think you need ctors/dtors on OSX if you already have the dylib callback.
I'm not sure if there is any other solution. There is one private function, "dyld_register_image_state_change_handler", that should work. I think it works because the image is never unloaded. I have not seen any function for deregistering a callback, private or public.
The I think Martin is right. We don't need ctors/dtors or any compiler fanciness. All we need is the two callbacks, which can be registered when druntime is initialized. _dyld_register_func_for_add_image _dyld_register_func_for_remove_image At this point, we would only be registering the callbacks once in the main image, and not from the shared library. Since all global functions and symbols are shared between images anyways, receiving the callback in the main image would be fine. So in this case, unregistering the callbacks is no longer needed.
 Isn't it better to avoid private undocumented functions?
Not only better, but mandatory, otherwise Apple will reject the app from the app store. I am certain this is the case for iOS, and I assume it would be the same for desktop. On Tue, 26 May 2015 02:30:51 -0400, Jacob Carlborg <doob me.com> wrote:
 What do you plan to do about TLS?
How would loading shared libraries change this? Couldn't TLS, however it's implemented now, be applied to shared libraries as well? Bit
May 26 2015
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-05-26 18:25, bitwise wrote:

 I think Martin is right. We don't need ctors/dtors or any compiler
 fanciness. All we need is the two callbacks, which can be registered
 when druntime is initialized.

 _dyld_register_func_for_add_image
 _dyld_register_func_for_remove_image

 At this point, we would only be registering the callbacks once in the
 main image, and not from the shared library. Since all global functions
 and symbols are shared between images anyways, receiving the callback in
 the main image would be fine. So in this case, unregistering the
 callbacks is no longer needed.
What about using a D dynamic library in a C application? The C application would initialize the runtime which would register the callback. Then it would be undefined to unload druntime?
 How would loading shared libraries change this? Couldn't TLS, however
 it's implemented now, be applied to shared libraries as well?
I'm not sure. The ___tls_get_addr function [1] is used when accessing a TLS variable on OS X. In all native implementations, both on OS X and Linux, the parameter is not just a void* but struct containing the image header as well. Looking at SectionGroup [2] and how its data is initialized [3] you can see that there's only one set of TLS data (__tls_data and __tlscoal_nt) in SectionGroup. It would be straight forward to change that to an array but then you would not know which index to access in getTLSBlockAlloc [4] which is used by ___tls_get_addr. [1] https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L115 [2] https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L26 [3] https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L227-L239 [4] https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L172 -- /Jacob Carlborg
May 26 2015
parent reply "Martin Nowak" <code dawg.eu> writes:
On Wednesday, 27 May 2015 at 06:45:49 UTC, Jacob Carlborg wrote:
 I'm not sure. The ___tls_get_addr function [1] is used when 
 accessing a TLS variable on OS X. In all native 
 implementations, both on OS X and Linux, the parameter is not 
 just a void* but struct containing the image header as well.
On Linux you call it with an dso index and an offset. The DSO index is assigned by the runtime linker (there is a special relocation for that). Something similar could be done manually if not natively supported on OSX.
May 27 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-05-27 15:38, Martin Nowak wrote:

 On Linux you call it with an dso index and an offset. The DSO index is
 assigned by the runtime linker (there is a special relocation for that).
 Something similar could be done manually if not natively supported on OSX.
It is natively support. -- /Jacob Carlborg
May 27 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Wed, 27 May 2015 02:45:57 -0400, Jacob Carlborg <doob me.com> wrote:
 What about using a D dynamic library in a C application? The C  
 application would initialize the runtime which would register the  
 callback. Then it would be undefined to unload druntime?
Good point. I've come up another solution and posted it here: http://dpaste.com/0DXBSNQ Basically, I've gone back to the idea of using the dylib ctor/dtors. I don't think we really even need the image-added callback, at least not for dylibs. I callback could still be used when the main application is written in D, but we could also replace it with my solution in the dpaste(the getThisImageInfo function). As far as TLS goes, I haven't looking into the details of actually initializing the images yet. Once we have a solution for when and how to initialize the dylibs, I'll move on to the specifics... pun intended ;) I think the solution above may be a winner though. Bit
May 27 2015
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-05-27 23:24, bitwise wrote:

 Good point.

 I've come up another solution and posted it here:
 http://dpaste.com/0DXBSNQ
BTW, I'm not sure what's the best solution but you can calculate the slide without iterating all images. Look at the first statements in "tlv_allocate_and_initialize_for_key" in [1]. Specifically where it sets the "slide" variable.
 Basically, I've gone back to the idea of using the dylib ctor/dtors. I
 don't think we really even need the image-added callback, at least not
 for dylibs.

 I callback could still be used when the main application is written in
 D, but we could also replace it with my solution in the dpaste(the
 getThisImageInfo function).
I think it would be simpler to only have one way to do this.
 As far as TLS goes, I haven't looking into the details of actually
 initializing the images yet. Once we have a solution for when and how to
 initialize the dylibs, I'll move on to the specifics... pun intended ;)
Fair enough.
 I think the solution above may be a winner though.
I agree, but I think ctor/dtors can be used for the executable as well. [1] http://opensource.apple.com/source/dyld/dyld-353.2.1/src/threadLocalVariables.c -- /Jacob Carlborg
May 27 2015
prev sibling next sibling parent "Martin Nowak" <code dawg.eu> writes:
On Wednesday, 27 May 2015 at 21:24:25 UTC, bitwise wrote:
 Basically, I've gone back to the idea of using the dylib 
 ctor/dtors. I don't think we really even need the image-added 
 callback, at least not for dylibs.
Looks good. The compiler could add a simple ctor/dtor to any D object.
May 29 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-05-27 23:24, bitwise wrote:

 Good point.

 I've come up another solution and posted it here:
 http://dpaste.com/0DXBSNQ
Will "dladdr" return different values (mach_header) for different dynamic libraries? Won't there only be one "init" function, as you said earlier. Or does it work somehow anyway? -- /Jacob Carlborg
May 29 2015
parent reply "bitwise" <bitwise.pvt gmail.com> writes:
On Friday, 29 May 2015 at 12:46:43 UTC, Jacob Carlborg wrote:
 On 2015-05-27 23:24, bitwise wrote:

 Good point.

 I've come up another solution and posted it here:
 http://dpaste.com/0DXBSNQ
Will "dladdr" return different values (mach_header) for different dynamic libraries? Won't there only be one "init" function, as you said earlier. Or does it work somehow anyway?
I'm going to test it out this weekend, but if I do get the wrong address, I think I can add a symbol with "__attribute__ ((visibility ("hidden")))" and use dladdr on that instead. I don't think hidden symbols should collide.
May 29 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-05-29 18:45, bitwise wrote:

 I'm going to test it out this weekend, but if I do get the wrong
 address, I think I can add a symbol with "__attribute__ ((visibility
 ("hidden")))" and use dladdr on that instead. I don't think hidden
 symbols should collide.
Doesn't "dlopen" open the current image if "null" is passed? And that void* which is returned is actually a "mach_header*", if I recall correctly. -- /Jacob Carlborg
May 29 2015
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Fri, 29 May 2015 14:05:10 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2015-05-29 18:45, bitwise wrote:

 I'm going to test it out this weekend, but if I do get the wrong
 address, I think I can add a symbol with "__attribute__ ((visibility
 ("hidden")))" and use dladdr on that instead. I don't think hidden
 symbols should collide.
Doesn't "dlopen" open the current image if "null" is passed?
No, it opens the image for the main program, even if called from a shared lib.
 And that void* which is returned is actually a "mach_header*", if I  
 recall correctly.
Not sure if this is true. This works for sure in terms of getting "a" mach_header*, but I still have to check if visibility(hidden) allows each shared lib to have it's own copy of "_symbol": __attribute__ ((visibility("hidden"))) static int _symbol = 1; Dl_info info; if(dladdr(&symbol, &info)) const struct mach_header* myHeader = cast(const struct mach_header*)info.dli_fbase https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dladdr.3.html Bit
May 29 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-05-30 01:03, bitwise wrote:

 No, it opens the image for the main program, even if called from  a
 shared lib.

 And that void* which is returned is actually a "mach_header*", if I
 recall correctly.
Not sure if this is true.
I see, it was mostly a guess. -- /Jacob Carlborg
May 30 2015
parent bitwise <bitwise.pvt gmail.com> writes:
On Sat, 30 May 2015 05:06:34 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2015-05-30 01:03, bitwise wrote:

 No, it opens the image for the main program, even if called from  a
 shared lib.

 And that void* which is returned is actually a "mach_header*", if I
 recall correctly.
Not sure if this is true.
I see, it was mostly a guess.
Not sure how accurate this code[1] is, but it seems dlopen returns a pointer to the image itself, which has a function machHeader(); http://www.opensource.apple.com/source/dyld/dyld-353.2.1/src/dyldAPIs.cpp dlopen returns magic numbers(RTLD_MAIN_ONLY or RTLD_DEFAULT) if you pass null, and sometimes it "|"s a 1 with the image pointer. So we can't really use it to get at the header. Bit
May 30 2015
prev sibling next sibling parent "Martin Nowak" <code dawg.eu> writes:
On Tuesday, 26 May 2015 at 16:25:52 UTC, bitwise wrote:
 Since all global functions and symbols are shared between 
 images anyways, receiving the callback in the main image would 
 be fine. So in this case, unregistering the callbacks is no 
 longer needed.
That only works when the host executable is linked against druntime, but falls when you load runtime dynamically, e.g. a D plugin for a C host.
May 27 2015
prev sibling parent "Martin Nowak" <code dawg.eu> writes:
On Tuesday, 26 May 2015 at 16:25:52 UTC, bitwise wrote:
 Isn't it better to avoid private undocumented functions?
Not only better, but mandatory, otherwise Apple will reject the app from the app store.
Calling back into an unloaded image without proving a mean to deregister the callback is a bug, so you might fix it. Looking at the code [1] again there is also an interesting sImagesToNotifyAboutOtherImages. [1]: http://opensource.apple.com/source/dyld/dyld-95.3/src/dyld.cpp
May 27 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-05-25 21:40, bitwise wrote:

 So then I think I have a full solution:
 1) _dyld_register_func_for_add_image should be taken care of with the
 above two fixes
 2) __attribute__((constructor/destructor)) can be added to druntime when
 building for osx like in the file dylib_fixes.c [1]
 3) copy paste rt_init/rt_term, rename them to dylib_init/dylib_term and
 remove everything except whats needed to initialize a shared lib's image.

 Does this make sense?
You plan to use __attribute__((constructor)) instead of _dyld_register_func_for_add_image should? As Marin said, you need to look at sections_osx.d and _d_dso_registry. What do you plan to do about TLS? -- /Jacob Carlborg
May 25 2015
prev sibling parent "Martin Nowak" <code dawg.eu> writes:
On Monday, 25 May 2015 at 14:33:43 UTC, bitwise wrote:
 At this point, my impression is that it would be very 
 impractical, if not impossible to have separate druntimes for 
 each shared library. Even when you do link separate runtimes, 
 dyld still treats all the exported symbols as shared.
Yes, you can't mix them with a D executable.
May 25 2015