www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - static ctors in shared libs

reply bitwise <bitwise.pvt gmail.com> writes:
I've been doing some work on shared libraries for OSX, and have 
come across a potential problem, which I'm not sure what to do 
with.

Currently, when a thread is spawned, that thread calls all the 
TLS ctors, then runs the thread entry point function, and finally 
calls the TLS dtors before the thread terminates.

Example, for windows:

https://github.com/D-Programming-Language/druntime/blob/15a227477a344583c4748d95492703901417f4f8/src/core/thread.d#L236

So, the question is, how do dynamic libraries interact here?

Example: A dynamic library contains D modules with TLS ctors. If 
I start a few threads, and then load a dynamic library, shouldn't 
the TLS ctors in the dynamic library be called for each running 
thread? If my assumption is correct, the next question is, how do 
you do this?

I don't think you can hijack each thread and have it run the TLS 
ctors, and you can run them all from the thread loading the 
shared library because of synchronization issues. So what's the 
solution? Should TLS ctors in dynamic libraries simply be 
specified not to run, or could they somehow be run lazily at the 
first TLS access in a dynamic library?

Any thoughts on this would be appreciated.

Thanks,
    Bit
Mar 17 2016
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 17 Mar 2016 20:27:39 +0000
schrieb bitwise <bitwise.pvt gmail.com>:

 I've been doing some work on shared libraries for OSX, and have 
 come across a potential problem, which I'm not sure what to do 
 with.
 
 Currently, when a thread is spawned, that thread calls all the 
 TLS ctors, then runs the thread entry point function, and finally 
 calls the TLS dtors before the thread terminates.
 
 Example, for windows:
 
 https://github.com/D-Programming-Language/druntime/blob/15a227477a344583c4748d95492703901417f4f8/src/core/thread.d#L236
 
 So, the question is, how do dynamic libraries interact here?
 
 Example: A dynamic library contains D modules with TLS ctors. If 
 I start a few threads, and then load a dynamic library, shouldn't 
 the TLS ctors in the dynamic library be called for each running 
 thread? If my assumption is correct, the next question is, how do 
 you do this?
 
 I don't think you can hijack each thread and have it run the TLS 
 ctors, and you can run them all from the thread loading the 
 shared library because of synchronization issues. So what's the 
 solution? Should TLS ctors in dynamic libraries simply be 
 specified not to run, or could they somehow be run lazily at the 
 first TLS access in a dynamic library?
 
 Any thoughts on this would be appreciated.
 
 Thanks,
     Bit
 
It's been some time since I looked at that code, but IIRC TLS ctors for newly loaded libraries are not run in old threads. There's nothing we can do about this without help from the C runtime. As TLS ctors are meant to initialize TLS storage the correct time to run these is after the per-thread TLS storage allocation. (The C runtime can actually allocate TLS memory lazily, on the first access to a TLS variable in a module). The C runtimes of course know when TLS storage is allocated, but they do not provide hooks we could use.
Mar 17 2016
parent reply Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Thursday, 17 March 2016 at 20:54:57 UTC, Johannes Pfau wrote:
 It's been some time since I looked at that code, but IIRC TLS 
 ctors for newly loaded libraries are not run in old threads. 
 There's nothing we can do about this without help from the C 
 runtime.
Hmm... are all module _destructors_ called when those threads exit? That would mean that some destructors may be called without the corresponding constructors having run...
Mar 18 2016
parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 18 Mar 2016 14:35:41 +0000
schrieb Marc Sch=C3=BCtz <schuetzm gmx.net>:

 On Thursday, 17 March 2016 at 20:54:57 UTC, Johannes Pfau wrote:
 It's been some time since I looked at that code, but IIRC TLS=20
 ctors for newly loaded libraries are not run in old threads.=20
 There's nothing we can do about this without help from the C=20
 runtime. =20
=20 Hmm... are all module _destructors_ called when those threads=20 exit? That would mean that some destructors may be called without=20 the corresponding constructors having run...
No. Each thread keeps a thread-local list of shared libraries it knows about. This list is inherited from the parent thread on thread creation. It also gets updated if a new library is loaded at runtime (but only for the loading thread!). The runtime only runs constructors and destructors for libraries in the list. Additionally the GC only scans TLS memory of libraries in the list. As an example: * Create 2 Threads, TA & TB * TB creates thread TB1 * TB loads library libA * TB creates new thread TB2 * TA creates new thread TA1 Threads TB and TB2 will run TLS constructors and destructors for libA. modules. These threads will also scan TLS memory from modules in libA. Threads TA, TA1 and TB1 will not run TLS constructors or destructors for libA. Additionally, the GC will not scan TLS memory for libA for these threads. So you can only safely use modules from libA in thread TB and TB2. Relevant links: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/secti= ons_elf_shared.d#L198 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/secti= ons_elf_shared.d#L281 https://github.com/D-Programming-Language/druntime/blob/master/src/core/thr= ead.d#L385 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/minfo= .d#L312 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/secti= ons_elf_shared.d#L49 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/secti= ons_elf_shared.d#L76
Mar 18 2016
parent bitwise <bitwise.pvt gmail.com> writes:
On Friday, 18 March 2016 at 16:23:00 UTC, Johannes Pfau wrote:
 Am Fri, 18 Mar 2016 14:35:41 +0000
 schrieb Marc Sch├╝tz <schuetzm gmx.net>:

 [...]
Thanks for the explanation! When I first looked at the elf implementation, I was confused by what was goign on with pinLoadedLibraries()/inheritLoadedLibraries(). With this problem in mind though, it all makes sense now. Bit
Mar 23 2016
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 17/03/16 21:27, bitwise wrote:
 I've been doing some work on shared libraries for OSX, and have come
 across a potential problem, which I'm not sure what to do with.

 Currently, when a thread is spawned, that thread calls all the TLS
 ctors, then runs the thread entry point function, and finally calls the
 TLS dtors before the thread terminates.

 Example, for windows:

 https://github.com/D-Programming-Language/druntime/blob/15a227477a344583c4748d95492703901417f4f8/src/core/thread.d#L236


 So, the question is, how do dynamic libraries interact here?

 Example: A dynamic library contains D modules with TLS ctors. If I start
 a few threads, and then load a dynamic library, shouldn't the TLS ctors
 in the dynamic library be called for each running thread? If my
 assumption is correct, the next question is, how do you do this?

 I don't think you can hijack each thread and have it run the TLS ctors,
 and you can run them all from the thread loading the shared library
 because of synchronization issues. So what's the solution? Should TLS
 ctors in dynamic libraries simply be specified not to run, or could they
 somehow be run lazily at the first TLS access in a dynamic library?

 Any thoughts on this would be appreciated.
How does it behave on Linux? It already supports dynamic libraries. -- /Jacob Carlborg
Mar 18 2016
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Friday, 18 March 2016 at 18:52:25 UTC, Jacob Carlborg wrote:
 On 17/03/16 21:27, bitwise wrote:
 I've been doing some work on shared libraries for OSX, and 
 have come
 across a potential problem, which I'm not sure what to do with.

 Currently, when a thread is spawned, that thread calls all the 
 TLS
 ctors, then runs the thread entry point function, and finally 
 calls the
 TLS dtors before the thread terminates.

 Example, for windows:

 https://github.com/D-Programming-Language/druntime/blob/15a227477a344583c4748d95492703901417f4f8/src/core/thread.d#L236


 So, the question is, how do dynamic libraries interact here?

 Example: A dynamic library contains D modules with TLS ctors. 
 If I start
 a few threads, and then load a dynamic library, shouldn't the 
 TLS ctors
 in the dynamic library be called for each running thread? If my
 assumption is correct, the next question is, how do you do 
 this?

 I don't think you can hijack each thread and have it run the 
 TLS ctors,
 and you can run them all from the thread loading the shared 
 library
 because of synchronization issues. So what's the solution? 
 Should TLS
 ctors in dynamic libraries simply be specified not to run, or 
 could they
 somehow be run lazily at the first TLS access in a dynamic 
 library?

 Any thoughts on this would be appreciated.
How does it behave on Linux? It already supports dynamic libraries.
After looking at the elf implementation, Johannes's explanation seems to be correct. I don't think it's the best solution though. I'm wondering if the _exact_ behavior of the current solution was intentional. It seems like everything will work fine as long as you only use statically linked shared libs, so I'm wondering if dynamic linking was fully considered at the time the current solution was written. As far as dynamic linking goes, I don't like how the current solution works. It can lead to very confusing problems, and I think it would be better to do an all-or-nothing approach: -statically linked shared libs always have shared static ctors called -statically linked shared libs always have TLS static ctors called for all new threads -dynamically linked shared libs always have shared static ctors called -dynamically linked shared libs NEVER have TLS static ctors called for ANY thread I can see how one could argue for benefits of the current approach, but I don't think it's worth exposing people to that kind of confusion, to have partially working TLS static ctors. Thoughts? Bit
Mar 23 2016
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Thursday, 24 March 2016 at 00:56:33 UTC, bitwise wrote:
 Thoughts?

     Bit
Ok..How about "Thought"? One thought will do. I'll take what I can get ;) Bit
Mar 25 2016
parent reply Joakim <dlang joakim.fea.st> writes:
On Friday, 25 March 2016 at 16:33:26 UTC, bitwise wrote:
 On Thursday, 24 March 2016 at 00:56:33 UTC, bitwise wrote:
 Thoughts?

     Bit
Ok..How about "Thought"? One thought will do. I'll take what I can get ;) Bit
Martin is the guy to talk to, he wrote all that code. David might have an opinion, as he adapted it for ldc. Most everybody else doesn't go into those weeds.
Mar 25 2016
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Saturday, 26 March 2016 at 03:08:13 UTC, Joakim wrote:
 On Friday, 25 March 2016 at 16:33:26 UTC, bitwise wrote:
 On Thursday, 24 March 2016 at 00:56:33 UTC, bitwise wrote:
 Thoughts?

     Bit
Ok..How about "Thought"? One thought will do. I'll take what I can get ;) Bit
Martin is the guy to talk to, he wrote all that code. David might have an opinion, as he adapted it for ldc. Most everybody else doesn't go into those weeds.
I was hoping to have some more people weigh in with their experience with plugins, whether or not they're expected to be multi-threaded, thread-safe, etc.. I think the current design is fragile, and given the limited usage of D shared libraries atm, I think it's a good time to come up with something a bit more solid/predictable. Anyways, I guess I'll have to email Martin and David and see what they say. Bit
Mar 26 2016
parent reply Guillaume Piolat <contact gam3sfrommars.fr> writes:
On Sunday, 27 March 2016 at 03:28:31 UTC, bitwise wrote:
 I was hoping to have some more people weigh in with their 
 experience with plugins, whether or not they're expected to be 
 multi-threaded, thread-safe, etc.. I think the current design 
 is fragile, and given the limited usage of D shared libraries 
 atm, I think it's a good time to come up with something a bit 
 more solid/predictable.
Using OSX shared libraries with both DMD and LDC for plugins, I'm trying to avoid all TLS, with LDC the only TLS I have is cached Obj-C selectors (pointers) which do not have constructors IIRC. A real concern is then that Phobos would use TLS internally. Apart from that I don't really have an opinion.
Mar 27 2016
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Sunday, 27 March 2016 at 11:38:15 UTC, Guillaume Piolat wrote:
 On Sunday, 27 March 2016 at 03:28:31 UTC, bitwise wrote:
 I was hoping to have some more people weigh in with their 
 experience with plugins, whether or not they're expected to be 
 multi-threaded, thread-safe, etc.. I think the current design 
 is fragile, and given the limited usage of D shared libraries 
 atm, I think it's a good time to come up with something a bit 
 more solid/predictable.
Using OSX shared libraries with both DMD and LDC for plugins, I'm trying to avoid all TLS, with LDC the only TLS I have is cached Obj-C selectors (pointers) which do not have constructors IIRC. A real concern is then that Phobos would use TLS internally. Apart from that I don't really have an opinion.
Ok, thanks! Phobos should be linked statically even when it's a shared library, so it should still have all static ctors called properly. Bit
Mar 27 2016
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Sunday, 27 March 2016 at 16:54:53 UTC, bitwise wrote:

 Phobos should be linked statically even when it's a shared 
 library, so it should still have all static ctors called 
 properly.

     Bit
Correction...Phobos would be statically linked to a dynamically loaded shared library. This could be a problem.
Mar 27 2016
parent reply Guillaume Piolat <name.lastname gmail.com> writes:
On Sunday, 27 March 2016 at 17:58:01 UTC, bitwise wrote:
 On Sunday, 27 March 2016 at 16:54:53 UTC, bitwise wrote:

 Phobos should be linked statically even when it's a shared 
 library, so it should still have all static ctors called 
 properly.

     Bit
Correction...Phobos would be statically linked to a dynamically loaded shared library. This could be a problem.
Hasn't seen any practical problem with that, not sure why people say it's a bad idea TBH. I like that everything is in a single static binary. The way bigger problem being OSX shared libraries unloading (with the leaking workaround), I haven't really followed your work recently about it!
Mar 27 2016
parent reply bitwise <bitwise.pvt gmail.com> writes:
On Sunday, 27 March 2016 at 19:31:46 UTC, Guillaume Piolat wrote:
 On Sunday, 27 March 2016 at 17:58:01 UTC, bitwise wrote:
 On Sunday, 27 March 2016 at 16:54:53 UTC, bitwise wrote:

 Phobos should be linked statically even when it's a shared 
 library, so it should still have all static ctors called 
 properly.

     Bit
Correction...Phobos would be statically linked to a dynamically loaded shared library. This could be a problem.
Hasn't seen any practical problem with that, not sure why people say it's a bad idea TBH. I like that everything is in a single static binary.
I'm not sure I understand what you're saying here. What I'm suggesting is that TLS static ctors not be run at all for dynamically loaded shared libraries, because at present, they are only run in certain circumstances, which can be confusing and unreliable. For example, if a program launched a bunch of worker threads, and then reloaded a D shared library, the TLS static ctors wouldn't be run after the second reload for those worker threads. So if the worker threads started accessing the shared library, they could encounter uninitialized data. If TLS static ctors simply weren't run at all for dynamically loaded libraries, there would be no confusion. Any D user that tried to explore the use of shared libraries, would quickly learn that TLS ctors are not called at all for dynamically loaded libraries. I suppose then, that an API like Runtime.AttachToThread() could be used to explicitly call the TLS ctors.
 The way bigger problem being OSX shared libraries unloading 
 (with the leaking workaround), I haven't really followed your 
 work recently about it!
I successfully modified DMD to output ctor/dtors which replace the need for the problematic callback, but I have to wait for Jacob's work on native TLS to be merged before I can continue. Bit
Mar 27 2016
parent Guillaume Piolat <name.lastname gmail.com> writes:
On Sunday, 27 March 2016 at 22:08:57 UTC, bitwise wrote:
 If TLS static ctors simply weren't run at all for dynamically 
 loaded libraries, there would be no confusion. Any D user that 
 tried to explore the use of shared libraries, would quickly 
 learn that TLS ctors are not called at all for dynamically 
 loaded libraries. I suppose then, that an API like 
 Runtime.AttachToThread() could be used to explicitly call the 
 TLS ctors.
That looks more sensible indeed.
Mar 27 2016