www.digitalmars.com         C & C++   DMDScript  

D.gnu - Getting tls symbols out of the compiler.

reply "Iain Buclaw" <ibuclaw ubuntu.com> writes:
OK, from today till the end of the week I'm going to be sitting 
down and thinking through this, as I'd really like the emitting 
of _tlsstart and _tlsend out of the compiler by hook or by crook.


Phase one - start:

We have rt/tls.S.  Have added this to the libdruntime builds, and 
I see no immediate problems running this through the testsuite  
(though - that is never really a good indicator of anything).  
Drawbacks, this is only available for linux at the moment. But 
that's fine if we keep this temporary for the week.


Phase two - plan:

The GC has a hook gc_addRoot used for the purpose of tracking GC 
allocated memory in C-land.  The idea I've got turning over in my 
head at the moment is that any thread local decls that are 
'new-able' (classes, pointers, d arrays) are added as a root upon 
'new' declaration,  this is safe-guarded by a thread-local static 
to prevent multiple calls to gc_addRoot.

eg:
---
var = new Object();
---
if (!var.guard)
{
   ++var.guard;
   gc_addRoot (&var);
}
var = _d_new_class (&Object_Class);
---

Then in the destruction of the thread (eg: atexit, or through a 
destructor called in .fini section)

---
if (var.guard)
   gc_removeRoot (&var);
---

Though this should not be required if we have a proper TLS GC in 
place.

Does this seem like a reasonable job?  Or have I completely lost 
the plot at half past 2 in the morning?  :o)

Regards
Iain.
May 29 2013
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 30 May 2013 01:19:10 +0200
schrieb "Iain Buclaw" <ibuclaw ubuntu.com>:

 OK, from today till the end of the week I'm going to be sitting 
 down and thinking through this, as I'd really like the emitting 
 of _tlsstart and _tlsend out of the compiler by hook or by crook.
 
 
 Phase one - start:
 
 We have rt/tls.S.  Have added this to the libdruntime builds, and 
 I see no immediate problems running this through the testsuite  
 (though - that is never really a good indicator of anything).  
 Drawbacks, this is only available for linux at the moment. But 
 that's fine if we keep this temporary for the week.

Do you know what that comment in that file "Sadly, this does not work because ld orders [...]" is about?
 Phase two - plan:
 
 The GC has a hook gc_addRoot used for the purpose of tracking GC 
 allocated memory in C-land.  The idea I've got turning over in my 
 head at the moment is that any thread local decls that are 
 'new-able' (classes, pointers, d arrays) are added as a root upon 
 'new' declaration,  this is safe-guarded by a thread-local static 
 to prevent multiple calls to gc_addRoot.
 

The good part part is that this will work with non-contiguous TLS, i.e GCC emulated TLS. The bad part is that it'll be very slow so it should really only be a fallback. And what about non-newable types? Why don't we register all thread local variables in a module constructor instead (in the C one or in the D one)? static int a; static void* b; static this() { gc_addRoot(&a); gc_addRoot(&b); } Then unload in the module destructor. We could try to optimize that, although we have to be careful: static this() //pseudo code { void*[2] roots; roots[0] = &a; roots[1] = &b; if(isContiguous(roots[])) //Sort. need to allow alignment though gc_addRange(roots[0], roots[$] + size) //probably need to keep a size array as well else foreach(ptr; roots) gc_addRoot(ptr); } This can detect if the TLS memory is contiguous and then only add one range per module. Maybe it's not worth the effort as long as it's only a slow fallback (and even with the optimization it'll still be slow)
 Though this should not be required if we have a proper TLS GC in 
 place.

Do you mean Martin Nowak's shared library/TLS work? That indeed sounds like the proper solution. (And maybe the D community should develop a sane standard interface to the runtime linker to access sections. Then go lobbying all major libcs out there...)
May 30 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-05-30 12:13, Johannes Pfau wrote:

 AFAIK he also changed how TLS sections are looked up for the main
 executable. _tlsstart and _tlsend are not used anymore. Instead some
 mainly undocumented obscure glibc interface is used to ask the runtime
 linker for the start and end of the TLS section.

He uses "dl_iterate_phdr" to iterate all shared libraries. Then iterates all segments/sections to find the ones used for TLS. Documented here, no secrets :) http://linux.die.net/man/3/dl_iterate_phdr -- /Jacob Carlborg
May 31 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-05-30 11:42, Iain Buclaw wrote:

 It's the same problem with what we currently have.  LD when it gets
 all reference symbols that are to go into the tls section (for Linux,
 at least) has the right to reorder the symbols.  As such, trying to
 push _tlsstart as the first symbol in the compiler has no guarantees
 it will be the first symbol in the object file.

For Mac OS X there's a function to get a section form an image: getsectbynamefromheader getsectbynamefromheader_64 Documented here: https://developer.apple.com/library/mac/#documentation/developertools/Reference/MachOReference/Reference/reference.html You can get the mach_header (image) using this callback function: _dyld_register_func_for_add_image Or using the same approach Martin Nowak used for Linux. The problem with the above function is that you cannot unregister the callback. If you have a dynamic library that registers a callback, i.e. druntime. If the dynamic library gets unloaded you will have a crash the next type "dlopen" is used. A solution to that problem would be to either use the same approach Martin Nowak used or any of these undocumented functions from the dynamic linker: void dyld_register_image_state_change_handler(dyld_image_states state, bool batch, dyld_image_state_change_handler handler) Works like "_dyld_register_func_for_add_image" but will force the image using it to not be unloaded, or something like that. Or: void dyld_enumerate_tlv_storage(dyld_tlv_state_change_handler handler) If I recall correctly, LDC is using "dyld_enumerate_tlv_storage". -- /Jacob Carlborg
May 31 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-06-01 11:53, David Nadlinger wrote:

 Yep. It's a bit cumbersome to handle, though, as it relies on Blocks
 support:

 https://github.com/ldc-developers/druntime/blob/2c78290fff65f2fce763da4b077856a2fc7596fb/src/ldc/osx_tls.c#L36

Oh, I didn't noticed it used blocks. -- /Jacob Carlborg
Jun 02 2013
prev sibling next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
On 30 May 2013 09:25, Johannes Pfau <nospam example.com> wrote:
 Am Thu, 30 May 2013 01:19:10 +0200
 schrieb "Iain Buclaw" <ibuclaw ubuntu.com>:

 OK, from today till the end of the week I'm going to be sitting
 down and thinking through this, as I'd really like the emitting
 of _tlsstart and _tlsend out of the compiler by hook or by crook.


 Phase one - start:

 We have rt/tls.S.  Have added this to the libdruntime builds, and
 I see no immediate problems running this through the testsuite
 (though - that is never really a good indicator of anything).
 Drawbacks, this is only available for linux at the moment. But
 that's fine if we keep this temporary for the week.

Do you know what that comment in that file "Sadly, this does not work because ld orders [...]" is about?

It's the same problem with what we currently have. LD when it gets all reference symbols that are to go into the tls section (for Linux, at least) has the right to reorder the symbols. As such, trying to push _tlsstart as the first symbol in the compiler has no guarantees it will be the first symbol in the object file.
 Phase two - plan:

 The GC has a hook gc_addRoot used for the purpose of tracking GC
 allocated memory in C-land.  The idea I've got turning over in my
 head at the moment is that any thread local decls that are
 'new-able' (classes, pointers, d arrays) are added as a root upon
 'new' declaration,  this is safe-guarded by a thread-local static
 to prevent multiple calls to gc_addRoot.

The good part part is that this will work with non-contiguous TLS, i.e GCC emulated TLS. The bad part is that it'll be very slow so it should really only be a fallback. And what about non-newable types?

Non-newable types aren't collected because they never reference new memory... This is true for basic types (int, float, complex, vectors). The same is also true for static arrays (although array[] = [1,2,3,4] calls _d_arrayliteral, the allocated memory is copied so can be free'd immediately afterwards). For structures we can check ((TypeStruct *) t)->hasPointers() at compile time - if false then we can be safely assured that this will never be referencing allocated memory. Classes, pointers and D arrays I've already mentioned. Forgot to mention associative arrays, which would also be added as a root upon initialisation.
 Why don't we register all thread local variables in a module
 constructor instead (in the C one or in the D one)?

 static int a;
 static void* b;

 static this()
 {
     gc_addRoot(&a);
     gc_addRoot(&b);
 }

This is more of a lazy init that won't affect start-up speed. Note: this idea is based off what C++ (g++) does for say - static A a = new A();
 Then unload in the module destructor. We could try to optimize that,
 although we have to be careful:

 static this() //pseudo code
 {
     void*[2] roots;
     roots[0] = &a;
     roots[1] = &b;
     if(isContiguous(roots[])) //Sort. need to allow alignment though
        gc_addRange(roots[0], roots[$] + size) //probably need to keep a
        size array as well
     else
         foreach(ptr; roots)
               gc_addRoot(ptr);
 }

This is a lot more work to describe in the compiler. :)
 This can detect if the TLS memory is contiguous and then only add one
 range per module. Maybe it's not worth the effort as long as it's only
 a slow fallback (and even with the optimization it'll still be slow)

I don't think it would have much slow down. Albeit the first initialisation would jump through the druntime library twice, but there after it's a single/two instruction test. Pretty negligible - but I'm not a speeeeeeeeed demon or freak who wants everything compiled with -fOMG-fast. =)
 Though this should not be required if we have a proper TLS GC in
 place.

Do you mean Martin Nowak's shared library/TLS work? That indeed sounds like the proper solution.

It should certainly mean that we don't have to worry about removing roots once they've been added.
 (And maybe the D community should develop a sane standard interface to
 the runtime linker to access sections. Then go lobbying all major libcs
 out there...)

You mean binutils? :-) See binutils/ld/scripttempl for the ldscripts used to lay out the tls data sections. (Note, only a few actually have a TLS section - and only winpe defines a _tls_start__ and _tls_end__ symbol). Regards -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0';
May 30 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 30 May 2013 10:42:17 +0100
schrieb Iain Buclaw <ibuclaw ubuntu.com>:

 
 This is more of a lazy init that won't affect start-up speed.  Note:
 this idea is based off what C++ (g++) does for say - static A a = new
 A();
 

I see.
 
 I don't think it would have much slow down.  Albeit the first
 initialisation would jump through the druntime library twice, but
 there after it's a single/two instruction test.  Pretty negligible -
 but I'm not a speeeeeeeeed demon or freak who wants everything
 compiled with -fOMG-fast. =)

I'm more worried about adding many roots to the GC. The code which checks the guard variable is probably neglectable.
 
 
 Though this should not be required if we have a proper TLS GC in
 place.

Do you mean Martin Nowak's shared library/TLS work? That indeed sounds like the proper solution.

It should certainly mean that we don't have to worry about removing roots once they've been added.

AFAIK he also changed how TLS sections are looked up for the main executable. _tlsstart and _tlsend are not used anymore. Instead some mainly undocumented obscure glibc interface is used to ask the runtime linker for the start and end of the TLS section. https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_linux.d#L122 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_linux.d#L164 https://github.com/D-Programming-Language/dmd/commit/313132b6e20c119fa64c6164574818421cb522ce
 
 (And maybe the D community should develop a sane standard interface
 to the runtime linker to access sections. Then go lobbying all
 major libcs out there...)

You mean binutils? :-) See binutils/ld/scripttempl for the ldscripts used to lay out the tls data sections. (Note, only a few actually have a TLS section - and only winpe defines a _tls_start__ and _tls_end__ symbol).

Well the start/end symbols could be introduced in binutils linker scripts. But the elf format already has all the information about section size and the runtime linker (ld.so) knows where every section starts and it also knows its size. There's just no standard interface to receive that information. (I had a quick look at the FreeBSD runtime linker some time ago. With some C libraries you can even get the elf header, but it's all non-standard and undocumented)
May 30 2013
prev sibling next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
On 30 May 2013 11:13, Johannes Pfau <nospam example.com> wrote:
 Am Thu, 30 May 2013 10:42:17 +0100
 schrieb Iain Buclaw <ibuclaw ubuntu.com>:

 This is more of a lazy init that won't affect start-up speed.  Note:
 this idea is based off what C++ (g++) does for say - static A a = new
 A();

I see.
 I don't think it would have much slow down.  Albeit the first
 initialisation would jump through the druntime library twice, but
 there after it's a single/two instruction test.  Pretty negligible -
 but I'm not a speeeeeeeeed demon or freak who wants everything
 compiled with -fOMG-fast. =)

I'm more worried about adding many roots to the GC. The code which checks the guard variable is probably neglectable.
 Though this should not be required if we have a proper TLS GC in
 place.

Do you mean Martin Nowak's shared library/TLS work? That indeed sounds like the proper solution.

It should certainly mean that we don't have to worry about removing roots once they've been added.

AFAIK he also changed how TLS sections are looked up for the main executable. _tlsstart and _tlsend are not used anymore. Instead some mainly undocumented obscure glibc interface is used to ask the runtime linker for the start and end of the TLS section. https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_linux.d#L122 https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_linux.d#L164 https://github.com/D-Programming-Language/dmd/commit/313132b6e20c119fa64c6164574818421cb522ce
 (And maybe the D community should develop a sane standard interface
 to the runtime linker to access sections. Then go lobbying all
 major libcs out there...)

You mean binutils? :-) See binutils/ld/scripttempl for the ldscripts used to lay out the tls data sections. (Note, only a few actually have a TLS section - and only winpe defines a _tls_start__ and _tls_end__ symbol).

Well the start/end symbols could be introduced in binutils linker scripts. But the elf format already has all the information about section size and the runtime linker (ld.so) knows where every section starts and it also knows its size. There's just no standard interface to receive that information. (I had a quick look at the FreeBSD runtime linker some time ago. With some C libraries you can even get the elf header, but it's all non-standard and undocumented)

Yes, I've noticed (and even gave libdruntime a test on gdc shortly after the conference) - it's broken everything ... literally ... -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0';
May 30 2013
prev sibling next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
On 30 May 2013 11:13, Johannes Pfau <nospam example.com> wrote:
 Am Thu, 30 May 2013 10:42:17 +0100
 schrieb Iain Buclaw <ibuclaw ubuntu.com>:

 This is more of a lazy init that won't affect start-up speed.  Note:
 this idea is based off what C++ (g++) does for say - static A a = new
 A();

I see.
 I don't think it would have much slow down.  Albeit the first
 initialisation would jump through the druntime library twice, but
 there after it's a single/two instruction test.  Pretty negligible -
 but I'm not a speeeeeeeeed demon or freak who wants everything
 compiled with -fOMG-fast. =)

I'm more worried about adding many roots to the GC. The code which checks the guard variable is probably neglectable.

Speed won't at all be a problem in that aspect. All roots are kept in an array, so during collection, rather than the following: --- mark (&_tlsstart, &_tlsend - &_tlsstart); --- It will be doing: --- mark (this.roots, this.roots + this.nroots); --- Regards -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0';
May 30 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 31 May 2013 09:42:10 +0200
schrieb Jacob Carlborg <doob me.com>:

 On 2013-05-30 12:13, Johannes Pfau wrote:
 
 AFAIK he also changed how TLS sections are looked up for the main
 executable. _tlsstart and _tlsend are not used anymore. Instead some
 mainly undocumented obscure glibc interface is used to ask the
 runtime linker for the start and end of the TLS section.

He uses "dl_iterate_phdr" to iterate all shared libraries. Then iterates all segments/sections to find the ones used for TLS. Documented here, no secrets :) http://linux.die.net/man/3/dl_iterate_phdr

OK, then I was wrong about that :-)
May 31 2013
prev sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Friday, 31 May 2013 at 07:58:25 UTC, Jacob Carlborg wrote:
 void dyld_enumerate_tlv_storage(dyld_tlv_state_change_handler 
 handler)

 If I recall correctly, LDC is using 
 "dyld_enumerate_tlv_storage".

Yep. It's a bit cumbersome to handle, though, as it relies on Blocks support: https://github.com/ldc-developers/druntime/blob/2c78290fff65f2fce763da4b077856a2fc7596fb/src/ldc/osx_tls.c#L36 David
Jun 01 2013