www.digitalmars.com         C & C++   DMDScript  

D.gnu - This week's experiment build gdc with tls disabled.

reply "Iain Buclaw" <ibuclaw ubuntu.com> writes:
We really need to understand this problem first before can start 
merging Martin's shared library work into gdc.

Running testsuite:

=== gdc Summary ===

# of expected passes		13841
# of unexpected failures	1

FAIL: runnable/testaa.d execution test


Running unittests:

./libdruntime/unittest
Success!

./src/unittest
Floating point exception (core dumped)


This has me surprised.
Dec 16 2013
next sibling parent "Iain Buclaw" <ibuclaw ubuntu.com> writes:
On Monday, 16 December 2013 at 15:46:59 UTC, Iain Buclaw wrote:
 We really need to understand this problem first before can 
 start merging Martin's shared library work into gdc.

 Running testsuite:

 === gdc Summary ===

 # of expected passes		13841
 # of unexpected failures	1

 FAIL: runnable/testaa.d execution test
OF course, the problem that I am running into here is in the same spirit as: http://d.puremagic.com/issues/show_bug.cgi?id=11139
Dec 16 2013
prev sibling parent reply "Iain Buclaw" <ibuclaw ubuntu.com> writes:
Gthreads and Emulated TLS are now in GDC!!!

https://github.com/D-Programming-GDC/GDC/commit/62554bfe1b35ee4f586634a76c65d83ebfa871ef

The bug noted above still exists, however what I am going to do 
is update and hook emulated TLS into the GC.
Dec 17 2013
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 17 December 2013 at 17:48:40 UTC, Iain Buclaw wrote:
 Gthreads and Emulated TLS are now in GDC!!!

 https://github.com/D-Programming-GDC/GDC/commit/62554bfe1b35ee4f586634a76c65d83ebfa871ef

 The bug noted above still exists, however what I am going to do 
 is update and hook emulated TLS into the GC.
Warning, n00b question ahead: What are the wider implications for this? Will this fix the problems with tls for embedded targets, windows xp and os x < 10.7?
Dec 17 2013
next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On 17 December 2013 18:35, John Colvin <john.loughran.colvin gmail.com> wrote:
 On Tuesday, 17 December 2013 at 17:48:40 UTC, Iain Buclaw wrote:
 Gthreads and Emulated TLS are now in GDC!!!


 https://github.com/D-Programming-GDC/GDC/commit/62554bfe1b35ee4f586634a76c65d83ebfa871ef

 The bug noted above still exists, however what I am going to do is update
 and hook emulated TLS into the GC.
Warning, n00b question ahead: What are the wider implications for this? Will this fix the problems with tls for embedded targets, windows xp and os x < 10.7?
Yes.
Dec 17 2013
prev sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 17 December 2013 19:08, Iain Buclaw <ibuclaw gdcproject.org> wrote:
 On 17 December 2013 18:35, John Colvin <john.loughran.colvin gmail.com> wrote:
 On Tuesday, 17 December 2013 at 17:48:40 UTC, Iain Buclaw wrote:
 Gthreads and Emulated TLS are now in GDC!!!


 https://github.com/D-Programming-GDC/GDC/commit/62554bfe1b35ee4f586634a76c65d83ebfa871ef

 The bug noted above still exists, however what I am going to do is update
 and hook emulated TLS into the GC.
Warning, n00b question ahead: What are the wider implications for this? Will this fix the problems with tls for embedded targets, windows xp and os x < 10.7?
Yes.
And will also make them primed and ready for shared library support too. Native TLS support on linux will have to wait a while before I get that working... (but you could always build with --disable-tls)
Dec 17 2013
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 17 December 2013 at 19:09:31 UTC, Iain Buclaw wrote:
 Native TLS support on linux will have to wait a while before I 
 get
 that working... (but you could always build with --disable-tls)
Wait, so GDC doesn't (entirely) support native TLS on Linux?! Or are you just talking about shared libraries? David
Dec 17 2013
parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 17 December 2013 19:34, David Nadlinger <code klickverbot.at> wrote:
 On Tuesday, 17 December 2013 at 19:09:31 UTC, Iain Buclaw wrote:
 Native TLS support on linux will have to wait a while before I get
 that working... (but you could always build with --disable-tls)
Wait, so GDC doesn't (entirely) support native TLS on Linux?! Or are you just talking about shared libraries?
Shared libraries (sorry, it made sense to me when I wrote it)... The hidden subtext that you didn't understand being, I'm holding back on Martin's patches for now.
Dec 17 2013
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 17 December 2013 at 20:07:41 UTC, Iain Buclaw wrote:
 The hidden subtext that you didn't understand being, I'm 
 holding back on Martin's patches for now.
That subtext isn't exactly hidden, given the first sentence in your first post. ;) Do you have a plan yet regarding how to implement module discovery for shared libraries? Would be a good idea to coordinate efforts and find a solution that works for all the backends, as Martin has been suggesting as well. David
Dec 17 2013
next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On 17 December 2013 20:12, David Nadlinger <code klickverbot.at> wrote:
 On Tuesday, 17 December 2013 at 20:07:41 UTC, Iain Buclaw wrote:
 The hidden subtext that you didn't understand being, I'm holding back on
 Martin's patches for now.
That subtext isn't exactly hidden, given the first sentence in your first post. ;) Do you have a plan yet regarding how to implement module discovery for shared libraries? Would be a good idea to coordinate efforts and find a solution that works for all the backends, as Martin has been suggesting as well.
Not yet, but that will be my focus after getting shared libs with emulated TLS working.
Dec 17 2013
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 17 Dec 2013 21:12:41 +0100
schrieb "David Nadlinger" <code klickverbot.at>:

 On Tuesday, 17 December 2013 at 20:07:41 UTC, Iain Buclaw wrote:
 The hidden subtext that you didn't understand being, I'm 
 holding back on Martin's patches for now.
That subtext isn't exactly hidden, given the first sentence in your first post. ;) Do you have a plan yet regarding how to implement module discovery for shared libraries? Would be a good idea to coordinate efforts and find a solution that works for all the backends, as Martin has been suggesting as well. David
I hope I'm not talking bullshit here as I'm not 100% sure what's meant with 'module discovery'. But if the question is how to get all ModuleInfos in a library, we should probably not forget the way C handles it's 'module constructors', aka the '.ctors' / '.dtors' section. They use linker scripts to make sure these sections are always correctly bracketed by _start and _end symbols and we could do the same for a '.moduleinfo' section and for the standard tls sections. Here's the relevant part of a linker script: ------------------------ .ctors : { /* gcc uses crtbegin.o to find the start of the constructors, so we make sure it is first. Because this is a wildcard, it doesn't matter if the user does not actually link against crtbegin.o; the linker won't look for a file to match a wildcard. The wildcard also means that it doesn't matter which directory crtbegin.o is in. */ KEEP (*crtbegin.o(.ctors)) KEEP (*crtbegin?.o(.ctors)) /* We don't want to include the .ctor section from the crtend.o file until after the sorted ctors. The .ctor section from the crtend file contains the end of ctors marker and it must be last */ KEEP (*(EXCLUDE_FILE (*crtend.o *crtend?.o ) .ctors)) KEEP (*(SORT(.ctors.*))) KEEP (*(.ctors)) } ------------------------ We can then access the bracket-symbols from a module-constructor (or a c-like .ctor constructor) in the library? So we'd have to add drtbegin.o / drtend.o files to every d library or executable. Ideally we'd get the default linker scripts changed, but that may take some time if the binutils guys even agree. It's also possible to 1) give a linker script to ld to replace the default script (not so good for us) 2) give ld a script to extend the default script (could work for .moduleinfo section, I'm not sure about TLS) One big problem is of course portability. But it seems such a solution might be more portable than a solution relying on runtime linkers (especially considering gcc already uses this internally for c++ support)
Dec 18 2013
next sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 18 December 2013 15:39, Johannes Pfau <nospam example.com> wrote:
 Am Tue, 17 Dec 2013 21:12:41 +0100
 schrieb "David Nadlinger" <code klickverbot.at>:

 On Tuesday, 17 December 2013 at 20:07:41 UTC, Iain Buclaw wrote:
 The hidden subtext that you didn't understand being, I'm
 holding back on Martin's patches for now.
That subtext isn't exactly hidden, given the first sentence in your first post. ;) Do you have a plan yet regarding how to implement module discovery for shared libraries? Would be a good idea to coordinate efforts and find a solution that works for all the backends, as Martin has been suggesting as well. David
I hope I'm not talking bullshit here as I'm not 100% sure what's meant with 'module discovery'.
I'm not so sure about 'module discovery' either. At least, in emulated TLS, it has a completely different concept - each thread has a dynamically allocated range (effectively, a void**[] on the heap that gets destroyed upon thread termination) which is shared amongst all modules / loaded libraries in D for the duration of the thread. So when it comes to calling getTLSRange() - what is effectively happening is: void**[] tlsarray = gthread_getspecific(emutls_key); return cast(void[]) tlsarray[0 .. $]; What I'm hoping is that whatever Martin has done, it is compatible with this way of doing things...
Dec 18 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-12-18 18:36, Iain Buclaw wrote:

 I'm not so sure about 'module discovery' either.  At least, in
 emulated TLS, it has a completely different concept - each thread has
 a dynamically allocated range (effectively, a void**[] on the heap
 that gets destroyed upon thread termination) which is shared amongst
 all modules / loaded libraries in D for the duration of the thread.
 So when it comes to calling getTLSRange() - what is effectively
 happening is:

    void**[] tlsarray = gthread_getspecific(emutls_key);
    return cast(void[]) tlsarray[0 .. $];

 What I'm hoping is that whatever Martin has done, it is compatible
 with this way of doing things...
After the changes Margin has done DMD now emits .ctors and .dtors sections to to every executable and shared library. These will call "_d_dso_registry", passing in, among other things, the start and end of the module infos. To get the TLS data the runtime iterates the sections of the executable/shared library and collections these. I don't know if that will be compatible with how it's done in GDC. This is only for Linux, FreeBSD and Mac OS X behaves as it did before. FreeBSD uses bracked sections and Mac OS X doing something similar as Linux does now, the dynamic linker has API's for this. -- /Jacob Carlborg
Dec 18 2013
prev sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 18 Dec 2013 17:36:53 +0000
schrieb Iain Buclaw <ibuclaw gdcproject.org>:

 On 18 December 2013 15:39, Johannes Pfau <nospam example.com> wrote:
 Am Tue, 17 Dec 2013 21:12:41 +0100
 schrieb "David Nadlinger" <code klickverbot.at>:

 On Tuesday, 17 December 2013 at 20:07:41 UTC, Iain Buclaw wrote:
 The hidden subtext that you didn't understand being, I'm
 holding back on Martin's patches for now.
That subtext isn't exactly hidden, given the first sentence in your first post. ;) Do you have a plan yet regarding how to implement module discovery for shared libraries? Would be a good idea to coordinate efforts and find a solution that works for all the backends, as Martin has been suggesting as well. David
I hope I'm not talking bullshit here as I'm not 100% sure what's meant with 'module discovery'.
I'm not so sure about 'module discovery' either. At least, in emulated TLS, it has a completely different concept - each thread has a dynamically allocated range (effectively, a void**[] on the heap that gets destroyed upon thread termination) which is shared amongst all modules / loaded libraries in D for the duration of the thread. So when it comes to calling getTLSRange() - what is effectively happening is: void**[] tlsarray = gthread_getspecific(emutls_key); return cast(void[]) tlsarray[0 .. $]; What I'm hoping is that whatever Martin has done, it is compatible with this way of doing things...
I guess you can add the scan code here: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/tlsgc.d#L62
Dec 19 2013
prev sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Wednesday, 18 December 2013 at 15:39:30 UTC, Johannes Pfau 
wrote:
 I hope I'm not talking bullshit here as I'm not 100% sure 
 what's meant
 with 'module discovery'.

 But if the question is how to get all ModuleInfos in a library, 
 […]
Yes, this was the idea.
 We can then access the bracket-symbols from a 
 module-constructor (or a
 c-like .ctor constructor) in the library? So we'd have to add
 drtbegin.o / drtend.o files to every d library or executable.
That is pretty much how it is done in DMD and LDC as well. The only difference is that instead of modifying the linker scripts to accommodate this, we emit the ModuleInfo references to custom sections. The GNU toolchain (and we are in highly system-specific territory here anyway) never changes the order of custom sections, which you can also verify using __attribute__((section("...))) in GCC. Thus, if you emit your relevant symbols into three sections like this, a.o --- _minfo_beg: moduleInfoBeginTag _minfo: moduleInfoForA _minfo_end: moduleInfoEndTag b.o --- _minfo_beg: moduleInfoBeginTag _minfo: moduleInfoForB _minfo_end: moduleInfoEndTag the linked result will be _minfo_beg: moduleInfoBeginTag _minfo: moduleInfoForA moduleInfoForB _minfo_end: moduleInfoEndTag and you can just use [moduleInfoBeginTag, moduleInfoEndTag) to get a list of all available ModuleInfos. Of course, once shared libraries come into play, you have to consider quite a few subtleties regarding symbol visibility and so on. Also, due to a bug in LLVM, LDC can't emit weak symbols with custom section names, so we end up with multiple copies of the begin/end symbols and the C .ctor/.dtor table entries. But in general, this works quite well, and is certainly much more portable than requiring changes to the linker script. David
Dec 19 2013
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 19 Dec 2013 18:59:42 +0100
schrieb "David Nadlinger" <code klickverbot.at>:

 Yes, this was the idea.
 
 We can then access the bracket-symbols from a 
 module-constructor (or a
 c-like .ctor constructor) in the library? So we'd have to add
 drtbegin.o / drtend.o files to every d library or executable.
That is pretty much how it is done in DMD and LDC as well. The only difference is that instead of modifying the linker scripts to accommodate this, we emit the ModuleInfo references to custom sections. The GNU toolchain (and we are in highly system-specific territory here anyway) never changes the order of custom sections, which you can also verify using __attribute__((section("...))) in GCC. Thus, if you emit your relevant symbols into three sections like this, a.o --- _minfo_beg: moduleInfoBeginTag _minfo: moduleInfoForA _minfo_end: moduleInfoEndTag b.o --- _minfo_beg: moduleInfoBeginTag _minfo: moduleInfoForB _minfo_end: moduleInfoEndTag the linked result will be _minfo_beg: moduleInfoBeginTag _minfo: moduleInfoForA moduleInfoForB _minfo_end: moduleInfoEndTag and you can just use [moduleInfoBeginTag, moduleInfoEndTag) to get a list of all available ModuleInfos. Of course, once shared libraries come into play, you have to consider quite a few subtleties regarding symbol visibility and so on.
Yes, I got that part. Begin/End symbols should be private to the library and accessed from a constructor function in that library which then passes the addresses of these symbols to the _d_dso_registry function. (And for TLS if we had these symbols tey would also be 'private' to different threads, so we'd have to register them at thread startup for every dso but AFAICS the currect DMD code basically does that already) However, I'd prefer to pass an additional object file to ld which contains this constructor instead of placing this code into every object file, but that's probably just a personal preference.
 Also, due to a bug in LLVM, LDC can't emit weak symbols 
 with custom section names, so we end up with multiple copies of 
 the begin/end symbols and the C .ctor/.dtor table entries.
 
 But in general, this works quite well, and is certainly much more 
 portable than requiring changes to the linker script.
 
 David
Sounds like that could work. But as the module section is a custom section anyway we wouldn't have to replace/modify the default linker script - we can pass a custom script to ld which just handles the ".minfo" section. That should be just as portable as relying on the "don't reorder sections" behavior: Works everywhere where GNU Binutils LD/GOLD are used. (Emitting 3 sections is a clever trick, but it feels like a hack imho. I'm also not sure if we can control the order in which sections are emitted in GCC) Even if we don't want to go that route I'll try to write a proof of concept this weekend as I really want to know now whether this will work. It seems like getting the TLS section is the more interesting part. We can't emit sections around the TLS section so IIRC the current dmd implementation therefore relies on the runtime linker and libc specific interfaces? I think asking the binutils maintainers to add __tdata_begin, __tdata_end, __tbss_begin and __tbss_end markers to the tdata and tbss sections would be a nice long-term solution, or is there some issue with that? If the maintainers don't want to add these extra symbols for every library we might ask for special treatment of dso_start.o/dso_end.o files, similar to what they do for crtbegin.o/crtend.o for C++. I'm wondering though if ld actually reorders symbols in a section by default? I know there are some --sort-section options, but does it sort by default? IIRC we used to use TLS bracketing in GDC and Iain said there were some issues. But if we placed the begin/end markers into extra object files and supplied them as first/last file to the linker, wouldn't that work for now?
Dec 19 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-19 20:51, Johannes Pfau wrote:

 Sounds like that could work. But as the module section is a custom
 section anyway we wouldn't have to replace/modify the default linker
 script - we can pass a custom script to ld which just handles the
 ".minfo" section. That should be just as portable as relying on the
 "don't reorder sections" behavior: Works everywhere where GNU Binutils
 LD/GOLD are used. (Emitting 3 sections is a clever trick, but it feels
 like a hack imho. I'm also not sure if we can control the order in
 which sections are emitted in GCC)
The trick with brackting sections doesn't work on Mac OS X. DMD had some problems with that. It basically broke on every new major release of Mac OS X, the linker changed all the time. If I recall correctly it either removed the empty sections or reordered the sections.
 It seems like getting the TLS section is the more interesting part. We
 can't emit sections around the TLS section so IIRC the current dmd
 implementation therefore relies on the runtime linker and libc specific
 interfaces?
Yes, using dl_iterate_phdr: http://linux.die.net/man/3/dl_iterate_phdr.
 I think asking the binutils maintainers to add __tdata_begin,
 __tdata_end, __tbss_begin and __tbss_end markers to the tdata and tbss
 sections would be a nice long-term solution, or is there some issue
 with that?
How does that work? Do people need to update their linkers for that to work? -- /Jacob Carlborg
Dec 19 2013
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 20 Dec 2013 08:40:28 +0100
schrieb Jacob Carlborg <doob me.com>:

 I think asking the binutils maintainers to add __tdata_begin,
 __tdata_end, __tbss_begin and __tbss_end markers to the tdata and
 tbss sections would be a nice long-term solution, or is there some
 issue with that?
How does that work? Do people need to update their linkers for that to work?
Likely yes, that's why I said 'longterm' solution. You need to edit the default linker script. Then there are 2 options: We can force certain object files to be at the start or the end of the section. So we could introduce dso_start.o and dso_end.o files containing the start/end sections and those would always be placed as the first/last objects in a section. (This is how the C++/C runtime calls the constructors in the .ctors section: crtbegin.o contains a __CTOR_LIST__ symbol) It's also possible to directly instruct the linker to insert a symbol at the start of the section and at the end of the section. This way you don't need special object files but the symbols will appear always, even for C/C++ programs. In theory it's always possible to use your own linker script without updating the compiler, but that's not a practical solution. Implicit linker-script which only extend the default script can be used for custom sections, but AFAIK this doesn't work for standard sections.
Dec 20 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-12-20 10:13, Johannes Pfau wrote:

 Likely yes, that's why I said 'longterm' solution. You need to edit the
 default linker script. Then there are 2 options: We can force certain
 object files to be at the start or the end of the section. So we could
 introduce dso_start.o and dso_end.o files containing the start/end
 sections and those would always be placed as the first/last objects in
 a section. (This is how the C++/C runtime calls the constructors in the
 .ctors section: crtbegin.o contains a __CTOR_LIST__ symbol)

 It's also possible to directly instruct the linker to insert a symbol
 at the start of the section and at the end of the section. This way you
 don't need special object files but the symbols will appear always,
 even for C/C++ programs.

 In theory it's always possible to use your own linker script without
 updating the compiler, but that's not a practical solution. Implicit
 linker-script which only extend the default script can be used for
 custom sections, but AFAIK this doesn't work for standard sections.
Ok, I see. -- /Jacob Carlborg
Dec 20 2013
prev sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 20 December 2013 07:40, Jacob Carlborg <doob me.com> wrote:
 On 2013-12-19 20:51, Johannes Pfau wrote:

 Sounds like that could work. But as the module section is a custom
 section anyway we wouldn't have to replace/modify the default linker
 script - we can pass a custom script to ld which just handles the
 ".minfo" section. That should be just as portable as relying on the
 "don't reorder sections" behavior: Works everywhere where GNU Binutils
 LD/GOLD are used. (Emitting 3 sections is a clever trick, but it feels
 like a hack imho. I'm also not sure if we can control the order in
 which sections are emitted in GCC)
The trick with brackting sections doesn't work on Mac OS X. DMD had some problems with that. It basically broke on every new major release of Mac OS X, the linker changed all the time. If I recall correctly it either removed the empty sections or reordered the sections.
If using GDC on OSX, then you be going through the emutls module I've got set-up. As for module discovery, we already generate this in the GDC backend, which makes it then the job of runtime to pick-up, sort and run the ctors on all modules: attribute("constructor") void __modinit() { extern (C) __gshared ModuleReference* _Dmodule_ref; static private ModuleReference __mymod = ModuleReference(null, minfo); __mymod.next = _Dmodule_ref; _Dmodule_ref = &__mymod; } Unless I'm missing something in how module discovery is supposed to work, this could be instead amended to: attribute("constructor") void __modinit() { extern (C) void __register_module(ModuleInfo *); __register_module(&minfo); } So that any lazily loaded modules would be recorded to an existing list and trigger a run-once ctor run. Regards Iain
Dec 20 2013
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Friday, 20 December 2013 at 10:54:56 UTC, Iain Buclaw wrote:
 As for module discovery, we already generate this in the GDC 
 backend,
 which makes it then the job of runtime to pick-up, sort and run 
 the
 ctors on all modules:

  attribute("constructor")
 void __modinit()
 {
   extern (C) __gshared ModuleReference* _Dmodule_ref;
   static private ModuleReference __mymod = 
 ModuleReference(null, minfo);

   __mymod.next = _Dmodule_ref;
   _Dmodule_ref = &__mymod;
 }



 Unless I'm missing something in how module discovery is 
 supposed to
 work, this could be instead amended to:

  attribute("constructor")
 void __modinit()
 {
   extern (C) void __register_module(ModuleInfo *);

   __register_module(&minfo);
 }

 So that any lazily loaded modules would be recorded to an 
 existing
 list and trigger a run-once ctor run.
The _Dmodule_ref approach is virtually the same as we have been doing in LDC up to now. The problem with regard to runtime loading of shared libraries is that you need to run certain code (i.e. the current _d_dso_registry in DMD's/LDC's druntime) when you know that you have collected all the modules. You can't do this from a .ctor, because if you also use them for constructing the module list, you'd need to call the registry function in the last .ctor to run. And at least if you don't know the order in which they are written to the executable, you can't know that. It might be possible to work around this requirement by rewriting the druntime internals to perform all the collision checking, etc. one-by-one, but I didn't investigate this in more detail. David
Dec 20 2013
parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 20 December 2013 13:06, David Nadlinger <code klickverbot.at> wrote:
 On Friday, 20 December 2013 at 10:54:56 UTC, Iain Buclaw wrote:
 As for module discovery, we already generate this in the GDC backend,
 which makes it then the job of runtime to pick-up, sort and run the
 ctors on all modules:

  attribute("constructor")
 void __modinit()
 {
   extern (C) __gshared ModuleReference* _Dmodule_ref;
   static private ModuleReference __mymod = ModuleReference(null, minfo);

   __mymod.next = _Dmodule_ref;
   _Dmodule_ref = &__mymod;
 }



 Unless I'm missing something in how module discovery is supposed to
 work, this could be instead amended to:

  attribute("constructor")
 void __modinit()
 {
   extern (C) void __register_module(ModuleInfo *);

   __register_module(&minfo);
 }

 So that any lazily loaded modules would be recorded to an existing
 list and trigger a run-once ctor run.
The _Dmodule_ref approach is virtually the same as we have been doing in LDC up to now. The problem with regard to runtime loading of shared libraries is that you need to run certain code (i.e. the current _d_dso_registry in DMD's/LDC's druntime) when you know that you have collected all the modules. You can't do this from a .ctor, because if you also use them for constructing the module list, you'd need to call the registry function in the last .ctor to run. And at least if you don't know the order in which they are written to the executable, you can't know that. It might be possible to work around this requirement by rewriting the druntime internals to perform all the collision checking, etc. one-by-one, but I didn't investigate this in more detail. David
It should be possible if it's druntime handling all module loading (if you circumvent the module load handlers, don't expect it to work properly). eg: loadModule(mod); // .ctors are ran and modules self register themselves to 'mod' mod.sortCtors(); mod.runSharedCtors(); // etc...
Dec 20 2013
parent "David Nadlinger" <code klickverbot.at> writes:
On Friday, 20 December 2013 at 14:55:46 UTC, Iain Buclaw wrote:
 It should be possible if it's druntime handling all module 
 loading (if
 you circumvent the module load handlers, don't expect it to work
 properly).

 eg:

 loadModule(mod);  // .ctors are ran and modules self register
 themselves to 'mod'
 mod.sortCtors();
 mod.runSharedCtors();
 // etc...
If you just load modules using druntime (or another helper function we control), then you can of course do the initialization like this. You'd lose the benefit of seamlessly being able to load D libraries from e.g. C/C++ plugin hosts, though. David
Dec 20 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-19 18:59, David Nadlinger wrote:

 That is pretty much how it is done in DMD and LDC as well. The only
 difference is that instead of modifying the linker scripts to
 accommodate this, we emit the ModuleInfo references to custom sections.
 The GNU toolchain (and we are in highly system-specific territory here
 anyway) never changes the order of custom sections, which you can also
 verify using __attribute__((section("...))) in GCC. Thus, if you emit
 your relevant symbols into three sections like this,
Does that really work reliably? As far as I remember, the trick with brackting sections doesn't work on Mac OS X. DMD had some problems with that. It basically broke on every new major release of Mac OS X, the linker changed all the time. If I recall correctly it either removed the empty sections or reordered the sections. But the dynamic linker on Mac OS X has an API to access these sections easily anyway. -- /Jacob Carlborg
Dec 19 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 20 Dec 2013 08:43:12 +0100
schrieb Jacob Carlborg <doob me.com>:

 On 2013-12-19 18:59, David Nadlinger wrote:
 
 That is pretty much how it is done in DMD and LDC as well. The only
 difference is that instead of modifying the linker scripts to
 accommodate this, we emit the ModuleInfo references to custom
 sections. The GNU toolchain (and we are in highly system-specific
 territory here anyway) never changes the order of custom sections,
 which you can also verify using __attribute__((section("...))) in
 GCC. Thus, if you emit your relevant symbols into three sections
 like this,
Does that really work reliably? As far as I remember, the trick with brackting sections doesn't work on Mac OS X. DMD had some problems with that. It basically broke on every new major release of Mac OS X, the linker changed all the time. If I recall correctly it either removed the empty sections or reordered the sections. But the dynamic linker on Mac OS X has an API to access these sections easily anyway.
Seems like the OSX linker is not based on the GNU linker and doesn't have linker scripts either. There's some "sectorder" argument which might be useful though. But relying on the runtime on OSX isn't bad as there's only one runtime anyway. On linux we have different libcs and non-standard interfaces which isn't optimal.
Dec 20 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-12-20 10:27, Johannes Pfau wrote:

 Seems like the OSX linker is not based on the GNU linker and doesn't
 have linker scripts either. There's some "sectorder" argument which
 might be useful though.
It's probably BSD.
 But relying on the runtime on OSX isn't bad as there's only one runtime
 anyway. On linux we have different libcs and non-standard interfaces
 which isn't optimal.
Ok, I see. -- /Jacob Carlborg
Dec 20 2013
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 17 Dec 2013 18:48:38 +0100
schrieb "Iain Buclaw" <ibuclaw ubuntu.com>:

 Gthreads and Emulated TLS are now in GDC!!!
 
 https://github.com/D-Programming-GDC/GDC/commit/62554bfe1b35ee4f586634a76c65d83ebfa871ef
 
 The bug noted above still exists, however what I am going to do 
 is update and hook emulated TLS into the GC.
A question about the emutls support: I guess it's not compatible with C/C++ emutls right now? So if we have: test.c: __thread int a; and test.d: extern(C) int a; This wouldn't work right now? I think it could be possible to treat thread local variables in extern(C) code specially and hook it up with the standard GCC emutls. This way we could make it work, however we likely can't make the GC scan extern(C) thread local variables then.
Dec 19 2013
parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 19 December 2013 11:47, Johannes Pfau <nospam example.com> wrote:
 Am Tue, 17 Dec 2013 18:48:38 +0100
 schrieb "Iain Buclaw" <ibuclaw ubuntu.com>:

 Gthreads and Emulated TLS are now in GDC!!!

 https://github.com/D-Programming-GDC/GDC/commit/62554bfe1b35ee4f586634a76c65d83ebfa871ef

 The bug noted above still exists, however what I am going to do
 is update and hook emulated TLS into the GC.
A question about the emutls support: I guess it's not compatible with C/C++ emutls right now? So if we have: test.c: __thread int a; and test.d: extern(C) int a; This wouldn't work right now? I think it could be possible to treat thread local variables in extern(C) code specially and hook it up with the standard GCC emutls. This way we could make it work, however we likely can't make the GC scan extern(C) thread local variables then.
It's compatible with libgcc emutls.
Dec 19 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 19 Dec 2013 15:39:28 +0000
schrieb Iain Buclaw <ibuclaw gdcproject.org>:

 On 19 December 2013 11:47, Johannes Pfau <nospam example.com> wrote:
 Am Tue, 17 Dec 2013 18:48:38 +0100
 schrieb "Iain Buclaw" <ibuclaw ubuntu.com>:
It's compatible with libgcc emutls.
Of course! I totally missed the extern(C) in emutls.d. So emutls.d just overwrites the default emutls implementation in libgcc? I guess this works fine for D apps as long as libgcc is a static library. What about a shared libgcc? Wouldn't the compiler then pickup the emutls function in libgcc instead of libgdruntme? The most difficult case is probably a C app loading a shared D library?
Dec 19 2013
parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On 19 December 2013 16:34, Johannes Pfau <nospam example.com> wrote:
 Am Thu, 19 Dec 2013 15:39:28 +0000
 schrieb Iain Buclaw <ibuclaw gdcproject.org>:

 On 19 December 2013 11:47, Johannes Pfau <nospam example.com> wrote:
 Am Tue, 17 Dec 2013 18:48:38 +0100
 schrieb "Iain Buclaw" <ibuclaw ubuntu.com>:
It's compatible with libgcc emutls.
Of course! I totally missed the extern(C) in emutls.d. So emutls.d just overwrites the default emutls implementation in libgcc? I guess this works fine for D apps as long as libgcc is a static library. What about a shared libgcc? Wouldn't the compiler then pickup the emutls function in libgcc instead of libgdruntme?
It picks up the libgdruntime function in both cases.
 The most difficult case is probably a C app loading a
 shared D library?
That I will worry about when we get to that point.
Dec 19 2013