www.digitalmars.com         C & C++   DMDScript  

D.gnu - Final steps for shared library support

reply Johannes Pfau <nospam example.com> writes:
I've opened this pull request to finish the shared library support:
https://github.com/D-Programming-GDC/GDC/pull/214

These are the remaining problems:

1) We have to find all ModuleInfos in any shared library or in the main
   executable (in the main executable even for statically linked
   libdruntime). We can use a nice binutils feature for this: If we
   place all symbols into a named section _not starting with a dot_
   binutils will provide __start_SECTION and __stop_SECTION symbols for
   us:
   https://sourceware.org/binutils/docs-2.26/ld/Orphan-Sections.html#Orphan-Sections
   https://github.com/jpf91/GDC/blob/shared3/libphobos/libdruntime/__dshared.c#L12

   This should work for all targets using the binutils linker, not just
   for linux. OTOH there are targets without binutils where we could
   also output ModuleInfos into a special section and use runtime
   functions to find the section (e.g. OSX):
   https://github.com/dlang/druntime/blob/master/src/rt/sections_osx_x86_64.d#L131

   I guess we want to keep our platform-independent C constructor based
   approach for finding ModuleInfos for all targets not supported
   directly by the druntime section code. So the difficult question we
   have to answer now is: When does the compiler place the ModuleInfo
   into a section, when does it use the generic constructor code?

   I think this is the best solution: In the druntime configure script,
   run a small test to check for __start/stop_minfo symbols. These
   could be provided by the binutils feature or by explicit linker
   scripts. Then set a variable in gcc.config accordingly.
   In the compiler, search for a useModuleInfoSection manifest constant
   in rt.sections. If rt.sections does not exists or
   useModuleInfoSection is not defined, default to the old constructor
   based code. Otherwise choose accordingly to useModuleInfoSection.
   useModuleInfoSection is then set using static if & CTFE according to
   information from gcc.config and OS information (= true if
   gcc.config.minfoStartStop || version(OSX) ...)


2) We have to call _d_dso_registry once from every shared library and
   from the main executable:
   * DMD emits a weak function _d_dso_init into every object file.
     Additionally it adds a weak reference to this into the .ctors
     section for every object. We could probably do this manually. But
     this is not really portable and I don't think we can get the
     portable DECL_CONSTRUCTOR to emit weak references.
   * LDC also emits the weak _d_dso_init function. It emits normal
     constructors so _d_dso_init will be called multiple times. It uses
     a static variable in _d_dso_init to detect repeated calls.
     We could probably do this. Disadvantages: many useless
     constructors, the source code for _d_dso_init is hardcoded in the
     compiler and we have to hardcode the OS detection (i.e. emit
     _d_dso_init for linux, freebsd, ...)
   * We use the 'GCC way': The way C constructors are implemented is
     using startup files. These are special object files linked once
     into every shared library and executable. I've implemented &
     tested this approach in the pull request:
     If we link against libphobos, load
     https://github.com/jpf91/GDC/blob/shared3/libphobos/libdruntime/libdruntime.spec
     libdruntime.spec will then add __dshared.o to the start files, if
     the file exists. This means:
     * When linking with nophoboslib, __dshared.o and libdruntime.spec
       are not used and do not have to exist (good for bare metal code)
     * The spec can be redefined using GCCs spec file mechanism.
     * If __dshared.o does not exist (maybe not needed for some OSs)
       linking is done as usual.
     * We can add any code we want to __dshared.o/c/d. It will be linked
       once into every shared library and executable. OS detection can
       be done in C or D source code instead of in the compiler.
     * Main drawback: Linking needs to be done with gdc. If a user
       needs to link with gcc or g++ this is still possible using
       -specs=libdruntime.spec
     * As we need to link phobos/druntime with -nophoboslib we also
       have to use -specs=libdruntime.spec here.

3) Building the sections code triggers issue #211. I guess we have to
   add a flag to ignore aggressive-loop-optimizations errors when
   building libdruntime?

4) Unit tests & test suite pass for shared libraries, except for
   one codegen/linker issue in the test suite. OTOH as this does not
   affect static linking we can handle this even after merging the
   initial shared library support code.
May 22 2016
parent Johannes Pfau <nospam example.com> writes:
Am Sun, 22 May 2016 20:41:03 +0200
schrieb Johannes Pfau <nospam example.com>:
 
 2) We have to call _d_dso_registry once from every shared library and
    from the main executable:
    * DMD emits a weak function _d_dso_init into every object file.
      Additionally it adds a weak reference to this into the .ctors
      section for every object. We could probably do this manually. But
      this is not really portable and I don't think we can get the
      portable DECL_CONSTRUCTOR to emit weak references.
    * LDC also emits the weak _d_dso_init function. It emits normal
      constructors so _d_dso_init will be called multiple times. It
 uses a static variable in _d_dso_init to detect repeated calls.
      We could probably do this. Disadvantages: many useless
      constructors, the source code for _d_dso_init is hardcoded in the
      compiler and we have to hardcode the OS detection (i.e. emit
      _d_dso_init for linux, freebsd, ...)
    * We use the 'GCC way': The way C constructors are implemented is
      using startup files. These are special object files linked once
      into every shared library and executable. I've implemented &
      tested this approach in the pull request:
      If we link against libphobos, load
      https://github.com/jpf91/GDC/blob/shared3/libphobos/libdruntime/libdruntime.spec
      libdruntime.spec will then add __dshared.o to the start files, if
      the file exists. This means:
      * When linking with nophoboslib, __dshared.o and libdruntime.spec
        are not used and do not have to exist (good for bare metal
 code)
      * The spec can be redefined using GCCs spec file mechanism.
      * If __dshared.o does not exist (maybe not needed for some OSs)
        linking is done as usual.
      * We can add any code we want to __dshared.o/c/d. It will be
 linked once into every shared library and executable. OS detection can
        be done in C or D source code instead of in the compiler.
      * Main drawback: Linking needs to be done with gdc. If a user
        needs to link with gcc or g++ this is still possible using
        -specs=libdruntime.spec
      * As we need to link phobos/druntime with -nophoboslib we also
        have to use -specs=libdruntime.spec here.
OTOH the LDC approach doesn't sound too bad either. There probably won't be more than ~1000 modules in a shared lib and the ctor stubs we call should be pretty cheap. I don't want the OS detection in the compiler though. Right now the function is quite simple, but who knows what it will look like once we support more OSs. We could probably adapt the __entrypoint.di approach: We need to add support for attribute(constructor, destructor, visibility) first. Then we can translate __dshared.c into D. We'd just ship the source file and compile it into every object file (except for -betterC). We also have to make sure the compiler does not emit ModuleInfo for this module. Then we can simply do this: __dshared.d: // No ModuleInfo for this module module sharedinclude; import rt.sections : useDSORegistry; static if (useDSORegistry) { weak hidden void __d_dso_init() {_d_dso_registry(...)} weak hidden __gshared bool started, stopped; // Have to somehow make sure this function does not clash with the // same function in other object files (i.e. C-like static) private ctor weak void startDSO() { if(started) return; started = true; __d_dso_init(); } [...] } I think both solutions are OK. Iain which one do you prefer?
May 23 2016