www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 4071] New: Missing support to share memory and objects between DLLs and executable

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4071

           Summary: Missing support to share memory and objects between
                    DLLs and executable
           Product: D
           Version: unspecified
          Platform: Other
        OS/Version: Windows
            Status: NEW
          Severity: normal
          Priority: P2
         Component: druntime
        AssignedTo: sean invisibleduck.org
        ReportedBy: r.sagitario gmx.de


--- Comment #0 from Rainer Schuetze <r.sagitario gmx.de> 2010-04-07 11:35:29
PDT ---
Sharing gc-allocated or c-runtime-allocated memory, files, threads and other
objects between different D-DLLs and the D-executable can be desirable,
especially when working in a larger environment with dynamically loaded DLLs.

The current support for this is very limited, supplying a proxy for the GC, but
its implementation is incomplete, e.g. with respect to TLS memory and new
threads.

The following comments will show an implementation of a phobos.dll that can be
accessed by a number of DLLs and the executable to create objects than can be
shared freely. It also allows the usage of a single GC.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Apr 07 2010
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #1 from Rainer Schuetze <r.sagitario gmx.de> 2010-04-07 11:40:36
PDT ---
Created an attachment (id=600)
add options -exportall and -sharedlib to dmd

Let's start with a few patches to dmd:

1. add option -exportall to export any mangled symbol defined by an object file
  (a workaround for #3956 is needed, patching obj_export to avoid
indeterministic removal of leading '_' from exported symbols by optlink)

2. add option -sharedlib to switch to different default libraries
phobos_shared.lib and snn_shared.lib instead of phobos.lib and snn.lib

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Apr 07 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #2 from Rainer Schuetze <r.sagitario gmx.de> 2010-04-07 11:47:04
PDT ---
Created an attachment (id=601)
druntime support for building phobos.dll

The major changes are in druntime and involve splitting files into the part
that can be shared in a single DLL and the functions that must exist per
binary:

- move moduleCtor/Dtor, etc. from object_.di into new file moduleinit.d (had to
copy some ModuleInfo (back?) declarations to object.di to compile)
- move extern(C) main() from dmain2.d into new file cmain.d
- exclude __LDBLULLNG() in llmath.d because it is already in snn.lib
- build druntime.obj with "-exportall" instead of druntime.lib excluding
moduleinit.d, cmain.d and a few more
- extracted stuff from dll_helper.d into new file shared_dll_helper.d and added
a few more functions to support patching relocations

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Apr 07 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #3 from Rainer Schuetze <r.sagitario gmx.de> 2010-04-07 11:54:49
PDT ---
Created an attachment (id=602)
changes to phobos to build phobos.dll

The patches to phobos are more-or-less limited to the makefile and adding 2
files:

- phobos: build phobos.obj with "-exportall" instead of phobos.lib
- add def-file exporting any sensible symbol from the C compiled modules and
snn.lib
- add dllmain.d that works along the line of the new dll-support in dmd 2.042
- build phobos.dll from druntime.obj, C compiled modules and the object files
not included in druntime.obj using the def-file
- use implib to create import library phobos_shared.lib (the optlink /IMLIB
creates a corrupted lib)
- add modules to the lib that must exist in each binary: minit.obj,
moduleinit.obj, cmain.obj, dll_helper.obj, etc.
- extracted some modules from snn.lib (constart, dllstart, winstart, excptlst,
cinit, ehinit, setargv, tlsdata, tlsseg, clearbss) and put them in new lib
snn_shared.lib

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Apr 07 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #4 from Rainer Schuetze <r.sagitario gmx.de> 2010-04-07 12:00:45
PDT ---
Created an attachment (id=603)
example using the shared dll

This is the example from dll.html modified to work with the shared DLL.

Please note that an executable/DLL needs a slightly different initialization:
- patch relocations to stubs to import table to the destination value to allow
data symbols to be accessed correctly
- do not init gc
- inform gc about TLS usage
- add check to module(tls)ctors to avoid calling modules in other dlls

These are already covered by the patches to druntime.

If you apply the patches, you might need to check the path to the tools at the
top of the makefiles.

current restrictions:
- there is no global moduleinfo-array spanning all binaries
- trace module not included (always produces output)
- no version check, so welcome to DLL hell!

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Apr 07 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #5 from Rainer Schuetze <r.sagitario gmx.de> 2010-04-07 12:15:59
PDT ---
forgot to mention that the patches are against dmd rev 431, druntime rev 282
and phobos rev 1477.

you can build the dll by executing

   make -f win32.mak dll

in the phobos directory.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Apr 07 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4071


Jacob Carlborg <doob me.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |doob me.com


--- Comment #6 from Jacob Carlborg <doob me.com> 2010-04-07 13:08:02 PDT ---
I have no experiences of DLLs but here are my experiences on converting Tango
to a dynamic library on Mac OS X. Don't know if any of this would work on
Windows.

Comment 1
An option to build as an dynamic library would be nice

Comment 2
The way I solved the problems here was to declare the D main function as a weak
symbol in a C file. The module constructors are handled by looping through all
loaded images (binaries, dynamic libraries) and collect the module info arrays.
Then combining all the arrays into one and running all the module constructors.
The same is done with the exception handling tables.

The following have I not done yet, but I hope it will work:
In a C file, declare a function to be a module initializer that initializes the
D runtime if no D main function is present.

Don't know if this helps anything.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Apr 07 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #7 from Rainer Schuetze <r.sagitario gmx.de> 2010-04-08 14:29:59
PDT ---
I'm not sure the way you did it on OSX is feasable on Windows aswell, because
you might not have access to all the info in the DLLs (maybe it's possible if
you export all necessary symbols in each DLL). But I guess it will not work for
dynamically loaded DLLs. My implementation does not need a "supervising"
function to run initializers, each binary just initializes the part it
contains.

I haven't done anything with respect to exception handling, but if I understand
it correctly, the exception frames are local to the functions and binary they
belong to, so they still are unwinded correctly. A simple test (catching an
exception from inside phobos.dll) was working correctly.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Apr 08 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #8 from Jacob Carlborg <doob me.com> 2010-04-09 02:26:03 PDT ---
That may actually a better idea, to let every binary handle its own
initialization. Then I guess there will be no problem with module constructors
that are run when they shouldn't.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Apr 09 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #9 from Jacob Carlborg <doob me.com> 2010-04-11 03:56:04 PDT ---
Another question: since you have moved the module initialization into a new
file and let every binary take care of its own initialization, couldn't that
cause problems like circular dependencies that aren't detected? For example,
two modules from two different libraries, each compiled into its own dll, which
import the other module.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Apr 11 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #10 from Rainer Schuetze <r.sagitario gmx.de> 2010-04-11 05:52:26
PDT ---
Circular dependencies between binaries are detected at link time, because
you'll have the circular reference issue when creating the DLLs.

moduleCtor() checks whether imported modules are in the same binary, and if
not, intializers are not called assuming that they are initialized already,
because the DLL they are in is already fully loaded.

The proposed implementation is meant to produce a shared phobos.dll. If you
want to apply it to other libraries, some functions need to be extended
(especially patching relocations to data values which currently works only for
a single DLL).

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Apr 11 2010
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4071


dawg dawgfoto.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dawg dawgfoto.de


--- Comment #11 from dawg dawgfoto.de 2012-01-17 23:56:24 PST ---
I have recently done a ModuleInfo refactoring along this line.

I'd like to avoid having to link in a static library.
The planned ELF mechanism works like the following:

 - The compiler emits a static init function into each
   object, library or executable. This function is a
   comdat so every DLL or EXE will have a single init
   function.

 - _minfo_beg/_minfo_end and _deh_beg/_deh_end are made
    static symbols.

 - The init function registers it's modules/EH tables
   with a global function in druntime. There we create
   an entry and store additional data like TLS index
   and writeable segments for GC.

 - The module initialization must now only iterate over
   all dlls and initialize them in the order of registration.

If we can figure out how to create a static init function
the same mechanism should work for Windows.
By splitting registration and initialization we should
be able to perform module construction outside DllMain of.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jan 17 2012