www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Re: D hates to be dynamic linked

reply Eldar Insafutdinov <e.insafutdinov gmail.com> writes:
Steve Teale Wrote:

 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:
 
 
 I tried to get some attention for this problem again a couple of weeks 
 ago (see the "special treat" thread), and everyone who replied said "yes, 
 we really need this", but Walter does not want to go there.
 

Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Feb 21 2010
parent reply Justin Johansson <no spam.com> writes:
Eldar Insafutdinov wrote:
 Steve Teale Wrote:
 
 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:


 I tried to get some attention for this problem again a couple of weeks 
 ago (see the "special treat" thread), and everyone who replied said "yes, 
 we really need this", but Walter does not want to go there.

Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.

Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.
Feb 21 2010
next sibling parent reply Lutger <lutger.blijdestijn gmail.com> writes:
Justin Johansson wrote:

 Eldar Insafutdinov wrote:
 Steve Teale Wrote:
 
 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:


 I tried to get some attention for this problem again a couple of weeks
 ago (see the "special treat" thread), and everyone who replied said
 "yes, we really need this", but Walter does not want to go there.

Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.

Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.

At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?
Feb 21 2010
next sibling parent reply Justin Johansson <no spam.com> writes:
Lutger wrote:
 Justin Johansson wrote:
 
 Eldar Insafutdinov wrote:
 Steve Teale Wrote:

 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:


 I tried to get some attention for this problem again a couple of weeks
 ago (see the "special treat" thread), and everyone who replied said
 "yes, we really need this", but Walter does not want to go there.

plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.

You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.

At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?

Specifically my roadblock was the inability to create shared objects (.so files) on Linux platform (analogous to DLLs in Windows). This was required for the plug-in architecture that I was targeting.
Feb 21 2010
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Another important question: does the application still blow up if an
exception gets thrown across the DLL boundary?
Feb 24 2010
parent Sean Kelly <sean invisibleduck.org> writes:
Daniel Keep Wrote:

 
 Another important question: does the application still blow up if an
 exception gets thrown across the DLL boundary?

If you called Runtime.loadLibrary() to load the DLL then it shouldn't.
Feb 24 2010
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Hi,

as I'm also hitting some problems with DLLs, here are some issues that I 
am now aware of (sorry, can't tell for linux shared objects, but I guess 
the situation is similar):

1. For D2, there is a major blocker with DLLs loaded after intialization 
on XP because of no TLS support from the OS. There is a simple 
workaround for single-threaded application (just setting FS:2c to a 
pointer to _tlsstart), but I'm considering a full emulation of the TLS 
initialization.

2. No multi-threading support: I've added a function to std.thread that 
allows adding a thread to the Thread.allThreads array, so it can be 
called from DLLMain/THREAD_ATTACH. That way the GC can suspend these, 
and allocations from different threads don't trample onto each other. Is 
there anything else needed for full multi-threading support?

My patch is currently D1/Win32 only, but should not be too hard to 
extend to other platforms. Attaching to already existing threads is a 
bit harder, though, and it might not always be desired, because these 
threads might never call into D-code.

3. To try things out, I am playing around with the mydll-example, and it 
shows some quirks that you need to get around: the implementation-file 
for the DLL needs to use a file name different from the module name, 
because you need another file specifying the exports. Unfortunately this 
cannot be the di-file created from the source, because it contains two 
much information that will cause references to symbols not actually 
exported.

Maybe some command line switch is needed to just write the exports into 
the di file without any implementations. Even better: import the 
implementation file, but don't create unnecessary references. (I think 
it's the module-initialization that's been called because of the import 
- maybe some modifier to the import could remove it).

4. The documentation on the website states, that you should use

	DATA            PRELOAD SINGLE

in the def-file. That probably is still there as a historical note to 
win 3.1, it will cause different processes to trample on each others 
data segments. This has caused a few hours of debugging, so please 
remove it, so others don't fall into the same trap. The mydll-example 
does not use the statement.

5. To share gc-collected objects between different DLLs, a common 
phobos-DLL seems necessary. Extracting the GC into a separate DLL and 
using the proxy-mechanism to attach any other client-DLL to it seems 
feasable, but are there other things that need to be shared between 
different phobos-instances? What about exception-handling?

I'd say, if there is a way to put all public symbols into a def-file, 
then compile phobos into a DLL and create the import library and use 
this instead of a static library. The multi-threading-issue for DLLs 
needs to be solved before, though.

I'm currently attacking 1. and 2., but only on windows at the moment. 
I'll add patches and reports into bugzilla later...

Rainer

Lutger wrote:
 Justin Johansson wrote:
 
 Eldar Insafutdinov wrote:
 Steve Teale Wrote:

 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:


 I tried to get some attention for this problem again a couple of weeks
 ago (see the "special treat" thread), and everyone who replied said
 "yes, we really need this", but Walter does not want to go there.

plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.

You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.

At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?

Feb 21 2010
next sibling parent Justin Johansson <no spam.com> writes:
Rainer Schuetze wrote:
 Hi,

 as I'm also hitting some problems with DLLs, here are some issues 

I guess the situation is similar):
 1. For D2, there is a major blocker with DLLs loaded after 

simple workaround for single-threaded application (just setting FS:2c to a pointer to _tlsstart), but I'm considering a full emulation of the TLS initialization.
 2. No multi-threading support: I've added a function to std.thread 

called from DLLMain/THREAD_ATTACH. That way the GC can suspend these, and allocations from different threads don't trample onto each other. Is there anything else needed for full multi-threading support?
 My patch is currently D1/Win32 only, but should not be too hard to 

bit harder, though, and it might not always be desired, because these threads might never call into D-code.
 3. To try things out, I am playing around with the mydll-example, and 

implementation-file for the DLL needs to use a file name different from the module name, because you need another file specifying the exports. Unfortunately this cannot be the di-file created from the source, because it contains two much information that will cause references to symbols not actually exported.
 Maybe some command line switch is needed to just write the exports 

implementation file, but don't create unnecessary references. (I think it's the module-initialization that's been called because of the import - maybe some modifier to the import could remove it).
 4. The documentation on the website states, that you should use

     DATA            PRELOAD SINGLE

 in the def-file. That probably is still there as a historical note to 

data segments. This has caused a few hours of debugging, so please remove it, so others don't fall into the same trap. The mydll-example does not use the statement.
 5. To share gc-collected objects between different DLLs, a common 

using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling?

 I'd say, if there is a way to put all public symbols into a def-file, 

this instead of a static library. The multi-threading-issue for DLLs needs to be solved before, though. Hi Rainer, Congrats. Sounds like you are giving the problem a lot of thought. Just regarding the public symbols though, I hazard a guess that you might run into a name-mangling problem here. Don't want to put a damper on your effort though; just a forewarning that this might be an issue. Perhaps someone else who knows about exporting DLL symbols might be gracious enough to chime in. Cheers Justin
Feb 22 2010
prev sibling next sibling parent reply Justin Johansson <no spam.com> writes:
Rainer Schuetze wrote:

 5. To share gc-collected objects between different DLLs, a common 
 phobos-DLL seems necessary. Extracting the GC into a separate DLL and 
 using the proxy-mechanism to attach any other client-DLL to it seems 
 feasable, but are there other things that need to be shared between 
 different phobos-instances? What about exception-handling?
 

Sounds like you have the right idea; that sounds similar to the way Microsoft does it with MFC DLL's. If I recall correctly, when MFC applications are DLL based, you link with a common C runtime DLL as well. This way all memory allocations and frees are handled by the common C runtime DLL (i.e. single point of responsibility). Presumably a similar regime for D and the GC would be necessary.
Feb 22 2010
parent reply Roald Ribe <rr.nospam nospam.teikom.no> writes:
Justin Johansson wrote:
 Rainer Schuetze wrote:
 
 5. To share gc-collected objects between different DLLs, a common 
 phobos-DLL seems necessary. Extracting the GC into a separate DLL and 
 using the proxy-mechanism to attach any other client-DLL to it seems 
 feasable, but are there other things that need to be shared between 
 different phobos-instances? What about exception-handling?

Sounds like you have the right idea; that sounds similar to the way Microsoft does it with MFC DLL's. If I recall correctly, when MFC applications are DLL based, you link with a common C runtime DLL as well. This way all memory allocations and frees are handled by the common C runtime DLL (i.e. single point of responsibility). Presumably a similar regime for D and the GC would be necessary.

Not only MFC. Plain C and C++ programs as well. This is why most MS- Windows based compiled languages comes with two versions of the runtime lib, one for static link and one for dynamic link (DLL). If not, there will always be problems with DLL's in that language sharing file handles, memory, sockets, threads and so on. The flip side of the coin is "DLL hell" (use Google). It is less of a problem in newer versions of MS-Windows than it used to be, but necessary to consider when supporting older platforms. The Linux/BSD/unix side I know less about, but since they have a common location for shared libs and support multiple versions of the same lib (AFAIK) the problems are less there. Roald
Feb 22 2010
parent Bane <branimir.milosavljevic gmail.com> writes:
Just to get it righ: this means GC will be in dl tool so people that complain
about exe size and gc bloat will be happy about it?

Roald Ribe Wrote:

 Justin Johansson wrote:
 Rainer Schuetze wrote:
 
 5. To share gc-collected objects between different DLLs, a common 
 phobos-DLL seems necessary. Extracting the GC into a separate DLL and 
 using the proxy-mechanism to attach any other client-DLL to it seems 
 feasable, but are there other things that need to be shared between 
 different phobos-instances? What about exception-handling?

Sounds like you have the right idea; that sounds similar to the way Microsoft does it with MFC DLL's. If I recall correctly, when MFC applications are DLL based, you link with a common C runtime DLL as well. This way all memory allocations and frees are handled by the common C runtime DLL (i.e. single point of responsibility). Presumably a similar regime for D and the GC would be necessary.

Not only MFC. Plain C and C++ programs as well. This is why most MS- Windows based compiled languages comes with two versions of the runtime lib, one for static link and one for dynamic link (DLL). If not, there will always be problems with DLL's in that language sharing file handles, memory, sockets, threads and so on. The flip side of the coin is "DLL hell" (use Google). It is less of a problem in newer versions of MS-Windows than it used to be, but necessary to consider when supporting older platforms. The Linux/BSD/unix side I know less about, but since they have a common location for shared libs and support multiple versions of the same lib (AFAIK) the problems are less there. Roald

Feb 23 2010
prev sibling next sibling parent Lutger <lutger.blijdestijn gmail.com> writes:
Thank you and everybody else for the explanation. Perhaps a shared phobos 
lib distributed with dmd will become a reality after all, not too far in the 
future.  
Feb 22 2010
prev sibling next sibling parent Rainer Schuetze <r.sagitario gmx.de> writes:
I've seen that, too, but have not tried it yet. As Roald has also 
pointed out, there are other things you might want to share between dlls 
(threads, file handles, etc.).

Robert Jacques wrote:
 On Mon, 22 Feb 2010 02:46:31 -0500, Rainer Schuetze <r.sagitario gmx.de> 
 wrote:
 5. To share gc-collected objects between different DLLs, a common 
 phobos-DLL seems necessary. Extracting the GC into a separate DLL and 
 using the proxy-mechanism to attach any other client-DLL to it seems 
 feasable, but are there other things that need to be shared between 
 different phobos-instances? What about exception-handling?

druntime (D2) already supports sharing the GC between DLLs.

Feb 23 2010
prev sibling parent reply Don <nospam nospam.com> writes:
Rainer Schuetze wrote:
 Hi,
 
 as I'm also hitting some problems with DLLs, here are some issues that I 
 am now aware of (sorry, can't tell for linux shared objects, but I guess 
 the situation is similar):
 
 1. For D2, there is a major blocker with DLLs loaded after intialization 
 on XP because of no TLS support from the OS. There is a simple 
 workaround for single-threaded application (just setting FS:2c to a 
 pointer to _tlsstart), but I'm considering a full emulation of the TLS 
 initialization.

That's incredibly difficult. Have you read this? http://www.nynaeve.net/?p=189 It'd be great if you can get it to work. But I fear that for Win32, DMD is going to have to switch to explicit TLS.
Feb 24 2010
next sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Don wrote:
 Rainer Schuetze wrote:
 Hi,

 as I'm also hitting some problems with DLLs, here are some issues that 
 I am now aware of (sorry, can't tell for linux shared objects, but I 
 guess the situation is similar):

 1. For D2, there is a major blocker with DLLs loaded after 
 intialization on XP because of no TLS support from the OS. There is a 
 simple workaround for single-threaded application (just setting FS:2c 
 to a pointer to _tlsstart), but I'm considering a full emulation of 
 the TLS initialization.

That's incredibly difficult. Have you read this? http://www.nynaeve.net/?p=189 It'd be great if you can get it to work. But I fear that for Win32, DMD is going to have to switch to explicit TLS.

I talked to the guy who wrote that. He wrote some workaround code to make TLS work on older Windows versions, and I asked him if we could license it. He was reluctant, saying it was experimental only. So yeah, rolling our own may be the only viable option.
Feb 24 2010
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
I knew that blog, but I expected to find the necessary data somewhere in 
the "undocumented" (but still available) structures.

But I had to dig a little deeper into ntdll myself and came up with a 
patch that is part of the multithread-support for DLLs:

http://d.puremagic.com/issues/show_bug.cgi?id=3885

It emulates the behaviour at process startup time, and modifies the data 
structures so that windows handles the DLL as if it was loaded with the 
process. This has the nice benefit that you don't need to handle the 
creation of new threads.

The downsides are:
- DLL-unload is disabled, but I think unloading a DLL that uses TLS is 
problematic anyway)
- It will leak some pointer-arrays in favor of not causing crashes due 
to other threads using TLS at same time as reallocating these arrays.

As XP is not a fast moving target anymore, I hope that ntdll will not 
change a lot, so it will not need adjustment for every windows-update. 
I've checked SP2 and SP3, which look very much the same with respect to 
TLS implementation.

Rainer

Don wrote:
 Rainer Schuetze wrote:
 Hi,

 as I'm also hitting some problems with DLLs, here are some issues that 
 I am now aware of (sorry, can't tell for linux shared objects, but I 
 guess the situation is similar):

 1. For D2, there is a major blocker with DLLs loaded after 
 intialization on XP because of no TLS support from the OS. There is a 
 simple workaround for single-threaded application (just setting FS:2c 
 to a pointer to _tlsstart), but I'm considering a full emulation of 
 the TLS initialization.

That's incredibly difficult. Have you read this? http://www.nynaeve.net/?p=189 It'd be great if you can get it to work. But I fear that for Win32, DMD is going to have to switch to explicit TLS.

Mar 06 2010
parent reply Don <nospam nospam.com> writes:
Rainer Schuetze wrote:
 I knew that blog, but I expected to find the necessary data somewhere in 
 the "undocumented" (but still available) structures.
 
 But I had to dig a little deeper into ntdll myself and came up with a 
 patch that is part of the multithread-support for DLLs:
 
 http://d.puremagic.com/issues/show_bug.cgi?id=3885
 
 It emulates the behaviour at process startup time, and modifies the data 
 structures so that windows handles the DLL as if it was loaded with the 
 process. This has the nice benefit that you don't need to handle the 
 creation of new threads.
 
 The downsides are:
 - DLL-unload is disabled, but I think unloading a DLL that uses TLS is 
 problematic anyway)
 - It will leak some pointer-arrays in favor of not causing crashes due 
 to other threads using TLS at same time as reallocating these arrays.
 
 As XP is not a fast moving target anymore, I hope that ntdll will not 
 change a lot, so it will not need adjustment for every windows-update. 
 I've checked SP2 and SP3, which look very much the same with respect to 
 TLS implementation.
 
 Rainer

You're totally awesome. This is a game-changer.
Mar 09 2010
parent Rainer Schuetze <r.sagitario gmx.de> writes:
Don wrote:
 Rainer Schuetze wrote:
 I knew that blog, but I expected to find the necessary data somewhere 
 in the "undocumented" (but still available) structures.

 But I had to dig a little deeper into ntdll myself and came up with a 
 patch that is part of the multithread-support for DLLs:

 http://d.puremagic.com/issues/show_bug.cgi?id=3885

 It emulates the behaviour at process startup time, and modifies the 
 data structures so that windows handles the DLL as if it was loaded 
 with the process. This has the nice benefit that you don't need to 
 handle the creation of new threads.

 The downsides are:
 - DLL-unload is disabled, but I think unloading a DLL that uses TLS is 
 problematic anyway)
 - It will leak some pointer-arrays in favor of not causing crashes due 
 to other threads using TLS at same time as reallocating these arrays.

 As XP is not a fast moving target anymore, I hope that ntdll will not 
 change a lot, so it will not need adjustment for every windows-update. 
 I've checked SP2 and SP3, which look very much the same with respect 
 to TLS implementation.

 Rainer

You're totally awesome. This is a game-changer.

Thanks :-) The patch might still need a little polishing before finding it's way into druntime, though...
Mar 10 2010
prev sibling next sibling parent Steve Teale <steve.teale britseyeview.com> writes:
On Sun, 21 Feb 2010 14:06:44 +0100, Lutger wrote:

 Justin Johansson wrote:
 
 Eldar Insafutdinov wrote:
 Steve Teale Wrote:
 
 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:


 I tried to get some attention for this problem again a couple of
 weeks ago (see the "special treat" thread), and everyone who replied
 said "yes, we really need this", but Walter does not want to go
 there.

plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.

Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.

At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?

I have not got to the bottom of this yet - been doing building work to get a house finished that was deemed to have higher priority. Just to keep things basic, I did quite a bit of investigation on the creation of Linux shared libraries. I could not do it cleanly with either DMD or GDC. There's an account of what I tried on the blog section of http:// www.britseyeview.com/GDC-newbie.html. I think a major stumbling block is that Phobos is not a shared library, so the dynamic linking process can't just pull in stuff it needs when loading your shared library plugin. In addition though I think there may be a problem with the runtime. I had difficulties with a symbol called __data_start (or similar). I could see this in the host program, but it was not available to the dynamically loaded object. Also it is not 100% clear whether DMD can actually generate position independent code. I had previously tried to simplify DDL to make it at least be able to load a single object file, and in Windows, I had limited success, but it seemed to me that throwing a lot of effort into the ancient OMF format was not the way to go. I have not had time yet to attempt a similar thing with ELF. So there's more than just dynamic loading, the DMD Windows back end is a bit of a blast from the past anyway. I'm sorry this is rather vague, but I'm not just bleating without having tried some things. Steve
Feb 21 2010
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 21 Feb 2010 08:06:44 -0500, Lutger <lutger.blijdestijn gmail.com>  
wrote:
 Justin Johansson wrote:

 Eldar Insafutdinov wrote:
 Steve Teale Wrote:

 On Fri, 19 Feb 2010 16:41:20 -0500, g wrote:


 I tried to get some attention for this problem again a couple of weeks
 ago (see the "special treat" thread), and everyone who replied said
 "yes, we really need this", but Walter does not want to go there.

Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.

Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.

At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?

I know one issue in D1 is the GC. That's been fixed in D2 with druntime.
Feb 21 2010
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 22 Feb 2010 02:46:31 -0500, Rainer Schuetze <r.sagitario gmx.de>  
wrote:
 5. To share gc-collected objects between different DLLs, a common  
 phobos-DLL seems necessary. Extracting the GC into a separate DLL and  
 using the proxy-mechanism to attach any other client-DLL to it seems  
 feasable, but are there other things that need to be shared between  
 different phobos-instances? What about exception-handling?

druntime (D2) already supports sharing the GC between DLLs.
Feb 23 2010
prev sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Wed, 24 Feb 2010 02:38:58 -0500, Rainer Schuetze <r.sagitario gmx.de>  
wrote:

 I've seen that, too, but have not tried it yet. As Roald has also  
 pointed out, there are other things you might want to share between dlls  
 (threads, file handles, etc.).

Given the GC has to know about all threads, I assume those are shared as well. As for file handles, those are just data variables, which can be easily shared between DLLs. The bug problem, as I understand it, with DLLs is global data requires manual re-routing, etc. to work.
Feb 24 2010