digitalmars.D - Re: D hates to be dynamic linked
- Eldar Insafutdinov <e.insafutdinov gmail.com> Feb 21 2010
- Justin Johansson <no spam.com> Feb 21 2010
- Lutger <lutger.blijdestijn gmail.com> Feb 21 2010
- Justin Johansson <no spam.com> Feb 21 2010
- Daniel Keep <daniel.keep.lists gmail.com> Feb 24 2010
- Sean Kelly <sean invisibleduck.org> Feb 24 2010
- Rainer Schuetze <r.sagitario gmx.de> Feb 21 2010
- Justin Johansson <no spam.com> Feb 22 2010
- Justin Johansson <no spam.com> Feb 22 2010
- Roald Ribe <rr.nospam nospam.teikom.no> Feb 22 2010
- Bane <branimir.milosavljevic gmail.com> Feb 23 2010
- Lutger <lutger.blijdestijn gmail.com> Feb 22 2010
- Rainer Schuetze <r.sagitario gmx.de> Feb 23 2010
- Don <nospam nospam.com> Feb 24 2010
- Walter Bright <newshound1 digitalmars.com> Feb 24 2010
- Rainer Schuetze <r.sagitario gmx.de> Mar 06 2010
- Don <nospam nospam.com> Mar 09 2010
- Rainer Schuetze <r.sagitario gmx.de> Mar 10 2010
- Steve Teale <steve.teale britseyeview.com> Feb 21 2010
- "Robert Jacques" <sandford jhu.edu> Feb 21 2010
- "Robert Jacques" <sandford jhu.edu> Feb 23 2010
- "Robert Jacques" <sandford jhu.edu> Feb 24 2010
Steve Teale Wrote:On Fri, 19 Feb 2010 16:41:20 -0500, g wrote: I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there.
Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Feb 21 2010
Eldar Insafutdinov wrote:Steve Teale Wrote:On Fri, 19 Feb 2010 16:41:20 -0500, g wrote: I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there.
Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.
Feb 21 2010
Justin Johansson wrote:Eldar Insafutdinov wrote:Steve Teale Wrote:On Fri, 19 Feb 2010 16:41:20 -0500, g wrote: I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there.
Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.
At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?
Feb 21 2010
Lutger wrote:Justin Johansson wrote:Eldar Insafutdinov wrote:Steve Teale Wrote:On Fri, 19 Feb 2010 16:41:20 -0500, g wrote: I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there.
plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.
At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?
Specifically my roadblock was the inability to create shared objects (.so files) on Linux platform (analogous to DLLs in Windows). This was required for the plug-in architecture that I was targeting.
Feb 21 2010
Another important question: does the application still blow up if an exception gets thrown across the DLL boundary?
Feb 24 2010
Daniel Keep Wrote:Another important question: does the application still blow up if an exception gets thrown across the DLL boundary?
If you called Runtime.loadLibrary() to load the DLL then it shouldn't.
Feb 24 2010
Hi, as I'm also hitting some problems with DLLs, here are some issues that I am now aware of (sorry, can't tell for linux shared objects, but I guess the situation is similar): 1. For D2, there is a major blocker with DLLs loaded after intialization on XP because of no TLS support from the OS. There is a simple workaround for single-threaded application (just setting FS:2c to a pointer to _tlsstart), but I'm considering a full emulation of the TLS initialization. 2. No multi-threading support: I've added a function to std.thread that allows adding a thread to the Thread.allThreads array, so it can be called from DLLMain/THREAD_ATTACH. That way the GC can suspend these, and allocations from different threads don't trample onto each other. Is there anything else needed for full multi-threading support? My patch is currently D1/Win32 only, but should not be too hard to extend to other platforms. Attaching to already existing threads is a bit harder, though, and it might not always be desired, because these threads might never call into D-code. 3. To try things out, I am playing around with the mydll-example, and it shows some quirks that you need to get around: the implementation-file for the DLL needs to use a file name different from the module name, because you need another file specifying the exports. Unfortunately this cannot be the di-file created from the source, because it contains two much information that will cause references to symbols not actually exported. Maybe some command line switch is needed to just write the exports into the di file without any implementations. Even better: import the implementation file, but don't create unnecessary references. (I think it's the module-initialization that's been called because of the import - maybe some modifier to the import could remove it). 4. The documentation on the website states, that you should use DATA PRELOAD SINGLE in the def-file. That probably is still there as a historical note to win 3.1, it will cause different processes to trample on each others data segments. This has caused a few hours of debugging, so please remove it, so others don't fall into the same trap. The mydll-example does not use the statement. 5. To share gc-collected objects between different DLLs, a common phobos-DLL seems necessary. Extracting the GC into a separate DLL and using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling? I'd say, if there is a way to put all public symbols into a def-file, then compile phobos into a DLL and create the import library and use this instead of a static library. The multi-threading-issue for DLLs needs to be solved before, though. I'm currently attacking 1. and 2., but only on windows at the moment. I'll add patches and reports into bugzilla later... Rainer Lutger wrote:Justin Johansson wrote:Eldar Insafutdinov wrote:Steve Teale Wrote:On Fri, 19 Feb 2010 16:41:20 -0500, g wrote: I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there.
plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.
At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?
Feb 21 2010
Rainer Schuetze wrote:Hi, as I'm also hitting some problems with DLLs, here are some issues
I guess the situation is similar):1. For D2, there is a major blocker with DLLs loaded after
simple workaround for single-threaded application (just setting FS:2c to a pointer to _tlsstart), but I'm considering a full emulation of the TLS initialization.2. No multi-threading support: I've added a function to std.thread
called from DLLMain/THREAD_ATTACH. That way the GC can suspend these, and allocations from different threads don't trample onto each other. Is there anything else needed for full multi-threading support?My patch is currently D1/Win32 only, but should not be too hard to
bit harder, though, and it might not always be desired, because these threads might never call into D-code.3. To try things out, I am playing around with the mydll-example, and
implementation-file for the DLL needs to use a file name different from the module name, because you need another file specifying the exports. Unfortunately this cannot be the di-file created from the source, because it contains two much information that will cause references to symbols not actually exported.Maybe some command line switch is needed to just write the exports
implementation file, but don't create unnecessary references. (I think it's the module-initialization that's been called because of the import - maybe some modifier to the import could remove it).4. The documentation on the website states, that you should use DATA PRELOAD SINGLE in the def-file. That probably is still there as a historical note to
data segments. This has caused a few hours of debugging, so please remove it, so others don't fall into the same trap. The mydll-example does not use the statement.5. To share gc-collected objects between different DLLs, a common
using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling?
I'd say, if there is a way to put all public symbols into a def-file,
this instead of a static library. The multi-threading-issue for DLLs needs to be solved before, though. Hi Rainer, Congrats. Sounds like you are giving the problem a lot of thought. Just regarding the public symbols though, I hazard a guess that you might run into a name-mangling problem here. Don't want to put a damper on your effort though; just a forewarning that this might be an issue. Perhaps someone else who knows about exporting DLL symbols might be gracious enough to chime in. Cheers Justin
Feb 22 2010
Rainer Schuetze wrote:5. To share gc-collected objects between different DLLs, a common phobos-DLL seems necessary. Extracting the GC into a separate DLL and using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling?
Sounds like you have the right idea; that sounds similar to the way Microsoft does it with MFC DLL's. If I recall correctly, when MFC applications are DLL based, you link with a common C runtime DLL as well. This way all memory allocations and frees are handled by the common C runtime DLL (i.e. single point of responsibility). Presumably a similar regime for D and the GC would be necessary.
Feb 22 2010
Justin Johansson wrote:Rainer Schuetze wrote:5. To share gc-collected objects between different DLLs, a common phobos-DLL seems necessary. Extracting the GC into a separate DLL and using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling?
Sounds like you have the right idea; that sounds similar to the way Microsoft does it with MFC DLL's. If I recall correctly, when MFC applications are DLL based, you link with a common C runtime DLL as well. This way all memory allocations and frees are handled by the common C runtime DLL (i.e. single point of responsibility). Presumably a similar regime for D and the GC would be necessary.
Not only MFC. Plain C and C++ programs as well. This is why most MS- Windows based compiled languages comes with two versions of the runtime lib, one for static link and one for dynamic link (DLL). If not, there will always be problems with DLL's in that language sharing file handles, memory, sockets, threads and so on. The flip side of the coin is "DLL hell" (use Google). It is less of a problem in newer versions of MS-Windows than it used to be, but necessary to consider when supporting older platforms. The Linux/BSD/unix side I know less about, but since they have a common location for shared libs and support multiple versions of the same lib (AFAIK) the problems are less there. Roald
Feb 22 2010
Just to get it righ: this means GC will be in dl tool so people that complain about exe size and gc bloat will be happy about it? Roald Ribe Wrote:Justin Johansson wrote:Rainer Schuetze wrote:5. To share gc-collected objects between different DLLs, a common phobos-DLL seems necessary. Extracting the GC into a separate DLL and using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling?
Sounds like you have the right idea; that sounds similar to the way Microsoft does it with MFC DLL's. If I recall correctly, when MFC applications are DLL based, you link with a common C runtime DLL as well. This way all memory allocations and frees are handled by the common C runtime DLL (i.e. single point of responsibility). Presumably a similar regime for D and the GC would be necessary.
Not only MFC. Plain C and C++ programs as well. This is why most MS- Windows based compiled languages comes with two versions of the runtime lib, one for static link and one for dynamic link (DLL). If not, there will always be problems with DLL's in that language sharing file handles, memory, sockets, threads and so on. The flip side of the coin is "DLL hell" (use Google). It is less of a problem in newer versions of MS-Windows than it used to be, but necessary to consider when supporting older platforms. The Linux/BSD/unix side I know less about, but since they have a common location for shared libs and support multiple versions of the same lib (AFAIK) the problems are less there. Roald
Feb 23 2010
Thank you and everybody else for the explanation. Perhaps a shared phobos lib distributed with dmd will become a reality after all, not too far in the future.
Feb 22 2010
I've seen that, too, but have not tried it yet. As Roald has also pointed out, there are other things you might want to share between dlls (threads, file handles, etc.). Robert Jacques wrote:On Mon, 22 Feb 2010 02:46:31 -0500, Rainer Schuetze <r.sagitario gmx.de> wrote:5. To share gc-collected objects between different DLLs, a common phobos-DLL seems necessary. Extracting the GC into a separate DLL and using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling?
druntime (D2) already supports sharing the GC between DLLs.
Feb 23 2010
Rainer Schuetze wrote:Hi, as I'm also hitting some problems with DLLs, here are some issues that I am now aware of (sorry, can't tell for linux shared objects, but I guess the situation is similar): 1. For D2, there is a major blocker with DLLs loaded after intialization on XP because of no TLS support from the OS. There is a simple workaround for single-threaded application (just setting FS:2c to a pointer to _tlsstart), but I'm considering a full emulation of the TLS initialization.
That's incredibly difficult. Have you read this? http://www.nynaeve.net/?p=189 It'd be great if you can get it to work. But I fear that for Win32, DMD is going to have to switch to explicit TLS.
Feb 24 2010
Don wrote:Rainer Schuetze wrote:Hi, as I'm also hitting some problems with DLLs, here are some issues that I am now aware of (sorry, can't tell for linux shared objects, but I guess the situation is similar): 1. For D2, there is a major blocker with DLLs loaded after intialization on XP because of no TLS support from the OS. There is a simple workaround for single-threaded application (just setting FS:2c to a pointer to _tlsstart), but I'm considering a full emulation of the TLS initialization.
That's incredibly difficult. Have you read this? http://www.nynaeve.net/?p=189 It'd be great if you can get it to work. But I fear that for Win32, DMD is going to have to switch to explicit TLS.
I talked to the guy who wrote that. He wrote some workaround code to make TLS work on older Windows versions, and I asked him if we could license it. He was reluctant, saying it was experimental only. So yeah, rolling our own may be the only viable option.
Feb 24 2010
I knew that blog, but I expected to find the necessary data somewhere in the "undocumented" (but still available) structures. But I had to dig a little deeper into ntdll myself and came up with a patch that is part of the multithread-support for DLLs: http://d.puremagic.com/issues/show_bug.cgi?id=3885 It emulates the behaviour at process startup time, and modifies the data structures so that windows handles the DLL as if it was loaded with the process. This has the nice benefit that you don't need to handle the creation of new threads. The downsides are: - DLL-unload is disabled, but I think unloading a DLL that uses TLS is problematic anyway) - It will leak some pointer-arrays in favor of not causing crashes due to other threads using TLS at same time as reallocating these arrays. As XP is not a fast moving target anymore, I hope that ntdll will not change a lot, so it will not need adjustment for every windows-update. I've checked SP2 and SP3, which look very much the same with respect to TLS implementation. Rainer Don wrote:Rainer Schuetze wrote:Hi, as I'm also hitting some problems with DLLs, here are some issues that I am now aware of (sorry, can't tell for linux shared objects, but I guess the situation is similar): 1. For D2, there is a major blocker with DLLs loaded after intialization on XP because of no TLS support from the OS. There is a simple workaround for single-threaded application (just setting FS:2c to a pointer to _tlsstart), but I'm considering a full emulation of the TLS initialization.
That's incredibly difficult. Have you read this? http://www.nynaeve.net/?p=189 It'd be great if you can get it to work. But I fear that for Win32, DMD is going to have to switch to explicit TLS.
Mar 06 2010
Rainer Schuetze wrote:I knew that blog, but I expected to find the necessary data somewhere in the "undocumented" (but still available) structures. But I had to dig a little deeper into ntdll myself and came up with a patch that is part of the multithread-support for DLLs: http://d.puremagic.com/issues/show_bug.cgi?id=3885 It emulates the behaviour at process startup time, and modifies the data structures so that windows handles the DLL as if it was loaded with the process. This has the nice benefit that you don't need to handle the creation of new threads. The downsides are: - DLL-unload is disabled, but I think unloading a DLL that uses TLS is problematic anyway) - It will leak some pointer-arrays in favor of not causing crashes due to other threads using TLS at same time as reallocating these arrays. As XP is not a fast moving target anymore, I hope that ntdll will not change a lot, so it will not need adjustment for every windows-update. I've checked SP2 and SP3, which look very much the same with respect to TLS implementation. Rainer
You're totally awesome. This is a game-changer.
Mar 09 2010
Don wrote:Rainer Schuetze wrote:I knew that blog, but I expected to find the necessary data somewhere in the "undocumented" (but still available) structures. But I had to dig a little deeper into ntdll myself and came up with a patch that is part of the multithread-support for DLLs: http://d.puremagic.com/issues/show_bug.cgi?id=3885 It emulates the behaviour at process startup time, and modifies the data structures so that windows handles the DLL as if it was loaded with the process. This has the nice benefit that you don't need to handle the creation of new threads. The downsides are: - DLL-unload is disabled, but I think unloading a DLL that uses TLS is problematic anyway) - It will leak some pointer-arrays in favor of not causing crashes due to other threads using TLS at same time as reallocating these arrays. As XP is not a fast moving target anymore, I hope that ntdll will not change a lot, so it will not need adjustment for every windows-update. I've checked SP2 and SP3, which look very much the same with respect to TLS implementation. Rainer
You're totally awesome. This is a game-changer.
Thanks :-) The patch might still need a little polishing before finding it's way into druntime, though...
Mar 10 2010
On Sun, 21 Feb 2010 14:06:44 +0100, Lutger wrote:Justin Johansson wrote:Eldar Insafutdinov wrote:Steve Teale Wrote:On Fri, 19 Feb 2010 16:41:20 -0500, g wrote: I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there.
plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.
At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?
I have not got to the bottom of this yet - been doing building work to get a house finished that was deemed to have higher priority. Just to keep things basic, I did quite a bit of investigation on the creation of Linux shared libraries. I could not do it cleanly with either DMD or GDC. There's an account of what I tried on the blog section of http:// www.britseyeview.com/GDC-newbie.html. I think a major stumbling block is that Phobos is not a shared library, so the dynamic linking process can't just pull in stuff it needs when loading your shared library plugin. In addition though I think there may be a problem with the runtime. I had difficulties with a symbol called __data_start (or similar). I could see this in the host program, but it was not available to the dynamically loaded object. Also it is not 100% clear whether DMD can actually generate position independent code. I had previously tried to simplify DDL to make it at least be able to load a single object file, and in Windows, I had limited success, but it seemed to me that throwing a lot of effort into the ancient OMF format was not the way to go. I have not had time yet to attempt a similar thing with ELF. So there's more than just dynamic loading, the DMD Windows back end is a bit of a blast from the past anyway. I'm sorry this is rather vague, but I'm not just bleating without having tried some things. Steve
Feb 21 2010
On Sun, 21 Feb 2010 08:06:44 -0500, Lutger <lutger.blijdestijn gmail.com> wrote:Justin Johansson wrote:Eldar Insafutdinov wrote:Steve Teale Wrote:On Fri, 19 Feb 2010 16:41:20 -0500, g wrote: I tried to get some attention for this problem again a couple of weeks ago (see the "special treat" thread), and everyone who replied said "yes, we really need this", but Walter does not want to go there.
Is it really true? That's a big shame. Being able to define and use plugins in a clean cross-platform way is on of the essential features of big software packages that D aims at to be used in. The solution to this problem should really be a top priority, at least I have hoped so.
Yes it is really true. You would have read of my dyna-link saga four posts above yours and, reiterating, that sadly was the final show-stopper for me to continue to using D.
At the risk of asking a dumb question, what is the major roadblock? Is it in the GC? Does it require changing the runtime only, or is it also required to make changes in the compiler (backend)?
I know one issue in D1 is the GC. That's been fixed in D2 with druntime.
Feb 21 2010
On Mon, 22 Feb 2010 02:46:31 -0500, Rainer Schuetze <r.sagitario gmx.de> wrote:5. To share gc-collected objects between different DLLs, a common phobos-DLL seems necessary. Extracting the GC into a separate DLL and using the proxy-mechanism to attach any other client-DLL to it seems feasable, but are there other things that need to be shared between different phobos-instances? What about exception-handling?
druntime (D2) already supports sharing the GC between DLLs.
Feb 23 2010
On Wed, 24 Feb 2010 02:38:58 -0500, Rainer Schuetze <r.sagitario gmx.de> wrote:I've seen that, too, but have not tried it yet. As Roald has also pointed out, there are other things you might want to share between dlls (threads, file handles, etc.).
Given the GC has to know about all threads, I assume those are shared as well. As for file handles, those are just data variables, which can be easily shared between DLLs. The bug problem, as I understand it, with DLLs is global data requires manual re-routing, etc. to work.
Feb 24 2010