www.digitalmars.com         C & C++   DMDScript  

D - dynamical linking

reply "Lloyd Dupont" <lloyd galador.net> writes:
I'm just wondering how friendly D is with DLL.
I haven't tryed yet but I'm aware you could write a D DLL.

My wonder is would it be a standart D feature. i.e. would phobos be shipped
as a DLL/.so ?
because, if not, even if I could write a D DLL, multiple D program will all
ship multiple copies of statically linked phobos library.

not only it's a waste of memory (when runned) but it also end with multiple
GC which do not play nicely all together...
Feb 16 2003
parent reply Michael Slater <mail effectivity.com> writes:
Lloyd Dupont wrote:
 I'm just wondering how friendly D is with DLL.
 I haven't tryed yet but I'm aware you could write a D DLL.
 
 My wonder is would it be a standart D feature. i.e. would phobos be shipped
 as a DLL/.so ?
 because, if not, even if I could write a D DLL, multiple D program will all
 ship multiple copies of statically linked phobos library.
 
 not only it's a waste of memory (when runned) but it also end with multiple
 GC which do not play nicely all together...

There is some basic information on creating DLL's on the Win32 page of Walter's D site: http://www.digitalmars.com/d/windows.html I would hazard the guess that Phobos would be available as a DLL as well as a static lib. This would parallel C/C++. In modern Windows, a DLL is loaded into the address space of each calling program. So we are comparing a single static lib vs. a single DLL, the memory usage difference is minimal. The disk space differs certainly, but disk space in the modern age is cheap, far cheaper than memory. DLL's introduce their own issues -- mostly dependencies, but often times you need to do some tweaking to have them load the way you want them to vs. the default. Most of the GC's that I've seen have problems when used with DLL's. For instance, if you have a DLL that is allocating memory for many different processes, you now need a GC that has to know about all the separate processes... more of an OS-level GC than an application-level GC. This is for a DLL that even knows about GC and uses the same GC as the calling app. You could also have the DLL rely on calling back into the calling app to use the app GC to alloc memory. Some clever system engineering can enable many different options. And then when calling functions in DLL's that are written in non-GC languages, it becomes even more of a puzzle. Moreover, when two applications are built with different versions of D, you want to make sure that each is built and tested with all the versions of runtime libs that it is expected to use. Once you get a body of code out there and the language evolves, there will be many runtime DLL's each with its own version of the GC, hopefully each runtime DLL will have a unique name. Theoretically with static linking, a smart linker could leave out everything that isn't used resulting in very small runtime image sizes. However, in practice this seems difficult to achieve. At Borland, I built a version of the C RTL as individual object modules (not a LIB) to see if TLink would do a better job with it. Well, it was smaller but not that much smaller. D uses OPTLink and I'm not familiar with how smart it is about not linking in unreferenced obj's/modules. I do remember, fuzzily, that OPTLink was a pretty good linker, so I hopeful. The sieve.d example works out to about 90K, but I don't know what's in there. I need to run TDump and take a peek. I don't know what will happen with modern Windows and DLL's as we know them today. Microsoft is dying to deprecate all the old styles of API's and go with .NET. However, much of the "OS" is still in COM -- DirectX for instance. And Microsoft got a lot of feedback on their attempt to back burner C++ (and replace it with C#) and is putting some more resources on properly bringing C++ into the .NET age. And please forgive my DLL ramble. I've spent far too much time with Windows DLL's. --ms
Feb 16 2003
next sibling parent reply Lloyd Dupont <lloyd galador.net> writes:
I will try to clarify my assumption to clarify my questions.

lets assume you have a stack/array/pool/set/something which hold the garbage 
collected values.

let's imaging that there is an internal function like this to access it:

Set getGCSet();


let's imaging you have a D executable called a.exe wich, of course have a 
statically linked phobos library withe the folowing function:
Set getGCSet() in its own private address space.

let's imaging that tis a.exe program use a DLL which happened to be a D DLL. 
As it's also statically linked this DLL has it own copy of 
Set getGCSet(); which also run its own address space.

now, there is th duplicate 'Set getGCSet()', why not ? but it also mean you 
have to take particular care when passing an object from the exe to the DLL 
and the other way round, because you should assume (rightfully) thatthe 
other exe/dll doesn't know how to handle you reference. and this is so 
sad....

Where as if phobos would be a DLL both this exe and this DLL would share the 
same Set getGCSet() function and would shared object reference without 
minding too much.

I had to admit there is the problem that D could link against C or D exe/DLL 
and that a C program could use a D DLL, so you have to take care anyway. 
But maybe there is a way to solve this problem ?
Or did I get it completly wrong ?

Michael Slater wrote:

 Lloyd Dupont wrote:
 I'm just wondering how friendly D is with DLL.
 I haven't tryed yet but I'm aware you could write a D DLL.
 
 My wonder is would it be a standart D feature. i.e. would phobos be
 shipped as a DLL/.so ?
 because, if not, even if I could write a D DLL, multiple D program will
 all ship multiple copies of statically linked phobos library.
 
 not only it's a waste of memory (when runned) but it also end with
 multiple GC which do not play nicely all together...

There is some basic information on creating DLL's on the Win32 page of Walter's D site: http://www.digitalmars.com/d/windows.html I would hazard the guess that Phobos would be available as a DLL as well as a static lib. This would parallel C/C++. In modern Windows, a DLL is loaded into the address space of each calling program. So we are comparing a single static lib vs. a single DLL, the memory usage difference is minimal. The disk space differs certainly, but disk space in the modern age is cheap, far cheaper than memory. DLL's introduce their own issues -- mostly dependencies, but often times you need to do some tweaking to have them load the way you want them to vs. the default. Most of the GC's that I've seen have problems when used with DLL's. For instance, if you have a DLL that is allocating memory for many different processes, you now need a GC that has to know about all the separate processes... more of an OS-level GC than an application-level GC. This is for a DLL that even knows about GC and uses the same GC as the calling app. You could also have the DLL rely on calling back into the calling app to use the app GC to alloc memory. Some clever system engineering can enable many different options. And then when calling functions in DLL's that are written in non-GC languages, it becomes even more of a puzzle. Moreover, when two applications are built with different versions of D, you want to make sure that each is built and tested with all the versions of runtime libs that it is expected to use. Once you get a body of code out there and the language evolves, there will be many runtime DLL's each with its own version of the GC, hopefully each runtime DLL will have a unique name. Theoretically with static linking, a smart linker could leave out everything that isn't used resulting in very small runtime image sizes. However, in practice this seems difficult to achieve. At Borland, I built a version of the C RTL as individual object modules (not a LIB) to see if TLink would do a better job with it. Well, it was smaller but not that much smaller. D uses OPTLink and I'm not familiar with how smart it is about not linking in unreferenced obj's/modules. I do remember, fuzzily, that OPTLink was a pretty good linker, so I hopeful. The sieve.d example works out to about 90K, but I don't know what's in there. I need to run TDump and take a peek. I don't know what will happen with modern Windows and DLL's as we know them today. Microsoft is dying to deprecate all the old styles of API's and go with .NET. However, much of the "OS" is still in COM -- DirectX for instance. And Microsoft got a lot of feedback on their attempt to back burner C++ (and replace it with C#) and is putting some more resources on properly bringing C++ into the .NET age. And please forgive my DLL ramble. I've spent far too much time with Windows DLL's. --ms

Feb 17 2003
parent Michael Slater <mail effectivity.com> writes:
Lloyd Dupont wrote:
 Where as if phobos would be a DLL both this exe and this DLL would share the 
 same Set getGCSet() function and would shared object reference without 
 minding too much.
 
 I had to admit there is the problem that D could link against C or D exe/DLL 
 and that a C program could use a D DLL, so you have to take care anyway. 
 But maybe there is a way to solve this problem ?
 Or did I get it completly wrong ?

Most if not all modern C/C++ compilers provide for both static and dynamic linking with the runtime library. I believe D will be the same. The default behavior under Windows is for a shared memory management DLL (such as a runtime library with non-GC malloc(), garbage collector with GC malloc(), etc.) is to manage memory on a per-process basis. For each process that calls a particular DLL, Windows maps that DLL's data segment into the address space of the calling process. Thus by default, no data is shared between DLL's. If a developer wants to share some information amongst various instances of a DLL, the DLL's own data segment can be marked as shared. This data segment must contain initialized data for this to work properly. And I believe if a DLL has a shared data segment, only one instance of a particular calling application can be run. So if we have: A.EXE (PHOBOS100.LIB) KOOLGFX1.DLL (PHOBOS100.DLL) Yes, we are going to have two GC's going. A doesn't know anything else other than to use the statically linked runtime lib. KOOLGFX1 of course was compiled to use the runtime DLL. I would not be surprised if you got runtime errors in this "mixed model" configuration. I don't remember the exact "why" of this, a downside to not having a Matt Pietrek to converse with. To facilitate the sharing that you desire, you'd want to build 'everything' to use the runtime library DLL: A.EXE (PHOBOS100.DLL) KOOLGFX1.DLL (PHOBOS100.DLL) How memory management usually works is for a runtime DLL to use VirtualAlloc(). This function reserves memory in the virtual address space of the calling process. This memory is easy to work with. For specific Windows-related tasks, it is important that a systems programming language provide a way to handle memory blocks that are outside of the usual runtime library functions and definitely outside the scope of the usual GC. When we need to share memory between our application process and DDE, the Clipboard, or to use OLE/COM uniform data transfer operations we have to allocate memory for this using GlobalAlloc() which allocates memory off the Windows heap. Note this memory cannot be directly shared between processes. To create a more complex system where memory is shared between processes, a couple things come to mind. One method would be to create a memory-mapped file that is shared by all the running processes. One could also use VirtualAllocEx() along with ReadProcessMemory() and WriteProcessMemory()... going into the land of functions that a debugger usually uses. This would be an entirely kludgy system, though. Everything in the modern programming world assumes each process has its own heap and that this heap is more or less sacrosanct. Is what I'm writing going in any helpful direction or are we wandering around in the weeds? --ms
Feb 17 2003
prev sibling parent Ilya Minkov <midiclub 8ung.at> writes:
A few comments.

Wonder why a consrvative GC for C is always a DLL?
There are a number of resons for this:
  - because a DLL gets notified of thread events, and can keep track of 
them without need for special code generation;
  - because a DLL has no limitations acessing the main programme's 
memory. All dlls run in the same space as the main programme.

DLL mutable sections are loaded into each application's space. However, 
immutable sections, such as code, remain shared. Thus a DLL works as if 
multiple copies were made of it. But since there are means of 
inter-process communication, a DLL which knows itself very well could 
also use it to provide functionality, where one application should 
depend on another. "Came first- served first" and similar things.

Another thing I remember, when designing a DLL with Delphi, it was often 
requiered to include a special "memory manager DLL", which handled 
strings and something. I'm not really sure how.

If loaded only once at a time, a static lib certainly takes less space 
because:
  - it only contains the functions which are really in use;
  - and besides, it doesn't even need to be loaded completely to run. 
"Real" OSs load executables by parts.

However, when an appliation gets started and closed very often, it makes 
sense to place much of its functionality into a DLL, since DLLs are 
retained in memory after all programs using them are closed, and only 
unloaded when OS decides. (e.g. when it's gotten short of memory.)

A comment of how good a linker eliminates unused functions is really 
irrelevant. Its job is simple: if a function is referenced somewhere in 
the code, it gets retained. If not, it's removed. BUT referenced does 
not necesarily mean called. An example of this are classes. Every class 
carries referencies to functions in its VTable, so that they cannot be 
removed, even if you never call them. Maybe a compiler could have some 
way to identify which functions are "really" used? I'm not very clear 
about that, but anyway the linker is not to be blamed. AFAIK, linker 
simply gets a list of "symbols" associated with "something", such as 
code and data, it starts inserting "something" into the executable. Now, 
whenever this "something" references a "symbol", another "something" 
associated with a symbol is placed into the executable, and the final 
adress of it lands onto the place of that original reference. This way, 
all unreferenced stuff gets ignored.

BTW, that's where some way to add functionality to already created 
object instances could be useful. For example, a usual GUI element such 
as a button may support tons of features, like it can accept leaves 
falling on it, track weather and somesuch. :) But usually it's not 
requiered, and would only bloat an executable. Wasn't there some 
discussion on mix-ins? Like extension on interfaces?

DLLs have always been and shall continue to be. They are a basis of not 
only w1ndow$, but also of any OS.

-i.
Feb 17 2003