www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Building several dynamic libraries with one shared GC

reply NonNull <non-null use.startmail.com> writes:
I am making a plug-in development system for a high performance 
Linux application that already exists and is written in C and 
will not be modified for this purpose. It is already has an API 
for plugins written in C. A plug-in simply makes new functions 
(e.g. manipulating text and numbers) available to be used inside 
the application. The D GC will intentionally be available inside 
a plugin.

I have written a D source file containing templates that do the 
work when a plugin is written by a third party in D. A plugin 
will expose a C API for the functions made available to the 
application, and any data in D made available to the application 
will be copied to the application's managed storage and vice 
versa, so there are no GC issues.

All of this is to make it as simple as possible for a third party 
to write such a plugin without worrying about any of these 
issues, and use D in a simplistic scripting language-like way. 
This is for non-experts. They just use the templates and build a 
dynamic library and all of the above is done for them.

If several plugins are built by different third parties, each 
dynamic library will have its own GC and copy of druntime right 
now. How can I organize that there is one separate dynamic 
library to share these among all plugins?
Sep 12 2021
next sibling parent reply frame <frame86 live.com> writes:
On Sunday, 12 September 2021 at 14:31:04 UTC, NonNull wrote:


 If several plugins are built by different third parties, each 
 dynamic library will have its own GC and copy of druntime right 
 now. How can I organize that there is one separate dynamic 
 library to share these among all plugins?


Shouldn't the runtime not already be shared on Linux? The 
`Runtime.loadLibrary` specs say
`If the library contains a D runtime it will be integrated with 
the current runtime.`

This should be true for the GC too. At least the memory is shared 
because as I remember I could access __gshared variables,

whereas on Windows nothing like this works and any DLL will spawn 
a new thread (for each thread you use too).
Sep 12 2021
parent NonNull <non-null use.startmail.com> writes:
On Sunday, 12 September 2021 at 16:23:13 UTC, frame wrote:

 Shouldn't the runtime not already be shared on Linux? The 
 `Runtime.loadLibrary` specs say
 `If the library contains a D runtime it will be integrated with 
 the current runtime.`

 This should be true for the GC too. At least the memory is 
 shared because as I remember I could access __gshared variables,

 whereas on Windows nothing like this works and any DLL will 
 spawn a new thread (for each thread you use too).


The plugin libraries expose a C API and are dynamically loaded by 
the application which is written in C, so presumably using 
dlopen. No D runtime there. Still your reply does suggest a way 
to proceed...
Sep 12 2021
prev sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 9/12/21 7:31 AM, NonNull wrote:

 I am making a plug-in development system for a high performance Linux
 application that already exists and is written in C and will not be
 modified for this purpose.


I've done something similar but in my case the main application is in D 
and the plugins are code-generated D (hand edited by non-D developers).

However, as is common in software, the requirements changed :) and we 
wanted to use it as a library as well; so, plugins became part of a main 
D library (which exposed C functions).

All initialization functions of the plugins were called automatically in 
my D test environment and all plugins were usable. The trouble started 
when the main library was being used in a foreign environment (something 
like Python loading Python loading C++ library loading our D library): 
Although the initialization function of the main library was being 
called, the 'shared static this' functions of the plugins were not being 
called.

(I tried dlopen after guessing intelligently the name of the 'shared 
static this' function (not obvious); it was not satisfactory and I don't 
remember exactly why not.)

Later I learned, this could be because merely loading a plugin might not 
be enough, and perhaps the main library might have to use a feature of 
the library as well:

   https://forum.dlang.org/post/sdb5jb$2rk3$1 digitalmars.com

If that link indeed explains the issue, I did not know about it when we 
worked around the initialization problem by running a D daemon per 
thread behind this main library.

Each thread of the library passes information to its daemon through 
shared memory and the daeman does it's thing and returns the result 
back. (The difference is, the plugins are linked to the daemon; avoiding 
the problem initialization scenario; 'shared static this' are called 
from a D environment.)


 If several plugins are built by different third parties, each dynamic
 library will have its own GC and copy of druntime right now.


Like the user 'frame', I don't think that's the case.

Going off topic, I started thinking about this "one daemon per library 
thread" idea: becaues I am under the impression that running multiple 
daemons does not put much stress on the system under Linux, this idea 
can automatically translate to "independent GCs for each thread". For 
example, the main library has 10 threads using 10 separate daemons on 
their backgrounds; each daemon handles its own GC needs without even 
knowing about the other daemons that are effectively working for the 
same main library. I haven't experimented with this yet.

Ali
Sep 12 2021
next sibling parent NonNull <non-null use.startmail.com> writes:
On Sunday, 12 September 2021 at 18:56:50 UTC, Ali Çehreli wrote:

 All initialization functions of the plugins were called 
 automatically in my D test environment and all plugins were 
 usable. The trouble started when the main library was being 
 used in a foreign environment (something like Python loading 
 Python loading C++ library loading our D library): Although the 
 initialization function of the main library was being called, 
 the 'shared static this' functions of the plugins were not 
 being called.

 (I tried dlopen after guessing intelligently the name of the 
 'shared static this' function (not obvious); it was not 
 satisfactory and I don't remember exactly why not.)

 Later I learned, this could be because merely loading a plugin 
 might not be enough, and perhaps the main library might have to 
 use a feature of the library as well:

   https://forum.dlang.org/post/sdb5jb$2rk3$1 digitalmars.com


[...]

 If several plugins are built by different third parties, each


 dynamic

 library will have its own GC and copy of druntime right now.


 Like the user 'frame', I don't think that's the case.



I hope you're right about this last. Not sure how static versus 
dynamic linking affects it. ldc2 seems to default to dynamic 
linking for phobos and have druntime in a dynamic library too. 
[Not clear what DMD does with druntime when it dynamically links 
to phobos.] In that ideal situation you seem to be right. Not 
sure how a plugin can be distributed as an executable only (to 
those without a D compiler installed) without static linking and 
then what? I have a mess to sort out.
Any info or suggestions?
Sep 12 2021
prev sibling parent reply NonNull <non-null use.startmail.com> writes:
On Sunday, 12 September 2021 at 18:56:50 UTC, Ali Çehreli wrote:

 All initialization functions of the plugins were called 
 automatically in my D test environment and all plugins were 
 usable. The trouble started when the main library was being 
 used in a foreign environment (something like Python loading 
 Python loading C++ library loading our D library): Although the 
 initialization function of the main library was being called, 
 the 'shared static this' functions of the plugins were not 
 being called.


So here, your main dynamic library in turn dynamically loads 
plugins. Did you try simply calling a function exported by a 
plugin from the static constructor in the main library after it 
had made the call to initialize druntime to see if that 
stimulated running the plugin's static constructors first? The 
problem you linked to suggests that might do the job. I haven't 
run into this problem yet myself. But I'm interested.
Sep 12 2021
parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 9/12/21 1:25 PM, NonNull wrote:

 On Sunday, 12 September 2021 at 18:56:50 UTC, Ali =C3=87ehreli wrote:

 All initialization functions of the plugins were called automatically =





 in my D test environment and all plugins were usable. The trouble=20
 started when the main library was being used in a foreign environment =





 (something like Python loading Python loading C++ library loading our =





 D library): Although the initialization function of the main library=20
 was being called, the 'shared static this' functions of the plugins=20
 were not being called.


=20
 So here, your main dynamic library in turn dynamically loads plugins.=20
 Did you try simply calling a function exported by a plugin from the=20
 static constructor in the main library after it had made the call to=20
 initialize druntime to see if that stimulated running the plugin's=20
 static constructors first? The problem you linked to suggests that migh=


t=20

 do the job. I haven't run into this problem yet myself. But I'm interes=


ted.

I did not try it because I learned about that potential cause after we=20
worked around the issue with the daemons.

Ali
Sep 12 2021