www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Emulating DLL

reply Craig <Craig Tonsils.com> writes:
Is it possible to create a D module that has functions in it, and 
then use those functions dynamically at run time emulating DLL 
like functionality?

e.g.,

module MyDL;

import Context;
export void foo(Context)
{
    ...
}

and

module UseDL;
import DL;
import Context;

void main()
{
     DL.Load("MyDL.dl");
     DL.foo(new Context);
     DL.Unload();

     // or
     DL.Loadable("MyDL.dl"); // monitors changes and reloads the 
function safely  when a change is detected. This might require 
some work to make safe.
     DL.foo(new Context);
     DL.Unload();
}


(pseudo)

The idea here is to simply be able to call external code that can 
be hotswapped and such. It requires dealing with relocatable 
code, GC, etc.


It would be portable and non-windows specific and just work.

dmd and ldc of course do not output anything that is reasonable 
as they use the traditional non-portable formats. This leads me 
to think that without modifying them, I'd have to extract the 
code from the DLL/SO and deal with all the problems and then 
generate the .dl file.

The idea is to create simple way to "script" D programs and use 
the D compiler to compile the functionality. It won't be as fast 
as an interpreter but it will have full speed.

I'm just not sure what would be the best way to generate the code 
and do the fixing up.

I'd be willing to hack on the dmd compiler to try and get it to 
generate a more sane file for this process if someone could point 
me to some details.
Mar 18
next sibling parent reply Ethan Watson <gooberman gmail.com> writes:
On Monday, 18 March 2019 at 22:50:57 UTC, Craig wrote:
 Is it possible to create a D module that has functions in it, 
 and then use those functions dynamically at run time emulating 
 DLL like functionality?
On Monday, 18 March 2019 at 22:50:57 UTC, Craig wrote:
 Is it possible to create a D module that has functions in it, 
 and then use those functions dynamically at run time emulating 
 DLL like functionality?
I've talked extensively on this topic at DConf over the last few years. http://dconf.org/2016/talks/watson.html http://dconf.org/2017/talks/watson.html http://dconf.org/2018/talks/watson.html https://github.com/GooberMan/binderoo It's not a simple thing by any means, there's plenty of little implementation details that you'll need to be aware of. But yes, it's quite possible.
Mar 18
parent reply Craig <Craig Tonsils.com> writes:
On Monday, 18 March 2019 at 22:59:12 UTC, Ethan Watson wrote:
 On Monday, 18 March 2019 at 22:50:57 UTC, Craig wrote:
 Is it possible to create a D module that has functions in it, 
 and then use those functions dynamically at run time emulating 
 DLL like functionality?
On Monday, 18 March 2019 at 22:50:57 UTC, Craig wrote:
 Is it possible to create a D module that has functions in it, 
 and then use those functions dynamically at run time emulating 
 DLL like functionality?
I've talked extensively on this topic at DConf over the last few years. http://dconf.org/2016/talks/watson.html http://dconf.org/2017/talks/watson.html http://dconf.org/2018/talks/watson.html https://github.com/GooberMan/binderoo It's not a simple thing by any means, there's plenty of little implementation details that you'll need to be aware of. But yes, it's quite possible.
This seems overly complex for what I want and not exactly what I need. I'm looking for simply that simply lets me call functions that exist in an external dynamically accessible file. In my host code I will use a function like "foo" but that foo will be a function pointer that can point to the external code. I don't need a bunch of bells and whistles or complex ways to accomplish this. 1. Load up external code, 2. Fixup whatever needs fixing up to make the code work properly 3. Point function pointer to code. 4. Call. For simple functions such as pure functions step 2 can be skipped and hence it is essentially just loading the code in to memory and calling it. I do get that in general it is more complex but other than relocatable code, initialization, and the GC, it shouldn't require much more work. For example, with windows I could simply compile to a dll then extract the code, or just use LoadLibrary and it effectively does all the work(steps 1 & 2 & 3). LoadLibrary is not portable though and seems excessive to do something that should be rather simple unless there is something I'm missing that has to be done that is complex. Relocation is easy if one has the appropriate locations to fix up which are generally included in the binary somewhere. The GC and rt can be initialized easily.
Mar 19
next sibling parent reply Ethan <gooberman gmail.com> writes:
On Tuesday, 19 March 2019 at 19:50:15 UTC, Craig wrote:
 For example, with windows I could simply compile to a dll then 
 extract the code, or just use LoadLibrary and it effectively 
 does all the work(steps 1 & 2 & 3).
LoadLibrary performs steps 1 and part of step 2. The DllMain function of a library is called separately by Windows for each thread in your system, which is where the bulk of step 2 is handled. Step 3 isn't handled at all by LoadLibrary, and is instead entirely up to you to deal with via the GetProcAddress function. If you want DLLs to operate in that step 1-2-3 manner, then the compiler can generate a static library that handles that all for you. But, as you might expect, that removes the hot reloading capability as that is all handled before WinMain is entered. If you expect hot reloading to work without effort, you're going to have to compile your DLLs with -betterC. The D Runtime is not yet in a standalone DLL (unless I've missed an announcement over the last couple of months). Each DLL you build will require the D runtime to be embedded. Things get really tricky from there. If you ever use your DLL for more than calling functions with basic types, or value aggregates (structs) containing basic types, then be aware that without -betterC that there is potential for the moduleinfo and typeinfo systems to go a bit screwy. Especially once druntime lives in its own DLL, as druntime expects those systems to be initialised once at startup. POSIX libraries will have the same issues, but I have nod had a need to investigate further over the last few months.
 LoadLibrary is not portable though and seems excessive to do 
 something that should be rather simple unless there is 
 something I'm missing that has to be done that is complex.
LoadLibrary is the equivalent to dlopen on POSIX systems, with one very important difference: Windows does not provide lazy symbol evaluation. As such, extra work is required to handle that. The average usecase is covered by the mentioned static lib, but any hot reloading is handled in a custom manner by every Windows codebase you'll encounter. The static lib I mention, in fact, redirects every cross-DLL call to a jump table. And cross-DLL data sharing gets hairy in Windows. Function parameters aren't a problem, but Windows explicitly disallows thread local variables to cross the DLL boundary for example (even if you expose it for export and perform a symbol lookup, it won't work and you'll get garbage back). As I said, it's not a simple thing by any means. My work has attempted to hide all that complexity by boiling it all down to "include these headers/modules, link this lib, and get on with life". If you find the resulting code too complex (and I freely admit that I haven't got the integration in to as foolproof a manner as I am aiming for) then very definitely read through the presentations to get a clear idea of why something that should be so simple really isn't.
Mar 20
parent reply Craig <Craig Tonsils.com> writes:
On Wednesday, 20 March 2019 at 14:27:54 UTC, Ethan wrote:
 On Tuesday, 19 March 2019 at 19:50:15 UTC, Craig wrote:
 For example, with windows I could simply compile to a dll then 
 extract the code, or just use LoadLibrary and it effectively 
 does all the work(steps 1 & 2 & 3).
LoadLibrary performs steps 1 and part of step 2. The DllMain function of a library is called separately by Windows for each thread in your system, which is where the bulk of step 2 is handled. Step 3 isn't handled at all by LoadLibrary, and is instead entirely up to you to deal with via the GetProcAddress function. If you want DLLs to operate in that step 1-2-3 manner, then the compiler can generate a static library that handles that all for you. But, as you might expect, that removes the hot reloading capability as that is all handled before WinMain is entered. If you expect hot reloading to work without effort, you're going to have to compile your DLLs with -betterC. The D Runtime is not yet in a standalone DLL (unless I've missed an announcement over the last couple of months). Each DLL you build will require the D runtime to be embedded. Things get really tricky from there. If you ever use your DLL for more than calling functions with basic types, or value aggregates (structs) containing basic types, then be aware that without -betterC that there is potential for the moduleinfo and typeinfo systems to go a bit screwy. Especially once druntime lives in its own DLL, as druntime expects those systems to be initialised once at startup. POSIX libraries will have the same issues, but I have nod had a need to investigate further over the last few months.
 LoadLibrary is not portable though and seems excessive to do 
 something that should be rather simple unless there is 
 something I'm missing that has to be done that is complex.
LoadLibrary is the equivalent to dlopen on POSIX systems, with one very important difference: Windows does not provide lazy symbol evaluation. As such, extra work is required to handle that. The average usecase is covered by the mentioned static lib, but any hot reloading is handled in a custom manner by every Windows codebase you'll encounter. The static lib I mention, in fact, redirects every cross-DLL call to a jump table. And cross-DLL data sharing gets hairy in Windows. Function parameters aren't a problem, but Windows explicitly disallows thread local variables to cross the DLL boundary for example (even if you expose it for export and perform a symbol lookup, it won't work and you'll get garbage back). As I said, it's not a simple thing by any means. My work has attempted to hide all that complexity by boiling it all down to "include these headers/modules, link this lib, and get on with life". If you find the resulting code too complex (and I freely admit that I haven't got the integration in to as foolproof a manner as I am aiming for) then very definitely read through the presentations to get a clear idea of why something that should be so simple really isn't.
You are making this more complicated because you don't understand that I don't need a general purpose solution. I simply need to call simple functions that do not do much, not full blown applications. For example, Suppose I want to write a fractal display application and instead of hard coding the function to display, it's external. e.g., import std.complex; alias C = Complex!double; C Mandelbrot(C z, C c) { return z*z + c; } This function does not require much of anything to run. The machine code can be copied and pasted at will to any executable memory location and ran... it can, in fact, be inlined with a little caution(reserve enough space and ignore the stack code). No initialization needs to be nor garbage collecting. For a little more robustness one would want to allow for GC, which should not be hard, and context passing so more useful functionality can be had. If D had a Dstring2Machine that compiled the above code to machine code then it could be used directly without issue. Because these functions are deterministic and the host determines their usage, there is little issues to worry about. Hot swapping is simply changing a pointer when the function is not being called. As long as certain guidelines are followed one can do this quite simply without too much issue and it allows for a scripting like solution for D apps. It allows one to inject functionality in to an application that runs at full speed and is relatively fast to compile. The idea is not to be able to connect two full blow external apps but to connect little pieces of functionality to an app that allow for vast flexibility. Since each unit of functionality is essentially pure, there are no major issues. It's this simple: 1. Take source and compile it in to machine code. 2. App extracts machine code and inserts it in to it's memory space to be able to execute. Do you agree that, at least in some cases, this is entirely feasible and easy? I'll give you a hint: 1. module simple; void foo() { }; compile directly in to machine code(the most raw form possible containing foo). 2. App load the machine code, execute, reload and execute if desired... Since foo does nothing it's just a nop. We are just copying around a ret. It's impossible for anything to ever go awry here excluding buggy code/compiler/etc. From here one can expand the model to include something that is more useful. The idea is to expand it enough to make it generally quite useful but not so much one has to include the kitchen sink. I don't need a heavy version of binderoo, I need a light one. Something that lets me do basic "scripting" so I can modify the application behavior at runtime without recompiling the application. If you think the example is stupid and useless then int age() { return 43; } and nothing changes. of course you will then think that example is stupid because we could just put it in a text file and read that rather than go through all this trouble. so then C calc(C z, C c) { return z*z*z*z*z*z + c*z + c; } and to get around this without leveraging the D compiler one has to make a full blown parser... even though it's still just relatively basic code like age. It just includes some calculations. It's no different from foo though. Just basic code. But we can go further by allowing more information to be passed to and from the function. The function can also be impure such as void foo() { writeln("I'm IMPURE!"); } which will work without much work(might require fixing up a few symbols/pointers). All it is, is delayed compilation. I'm simply wanting to delay the compilation of a piece of code in an app from the main compilation. All the other stuff about the OS and TLS are irrelevant in this context. One could sort of do this with the object files and just recompile the functions and relink everything. But with a little work this step can be avoided since it's not needed and slower(but more robust).
Mar 20
parent Ethan Watson <gooberman gmail.com> writes:
On Wednesday, 20 March 2019 at 19:44:52 UTC, Craig wrote:
 Do you agree that, at least in some cases, this is entirely 
 feasible and easy?
I literally answered in the affirmative in my first post. Note that all that Binderoo does is put function pointers in places that looks like real code, and does this automatically for DLL recompiles. Everything you just described in your above post overcomplicates it from there. Go wild. But you're not going to get far if you choose to expand it past your initial examples (as you've suggested you might do) if you don't take the above implementation details in to account. Like I literally said in my first post.
Mar 20
prev sibling parent reply SrMordred <patric.dexheimer gmail.com> writes:
On Tuesday, 19 March 2019 at 19:50:15 UTC, Craig wrote:

Take a look at my lib, its a simple hot-reload external dll lib:

https://github.com/SrMordred/reloaded

I did´nt use it extensively, but its simple enough to be tweaked 
at your own need :)
Mar 20
parent Craig <Craig Tonsils.com> writes:
On Thursday, 21 March 2019 at 02:27:46 UTC, SrMordred wrote:
 On Tuesday, 19 March 2019 at 19:50:15 UTC, Craig wrote:

 Take a look at my lib, its a simple hot-reload external dll lib:

 https://github.com/SrMordred/reloaded

 I did´nt use it extensively, but its simple enough to be 
 tweaked at your own need :)
Thanks, Looks much more like what I need. I'll have to mess around with it to see how it works though.
Mar 25
prev sibling parent reply Kagamin <spam here.lot> writes:
There was this old project: http://dsource.org/projects/ddl/
Mar 19
next sibling parent Craig <Craig Tonsils.com> writes:
On Tuesday, 19 March 2019 at 09:18:56 UTC, Kagamin wrote:
 There was this old project: http://dsource.org/projects/ddl/
Thanks, looks pretty similar to what I was thinking. I'll have to mess with it a bit but maybe I can tweak any issues to suit my agenda.
Mar 19
prev sibling parent Craig <Craig Tonsils.com> writes:
On Tuesday, 19 March 2019 at 09:18:56 UTC, Kagamin wrote:
 There was this old project: http://dsource.org/projects/ddl/
Seems it is too heavily dependent on mago and has a lot of depreciated content. Probably not worth trying to update. Probably much easier just to hack dmd to export a simpler dll format or use dll's directly.
Mar 19