www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Plugins and D programs

reply "Steve Teale" <steve.teale britseyeview.com> writes:
One of the primary uses of dynamic loading of libraries might 
well be to provide plugins. By plugins I mean extensions to an 
existing program that can be added to the program at run-time, 
and can be written by separate authors who don't necessarily have 
access to the source code of the program, but who do understand 
the rules provided by the program documentation as to what 
capabilities the plugin must have.

The most obvious way to allow a D program to cope with plugins, 
is to specify an interface to which the plugin is expected to 
conform. Communication between the program and the plugin, after 
the latter is loaded would then be restricted to calls provided 
by that interface.

One further library method would likely be necessary to allow for 
the acquisition of an instance of the plugin, and its attachment 
to the program.

So, we could have

module ifd;
interface I
{
    string saySomething();
}

The plugin could then be:

module plugin;
import ifd;
import std.stdio;

class Plugin: I
{
    this() { writeln("plugin ctor"); }

    string saySomething() { return "I am plugin"; }
}

I getInstance()
{
    return new Plugin();
}

And our program could be:

module main;
import core.runtime;
import std.stdio;
import ifd;

extern(C) void* dlsym(void*, const char*);
extern(C) void* dlopen(const char*, int);

alias I function() pfi;

I getPlugin(string name)
{
    // Take your pick from these two - makes no odds
    //void* lib = dlopen("plugin.so\0".ptr, 1);
    void* lib = Runtime.loadLibrary(name~".so");

    if (lib is null)
    {
       writeln("failed to load plugin shared object");
       return null;
    }

    void* vp = dlsym(lib, "_D6plugin11getInstanceFZC3ifd1I\0".ptr);
    if (vp is null)
    {
       writeln("plugin creator function not found");
       return null;
    }
    pfi f = cast(pfi) vp;
    I x = f();
	if (x is null)
	{
		writeln("creation of plugin failed");
		return null;
	}
	return x;
}

void main()
{
    I x = getPlugin("plugin");
    writeln(x.saySomething());
}

Unfortunately, the result of running the program is:

steve steve-desktop:~/scratch/piif$ ./main
plugin ctor

Segmentation fault (core dumped)

Which suggests that the library was loaded, the symbol found, and 
an instance of plugin created.

The two pieces were built using dmd2.064 with:

main : ifd.d main.d
	dmd -c ifd.d
	dmd -c main.d
	dmd main.o ifd.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.

plugin : plugin.d
	dmd -c -shared -fPIC plugin.d
	dmd plugin.o -shared -defaultlib=libphobos2.so -map

clean :
	rm *.o
	rm main
	rm plugin.so

Does anyone have any suggestions as to what might be going wrong 
here? I have further examples, but I guess I should do them one 
at a time.

Steve
Mar 13 2014
next sibling parent "Tolga Cakiroglu" <tcak pcak.com> writes:
Hi Steve. Well, I had so many experience with this and after many 
tries, I gave up unfortunately. It continuous giving Segmentation 
error.

What I have found is that while the main programme is closing, it 
tries to call "rt_finalize" to remove that object from memory, 
then it says that method is not defined. I complied programme in 
Debug version and used GDB to understand above situation.

(Linux x64, DMD 2.065)

On Thursday, 13 March 2014 at 13:22:55 UTC, Steve Teale wrote:
 One of the primary uses of dynamic loading of libraries might 
 well be to provide plugins. By plugins I mean extensions to an 
 existing program that can be added to the program at run-time, 
 and can be written by separate authors who don't necessarily 
 have access to the source code of the program, but who do 
 understand the rules provided by the program documentation as 
 to what capabilities the plugin must have.

 The most obvious way to allow a D program to cope with plugins, 
 is to specify an interface to which the plugin is expected to 
 conform. Communication between the program and the plugin, 
 after the latter is loaded would then be restricted to calls 
 provided by that interface.

 One further library method would likely be necessary to allow 
 for the acquisition of an instance of the plugin, and its 
 attachment to the program.

 So, we could have

 module ifd;
 interface I
 {
    string saySomething();
 }

 The plugin could then be:

 module plugin;
 import ifd;
 import std.stdio;

 class Plugin: I
 {
    this() { writeln("plugin ctor"); }

    string saySomething() { return "I am plugin"; }
 }

 I getInstance()
 {
    return new Plugin();
 }

 And our program could be:

 module main;
 import core.runtime;
 import std.stdio;
 import ifd;

 extern(C) void* dlsym(void*, const char*);
 extern(C) void* dlopen(const char*, int);

 alias I function() pfi;

 I getPlugin(string name)
 {
    // Take your pick from these two - makes no odds
    //void* lib = dlopen("plugin.so\0".ptr, 1);
    void* lib = Runtime.loadLibrary(name~".so");

    if (lib is null)
    {
       writeln("failed to load plugin shared object");
       return null;
    }

    void* vp = dlsym(lib, 
 "_D6plugin11getInstanceFZC3ifd1I\0".ptr);
    if (vp is null)
    {
       writeln("plugin creator function not found");
       return null;
    }
    pfi f = cast(pfi) vp;
    I x = f();
 	if (x is null)
 	{
 		writeln("creation of plugin failed");
 		return null;
 	}
 	return x;
 }

 void main()
 {
    I x = getPlugin("plugin");
    writeln(x.saySomething());
 }

 Unfortunately, the result of running the program is:

 steve steve-desktop:~/scratch/piif$ ./main
 plugin ctor

 Segmentation fault (core dumped)

 Which suggests that the library was loaded, the symbol found, 
 and an instance of plugin created.

 The two pieces were built using dmd2.064 with:

 main : ifd.d main.d
 	dmd -c ifd.d
 	dmd -c main.d
 	dmd main.o ifd.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.

 plugin : plugin.d
 	dmd -c -shared -fPIC plugin.d
 	dmd plugin.o -shared -defaultlib=libphobos2.so -map

 clean :
 	rm *.o
 	rm main
 	rm plugin.so

 Does anyone have any suggestions as to what might be going 
 wrong here? I have further examples, but I guess I should do 
 them one at a time.

 Steve

Mar 13 2014
prev sibling next sibling parent "Tolga Cakiroglu" <tcak pcak.com> writes:
 	dmd main.o ifd.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.


Hmm. I saw that you are using libphobos2.so. I wasn't using this.
Mar 13 2014
prev sibling next sibling parent reply "Sean Kelly" <sean invisibleduck.org> writes:
On Thursday, 13 March 2014 at 13:22:55 UTC, Steve Teale wrote:
 One of the primary uses of dynamic loading of libraries might 
 well be to provide plugins. By plugins I mean extensions to an 
 existing program that can be added to the program at run-time, 
 and can be written by separate authors who don't necessarily 
 have access to the source code of the program, but who do 
 understand the rules provided by the program documentation as 
 to what capabilities the plugin must have.

It might be purely a matter of historic interest at this point, but DDL was basically doing plugins in D ages ago: http://www.dsource.org/projects/ddl I /think/ this was the dynamic loading scheme h3r3tic used in his game engine back in the day. I'm sure someone with a better memory than me could explain it better.
Mar 13 2014
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-03-14 00:44, Sean Kelly wrote:

 It might be purely a matter of historic interest at this point,
 but DDL was basically doing plugins in D ages ago:

 http://www.dsource.org/projects/ddl

Yes, but DDL uses a custom linker.
 I /think/ this was the dynamic loading scheme h3r3tic used in his
 game engine back in the day.  I'm sure someone with a better
 memory than me could explain it better.

Yes, exactly. It basically allowed him to update code on the fly, when the application was running. He showed this on the Tango conference. -- /Jacob Carlborg
Mar 14 2014
prev sibling parent Martin Nowak <code dawg.eu> writes:
On 03/14/2014 12:44 AM, Sean Kelly wrote:
 It might be purely a matter of historic interest at this point,
 but DDL was basically doing plugins in D ages ago:

 http://www.dsource.org/projects/ddl

 I /think/ this was the dynamic loading scheme h3r3tic used in his
 game engine back in the day.  I'm sure someone with a better
 memory than me could explain it better.

But this use object with non-PIC and does full relocation while loading. Thus it also requires writable, executable memory which isn't available on many systems due to security issues.
Mar 15 2014
prev sibling next sibling parent "Steve Teale" <steve.teale britseyeview.com> writes:
On Thursday, 13 March 2014 at 21:20:10 UTC, Tolga Cakiroglu wrote:
 	dmd main.o ifd.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.


Hmm. I saw that you are using libphobos2.so. I wasn't using this.

Yes, you have to make the program and the plugin both use the same runtime, presumably so they are using the same memory allocation system, so you link both with libphobos2.so. It is not just a shut-down artefact, since if I repeat the exercise with an exemplar class instead of an interface, then it will work. But then if I make the exemplar class an instance of an interface, and call one of the methods that is part of the interface it goes back to crashing, so it does seem that this behaviour is linked to interfaces. I will get stuck in with GDB and see if I can find out more, but that's not an agreeable experience, so I thought I would ask first. Steve
Mar 13 2014
prev sibling next sibling parent "Steve Teale" <steve.teale britseyeview.com> writes:
On Thursday, 13 March 2014 at 23:44:23 UTC, Sean Kelly wrote:
 It might be purely a matter of historic interest at this point,
 but DDL was basically doing plugins in D ages ago:

 http://www.dsource.org/projects/ddl

 I /think/ this was the dynamic loading scheme h3r3tic used in 
 his
 game engine back in the day.  I'm sure someone with a better
 memory than me could explain it better.

Sean, Yes, been there years ago - maybe 2006. I could never get the full edifice working, so I extracted the parts that would enable me to just load a single .o file, and that would work most of the time, but then sometimes it would just crash. Maybe that would work with today's runtime, but the machine that particular code was on got stolen, so I'd have to start from scratch. Steve
Mar 14 2014
prev sibling next sibling parent "Mathias LANG" <pro.mathias.lang gmail.com> writes:
On Thursday, 13 March 2014 at 13:22:55 UTC, Steve Teale wrote:
 Does anyone have any suggestions as to what might be going 
 wrong here? I have further examples, but I guess I should do 
 them one at a time.

 Steve

Copied your files, added: -L-rpath=/home/geod/bin/dmd-master/linux/lib64/ as dmd is not on the system path for me. It was working. So try to update your compiler and check if that's the reason.
Mar 14 2014
prev sibling next sibling parent "Steve Teale" <steve.teale britseyeview.com> writes:
On Friday, 14 March 2014 at 06:54:34 UTC, Steve Teale wrote:
 On Thursday, 13 March 2014 at 21:20:10 UTC, Tolga Cakiroglu

 It is not just a shut-down artefact, since if I repeat the 
 exercise with an exemplar class instead of an interface, then 
 it will work.

OK, so here's an example using an exemplar base class instead of an interface. This includes making the exemplar class implement an interface. Here's the exemplar class, and another class that references an instance of it: module bc; import std.stdio; interface I { string saySomething(); } class Bc: I { string saySomething() { return null; }; } class Other { I target; this(I i) { target = i; } void invokeWithCast() { writeln((cast(Bc) target).saySomething()); } void invoke() { writeln(target.saySomething()); } } Then the plugin: module plugin; import bc; import std.stdio; class Plugin: Bc { this() { writeln("plugin ctor"); } override string saySomething() { return "I am plugin"; } } Bc getInstance() { return new Plugin(); } And the program: module main; import core.runtime; import std.stdio; import bc; extern(C) void* dlsym(void*, const char*); alias Bc function() pfi; Bc getPlugin(string name) { void* lib = Runtime.loadLibrary(name~".so"); void* vp = dlsym(lib, "_D6plugin11getInstanceFZC2bc2Bc\0".ptr); pfi f = cast(pfi) vp; Bc x = f(); return x; } void main() { Bc x = getPlugin("plugin"); writeln(x.saySomething()); // OK Other other = new Other(x); other.invokeWithCast(); // OK other.invoke(); // Crash } Built with: main : bc.d main.d dmd -c bc.d dmd -c main.d dmd main.o bc.o -L-L/usr/local/lib -L-ldl -L-lgtkd-2 -defaultlib=libphobos2.so -L-rpath=. plugin : plugin.d dmd -c -shared -fPIC plugin.d dmd plugin.o -shared -defaultlib=libphobos2.so -map clean : rm *.o rm main rm plugin.so Output: steve steve-desktop:~/scratch/pibc$ ./main plugin ctor I am plugin I am plugin Segmentation fault (core dumped) If you comment out the last call via the interface instance with no cast, the program exits cleanly. So the problem does seem to relate to interfaces. So you can do it, but if the base class conforms to some interface, then you have to special-case calls to the interface functions. That rather destroys the utility of classes like Other which provide services to the installed plugin. My gut-feeling question is "is the vtable in the program or the plugin, and does this vary when the plugin implements an interface?" I'm hoping Martin Nowak might join in this discussion. Steve
Mar 14 2014
prev sibling next sibling parent "Mathias LANG" <pro.mathias.lang gmail.com> writes:
On Friday, 14 March 2014 at 07:46:25 UTC, Steve Teale wrote:
 My gut-feeling question is "is the vtable in the program or the 
 plugin, and does this vary when the plugin implements an 
 interface?"

 I'm hoping Martin Nowak might join in this discussion.

 Steve

geod Barsoom:~/Work/BoapNet/Client/Native/SharedTemplate/test$ make plugin /home/geod/bin/dmd-2.064.2/linux/bin64/dmd -c -shared -fPIC plugin.d /home/geod/bin/dmd-2.064.2/linux/bin64/dmd plugin.o -shared -defaultlib=libphobos2.so -map geod Barsoom:~/Work/BoapNet/Client/Native/SharedTemplate/test$ make /home/geod/bin/dmd-2.064.2/linux/bin64/dmd -c bc.d /home/geod/bin/dmd-2.064.2/linux/bin64/dmd -c main.d /home/geod/bin/dmd-2.064.2/linux/bin64/dmd main.o bc.o -L-L/usr/local/lib -L-ldl -defaultlib=libphobos2.so -L-rpath=. -L-rpath=/home/geod/bin/dmd-2.064.2/linux/lib64/ geod Barsoom:~/Work/BoapNet/Client/Native/SharedTemplate/test$ ./main plugin ctor I am plugin I am plugin I am plugin Same result with master branch. Long story short: I can't reproduce it on my machine (Debian x86_64). Which platform are you working on ? 32 bits ?
Mar 14 2014
prev sibling next sibling parent "Steve Teale" <steve.teale britseyeview.com> writes:
On Friday, 14 March 2014 at 08:15:00 UTC, Mathias LANG wrote:
 ./main
 plugin ctor
 I am plugin
 I am plugin
 I am plugin


 Same result with master branch. Long story short: I can't 
 reproduce it on my machine (Debian x86_64). Which platform are 
 you working on ? 32 bits ?

Ubuntu 12.04 32 bit. I'll reboot into 64 bit and try there.
Mar 14 2014
prev sibling next sibling parent "Steve Teale" <steve.teale britseyeview.com> writes:
On Friday, 14 March 2014 at 08:56:25 UTC, Steve Teale wrote:
 On Friday, 14 March 2014 at 08:15:00 UTC, Mathias LANG wrote:
 ./main
 plugin ctor
 I am plugin
 I am plugin
 I am plugin


 Same result with master branch. Long story short: I can't 
 reproduce it on my machine (Debian x86_64). Which platform are 
 you working on ? 32 bits ?

Ubuntu 12.04 32 bit. I'll reboot into 64 bit and try there.

Sure enough, works OK on 12.04 64 bit, so I guess it's a bug. Probably not core.runtime, because I think the problem still occurs if I use dlopen() directly, so code generation? I'll work up a minimal test case just using an interface, then submit it, yes? Steve
Mar 14 2014
prev sibling next sibling parent "Steve Teale" <steve.teale britseyeview.com> writes:
On Friday, 14 March 2014 at 09:08:17 UTC, Steve Teale wrote:
 I'll work up a minimal test case just using an interface, then 
 submit it, yes?

that does not mention the fact that the problem is 32 bit specific. I updated it. Will install 2.065 and test with that.
Mar 14 2014
prev sibling next sibling parent "Steve Teale" <steve.teale britseyeview.com> writes:
On Friday, 14 March 2014 at 07:22:06 UTC, Mathias LANG wrote:
 On Thursday, 13 March 2014 at 13:22:55 UTC, Steve Teale wrote:
 Does anyone have any suggestions as to what might be going 
 wrong here? I have further examples, but I guess I should do 
 them one at a time.

 Steve

Copied your files, added: -L-rpath=/home/geod/bin/dmd-master/linux/lib64/ as dmd is not on the system path for me. It was working. So try to update your compiler and check if that's the reason.

Everything works OK with 64 bit anyway!
Mar 14 2014
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On 03/13/2014 02:22 PM, Steve Teale wrote:
 main : ifd.d main.d
      dmd -c ifd.d
      dmd -c main.d
      dmd main.o ifd.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.

 plugin : plugin.d
      dmd -c -shared -fPIC plugin.d
      dmd plugin.o -shared -defaultlib=libphobos2.so -map

 clean :
      rm *.o
      rm main
      rm plugin.so

One problem here is that main and the plugin share the definitions in ifd.d. Therefor you need to move the interface into it's own shared library and link both (main and any plugin) against the interface definition. main : main.d ifd dmd -c main.d dmd main.o -L-ldl -defaultlib=libphobos2.so -L-rpath=. -L./ifd.so plugin : plugin.d ifd dmd -c -shared -fPIC plugin.d dmd plugin.o -shared -defaultlib=libphobos2.so -L./ifd.so ifd : ifd.d dmd -c -shared -fPIC ifd.d dmd ifd.o -shared -defaultlib=libphobos2.so clean : rm *.o rm main ifd.so plugin.so Note that it was possible to load plugin.so even though it has undefined symbols because we link with -L--export-symbols (for backtrace symbols). This linker flag causes the main executable to export it's symbols, so the plugin will resolve it's undefined symbols by using the ones from main. I'm still investigating why it crashes when using --export-dynamic though.
Mar 15 2014
parent reply Martin Nowak <code dawg.eu> writes:
On 03/15/2014 09:50 PM, Martin Nowak wrote:
 I'm still investigating why it crashes when using --export-dynamic though.

Looks like a revenant of https://d.puremagic.com/issues/show_bug.cgi?id=9729. The problem is that nobody sets ebx before calling the interface thunk, therefor resolving the PLT entry fails. _TMP5 LABEL NEAR sub eax, 8 ; 0078 _ 83. E8, 08 jmp _D6plugin6Plugin12saySomethingMFZAya ; 007B _ E9, FFFFFFFC(PLT r)
Mar 15 2014
parent Martin Nowak <code dawg.eu> writes:
On 03/15/2014 11:04 PM, Martin Nowak wrote:
 On 03/15/2014 09:50 PM, Martin Nowak wrote:
 I'm still investigating why it crashes when using --export-dynamic
 though.

Looks like a revenant of https://d.puremagic.com/issues/show_bug.cgi?id=9729. The problem is that nobody sets ebx before calling the interface thunk, therefor resolving the PLT entry fails. _TMP5 LABEL NEAR sub eax, 8 ; 0078 _ 83. E8, 08 jmp _D6plugin6Plugin12saySomethingMFZAya ; 007B _ E9, FFFFFFFC(PLT r)

I prepared a fix here (https://github.com/MartinNowak/dmd/tree/thunkEBX) but it needs more work. Currently it violates the platform ABI because the interface thunk destroys EBX.
Mar 15 2014
prev sibling next sibling parent "Steve Teale" <steve.teale britseyeview.com> writes:
On Saturday, 15 March 2014 at 22:56:19 UTC, Martin Nowak wrote:
 On 03/15/2014 11:04 PM, Martin Nowak wrote:
 On 03/15/2014 09:50 PM, Martin Nowak wrote:
 I'm still investigating why it crashes when using 
 --export-dynamic
 though.

Looks like a revenant of https://d.puremagic.com/issues/show_bug.cgi?id=9729. The problem is that nobody sets ebx before calling the interface thunk, therefor resolving the PLT entry fails. _TMP5 LABEL NEAR sub eax, 8 ; 0078 _ 83. E8, 08 jmp _D6plugin6Plugin12saySomethingMFZAya ; 007B _ E9, FFFFFFFC(PLT r)

I prepared a fix here (https://github.com/MartinNowak/dmd/tree/thunkEBX) but it needs more work. Currently it violates the platform ABI because the interface thunk destroys EBX.

Thanks Martin - it's good to have you involved. I'd got as far as dissasembling it and stepping single instructions in GDB while looking at that, so I could have told you where it crashed, but had no idea what was going on
Mar 16 2014
prev sibling parent "Steve Teale" <steve.teale britseyeview.com> writes:
On Saturday, 15 March 2014 at 22:56:19 UTC, Martin Nowak wrote:
 On 03/15/2014 11:04 PM, Martin Nowak wrote:
 On 03/15/2014 09:50 PM, Martin Nowak wrote:
 I'm still investigating why it crashes when using 
 --export-dynamic
 though.



10719 as fixed ;=)
Mar 16 2014