www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Enhancing DLL/SO behavior with D

reply pragma <pragma_member pathlink.com> writes:
To make an already long story short, I'm looking for a way to dynamically
resolve symbols between dll/so files and their host executable.  This way, one
could unify the TypeInfo tree at runtime (and more) to achive a sort of
runtime-binding similar to what Java does with .class files.*

With portability in mind, I went about researching how to perform a dll detour
(also called a trampoline) in linux using a Shared Object.  

The general idea, is to perform a fixup on the dll after its loaded into memory,
as to circumvent the original call hook for the library.  I know that ELF shared
objects can have external symbols resolved via the ELF loader itself (which
searches for matching symbols provided by in-memory tables); which is kind of
backwards, but may be workable.

With that in mind, it left me with the following questions:

- Is there an ELF, COFF or OMF loader workable under windows out there in F/OSS
land?
- Is there any way to generate an ELF shared object under windows, as an analog
to a dll file (possibly via a coff to elf converter)?
- ELF loading is grand, but isn't there a guide to hacking away at the shared
object after its been loaded into memory (like what we do in windows with dll
detours)?
- Would anyone like more detail or background on the problem and why custom
library loading is important?

(* Java .class files contain a symbol table of what external symbols are needed
to run the class, and what symbols are available to the calling program.  The
exhange of this information, or binding is performed by the classloader at
runtime when the .class file is loaded.  This is in stark contrast to D or C++
where type information is held statically in a dll/so or executable and is not
shared with anything at runtime.)

Thanks in advance,

- EricAnderton at yahoo
Aug 09 2005
next sibling parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
"pragma" <pragma_member pathlink.com> wrote in message 
news:ddbn9q$143n$1 digitaldaemon.com...
 To make an already long story short, I'm looking for a way to dynamically
 resolve symbols between dll/so files and their host executable.  This way, 
 one
 could unify the TypeInfo tree at runtime (and more) to achive a sort of
 runtime-binding similar to what Java does with .class files.*

I was playing around with similar goals with dflect (I think... I'm not exactly sure what your goals are but I'm guessing). The work-flow was as follows: run dflect on the classes/modules you wish to expose (this requires some special coding practices like marking ctors as "export") to generate MetaInfo code. Compile the MetaInfo classes together with your actual classes and make the dll. In the main app load the dll and call the exported function getMetaClassCtors() in the dll which returns an AA indexed by class name and with value the ctor for the MetaInfo class for the class in question. Once you construct the MetaClass you can call it's methods to construct instances and get fields and methods etc etc. The AA mapping class names to ctors is initialized in the module ctors so the main app needs to be able to run the module ctors before getting the AA of MetaInfo. Dflect also has files for interfacing with C/C++ - like the ability to get a global ref to an object so that it doesn't get garbage collected while some C code owns the only reference. Dflect is on the back-burner now, though. I was hoping someone would redo std.loader first.
 With portability in mind, I went about researching how to perform a dll 
 detour
 (also called a trampoline) in linux using a Shared Object.

 The general idea, is to perform a fixup on the dll after its loaded into 
 memory,
 as to circumvent the original call hook for the library.  I know that ELF 
 shared
 objects can have external symbols resolved via the ELF loader itself 
 (which
 searches for matching symbols provided by in-memory tables); which is kind 
 of
 backwards, but may be workable.

 With that in mind, it left me with the following questions:

 - Is there an ELF, COFF or OMF loader workable under windows out there in 
 F/OSS
 land?
 - Is there any way to generate an ELF shared object under windows, as an 
 analog
 to a dll file (possibly via a coff to elf converter)?
 - ELF loading is grand, but isn't there a guide to hacking away at the 
 shared
 object after its been loaded into memory (like what we do in windows with 
 dll
 detours)?
 - Would anyone like more detail or background on the problem and why 
 custom
 library loading is important?

 (* Java .class files contain a symbol table of what external symbols are 
 needed
 to run the class, and what symbols are available to the calling program. 
 The
 exhange of this information, or binding is performed by the classloader at
 runtime when the .class file is loaded.  This is in stark contrast to D or 
 C++
 where type information is held statically in a dll/so or executable and is 
 not
 shared with anything at runtime.)

I don't know anything about trampolines so I can't help but it sounds interesting. What exactly would the trampoline do? You mention unify TypeInfos - is this so that, for example, int's TypeInfo is the same address in all the dlls and main executable? Can this stuff be built into std.loader?
 Thanks in advance,

 - EricAnderton at yahoo 

Aug 09 2005
parent pragma <pragma_member pathlink.com> writes:
In article <ddbrve$1bkm$1 digitaldaemon.com>, Ben Hinkle says...
"pragma" <pragma_member pathlink.com> wrote in message 
news:ddbn9q$143n$1 digitaldaemon.com...
 To make an already long story short, I'm looking for a way to dynamically
 resolve symbols between dll/so files and their host executable.  This way, 
 one
 could unify the TypeInfo tree at runtime (and more) to achive a sort of
 runtime-binding similar to what Java does with .class files.*

I was playing around with similar goals with dflect (I think... I'm not exactly sure what your goals are but I'm guessing). The work-flow was as follows: run dflect on the classes/modules you wish to expose (this requires some special coding practices like marking ctors as "export") to generate MetaInfo code. Compile the MetaInfo classes together with your actual classes and make the dll. In the main app load the dll and call the exported function getMetaClassCtors() in the dll which returns an AA indexed by class name and with value the ctor for the MetaInfo class for the class in question. Once you construct the MetaClass you can call it's methods to construct instances and get fields and methods etc etc. The AA mapping class names to ctors is initialized in the module ctors so the main app needs to be able to run the module ctors before getting the AA of MetaInfo. Dflect also has files for interfacing with C/C++ - like the ability to get a global ref to an object so that it doesn't get garbage collected while some C code owns the only reference. Dflect is on the back-burner now, though. I was hoping someone would redo std.loader first.

Excellent. That sounds like a worthwhile goal. I've been very keen on deflect since you first introduced it, and I'll be eager to see where this goes.
 <snip my discussion about trampolines and ELF loaders>

interesting. What exactly would the trampoline do?

Basically the concept is to load the dll into memory, treat it as data, rewrite portions of it and then treat it as you normally would. A trampoline, also known as a "detour" is explained here: http://research.microsoft.com/~galenh/Publications/HuntUsenixNt99.pdf
You mention unify 
TypeInfos - is this so that, for example, int's TypeInfo is the same address 
in all the dlls and main executable? Can this stuff be built into 
std.loader?

You hit the nail right on the head. Although I'd like to see it extended to all TypeInfo classes, standard and custom. In a perfect world, yes, it'd be written directly into std.loader or something similar. Such a custom loader would take advantage of the concepts outlined by Detours/Trampolines, so that it may overwrite pointers in the in-memory-dll so that they point to the 'correct' values. The catch is: how do we coerce a dll to create such a table of all in-code references to TypeInfo structures? Its either that, or somehow rewrite DMD to output code that changes indirect references to TypeInfo (say in typeid() and cast() calls) to instead call "getTypeinfo(foo)", which would then use a lookup table. The table could be overwritten on load to the correct set of values. FYI, I'm curious about custom loading COFF or OMF directly, simply to bypass all the cruft that win32 places on dlls in general. It be nice since these formats (by definition) already have all of their external references exposed for linking; we'd just be doing it at run-time instead of at compile-time. As a side-effect, it would do more than helping the current TypeInfo problem as it would naturally extend to every kind of symbol. (AFAIK, this is like emulating ELF loading under win32) - EricAnderton at yahoo
Aug 10 2005
prev sibling parent reply "Carlos Smith" <c_____.s____ sympatico.ca> writes:
"pragma" <pragma_member pathlink.com> wrote in message

 To make an already long story short, I'm looking for a way to dynamically
 resolve symbols between dll/so files and their host executable.

http://edll.sourceforge.net/index.html
Aug 10 2005
parent reply pragma <pragma_member pathlink.com> writes:
In article <ddcpuo$2m16$1 digitaldaemon.com>, Carlos Smith says...
"pragma" <pragma_member pathlink.com> wrote in message

 To make an already long story short, I'm looking for a way to dynamically
 resolve symbols between dll/so files and their host executable.

http://edll.sourceforge.net/index.html

Carlos, you are my hero. That's one heck of a starting point, even if it depends on the GCC/MinGW toolchain. - EricAnderton at yahoo
Aug 10 2005
parent reply "Carlos Smith" <c_____.s____ sympatico.ca> writes:
"pragma" <pragma_member pathlink.com> wrote in message

 Carlos, you are my hero.  That's one heck of a starting point, even if it
 depends on the GCC/MinGW toolchain.

for( i=1.0 ; i > 0 ; i += 1.0 ) LOL(); The greatest things about Free/OpenSource software, are: It promote/encourage collaboration instead of competition, It's "Un pour Tous, Tous pour Un", instead of "We will beat them all", It's the realisation of "L'Union fait la Force", It's a great way to make friends, And, when Bill Gates will try this, humanity will enter a new world...
Aug 10 2005
parent pragma <pragma_member pathlink.com> writes:
In article <ddd5h5$453$1 digitaldaemon.com>, Carlos Smith says...
"pragma" <pragma_member pathlink.com> wrote in message

 Carlos, you are my hero.  That's one heck of a starting point, even if it
 depends on the GCC/MinGW toolchain.

for( i=1.0 ; i > 0 ; i += 1.0 ) LOL();

You're welcome.
The greatest things about Free/OpenSource software, are:

It promote/encourage collaboration instead of competition,

It's "Un pour Tous, Tous pour Un", instead of "We will beat
them all",

It's the realisation of "L'Union fait la Force",

It's a great way to make friends,

And, when Bill Gates will try this, humanity will enter
a new world...

I couldn't have said it better myself. In my experience, when I can't just google for answers it means that either it's right under my nose or its rediculously obscure and possibly offline someplace. That's why I posted the question here since my research was going nowhere fast. If I could work with someone, drum up interest in a new project or add to an existing project in the process: all the better. Also, I'm long overdue for posting this to the DNG as I've been writing about it on my DSP project board for a while now. Anyway edll will give me a much needed foothold: it uses BFD (object file loader library for linux) under windows. I'm sure that it can be ported to DMC/DMD, but it lacks OMF support. Otherwise, I'll have to find a way to convert OMF to COFF or OMF to ELF. So, is anyone out there willing to donate an OMF loader before I try to write one? ;) - EricAnderton at yahoo
Aug 10 2005