www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Dynamic Class Loading Idea

reply "Craig Black" <craigblack2 cox.net> writes:
Dynamic class loading has the requirement that the class library can be 
loaded at run time.  Traditionally, this means that all functions had to be 
bound via function pointers or virtual functions.  This would mean that all 
the local functions would have to be virtual, and static/global functions 
would need to be bound via function pointers somehow.  This is obviously a 
performance issue.

However, self modifying code could be employed to load a class at run time 
without using function pointers and virtual functions.  For each function in 
a dynamically loadable class, we could have an array of pointers that would 
point to the addresses in code where that function is called (call sites). 
Before the class library is loaded, these call sites would simply throw 
exceptions.  When the class is loaded, the code at the call site to throw 
the exception could be overwritten to call the appropriate function.  This 
would eliminate the requirement to use function pointers/virtual functions 
and allow us to do high-performance dynamic class loading.

I don't know if it is possible to modify functions that reside in the code 
segment.  Does the operating system allow this?  If not, we could place 
functions with these dynamic call sites on the heap.

Thoughts?

-Craig 
Nov 02 2007
next sibling parent reply "Vladimir Panteleev" <thecybershadow gmail.com> writes:
On Sat, 03 Nov 2007 08:26:42 +0200, Craig Black <craigblack2 cox.net> wrote:

 Dynamic class loading has the requirement that the class library can be
 loaded at run time.  Traditionally, this means that all functions had to be
 bound via function pointers or virtual functions.  This would mean that all
 the local functions would have to be virtual, and static/global functions
 would need to be bound via function pointers somehow.  This is obviously a
 performance issue.

 However, self modifying code could be employed to load a class at run time
 without using function pointers and virtual functions.  For each function in
 a dynamically loadable class, we could have an array of pointers that would
 point to the addresses in code where that function is called (call sites).
 Before the class library is loaded, these call sites would simply throw
 exceptions.  When the class is loaded, the code at the call site to throw
 the exception could be overwritten to call the appropriate function.  This
 would eliminate the requirement to use function pointers/virtual functions
 and allow us to do high-performance dynamic class loading.

 I don't know if it is possible to modify functions that reside in the code
 segment.  Does the operating system allow this?  If not, we could place
 functions with these dynamic call sites on the heap.

 Thoughts?

I like DDL's approach, in which it acts as a run-time linker. It uses relocation data in OBJ files to find address references and fix them (thus "relocating" the code for any memory address) when loading the libraries. -- Best regards, Vladimir mailto:thecybershadow gmail.com
Nov 03 2007
parent reply "Craig Black" <craigblack2 cox.net> writes:
 I like DDL's approach, in which it acts as a run-time linker. It uses 
 relocation data in OBJ files to find address references and fix them (thus 
 "relocating" the code for any memory address) when loading the libraries.

Does this relocation use function pointers?
Nov 03 2007
next sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Craig Black wrote:
 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them
 (thus "relocating" the code for any memory address) when loading the
 libraries.

Does this relocation use function pointers?

Here's how I understand it. DDL is a linker. The only difference between DDL and dmd/gcc is that it works at run time instead of compile time. What you described is basically what a linker does, hence what DDL does. -- Daniel
Nov 03 2007
parent reply "Craig Black" <craigblack2 cox.net> writes:
"Daniel Keep" <daniel.keep.lists gmail.com> wrote in message 
news:fgi777$na4$2 digitalmars.com...
 Craig Black wrote:
 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them
 (thus "relocating" the code for any memory address) when loading the
 libraries.

Does this relocation use function pointers?

Here's how I understand it. DDL is a linker. The only difference between DDL and dmd/gcc is that it works at run time instead of compile time. What you described is basically what a linker does, hence what DDL does. -- Daniel

That's pretty cool then. Since we have DDL, can we effectively say that we have dynamic class loading? Or are features missing from DDL that make it not a full dynamic class loader? -Craig
Nov 03 2007
parent TomD <t_demmer nospam.web.de> writes:
Vladimir Panteleev Wrote:
[...]
 
 DDL is, at the moment, defunct when it comes to compatibility with newer
versions of the compiler - and it has been so for a while now. There have been
recent talks of someone getting their hands dirty and bringing it up-to-date.
 
 -- 
 Best regards,
  Vladimir                          mailto:thecybershadow gmail.com

the thing relies on (M/T)ango, at least for I/O (which should be quite easily adjustable). Are there further complications involved? Ciao Tom
Nov 04 2007
prev sibling parent "Craig Black" <craigblack2 cox.net> writes:
"Vladimir Panteleev" <thecybershadow gmail.com> wrote in message 
news:op.t0741htdm02fvl cybershadow...
 On Sat, 03 Nov 2007 15:40:34 +0200, Craig Black <craigblack2 cox.net> 
 wrote:

 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them 
 (thus
 "relocating" the code for any memory address) when loading the 
 libraries.

Does this relocation use function pointers?

"Relocation data" is just a list of addresses which have pointers. I didn't express myself correctly, relocation data isn't used to resolve cross-module references. It is only used to "move" that code at a random address. To do that, the linker/loader adds the address to which the code is relocated to each DWORD the address of which is specified in the relocation list. References are resolved by having a list of symbols, and each symbol having an address of where in this module it's referenced. The linker/loader then simply places the address of each symbol into the referenced locations. Since this is done before any code in the module is executed, it should be possible to implement on all platforms (except platforms where code and data is strictly separated). -- Best regards, Vladimir mailto:thecybershadow gmail.com

Thanks for the explanation. So then modules can be loaded at run time, and this system effectively eliminates function pointers?
Nov 03 2007
prev sibling next sibling parent "Vladimir Panteleev" <thecybershadow gmail.com> writes:
On Sat, 03 Nov 2007 15:40:34 +0200, Craig Black <craigblack2 cox.net> wrote:

 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them (thus
 "relocating" the code for any memory address) when loading the libraries.

Does this relocation use function pointers?

"Relocation data" is just a list of addresses which have pointers. I didn't express myself correctly, relocation data isn't used to resolve cross-module references. It is only used to "move" that code at a random address. To do that, the linker/loader adds the address to which the code is relocated to each DWORD the address of which is specified in the relocation list. References are resolved by having a list of symbols, and each symbol having an address of where in this module it's referenced. The linker/loader then simply places the address of each symbol into the referenced locations. Since this is done before any code in the module is executed, it should be possible to implement on all platforms (except platforms where code and data is strictly separated). -- Best regards, Vladimir mailto:thecybershadow gmail.com
Nov 03 2007
prev sibling next sibling parent "Vladimir Panteleev" <thecybershadow gmail.com> writes:
On Sat, 03 Nov 2007 18:28:13 +0200, Craig Black <craigblack2 cox.net> wrote:

 "Vladimir Panteleev" <thecybershadow gmail.com> wrote in message
 news:op.t0741htdm02fvl cybershadow...
 On Sat, 03 Nov 2007 15:40:34 +0200, Craig Black <craigblack2 cox.net>
 wrote:

 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them
 (thus
 "relocating" the code for any memory address) when loading the
 libraries.

Does this relocation use function pointers?

"Relocation data" is just a list of addresses which have pointers. I didn't express myself correctly, relocation data isn't used to resolve cross-module references. It is only used to "move" that code at a random address. To do that, the linker/loader adds the address to which the code is relocated to each DWORD the address of which is specified in the relocation list. References are resolved by having a list of symbols, and each symbol having an address of where in this module it's referenced. The linker/loader then simply places the address of each symbol into the referenced locations. Since this is done before any code in the module is executed, it should be possible to implement on all platforms (except platforms where code and data is strictly separated). -- Best regards, Vladimir mailto:thecybershadow gmail.com

Thanks for the explanation. So then modules can be loaded at run time, and this system effectively eliminates function pointers?

Yes, that's the goal and purpose of DDL. -- Best regards, Vladimir mailto:thecybershadow gmail.com
Nov 03 2007
prev sibling parent "Vladimir Panteleev" <thecybershadow gmail.com> writes:
On Sat, 03 Nov 2007 20:47:58 +0200, Craig Black <craigblack2 cox.net> wrote:

 "Daniel Keep" <daniel.keep.lists gmail.com> wrote in message
 news:fgi777$na4$2 digitalmars.com...
 Craig Black wrote:
 I like DDL's approach, in which it acts as a run-time linker. It uses
 relocation data in OBJ files to find address references and fix them
 (thus "relocating" the code for any memory address) when loading the
 libraries.

Does this relocation use function pointers?

Here's how I understand it. DDL is a linker. The only difference between DDL and dmd/gcc is that it works at run time instead of compile time. What you described is basically what a linker does, hence what DDL does. -- Daniel

That's pretty cool then. Since we have DDL, can we effectively say that we have dynamic class loading? Or are features missing from DDL that make it not a full dynamic class loader?

DDL is, at the moment, defunct when it comes to compatibility with newer versions of the compiler - and it has been so for a while now. There have been recent talks of someone getting their hands dirty and bringing it up-to-date. -- Best regards, Vladimir mailto:thecybershadow gmail.com
Nov 03 2007