www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Article first draft: The Joy and Gibbering Terror of Custom-Loading Executables

reply Burton Radons <burton-radons shaw.ca> writes:
I've made a first draft of an article on custom-loading executables, which can
be used for dynamic compilation, alternative (and potentially portable) plugin
structures to DLLs, scripting-style eval, and turning a compiler or most any
other program into a library.

There are four separate formats of the document but here's the HTML which
branches everywhere:
http://members.shaw.ca/burton-radons/The%20Joy%20and%20Gibbering%20Terror%20of%20Custom-Loading%20Executables.html

Or if that breaks your newsreader:
http://tinyurl.com/22a6x7

A decent library for Phobos is included
(http://members.shaw.ca/burton-radons/exelib.zip). It has actual (lean but
complete) documentation under docs/index.html; I'd particularly check out
exe.dynamism's RuntimeTemplate, which is suggestive of what this kind of thing
can do to coding.

The background is that I've been doing this off and on for eight years now
(starting with a library called Stonewheel for DJGPP, resurrected briefly with
a truly incomprehensible announcement in 2002 for D) and I recently sent an
e-mail to Walter saying that I'd implemented the scripting-language function
"eval" and found it to only take 20 ms per call with DMD, and I rambled about
the fine points of doing this kind of tomfoolery. By the time I said I'd loaded
DMD manually and it was now taking 1 ms per call, he told me to make an article
of it which I was planning to anyway.

As a draft and as someone who's always been a lousy teacher, I'd like input on
anything that seems confusing or you'd like expanded upon or even whole new
sections you feel deserve coverage, like the D side of runtime templating. I
know I race through the process of loading and linking, but the poor teacher
part comes into play - I have no idea how to think like someone who doesn't
know what I know. Toodles!
Feb 13 2008
next sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Burton Radons wrote:

 There are four separate formats of the document but here's the HTML
 which branches everywhere: 
 http://members.shaw.ca/burton-radons/The%20Joy%20and%20Gibbering%20Terror%20of%20Custom-Loading%20Executables.html


Can you please put the docs up from exelib.zip there too?

Could also use some example code.
Feb 14 2008
prev sibling next sibling parent reply Bjoern <nanali nospam-wanadoo.fr> writes:
I will read your article a third time, but I think it means : No need 
for CTFE anymore, DDL is dead. Furthermore we can use D as ultra fast 
scripting on-the-fly-compiled language. (using eval)

Sorry I must be wrong!

B
Feb 14 2008
next sibling parent "Craig Black" <cblack ara.com> writes:
"Bjoern" <nanali nospam-wanadoo.fr> wrote in message 
news:fp2apd$d82$1 digitalmars.com...

I will read your article a third time, but I think it means : No need for 
CTFE anymore, DDL is dead. Furthermore we can use D as ultra fast scripting 
on-the-fly-compiled language. (using eval)

 Sorry I must be wrong!

 B


This approach seems to have its merits but not without caveats.  I don't 
think it replaces anything you mentioned, but perhaps complements them.

-Craig
Feb 14 2008
prev sibling parent reply Burton Radons <burton-radons shaw.ca> writes:
Bjoern Wrote:


 I will read your article a third time, but I think it means : No need 
 for CTFE anymore, DDL is dead. Furthermore we can use D as ultra fast 
 scripting on-the-fly-compiled language. (using eval)
 
 Sorry I must be wrong!


It can take a big chunk out of any situation where you need to use a switch to
select between templates, but it'll have little effect on anything which uses
templating and CTFE for expressiveness. Or maybe it will - it's a diverse
concept that I haven't fully explored.

As to DDL, it looks like a great utility that could easily be used for many of
the same things my library could, and is portable and supports D modules more
thoroughly to boot. I didn't know of it until you mentioned it, but I remember
Eric Anderton talking about it years ago. My code is much more ramshackle and
driven for a singular purpose.
Feb 14 2008
parent reply Don Clugston <dac nospam.com.au> writes:
Burton Radons wrote:

 Bjoern Wrote:
 

 I will read your article a third time, but I think it means : No need 
 for CTFE anymore, DDL is dead. Furthermore we can use D as ultra fast 
 scripting on-the-fly-compiled language. (using eval)

 Sorry I must be wrong!


 
 It can take a big chunk out of any situation where you need to use a switch to
select between templates, but it'll have little effect on anything which uses
templating and CTFE for expressiveness. Or maybe it will - it's a diverse
concept that I haven't fully explored.


The back-end of my BLADE code uses CTFE to generates asm for floating point 
array operations at compile-time; but the same CTFE can be used without 
modification for code generation at run-time. The advantage at run-time (or at 
least, at install time) is that you know the CPU, number of cores, and cache
size.

I think that in your article you should include a brief comparison with the JIT 
techniques of Java/.NET. To a naive reading, it sounds similar; but they are 
fundamentally different. .NET's last-minute conversion of intermediate form to 
native code doesn't buy you very much. JIT source code generation is something 
else entirely. Using your library, you can have JIT selection of the 
_algorithm_. FFTW (www.fftw.org) is an interesting example of this sort of
thing.
Feb 18 2008
next sibling parent reply Burton Radons <burton-radons shaw.ca> writes:
Don Clugston Wrote:


 Burton Radons wrote:

 Bjoern Wrote:
 

 I will read your article a third time, but I think it means : No need 
 for CTFE anymore, DDL is dead. Furthermore we can use D as ultra fast 
 scripting on-the-fly-compiled language. (using eval)

 Sorry I must be wrong!


 
 It can take a big chunk out of any situation where you need to use a switch to
select between templates, but it'll have little effect on anything which uses
templating and CTFE for expressiveness. Or maybe it will - it's a diverse
concept that I haven't fully explored.


 
 The back-end of my BLADE code uses CTFE to generates asm for floating point 
 array operations at compile-time; but the same CTFE can be used without 
 modification for code generation at run-time. The advantage at run-time (or at 
 least, at install time) is that you know the CPU, number of cores, and cache
size.


This can also work at a slightly higher level, where your object file has
multiple versions of a function for different feature sets and the custom
linker chooses the ideal one at runtime. Another use would be to make the
inliner work better; I've noticed that sometimes it's better to avoid asm
statements because they prevent the optimiser from inlining the function and
applying further optimisations to it. So you could compile the code with a
regular D function, then link whatever doesn't inline the regular function to a
custom-written asm function, improving execution speed in both scenarios.

However that won't be helpful if the compiler's only reasoning about inlining
is the number of statements in the function; I haven't really tested this.


 I think that in your article you should include a brief comparison with the
JIT 
 techniques of Java/.NET. To a naive reading, it sounds similar; but they are 
 fundamentally different. .NET's last-minute conversion of intermediate form to 
 native code doesn't buy you very much. JIT source code generation is something 
 else entirely. Using your library, you can have JIT selection of the 
 _algorithm_. FFTW (www.fftw.org) is an interesting example of this sort of
thing.


I don't know much about Sun's Java or Microsoft's .NET JITs; I'll read about
them when I have the time.

From my initial perusal, dynamic recompilation is certainly something you could
do, where you profile code then generate a more optimal version of the function
based upon how it's being used or the environment. However, I'd like a more
solid case for where that could be useful in D before putting it in the article.
Feb 18 2008
parent John Reimer <terminal.node gmail.com> writes:
Burton Radons wrote:


 I think that in your article you should include a brief comparison with the
JIT 
 techniques of Java/.NET. To a naive reading, it sounds similar; but they are 
 fundamentally different. .NET's last-minute conversion of intermediate form to 
 native code doesn't buy you very much. JIT source code generation is something 
 else entirely. Using your library, you can have JIT selection of the 
 _algorithm_. FFTW (www.fftw.org) is an interesting example of this sort of
thing.


 
 I don't know much about Sun's Java or Microsoft's .NET JITs; I'll read about
them when I have the time.
 
 From my initial perusal, dynamic recompilation is certainly something you
could do, where you profile code then generate a more optimal version of the
function based upon how it's being used or the environment. However, I'd like a
more solid case for where that could be useful in D before putting it in the
article.



Just out of curiousity, isn't such a dynamic recompilation technique 
used is some emulator systems (CPU instruction translators like the old 
680x0 emulators for PowerPC systems).  I believe some of these ideas 
were used to compile, profile, and cache runs of instructions (saving 
the optimal results)... or something to that effect.

-JJR
Feb 18 2008
prev sibling parent renoX <renosky free.fr> writes:
Don Clugston a écrit :

 Burton Radons wrote:

 Bjoern Wrote:


 I will read your article a third time, but I think it means : No need 
 for CTFE anymore, DDL is dead. Furthermore we can use D as ultra fast 
 scripting on-the-fly-compiled language. (using eval)

 Sorry I must be wrong!


 It can take a big chunk out of any situation where you need to use a 
 switch to select between templates, but it'll have little effect on 
 anything which uses templating and CTFE for expressiveness. Or maybe 
 it will - it's a diverse concept that I haven't fully explored.


 
 The back-end of my BLADE code uses CTFE to generates asm for floating 
 point array operations at compile-time; but the same CTFE can be used 
 without modification for code generation at run-time. The advantage at 
 run-time (or at least, at install time) is that you know the CPU, number 
 of cores, and cache size.
 
 I think that in your article you should include a brief comparison with 
 the JIT techniques of Java/.NET. To a naive reading, it sounds similar; 
 but they are fundamentally different. .NET's last-minute conversion of 
 intermediate form to native code doesn't buy you very much. JIT source 
 code generation is something else entirely. Using your library, you can 
 have JIT selection of the _algorithm_. FFTW (www.fftw.org) is an 
 interesting example of this sort of thing.



Very good remark, thanks.
renoX
Feb 18 2008
prev sibling next sibling parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
Amazing.

I am not a native English speaker, but "here me out" should probably be 
"hear me out". And later, have a look at "It would've been easier for me 
to have done it than to written about it".

Cheers,
Bastiaan.


Burton Radons wrote:

 I've made a first draft of an article on custom-loading executables, which can
be used for dynamic compilation, alternative (and potentially portable) plugin
structures to DLLs, scripting-style eval, and turning a compiler or most any
other program into a library.
 
 There are four separate formats of the document but here's the HTML which
branches everywhere:
 http://members.shaw.ca/burton-radons/The%20Joy%20and%20Gibbering%20Terror%20of%20Custom-Loading%20Executables.html
 
 Or if that breaks your newsreader:
 http://tinyurl.com/22a6x7
 
 A decent library for Phobos is included
(http://members.shaw.ca/burton-radons/exelib.zip). It has actual (lean but
complete) documentation under docs/index.html; I'd particularly check out
exe.dynamism's RuntimeTemplate, which is suggestive of what this kind of thing
can do to coding.
 
 The background is that I've been doing this off and on for eight years now
(starting with a library called Stonewheel for DJGPP, resurrected briefly with
a truly incomprehensible announcement in 2002 for D) and I recently sent an
e-mail to Walter saying that I'd implemented the scripting-language function
"eval" and found it to only take 20 ms per call with DMD, and I rambled about
the fine points of doing this kind of tomfoolery. By the time I said I'd loaded
DMD manually and it was now taking 1 ms per call, he told me to make an article
of it which I was planning to anyway.
 
 As a draft and as someone who's always been a lousy teacher, I'd like input on
anything that seems confusing or you'd like expanded upon or even whole new
sections you feel deserve coverage, like the D side of runtime templating. I
know I race through the process of loading and linking, but the poor teacher
part comes into play - I have no idea how to think like someone who doesn't
know what I know. Toodles!
Feb 15 2008
parent Burton Radons <burton-radons shaw.ca> writes:
Bastiaan Veelo Wrote:


 Amazing.
 
 I am not a native English speaker, but "here me out" should probably be 
 "hear me out". And later, have a look at "It would've been easier for me 
 to have done it than to written about it".


Ah, it's been a weird quirk as I grow older that I've started to write the
wrong homonym on occasion; I guess the neural pathways, previously clear
towards each word, get intertwined as you hear both words over time and have to
sort them out from some difficult contexts. Thanks!

The latter part should disappear soon, as in in two minutes.
Feb 15 2008
prev sibling next sibling parent reply Dejan Lekic <dejan.lekic gmail.com> writes:
Mr. Radons, this is *AMAZING* ! Thank you for sharing it with us! I 
would (seriously) like to see this in Phobos one day (hopefully Phobos 
developers are reading this thread). It deserves to go into Phobos, 
definitely.

Kind regards
Feb 15 2008
next sibling parent Dejan Lekic <dejan.lekic gmail.com> writes:
Naturally, a _portable_ version of your code should go into Phobos. :)
Feb 15 2008
prev sibling parent reply Burton Radons <burton-radons shaw.ca> writes:
Dejan Lekic Wrote:


 Mr. Radons, this is *AMAZING* ! Thank you for sharing it with us! I 
 would (seriously) like to see this in Phobos one day (hopefully Phobos 
 developers are reading this thread). It deserves to go into Phobos, 
 definitely.


Sure. What I'd prefer actually would be language integration, so that it can
sort out the details of integrating runtime types into the rest of the type
hierarchy, and it would be able to statically test the template. That can be
done automatically (So some template parameter can't be const-folded? Okay,
we'll compile it when we know what the value is!) or with subtle hints like a
dynamic attribute (that, and here's the very important part, act like regular
template parameters if they ARE const). But that seems unlikely as it would put
a comparatively massive burden on the runtime* and make porting to new
environments quite a bit more difficult. I haven't used GDC, but if it's as
terrifyingly slow as GCC (at no fault of GDC I'll add) I wouldn't consider it
suitable for dynamic compilation.

* In which case, hey, it's optional for restrictive environments!
Feb 15 2008
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Burton Radons wrote:

 Dejan Lekic Wrote:
 

 Mr. Radons, this is *AMAZING* ! Thank you for sharing it with us! I 
 would (seriously) like to see this in Phobos one day (hopefully Phobos 
 developers are reading this thread). It deserves to go into Phobos, 
 definitely.


 
 Sure. What I'd prefer actually would be language integration, so that it can
sort out the details of integrating runtime types into the rest of the type
hierarchy, and it would be able to statically test the template. That can be
done automatically (So some template parameter can't be const-folded? Okay,
we'll compile it when we know what the value is!) or with subtle hints like a
dynamic attribute (that, and here's the very important part, act like regular
template parameters if they ARE const). But that seems unlikely as it would put
a comparatively massive burden on the runtime* and make porting to new
environments quite a bit more difficult. I haven't used GDC, but if it's as
terrifyingly slow as GCC (at no fault of GDC I'll add) I wouldn't consider it
suitable for dynamic compilation.
 
 * In which case, hey, it's optional for restrictive environments!


I wouldn't want it transparently doing runtime code generation. I think 
the library solution, as ugly as it is, is ideal since it lets people 
know "yeah, this is a weighty operation".
Feb 15 2008
parent reply Burton Radons <burton-radons shaw.ca> writes:
(Web-News decided the best place to put a reply to a post in
digitalmars.D.announce that was marked as being for digitalmars.D.announce was
actually digitalmars.D. I've noticed it doesn't like it when I take more than
eight hours or so to write a reply; I guess it's using cookies in silly ways.)

Robert Fraser Wrote:


 Burton Radons wrote:

 Dejan Lekic Wrote:
 

 Mr. Radons, this is *AMAZING* ! Thank you for sharing it with us! I 
 would (seriously) like to see this in Phobos one day (hopefully Phobos 
 developers are reading this thread). It deserves to go into Phobos, 
 definitely.


 
 Sure. What I'd prefer actually would be language integration, so that it can
sort out the details of integrating runtime types into the rest of the type
hierarchy, and it would be able to statically test the template. That can be
done automatically (So some template parameter can't be const-folded? Okay,
we'll compile it when we know what the value is!) or with subtle hints like a
dynamic attribute (that, and here's the very important part, act like regular
template parameters if they ARE const). But that seems unlikely as it would put
a comparatively massive burden on the runtime* and make porting to new
environments quite a bit more difficult. I haven't used GDC, but if it's as
terrifyingly slow as GCC (at no fault of GDC I'll add) I wouldn't consider it
suitable for dynamic compilation.
 
 * In which case, hey, it's optional for restrictive environments!


 
 I wouldn't want it transparently doing runtime code generation. I think 
 the library solution, as ugly as it is, is ideal since it lets people 
 know "yeah, this is a weighty operation".


D has absolutely no presence in limited-memory environments and I don't see
that ever happening, not just because of its design but because those
environments are dying out (micro-limited-memory environments are here to say).
Maybe someone with a 2 GHz processor and a gigabyte of RAM wants to control
every single byte, but their concern is invalid. Where they should be enjoying
their riches they're sitting there biting their nails over nothing like a rich
man obsessed with thieves, while Java and C# gladly take a few dozen megabytes
for simple programs. Even if I did think it worth courting them it would be
pointless; D has garbage collection and that's the end of the story.
Feb 16 2008
parent Robert Fraser <fraserofthenight gmail.com> writes:
Burton Radons wrote:

 (Web-News decided the best place to put a reply to a post in
digitalmars.D.announce that was marked as being for digitalmars.D.announce was
actually digitalmars.D. I've noticed it doesn't like it when I take more than
eight hours or so to write a reply; I guess it's using cookies in silly ways.)
 
 Robert Fraser Wrote:
 

 Burton Radons wrote:

 Dejan Lekic Wrote:


 Mr. Radons, this is *AMAZING* ! Thank you for sharing it with us! I 
 would (seriously) like to see this in Phobos one day (hopefully Phobos 
 developers are reading this thread). It deserves to go into Phobos, 
 definitely.


 Sure. What I'd prefer actually would be language integration, so that it can
sort out the details of integrating runtime types into the rest of the type
hierarchy, and it would be able to statically test the template. That can be
done automatically (So some template parameter can't be const-folded? Okay,
we'll compile it when we know what the value is!) or with subtle hints like a
dynamic attribute (that, and here's the very important part, act like regular
template parameters if they ARE const). But that seems unlikely as it would put
a comparatively massive burden on the runtime* and make porting to new
environments quite a bit more difficult. I haven't used GDC, but if it's as
terrifyingly slow as GCC (at no fault of GDC I'll add) I wouldn't consider it
suitable for dynamic compilation.

 * In which case, hey, it's optional for restrictive environments!


 I wouldn't want it transparently doing runtime code generation. I think 
 the library solution, as ugly as it is, is ideal since it lets people 
 know "yeah, this is a weighty operation".


 
 D has absolutely no presence in limited-memory environments and I don't see
that ever happening, not just because of its design but because those
environments are dying out (micro-limited-memory environments are here to say).
Maybe someone with a 2 GHz processor and a gigabyte of RAM wants to control
every single byte, but their concern is invalid. Where they should be enjoying
their riches they're sitting there biting their nails over nothing like a rich
man obsessed with thieves, while Java and C# gladly take a few dozen megabytes
for simple programs. Even if I did think it worth courting them it would be
pointless; D has garbage collection and that's the end of the story.


That must be why tabs make it so sad...
Feb 19 2008
prev sibling next sibling parent Chad J <gamerChad _spamIsBad_gmail.com> writes:
I like where this is going.

Now it just needs to become portable, and awesomeness will ensue.

Unfortunately I don't have time to play with this stuff, but I'm glad 
you explained how to make this stuff happen.  This kind of thing is 
ultimately rather useful to me.
Feb 15 2008
prev sibling next sibling parent reply Burton Radons <burton-radons shaw.ca> writes:
Okay, the article and the library have both been updated, with the little
tweaks suggested (here versus hear, doc code online) and support for Linux
(running Windows' DMD and SCPPN in Linux, that is); that proved a little bit
tricky so there's a section in the article on it now. Note that that's
specifically Linux - FreeBSD and BeOS would require a different syscall, but
I'm already burned out from Linux's poor documentation so I'm ignoring that for
now, if not forever.
Feb 15 2008
parent reply Burton Radons <burton-radons shaw.ca> writes:
Oh, and: I don't know why SCPPN (DigitalMars C) is twice as fast as DMD in
Windows and many times slower in Linux. It asks for some really weird
information (like GetTimeZoneInformation) that I just brush off rather than
implement, so maybe I'm sending it on some loopy journey that costs millions of
cycles. I might investigate that later.
Feb 15 2008
parent Walter Bright <newshound1 digitalmars.com> writes:
Burton Radons wrote:

 Oh, and: I don't know why SCPPN (DigitalMars C) is twice as fast as
 DMD in Windows and many times slower in Linux. It asks for some
 really weird information (like GetTimeZoneInformation) that I just
 brush off rather than implement, so maybe I'm sending it on some
 loopy journey that costs millions of cycles. I might investigate that
 later.


The compiler calls ctime() in order to set the macros __DATE__, 
__TIME__, and __TIMESTAMP__.
Feb 15 2008
prev sibling next sibling parent Burton Radons <burton-radons shaw.ca> writes:
Updated the library. It now compiles in D 1.0, D 2.0, and Tango without
changes. Tango's a pain for me to test so it might drift and break before the
article's finished.
Feb 15 2008
prev sibling parent reply Burton Radons <burton-radons shaw.ca> writes:
Just an update - I haven't let this die, and I'm not onto any other projects,
but the pub I cook at has been very busy the last few weeks because of a
city-wide off-season promotion designed to encourage inter-city touristry that
turns us into a packed house for three straight weeks. I also just moved. I
should be able to get back to devoting brain space to this by this Sunday.
Mar 06 2008
parent BCS <BCS pathlink.com> writes:
Burton Radons wrote:

 Just an update - I haven't let this die, and I'm not onto any other projects,
but the pub I cook at has been very busy the last few weeks because of a
city-wide off-season promotion designed to encourage inter-city touristry that
turns us into a packed house for three straight weeks. I also just moved. I
should be able to get back to devoting brain space to this by this Sunday.


Well aren't you just lucky. <G> I wish I would get that good an excuse 
once in a while.

(BTW that's suposed to be a joke, and it /is/ a good excuse)
Mar 06 2008