www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Idea: Run Time Compilation

reply "Craig Black" <cblack ara.com> writes:
Since Walter is in the habit of adding super advanced features to D, I 
thought it possible that this one might eventually make it in.  As always, 
please let me know if I'm way off base.

What if we could invoke the compiler from an application at run time and 
generate a function that would be instantiated on the heap?  The function 
would be defined by a text string and would be invoked via a function 
pointer or delegate.  This would allow a script-like capability with 
fully-optimized performance provided by the D compiler.  I think the D 
compiler would be ideal for this since it is so fast.  Since the method 
would be allocated on the heap dynamically, it could also be removed when it 
is done being used.

Thoughts?

-Craig
Mar 09 2007
next sibling parent reply Johan Granberg <lijat.meREM OVE.gmail.com> writes:
Craig Black wrote:

 Since Walter is in the habit of adding super advanced features to D, I
 thought it possible that this one might eventually make it in.  As always,
 please let me know if I'm way off base.
 
 What if we could invoke the compiler from an application at run time and
 generate a function that would be instantiated on the heap?  The function
 would be defined by a text string and would be invoked via a function
 pointer or delegate.  This would allow a script-like capability with
 fully-optimized performance provided by the D compiler.  I think the D
 compiler would be ideal for this since it is so fast.  Since the method
 would be allocated on the heap dynamically, it could also be removed when
 it is done being used.
 
 Thoughts?
 
 -Craig

I like this idea but it sounds a bit tricky to implement (not to mention possible license issues where the compiler is made a part of the programs it compiles).
Mar 09 2007
parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Johan Granberg wrote:
 Craig Black wrote:
 
 Since Walter is in the habit of adding super advanced features to D, I
 thought it possible that this one might eventually make it in.  As always,
 please let me know if I'm way off base.

 What if we could invoke the compiler from an application at run time and
 generate a function that would be instantiated on the heap?  The function
 would be defined by a text string and would be invoked via a function
 pointer or delegate.  This would allow a script-like capability with
 fully-optimized performance provided by the D compiler.  I think the D
 compiler would be ideal for this since it is so fast.  Since the method
 would be allocated on the heap dynamically, it could also be removed when
 it is done being used.


I've actually been doing something very similar to this already, but I just write some source code to a file, compile it to a shared library, and load it :P. Note, In my case, the generated source is C instead of D because http://dsource.org/projects/ddl/ says it's not compatible with current compilers, and the code I generate doesn't use any advanced features anyway. But once DDL is updated it should be quite possible to do this with D code as well. This does of course require that the user has the compiler installed, but in my case I'm the only one running it, so that's not a problem here :). This stuff is actually pretty easy to do as long as you're OK with using the compiler as an external program (and keep any source needed for compilation[1] around, obviously). [1]: i.e. .h files for C/C++, .d/.di for D
 I like this idea but it sounds a bit tricky to implement (not to mention
 possible license issues where the compiler is made a part of the programs
 it compiles).

I think many Common Lisp compilers already do something like this, and license issues don't seem to be a problem. Just consider it an expanded runtime library :P.
Mar 09 2007
parent BCS <BCS pathlink.com> writes:
Frits van Bommel wrote:
 Johan Granberg wrote:
 
 Craig Black wrote:

 What if we could invoke the compiler from an application at run time and
 generate a function that would be instantiated on the heap?  The 
 function
 would be defined by a text string and would be invoked via a function
 pointer or delegate.


I've actually been doing something very similar to this already, but I just write some source code to a file, compile it to a shared library, and load it :P.

Darn you all! I've been thinking of both of these ideas for some time. <G> I like both ideas by the way. However I wouldn't use text, I'd go with generated AST or something like that, maybe a mesh of basic blocks. I don't known exactly what, but it seems that the nature of memory being lots different than a text file would allow for some more fun (powerful) ways of doing things.
Mar 09 2007
prev sibling next sibling parent reply Russell Lewis <webmaster villagersonline.com> writes:
Ok, ignore the bootstrap issues for a second, and consider: If the 
compiler were primarily implemented as a set of functions in the 
standard library, then we could view "static compilation" as CTFE of 
those functions...while "dynamic compilation" (a.k.a. runtime compiler) 
as ordinary, runtime execution.

Russ

Craig Black wrote:
 Since Walter is in the habit of adding super advanced features to D, I 
 thought it possible that this one might eventually make it in.  As always, 
 please let me know if I'm way off base.
 
 What if we could invoke the compiler from an application at run time and 
 generate a function that would be instantiated on the heap?  The function 
 would be defined by a text string and would be invoked via a function 
 pointer or delegate.  This would allow a script-like capability with 
 fully-optimized performance provided by the D compiler.  I think the D 
 compiler would be ideal for this since it is so fast.  Since the method 
 would be allocated on the heap dynamically, it could also be removed when it 
 is done being used.
 
 Thoughts?
 
 -Craig

Mar 09 2007
parent Nicolai Waniek <no.spam thank.you> writes:
Russell Lewis wrote:
 Ok, ignore the bootstrap issues for a second, and consider: If the
 compiler were primarily implemented as a set of functions in the
 standard library, then we could view "static compilation" as CTFE of
 those functions...while "dynamic compilation" (a.k.a. runtime compiler)
 as ordinary, runtime execution.
 
 Russ
 

Even the .NET framework has something similar, though I don't know if you may call the compilers with a string containting the code you wish to execude. For more details, search for CodeDOM at msdn
Mar 17 2007
prev sibling next sibling parent reply janderson <askme me.com> writes:
Craig Black wrote:
 Since Walter is in the habit of adding super advanced features to D, I 
 thought it possible that this one might eventually make it in.  As always, 
 please let me know if I'm way off base.
 
 What if we could invoke the compiler from an application at run time and 
 generate a function that would be instantiated on the heap?  The function 
 would be defined by a text string and would be invoked via a function 
 pointer or delegate.  This would allow a script-like capability with 
 fully-optimized performance provided by the D compiler.  I think the D 
 compiler would be ideal for this since it is so fast.  Since the method 
 would be allocated on the heap dynamically, it could also be removed when it 
 is done being used.
 
 Thoughts?
 
 -Craig
 
 

I like this idea (and have thought about it myself, I haven't brought it up yet, because I'm still unsure what the best approach would be). Michael Ambrash, was doing some something of similar nature for Pixomatic (a very fast software renderer that looks like DirectX 7). I'm pretty sure his compiler would not map directly to DMD because it needed to be the fastest possible however he does point out some interesting ideas. The cool thing is, that its setup to optimise for the system it is running on (kinda like java's virtual machine) of which (in Ambrash's case) there are thousands of combinations. For many case, we could change the code for that users environment write the algorithm in the most otimal way for the given inputs and change dymamic data to constants ect... System cache can have a dramatic impact on performance which is why code-modifcation techniques have been avoided in the past. Ambrash found that one frame was enough to allow the newly generated code to propgage into cache. If you were to encapsulate the code in a function pointer or something I guess, you could have 2 stages. 1) compile the function and 2) run it. That way in step 1, DMD could push it into cache. Here are the articles for anyone who is interested. http://www.ddj.com/184405765 http://www.ddj.com/184405807 http://www.ddj.com/184405848 Anyhow, we do have DDL. I'm not exactly sure how the compiler/syntax could do a much better job yet (I'm sure its possible). -Joel
Mar 09 2007
parent janderson <askme me.com> writes:
janderson wrote:
 Craig Black wrote:
 Since Walter is in the habit of adding super advanced features to D, I 
 thought it possible that this one might eventually make it in.  As 
 always, please let me know if I'm way off base.

 What if we could invoke the compiler from an application at run time 
 and generate a function that would be instantiated on the heap?  The 
 function would be defined by a text string and would be invoked via a 
 function pointer or delegate.  This would allow a script-like 
 capability with fully-optimized performance provided by the D 
 compiler.  I think the D compiler would be ideal for this since it is 
 so fast.  Since the method would be allocated on the heap dynamically, 
 it could also be removed when it is done being used.

 Thoughts?

 -Craig

I like this idea (and have thought about it myself, I haven't brought it up yet, because I'm still unsure what the best approach would be). Michael Ambrash, was doing some something of similar nature for Pixomatic (a very fast software renderer that looks like DirectX 7). I'm pretty sure his compiler would not map directly to DMD because it needed to be the fastest possible however he does point out some interesting ideas. The cool thing is, that its setup to optimise for the system it is running on (kinda like java's virtual machine) of which (in Ambrash's case) there are thousands of combinations. For many case, we could change the code for that users environment write the algorithm in the most otimal way for the given inputs and change dymamic data to constants ect... System cache can have a dramatic impact on performance which is why code-modifcation techniques have been avoided in the past. Ambrash found that one frame was enough to allow the newly generated code to propgage into cache. If you were to encapsulate the code in a function pointer or something I guess, you could have 2 stages. 1) compile the function and 2) run it. That way in step 1, DMD could push it into cache. Here are the articles for anyone who is interested. http://www.ddj.com/184405765 http://www.ddj.com/184405807 http://www.ddj.com/184405848 Anyhow, we do have DDL. I'm not exactly sure how the compiler/syntax could do a much better job yet (I'm sure its possible). -Joel

Humm, why do I always spell Abrash's name wrong :(
Mar 09 2007
prev sibling next sibling parent reply Pragma <ericanderton yahoo.removeme.com> writes:
Craig Black wrote:
 Since Walter is in the habit of adding super advanced features to D, I 
 thought it possible that this one might eventually make it in.  As always, 
 please let me know if I'm way off base.
 
 What if we could invoke the compiler from an application at run time and 
 generate a function that would be instantiated on the heap?  The function 
 would be defined by a text string and would be invoked via a function 
 pointer or delegate.  This would allow a script-like capability with 
 fully-optimized performance provided by the D compiler.  I think the D 
 compiler would be ideal for this since it is so fast.  Since the method 
 would be allocated on the heap dynamically, it could also be removed when it 
 is done being used.
 
 Thoughts?

Plenty. I've tried my best to approach this kind of functionality for some time in library space. However I have my doubts as to having this included in D proper - since D is so minimal at it's core, it becomes very much a library support problem. (I'll leave the DSP/DDL/Flectioned discussion for another thread). I can say from experience that doing this from within library space is perfectly feasable, and yields some great results, at the cost of additional memory at runtime and a small (one-time per module) delay for compilation and runtime binding. I would have sooner considered drafting a runtime compiler if DMD wasn't so fast. -- - EricAnderton at yahoo
Mar 09 2007
parent "Craig Black" <cblack ara.com> writes:
"Pragma" <ericanderton yahoo.removeme.com> wrote in message
news:essjou$cks$1 digitalmars.com...
 Craig Black wrote:
 Since Walter is in the habit of adding super advanced features to D, I
 thought it possible that this one might eventually make it in.  As


 please let me know if I'm way off base.

 What if we could invoke the compiler from an application at run time and
 generate a function that would be instantiated on the heap?  The


 would be defined by a text string and would be invoked via a function
 pointer or delegate.  This would allow a script-like capability with
 fully-optimized performance provided by the D compiler.  I think the D
 compiler would be ideal for this since it is so fast.  Since the method
 would be allocated on the heap dynamically, it could also be removed


 is done being used.

 Thoughts?

Plenty. I've tried my best to approach this kind of functionality for

 doubts as to having this included in D proper - since D is so minimal at

 support problem.

 (I'll leave the DSP/DDL/Flectioned discussion for another thread).

 I can say from experience that doing this from within library space is

 results, at the cost of additional memory at runtime and a small (one-time

 binding.  I would have sooner considered drafting a runtime compiler if

 --
 - EricAnderton at yahoo

The run time library approach is feasible as you say, but not a efficient or elegant as using the heap. However, this may be possible without modifications to the D compiler. It would require compiling some code, then getting the result from a file, and loading it onto the heap. Not a trivial task as there would be many details to work through. But I am inclined to believe that it could be done. On the other hand, if Walter wanted to help, we could probably get the compiler to emit the code directly to the heap, which would be even better. A run time compiler would have many applications to be sure. One is gaming, where a good chunk of the code is usually script. Some game engines use .NET or Java instead of a scripting language for better performance. A run time compiler would be even better. Further, D is an ideal candidate for such a feature because of D's compilation speed and garbage collection. -Craig
Mar 11 2007
prev sibling next sibling parent KlausO <oberhofer users.sourceforge.net> writes:
Craig Black schrieb:
 Since Walter is in the habit of adding super advanced features to D, I 
 thought it possible that this one might eventually make it in.  As always, 
 please let me know if I'm way off base.
 
 What if we could invoke the compiler from an application at run time and 
 generate a function that would be instantiated on the heap?  The function 
 would be defined by a text string and would be invoked via a function 
 pointer or delegate.  This would allow a script-like capability with 
 fully-optimized performance provided by the D compiler.  I think the D 
 compiler would be ideal for this since it is so fast.  Since the method 
 would be allocated on the heap dynamically, it could also be removed when it 
 is done being used.
 
 Thoughts?
 
 -Craig
 
 

I remember the softwire project which does such things. It is currently hosted at https://gna.org/projects/softwire I also thought that the Win32 port of TinyCC could compile directly to the heap. KlausO
Mar 09 2007
prev sibling parent Andreas Kochenburger <akk nospam.org> writes:
Craig Black wrote:
 What if we could invoke the compiler from an application at run time and 
 generate a function that would be instantiated on the heap?  The function 
 would be defined by a text string and would be invoked via a function 
 pointer or delegate.  This would allow a script-like capability with 
 fully-optimized performance provided by the D compiler.  I think the D 
 compiler would be ideal for this since it is so fast.  Since the method 
 would be allocated on the heap dynamically, it could also be removed when it 
 is done being used.
 
 Thoughts?

Yep. You will like Forth Andreas
Mar 10 2007