• Craig Black (13/13) Mar 09 2007 Since Walter is in the habit of adding super advanced features to D, I
• Johan Granberg (4/20) Mar 09 2007 I like this idea but it sounds a bit tricky to implement (not to mention
• Frits van Bommel (18/35) Mar 09 2007 I've actually been doing something very similar to this already, but I
• BCS (7/22) Mar 09 2007 Darn you all! I've been thinking of both of these ideas for some time.
• Russell Lewis (7/23) Mar 09 2007 Ok, ignore the bootstrap issues for a second, and consider: If the
• Nicolai Waniek (5/13) Mar 17 2007 Even the .NET framework has something similar, though I don't know if
• janderson (27/45) Mar 09 2007 I like this idea (and have thought about it myself, I haven't brought it...
• janderson (2/55) Mar 09 2007 Humm, why do I always spell Abrash's name wrong :(
• Pragma (10/24) Mar 09 2007 Plenty. I've tried my best to approach this kind of functionality for s...
• Craig Black (24/48) Mar 11 2007 always,
• KlausO (7/25) Mar 09 2007 I remember the softwire project which does such things. It is currently
• Andreas Kochenburger (3/13) Mar 10 2007 Yep. You will like Forth

"Craig Black" <cblack ara.com> writes:

Since Walter is in the habit of adding super advanced features to D, I 
thought it possible that this one might eventually make it in.  As always, 
please let me know if I'm way off base.

What if we could invoke the compiler from an application at run time and 
generate a function that would be instantiated on the heap?  The function 
would be defined by a text string and would be invoked via a function 
pointer or delegate.  This would allow a script-like capability with 
fully-optimized performance provided by the D compiler.  I think the D 
compiler would be ideal for this since it is so fast.  Since the method 
would be allocated on the heap dynamically, it could also be removed when it 
is done being used.

Thoughts?

-Craig

Johan Granberg <lijat.meREM OVE.gmail.com> writes:

Craig Black wrote:

 Since Walter is in the habit of adding super advanced features to D, I
 thought it possible that this one might eventually make it in.  As always,
 please let me know if I'm way off base.
 
 What if we could invoke the compiler from an application at run time and
 generate a function that would be instantiated on the heap?  The function
 would be defined by a text string and would be invoked via a function
 pointer or delegate.  This would allow a script-like capability with
 fully-optimized performance provided by the D compiler.  I think the D
 compiler would be ideal for this since it is so fast.  Since the method
 would be allocated on the heap dynamically, it could also be removed when
 it is done being used.
 
 Thoughts?
 
 -Craig

I like this idea but it sounds a bit tricky to implement (not to mention
possible license issues where the compiler is made a part of the programs
it compiles).

Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:

Johan Granberg wrote:
 Craig Black wrote:
 
 Since Walter is in the habit of adding super advanced features to D, I
 thought it possible that this one might eventually make it in.  As always,
 please let me know if I'm way off base.

 What if we could invoke the compiler from an application at run time and
 generate a function that would be instantiated on the heap?  The function
 would be defined by a text string and would be invoked via a function
 pointer or delegate.  This would allow a script-like capability with
 fully-optimized performance provided by the D compiler.  I think the D
 compiler would be ideal for this since it is so fast.  Since the method
 would be allocated on the heap dynamically, it could also be removed when
 it is done being used.


I've actually been doing something very similar to this already, but I 
just write some source code to a file, compile it to a shared library, 
and load it :P.
Note, In my case, the generated source is C instead of D because 
http://dsource.org/projects/ddl/ says it's not compatible with current 
compilers, and the code I generate doesn't use any advanced features 
anyway. But once DDL is updated it should be quite possible to do this 
with D code as well.

This does of course require that the user has the compiler installed, 
but in my case I'm the only one running it, so that's not a problem here :).

This stuff is actually pretty easy to do as long as you're OK with using 
the compiler as an external program (and keep any source needed for 
compilation[1] around, obviously).


[1]: i.e. .h files for C/C++, .d/.di for D

 I like this idea but it sounds a bit tricky to implement (not to mention
 possible license issues where the compiler is made a part of the programs
 it compiles).

I think many Common Lisp compilers already do something like this, and 
license issues don't seem to be a problem. Just consider it an expanded 
runtime library :P.

BCS <BCS pathlink.com> writes:

Frits van Bommel wrote:
 Johan Granberg wrote:
 
 Craig Black wrote:

 What if we could invoke the compiler from an application at run time and
 generate a function that would be instantiated on the heap?  The 
 function
 would be defined by a text string and would be invoked via a function
 pointer or delegate.


 
 
 I've actually been doing something very similar to this already, but I 
 just write some source code to a file, compile it to a shared library, 
 and load it :P.

Darn you all! I've been thinking of both of these ideas for some time. 
<G> I like both ideas by the way.

However I wouldn't use text, I'd go with generated AST or something like 
that, maybe a mesh of basic blocks. I don't known exactly what, but it 
seems that the nature of memory being lots different than a text file 
would allow for some more fun (powerful) ways of doing things.

Russell Lewis <webmaster villagersonline.com> writes:

Ok, ignore the bootstrap issues for a second, and consider: If the 
compiler were primarily implemented as a set of functions in the 
standard library, then we could view "static compilation" as CTFE of 
those functions...while "dynamic compilation" (a.k.a. runtime compiler) 
as ordinary, runtime execution.

Russ

Craig Black wrote:
 Since Walter is in the habit of adding super advanced features to D, I 
 thought it possible that this one might eventually make it in.  As always, 
 please let me know if I'm way off base.
 
 What if we could invoke the compiler from an application at run time and 
 generate a function that would be instantiated on the heap?  The function 
 would be defined by a text string and would be invoked via a function 
 pointer or delegate.  This would allow a script-like capability with 
 fully-optimized performance provided by the D compiler.  I think the D 
 compiler would be ideal for this since it is so fast.  Since the method 
 would be allocated on the heap dynamically, it could also be removed when it 
 is done being used.
 
 Thoughts?
 
 -Craig

Nicolai Waniek <no.spam thank.you> writes:

Russell Lewis wrote:
 Ok, ignore the bootstrap issues for a second, and consider: If the
 compiler were primarily implemented as a set of functions in the
 standard library, then we could view "static compilation" as CTFE of
 those functions...while "dynamic compilation" (a.k.a. runtime compiler)
 as ordinary, runtime execution.
 
 Russ
 


Even the .NET framework has something similar, though I don't know if
you may call the compilers with a string containting the code you wish
to execude.

For more details, search for CodeDOM at msdn

janderson <askme me.com> writes:

Craig Black wrote:
 Since Walter is in the habit of adding super advanced features to D, I 
 thought it possible that this one might eventually make it in.  As always, 
 please let me know if I'm way off base.
 
 What if we could invoke the compiler from an application at run time and 
 generate a function that would be instantiated on the heap?  The function 
 would be defined by a text string and would be invoked via a function 
 pointer or delegate.  This would allow a script-like capability with 
 fully-optimized performance provided by the D compiler.  I think the D 
 compiler would be ideal for this since it is so fast.  Since the method 
 would be allocated on the heap dynamically, it could also be removed when it 
 is done being used.
 
 Thoughts?
 
 -Craig
 
 

I like this idea (and have thought about it myself, I haven't brought it 
up yet, because I'm still unsure what the best approach would be).

Michael Ambrash, was doing some something of similar nature for 
Pixomatic (a very fast software renderer that looks like DirectX 7). 
I'm pretty sure his compiler would not map directly to DMD because it 
needed to be the fastest possible however he does point out some 
interesting ideas.   The cool thing is, that its setup to optimise for 
the system it is running on (kinda like java's virtual machine) of which 
(in Ambrash's case) there are thousands of combinations.

For many case, we could change the code for that users environment write 
the algorithm in the most otimal way for the given inputs and change 
dymamic data to constants ect...

System cache can have a dramatic impact on performance which is why 
code-modifcation techniques have been avoided in the past.  Ambrash 
found that one frame was enough to allow the newly generated code to 
propgage into cache.

If you were to encapsulate the code in a function pointer or something I 
guess, you could have 2 stages.  1) compile the function and 2) run it. 
  That way in step 1, DMD could push it into cache.

Here are the articles for anyone who is interested.
http://www.ddj.com/184405765
http://www.ddj.com/184405807
http://www.ddj.com/184405848

Anyhow, we do have DDL.  I'm not exactly sure how the compiler/syntax 
could do a much better job yet (I'm sure its possible).

-Joel

janderson <askme me.com> writes:

janderson wrote:
 Craig Black wrote:
 Since Walter is in the habit of adding super advanced features to D, I 
 thought it possible that this one might eventually make it in.  As 
 always, please let me know if I'm way off base.

 What if we could invoke the compiler from an application at run time 
 and generate a function that would be instantiated on the heap?  The 
 function would be defined by a text string and would be invoked via a 
 function pointer or delegate.  This would allow a script-like 
 capability with fully-optimized performance provided by the D 
 compiler.  I think the D compiler would be ideal for this since it is 
 so fast.  Since the method would be allocated on the heap dynamically, 
 it could also be removed when it is done being used.

 Thoughts?

 -Craig

 
 I like this idea (and have thought about it myself, I haven't brought it 
 up yet, because I'm still unsure what the best approach would be).
 
 Michael Ambrash, was doing some something of similar nature for 
 Pixomatic (a very fast software renderer that looks like DirectX 7). I'm 
 pretty sure his compiler would not map directly to DMD because it needed 
 to be the fastest possible however he does point out some interesting 
 ideas.   The cool thing is, that its setup to optimise for the system it 
 is running on (kinda like java's virtual machine) of which (in Ambrash's 
 case) there are thousands of combinations.
 
 For many case, we could change the code for that users environment write 
 the algorithm in the most otimal way for the given inputs and change 
 dymamic data to constants ect...
 
 System cache can have a dramatic impact on performance which is why 
 code-modifcation techniques have been avoided in the past.  Ambrash 
 found that one frame was enough to allow the newly generated code to 
 propgage into cache.
 
 If you were to encapsulate the code in a function pointer or something I 
 guess, you could have 2 stages.  1) compile the function and 2) run it. 
  That way in step 1, DMD could push it into cache.
 
 Here are the articles for anyone who is interested.
 http://www.ddj.com/184405765
 http://www.ddj.com/184405807
 http://www.ddj.com/184405848
 
 Anyhow, we do have DDL.  I'm not exactly sure how the compiler/syntax 
 could do a much better job yet (I'm sure its possible).
 
 -Joel

Humm, why do I always spell Abrash's name wrong  :(

Pragma <ericanderton yahoo.removeme.com> writes:

Craig Black wrote:
 Since Walter is in the habit of adding super advanced features to D, I 
 thought it possible that this one might eventually make it in.  As always, 
 please let me know if I'm way off base.
 
 What if we could invoke the compiler from an application at run time and 
 generate a function that would be instantiated on the heap?  The function 
 would be defined by a text string and would be invoked via a function 
 pointer or delegate.  This would allow a script-like capability with 
 fully-optimized performance provided by the D compiler.  I think the D 
 compiler would be ideal for this since it is so fast.  Since the method 
 would be allocated on the heap dynamically, it could also be removed when it 
 is done being used.
 
 Thoughts?

Plenty.  I've tried my best to approach this kind of functionality for some
time in library space.  However I have my 
doubts as to having this included in D proper - since D is so minimal at it's
core, it becomes very much a library 
support problem.

(I'll leave the DSP/DDL/Flectioned discussion for another thread).

I can say from experience that doing this from within library space is
perfectly feasable, and yields some great 
results, at the cost of additional memory at runtime and a small (one-time per
module) delay for compilation and runtime 
binding.  I would have sooner considered drafting a runtime compiler if DMD
wasn't so fast.

-- 
- EricAnderton at yahoo

"Craig Black" <cblack ara.com> writes:

"Pragma" <ericanderton yahoo.removeme.com> wrote in message
news:essjou$cks$1 digitalmars.com...
 Craig Black wrote:
 Since Walter is in the habit of adding super advanced features to D, I
 thought it possible that this one might eventually make it in.  As


always,
 please let me know if I'm way off base.

 What if we could invoke the compiler from an application at run time and
 generate a function that would be instantiated on the heap?  The


function
 would be defined by a text string and would be invoked via a function
 pointer or delegate.  This would allow a script-like capability with
 fully-optimized performance provided by the D compiler.  I think the D
 compiler would be ideal for this since it is so fast.  Since the method
 would be allocated on the heap dynamically, it could also be removed


when it
 is done being used.

 Thoughts?

 Plenty.  I've tried my best to approach this kind of functionality for

some time in library space.  However I have my
 doubts as to having this included in D proper - since D is so minimal at

it's core, it becomes very much a library
 support problem.

 (I'll leave the DSP/DDL/Flectioned discussion for another thread).

 I can say from experience that doing this from within library space is

perfectly feasable, and yields some great
 results, at the cost of additional memory at runtime and a small (one-time

per module) delay for compilation and runtime
 binding.  I would have sooner considered drafting a runtime compiler if

DMD wasn't so fast.
 --
 - EricAnderton at yahoo

The run time library approach is feasible as you say, but not a efficient or
elegant as using the heap.  However, this may be possible without
modifications to the D compiler.  It would require compiling some code, then
getting the result from a file, and loading it onto the heap.  Not a trivial
task as there would be many details to work through.  But I am inclined to
believe that it could be done.  On the other hand, if Walter wanted to help,
we could probably get the compiler to emit the code directly to the heap,
which would be even better.

A run time compiler would have many applications to be sure.  One is gaming,
where a good chunk of the code is usually script.  Some game engines use
.NET or Java instead of a scripting language for better performance.  A run
time compiler would be even better.  Further, D is an ideal candidate for
such a feature because of D's compilation speed and garbage collection.

-Craig

KlausO <oberhofer users.sourceforge.net> writes:

Craig Black schrieb:
 Since Walter is in the habit of adding super advanced features to D, I 
 thought it possible that this one might eventually make it in.  As always, 
 please let me know if I'm way off base.
 
 What if we could invoke the compiler from an application at run time and 
 generate a function that would be instantiated on the heap?  The function 
 would be defined by a text string and would be invoked via a function 
 pointer or delegate.  This would allow a script-like capability with 
 fully-optimized performance provided by the D compiler.  I think the D 
 compiler would be ideal for this since it is so fast.  Since the method 
 would be allocated on the heap dynamically, it could also be removed when it 
 is done being used.
 
 Thoughts?
 
 -Craig
 
 

I remember the softwire project which does such things. It is currently 
hosted at

https://gna.org/projects/softwire

I also thought that the Win32 port of TinyCC could compile directly to 
the heap.

KlausO

Andreas Kochenburger <akk nospam.org> writes:

Craig Black wrote:
 What if we could invoke the compiler from an application at run time and 
 generate a function that would be instantiated on the heap?  The function 
 would be defined by a text string and would be invoked via a function 
 pointer or delegate.  This would allow a script-like capability with 
 fully-optimized performance provided by the D compiler.  I think the D 
 compiler would be ideal for this since it is so fast.  Since the method 
 would be allocated on the heap dynamically, it could also be removed when it 
 is done being used.
 
 Thoughts?


Yep. You will like Forth

Andreas