Digital Mars - D - D scripting (c++ integration)

↑ ↓ ← → Pragma <ericanderton yahoo.removeme.com> writes:

Johan Granberg wrote:
 bubu wrote:
 Hiya,

 im a game programmer. I want to use some D-scripted objects in my game 
 which
 is programmed in C++. Can I embed a D scripting host like I can do 
 with Lua,
 Phyton or Ruby? thx


 This newsgroup is not used sou much anymore try digitalmars.D instead.
 
 To answer your question.
 
 I don't know of any way to embed D as a script language but if you want 
 you could require script writers to compile their scripts using some D 
 compiler. I think DDL (dsource.org) can be of use here, alternativly 
 just build the scripts into a .dll/.so file.
 
 Hope that helped :)


Johan, thanks for the plug. :)

bubu,

DDL is going to work, but only if you have a D program to start.  You 
might be able to roll your own scripting host as a .dll or something 
similar, provided it's in D.  However, there's no embeded compiler 
product available, so users will still need the D compiler installed.

With regards to DDL, using D for scripting is the reason why I hatched 
the project in the first place.  If you go back and read my journal 
entries in the DSP forum (also on DSource), you'll get an idea of what 
is ahead of you for dynamic binding on windows, without such a 
technology.  Depending on how you proceed, you may or may not need to 
worry about the things I did.

FWIW, the following pattern can be used to make D flow a bit more like 
live scripting environments:

1) Launch the D compiler from within your app directly, to compile the 
"script" you want to run - std.process works well.
2) Use DDL to bind the .obj file from the compiler output.  The linker 
will return a DynamicLibrary object to use.
3) Use the DynamicLibrary to bind any functions, classes, or static 
fields that you know to exist in the compiled script.

That last point is kind of tricky.  While you can simply force script 
coders to adhere to a particular interface, I found it easier to wrap 
the whole script in a function (call that step #0) before compilation. 
This ensures that the starting point of the script was always known, at 
the cost of disallowing some D constructs (bang-line, imports, module 
static init, etc).  It also allows the script author to start from line 
1, much like one can in PHP or JavaScript.

-- 
- EricAnderton at yahoo

Oct 02 2006

↑ ↓ ← → J Duncan <jtd514 nospam.ameritech.net> writes:

Pragma wrote:
 Johan Granberg wrote:
 bubu wrote:
 Hiya,

 im a game programmer. I want to use some D-scripted objects in my 
 game which
 is programmed in C++. Can I embed a D scripting host like I can do 
 with Lua,
 Phyton or Ruby? thx


 This newsgroup is not used sou much anymore try digitalmars.D instead.

 To answer your question.

 I don't know of any way to embed D as a script language but if you 
 want you could require script writers to compile their scripts using 
 some D compiler. I think DDL (dsource.org) can be of use here, 
 alternativly just build the scripts into a .dll/.so file.

 Hope that helped :)


 Johan, thanks for the plug. :)
 
 bubu,
 
 DDL is going to work, but only if you have a D program to start.  You 
 might be able to roll your own scripting host as a .dll or something 
 similar, provided it's in D.  However, there's no embeded compiler 
 product available, so users will still need the D compiler installed.
 
 With regards to DDL, using D for scripting is the reason why I hatched 
 the project in the first place.  If you go back and read my journal 
 entries in the DSP forum (also on DSource), you'll get an idea of what 
 is ahead of you for dynamic binding on windows, without such a 
 technology.  Depending on how you proceed, you may or may not need to 
 worry about the things I did.
 
 FWIW, the following pattern can be used to make D flow a bit more like 
 live scripting environments:
 
 1) Launch the D compiler from within your app directly, to compile the 
 "script" you want to run - std.process works well.
 2) Use DDL to bind the .obj file from the compiler output.  The linker 
 will return a DynamicLibrary object to use.
 3) Use the DynamicLibrary to bind any functions, classes, or static 
 fields that you know to exist in the compiled script.
 
 That last point is kind of tricky.  While you can simply force script 
 coders to adhere to a particular interface, I found it easier to wrap 
 the whole script in a function (call that step #0) before compilation. 
 This ensures that the starting point of the script was always known, at 
 the cost of disallowing some D constructs (bang-line, imports, module 
 static init, etc).  It also allows the script author to start from line 
 1, much like one can in PHP or JavaScript.
 


Actually, I spent most of my weekend experimenting with such techniques. 
(btw pragma I am updating the COFF code but i keep getting sidetracked 
with DDL adventures :) Ive got a little IDE that combines the dparser 
and DDL - mostly for experimentation at the present time. I have a class 
I call a DMachine that loads, links, and manages a set of 
(object/library) modules. Im currently doing basic introspection of the 
mangled identifiers to obtain a catalog of symbols, but Ive started to 
integrate dparser to get a better symbol catalogs. im playing around 
with having a 'extension' source module that is compiled and linked into 
your running exe - at runtime (I even tried did the trick of embedding a 
'script' into a fake module function). Ive even added the expression 
parsing capabilities of dparser to parse out an expression from a string 
( such as 'doSomething("hello");' to a CallExp to call a function). 
basically, as we have discussed, DDL is still very 'raw' and we still 
probably need some sort of 'sdk' layer to make this technology much 
easier. But, we definitely have the capabilities to do what is being 
discussed in this thread. Im not sure i would call this 'scripting' but 
that makes sense. at the least these technologies can probably open our 
binaries or source up to automating the integration of external 
scripting solutions - automatic stub generation, etc...

Anyway, DDL gives us great potential in this area.....

Oct 02 2006

↑ ↓ ← → Pragma <ericanderton yahoo.removeme.com> writes:

J Duncan wrote:
 Pragma wrote:
 Johan Granberg wrote:
 bubu wrote:
 Hiya,

 im a game programmer. I want to use some D-scripted objects in my 
 game which
 is programmed in C++. Can I embed a D scripting host like I can do 
 with Lua,
 Phyton or Ruby? thx


 This newsgroup is not used sou much anymore try digitalmars.D instead.

 To answer your question.

 I don't know of any way to embed D as a script language but if you 
 want you could require script writers to compile their scripts using 
 some D compiler. I think DDL (dsource.org) can be of use here, 
 alternativly just build the scripts into a .dll/.so file.

 Hope that helped :)


 Johan, thanks for the plug. :)

 bubu,

 DDL is going to work, but only if you have a D program to start.  You 
 might be able to roll your own scripting host as a .dll or something 
 similar, provided it's in D.  However, there's no embeded compiler 
 product available, so users will still need the D compiler installed.

 With regards to DDL, using D for scripting is the reason why I hatched 
 the project in the first place.  If you go back and read my journal 
 entries in the DSP forum (also on DSource), you'll get an idea of what 
 is ahead of you for dynamic binding on windows, without such a 
 technology.  Depending on how you proceed, you may or may not need to 
 worry about the things I did.

 FWIW, the following pattern can be used to make D flow a bit more like 
 live scripting environments:

 1) Launch the D compiler from within your app directly, to compile the 
 "script" you want to run - std.process works well.
 2) Use DDL to bind the .obj file from the compiler output.  The linker 
 will return a DynamicLibrary object to use.
 3) Use the DynamicLibrary to bind any functions, classes, or static 
 fields that you know to exist in the compiled script.

 That last point is kind of tricky.  While you can simply force script 
 coders to adhere to a particular interface, I found it easier to wrap 
 the whole script in a function (call that step #0) before compilation. 
 This ensures that the starting point of the script was always known, 
 at the cost of disallowing some D constructs (bang-line, imports, 
 module static init, etc).  It also allows the script author to start 
 from line 1, much like one can in PHP or JavaScript.


 Actually, I spent most of my weekend experimenting with such techniques. 
 (btw pragma I am updating the COFF code but i keep getting sidetracked 
 with DDL adventures :) Ive got a little IDE that combines the dparser 
 and DDL - mostly for experimentation at the present time. I have a class 
 I call a DMachine that loads, links, and manages a set of 
 (object/library) modules. Im currently doing basic introspection of the 
 mangled identifiers to obtain a catalog of symbols, but Ive started to 
 integrate dparser to get a better symbol catalogs. im playing around 
 with having a 'extension' source module that is compiled and linked into 
 your running exe - at runtime (I even tried did the trick of embedding a 
 'script' into a fake module function). Ive even added the expression 
 parsing capabilities of dparser to parse out an expression from a string 
 ( such as 'doSomething("hello");' to a CallExp to call a function). 
 basically, as we have discussed, DDL is still very 'raw' and we still 
 probably need some sort of 'sdk' layer to make this technology much 
 easier. But, we definitely have the capabilities to do what is being 
 discussed in this thread. Im not sure i would call this 'scripting' but 
 that makes sense. at the least these technologies can probably open our 
 binaries or source up to automating the integration of external 
 scripting solutions - automatic stub generation, etc...
 
 Anyway, DDL gives us great potential in this area.....


Awesome.  I can't wait to see what you've cooked up.  The idea of using 
DDL for IDE extensions is extremely cool - reminds me of VB-macros for 
Visual Studio.

I've been trying to develop a DDL-based reflection interface myself.

I can't blame you for using a parser to obtain a symbol catalog.  After 
compilation, I've noticed that there are some things that get thrown out:

- non-static field information (it's just an offset)
- templates (template declarations - instances, however, *are* visible)
- enums

... which is going to get a big "duh" from the gallery. :)  Anyway, some 
of this stuff is likely recoverable via debug info, but that's never 
100% reliable, nor implemented (in DDL) as of yet.

Which leads me to a question I've had for a while now.  You mentioned 
stub generators: do you (Mr. Duncan) think that it's feasible to have a 
front end that generates runtime reflection data, suitable for build to 
fold in automatically?  Have you seen/done anything that would make this 
a bad move?

-- 
- EricAnderton at yahoo

Oct 02 2006

↑ ↓ ← → J Duncan <jtd514 nospam.ameritech.net> writes:

Pragma wrote:
 J Duncan wrote:
 Pragma wrote:
 Johan Granberg wrote:
 bubu wrote:
 Hiya,

 im a game programmer. I want to use some D-scripted objects in my 
 game which
 is programmed in C++. Can I embed a D scripting host like I can do 
 with Lua,
 Phyton or Ruby? thx


 This newsgroup is not used sou much anymore try digitalmars.D instead.

 To answer your question.

 I don't know of any way to embed D as a script language but if you 
 want you could require script writers to compile their scripts using 
 some D compiler. I think DDL (dsource.org) can be of use here, 
 alternativly just build the scripts into a .dll/.so file.

 Hope that helped :)


 Johan, thanks for the plug. :)

 bubu,

 DDL is going to work, but only if you have a D program to start.  You 
 might be able to roll your own scripting host as a .dll or something 
 similar, provided it's in D.  However, there's no embeded compiler 
 product available, so users will still need the D compiler installed.

 With regards to DDL, using D for scripting is the reason why I 
 hatched the project in the first place.  If you go back and read my 
 journal entries in the DSP forum (also on DSource), you'll get an 
 idea of what is ahead of you for dynamic binding on windows, without 
 such a technology.  Depending on how you proceed, you may or may not 
 need to worry about the things I did.

 FWIW, the following pattern can be used to make D flow a bit more 
 like live scripting environments:

 1) Launch the D compiler from within your app directly, to compile 
 the "script" you want to run - std.process works well.
 2) Use DDL to bind the .obj file from the compiler output.  The 
 linker will return a DynamicLibrary object to use.
 3) Use the DynamicLibrary to bind any functions, classes, or static 
 fields that you know to exist in the compiled script.

 That last point is kind of tricky.  While you can simply force script 
 coders to adhere to a particular interface, I found it easier to wrap 
 the whole script in a function (call that step #0) before 
 compilation. This ensures that the starting point of the script was 
 always known, at the cost of disallowing some D constructs 
 (bang-line, imports, module static init, etc).  It also allows the 
 script author to start from line 1, much like one can in PHP or 
 JavaScript.


 Actually, I spent most of my weekend experimenting with such 
 techniques. (btw pragma I am updating the COFF code but i keep getting 
 sidetracked with DDL adventures :) Ive got a little IDE that combines 
 the dparser and DDL - mostly for experimentation at the present time. 
 I have a class I call a DMachine that loads, links, and manages a set 
 of (object/library) modules. Im currently doing basic introspection of 
 the mangled identifiers to obtain a catalog of symbols, but Ive 
 started to integrate dparser to get a better symbol catalogs. im 
 playing around with having a 'extension' source module that is 
 compiled and linked into your running exe - at runtime (I even tried 
 did the trick of embedding a 'script' into a fake module function). 
 Ive even added the expression parsing capabilities of dparser to parse 
 out an expression from a string ( such as 'doSomething("hello");' to a 
 CallExp to call a function). basically, as we have discussed, DDL is 
 still very 'raw' and we still probably need some sort of 'sdk' layer 
 to make this technology much easier. But, we definitely have the 
 capabilities to do what is being discussed in this thread. Im not sure 
 i would call this 'scripting' but that makes sense. at the least these 
 technologies can probably open our binaries or source up to automating 
 the integration of external scripting solutions - automatic stub 
 generation, etc...

 Anyway, DDL gives us great potential in this area.....


 Awesome.  I can't wait to see what you've cooked up.  The idea of using 
 DDL for IDE extensions is extremely cool - reminds me of VB-macros for 
 Visual Studio.
 
 I've been trying to develop a DDL-based reflection interface myself.
 
 I can't blame you for using a parser to obtain a symbol catalog.  After 
 compilation, I've noticed that there are some things that get thrown out:
 
 - non-static field information (it's just an offset)
 - templates (template declarations - instances, however, *are* visible)
 - enums
 
 ... which is going to get a big "duh" from the gallery. :)  Anyway, some 
 of this stuff is likely recoverable via debug info, but that's never 
 100% reliable, nor implemented (in DDL) as of yet.
 
 Which leads me to a question I've had for a while now.  You mentioned 
 stub generators: do you (Mr. Duncan) think that it's feasible to have a 
 front end that generates runtime reflection data, suitable for build to 
 fold in automatically?  Have you seen/done anything that would make this 
 a bad move?
 


Yeah I think its completely feasible. In fact I think the dflect project 
does something similar with the C++ front end. it generates reflection 
data into a source module that gets compiled and linked into the final 
binary. something like this is more than feasible, we just need to come 
up with a good system. We need a good data representation of reflection 
info plus an api to access it. we could just generate source code to 
compile into a project - or even go all the way to produce an OMF binary 
file.... In my mind just about everything we want to do is right in 
front of us, we just have to choose a path. And I am not one that is 
good at defining systems for a community to use, so I have mostly stuck 
to my personal projects so far. My only prior experience at reflection 
is openc++ and their compile time system, which in theory is way cooler 
than in practice. Over the last year or so I have tried to instigate 
interest in the D community for both compile-time and runtime 
introspection systems. I dont have any experience with java reflection 
so I was hoping more informed people than myself would come forward with 
ideas and comments.

as for compile-time reflection it is best handled by the compiler; 
however we can do it outside the compiler by generating intermediate 
source code; something I have been experimenting with but not sure its 
really practical on a large project, and it causes debugging issues.

Anyway i tend to ramble - all the pieces are there for basic runtime 
reflection information; we just need a good system.

Oct 02 2006

↑ ↓ ← → mike <vertex gmx.at> writes:

Maybe for DMD 3.0 ...

' import std.compiler;
'
' char[] source =3D "writefln(\"Hello World!\");";
' void delegate() foo =3D compile(source);
' foo();

That would be just awesome :-)

- mike

 1) Launch the D compiler from within your app directly, to compile the=



 "script" you want to run - std.process works well.
 2) Use DDL to bind the .obj file from the compiler output.  The linker=



 will return a DynamicLibrary object to use.
 3) Use the DynamicLibrary to bind any functions, classes, or static  =


 fields that you know to exist in the compiled script.

 That last point is kind of tricky.  While you can simply force script =



 coders to adhere to a particular interface, I found it easier to wrap =



 the whole script in a function (call that step #0) before compilation.=



 This ensures that the starting point of the script was always known, a=



 the cost of disallowing some D constructs (bang-line, imports, module =



 static init, etc).  It also allows the script author to start from lin=



 1, much like one can in PHP or JavaScript.




-- =

Erstellt mit Operas revolution=E4rem E-Mail-Modul: http://www.opera.com/=
mail/

Oct 03 2006

↑ ↓ ← → Georg Wrede <georg.wrede nospam.org> writes:

mike wrote:
 Maybe for DMD 3.0 ...
 
 ' import std.compiler;
 '
 ' char[] source = "writefln(\"Hello World!\");";
 ' void delegate() foo = compile(source);
 ' foo();
 
 That would be just awesome :-)
 
 - mike


In addition to all the obvious, I'd say this is yet another thing that's 
not doable in C++. Compiling anything takes simply too long. But D, it's 
totally different.

And especially where the same program keeps compiling different stuff, 
then speed would become quite fast since the compiler wouldn't be 
discarded from memory.

"Blurring" this way the distinction and chasm between runtime and 
compile time should give us quite interesting new possibilities and avenues.

 1) Launch the D compiler from within your app directly, to compile 
 the  "script" you want to run - std.process works well.
 2) Use DDL to bind the .obj file from the compiler output.  The 
 linker  will return a DynamicLibrary object to use.
 3) Use the DynamicLibrary to bind any functions, classes, or static  
 fields that you know to exist in the compiled script.

 That last point is kind of tricky.  While you can simply force script  
 coders to adhere to a particular interface, I found it easier to wrap  
 the whole script in a function (call that step #0) before 
 compilation.  This ensures that the starting point of the script was 
 always known, at  the cost of disallowing some D constructs 
 (bang-line, imports, module  static init, etc).  It also allows the 
 script author to start from line  1, much like one can in PHP or 
 JavaScript.

Oct 03 2006

↑ ↓ ← → Lars Ivar Igesund <larsivar igesund.net> writes:

Georg Wrede wrote:

 mike wrote:
 Maybe for DMD 3.0 ...
 
 ' import std.compiler;
 '
 ' char[] source = "writefln(\"Hello World!\");";
 ' void delegate() foo = compile(source);
 ' foo();
 
 That would be just awesome :-)
 
 - mike


 In addition to all the obvious, I'd say this is yet another thing that's
 not doable in C++. Compiling anything takes simply too long. But D, it's
 totally different.
 
 And especially where the same program keeps compiling different stuff,
 then speed would become quite fast since the compiler wouldn't be
 discarded from memory.
 
 "Blurring" this way the distinction and chasm between runtime and
 compile time should give us quite interesting new possibilities and
 avenues.
 


I believe that the CS term for these possibilities are genetic programming
(which might or might not be used for genetic algorithms). Genetic
algorithms are quite common (although that term might not be used), and is
often seen as an AI technique. Genetic programming is probably not that
common, partially because it is not very obvious how it should be used;
What decides how sucessful a program is without it turning into an
algorithm itself? I know one company that say they use genetic programming
to find search queries (in a form of query language) for biological (DNA
searches and such) research, and then in an interactive environment where a
researcher can tune the generational environment while looking at the
search results unfold.

Definately very interesting :)

-- 
Lars Ivar Igesund
blog at http://larsivi.net
DSource & #D: larsivi

Oct 03 2006

↑ ↓ ← → Bill Baxter <dnewsgroup billbaxter.com> writes:

mike wrote:
 Maybe for DMD 3.0 ...
 
 ' import std.compiler;
 '
 ' char[] source = "writefln(\"Hello World!\");";
 ' void delegate() foo = compile(source);
 ' foo();
 
 That would be just awesome :-)
 


That's pretty much what SciPy's weave.blitz and weave.inline do for 
Python code.
Some example here: http://scipy.org/PerformancePython

Example:
     # put expression in a string
     expr = "u[1:-1, 1:-1] = ((u[0:-2, 1:-1] + u[2:, 1:-1])*dy2 + "\
                            "(u[1:-1,0:-2] + u[1:-1, 2:])*dx2)*dnr_inv"
     # compile the expression and run it
     weave.blitz(expr, check_size=0)

The first time you call this you take a hit, but the result is cached 
(and a DLL saved on disk too), so subsequent calls are fast, as well as 
subsequent runs of the program.   This is a big win for optimizing the 
inner loops of computations in Python code.  300x speed increase over 
the original Python loop is not uncommon.  The difference in speed would 
clearly be less impressive in D, since D is already very fast, but still 
it could be useful for building native-speed optimized expressions on 
the fly.  For instance, in a Photoshop type app you could imagine some 
sort of image filter that gets some parameters from the user then 
compiles the resulting parameters into to a binary format for fast 
execution.

I don't think there'd be much point in using it to call "writefln" 
though.  :-)

The thing is you have to have the entire compiler and linker sitting 
around to make this work.

But I don't see that there's anything about D that makes any of this 
more feasible than it would be with C++.  In fact scipy.weave is 
compiling C++ code on the fly.

--bb

Oct 03 2006