www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - How D addresses the problem of Extending and/or Embedding other

reply "Brost" <a1378499 trbvm.com> writes:
Hi,

any language that I know has its own features, sometimes is
"marketing", sometimes are really useful features, useful
algorithms and useful data structures .

Big problems arise when given an application written in the X
language, I want to plug in an Y scripting language, or a
language that is there to offer some level of interaction with
the end user.

In this scenario you tipically end up wiring everything into C
code, and use C as a bridge because both X and Y do not really
support anything other than C .

And yes there are libraries that sometimes help, but libraries
need maintenance too and libraries usually don't give strong
invariances like a good language usually does .

For example when you start from a C++ application, and you want
to expose an std::vector to your Y language, your only real
option with something like Python or Lua is to write C code that
will use a pointer to the chunk of memory where the meat of the
vector is, that contiguos portion of memory where things are
stored, and pass the size of the vector as a second argument too
because C doesn't even know what a std::vector is .

Et voilà, you basically have lost years and years of evolution in
language design, memory safety, type safety and all the other
good things, just to let someone else play with your std::vector
which no one know what it is except C++ itself .

This problem also makes things a lot more complicated with
concurrency and code with a complex behaviour; also I said
before, this entire concept is based on some basic invariances,
in the case of the vector the invariance is about having a
contiguos chunk of memory allocated, the same thing wouldn't be
possible with the cells of the array scattered everywhere; this
is for saying that with this way of doing "extensions" for my
applications I can't just use any library, I also need to check
things out everytime, see how they internally work, if they are
suitable and with an appropriate behaviour for this "C bridge"
that limits my original language, and by that time I can probably
write that Z library all by myself.

The only exception to the rule that I know is V8, which tries to
expose itself with C++ based APIs since it's being written in C++
in the first place, but when it comes to languages, compiled
languages, the entire Zoo of languages all point to the same
target: C .

Is D different in this regard ?
Aug 24 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 25/08/14 08:18, Brost wrote:


 Is D different in this regard ?


In addition to being ABI compatible with C, D is also compatible with 
C++ [1]. D recently got support for C++ templates and namespaces. 
Hopefully it will get support for Objective-C as well, in the not too 
far away future.

[1] http://dlang.org/cpp_interface.html

-- 
/Jacob Carlborg
Aug 24 2014
parent reply "Brost" <a1378499 trbvm.com> writes:
On Monday, 25 August 2014 at 06:44:25 UTC, Jacob Carlborg wrote:

 On 25/08/14 08:18, Brost wrote:


 Is D different in this regard ?


 In addition to being ABI compatible with C, D is also 
 compatible with C++ [1]. D recently got support for C++ 
 templates and namespaces. Hopefully it will get support for 
 Objective-C as well, in the not too far away future.

 [1] http://dlang.org/cpp_interface.html


thanks, but unfortunately neither of those is a scripting
language, I can't see any facility in the standard D library or
in the language that can help me retain the type safety, the
memory management, and all the other things I talked about while
interfacing with another language .

for example how do you correctly expose a given D data structure
to a scripting language of your choice ?
Aug 25 2014
next sibling parent Mike Parker <aldacron gmail.com> writes:
On 8/25/2014 4:18 PM, Brost wrote:


 for example how do you correctly expose a given D data structure
 to a scripting language of your choice ?


By using whatever C or C++ interface that scripting language provides. C 
is commonly used for this precisely because it has a fairly universal 
API. If a language can speak C, it can be used anywhere. I'm not sure 
what you would expect D to do in order to be usable from any and all 
scripting languages, other than be C ABI compatible as it is.

---
This email is free from viruses and malware because avast! Antivirus protection
is active.
http://www.avast.com
Aug 25 2014
prev sibling next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 25 Aug 2014 07:18:50 +0000
Brost via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 for example how do you correctly expose a given D data structure
 to a scripting language of your choice ?


using LuaD, for example: http://jakobovrum.github.io/LuaD/
Aug 25 2014
prev sibling next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 25 Aug 2014 07:18:50 +0000
Brost via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 for example how do you correctly expose a given D data structure
 to a scripting language of your choice ?


or Adam D. Ruppe's "script", which is written completely in D:
https://github.com/adamdruppe/script.git
Aug 25 2014
prev sibling next sibling parent reply "Ola Fosheim Gr" <ola.fosheim.grostad+dlang gmail.com> writes:
On Monday, 25 August 2014 at 07:18:51 UTC, Brost wrote:

 language, I can't see any facility in the standard D library or
 in the language that can help me retain the type safety, the
 memory management, and all the other things I talked about while
 interfacing with another language .

 for example how do you correctly expose a given D data structure
 to a scripting language of your choice ?


Wouldn't that depend on the scripting language? But you can 
control layout of structs and might be able to print out other 
information you need using reflection, so you might be able to 
create the stubs you need using traits and properties?

The gc accepts foreign root pointers, but the gc itself might be 
too slow. So you would rather use v8s gc and  nogc in D?
Aug 25 2014
parent reply "Brost" <a1378499 trbvm.com> writes:
It's really strange the fact that no one seems to feel about this
the same way I do, I mean you are happy with a C based bridge
between 2 languages, you are ok at throwing away all the good
stuff that both languages ( your X and your Y ) are offering ?

Thank you for posting various scripts and libraries but this is
not my point, I think that the game changer will be some kind of
support for this extensions and foreign languages built right
into the language itself.

Being ABI compatible it's not exactly like saying " I'm taking
all the invariants and features of D, translating your code in C,
and keeping everything functional and trustable just as it was in
D "

On Monday, 25 August 2014 at 07:37:28 UTC, Ola Fosheim Gr wrote:

 On Monday, 25 August 2014 at 07:18:51 UTC, Brost wrote:

 language, I can't see any facility in the standard D library or
 in the language that can help me retain the type safety, the
 memory management, and all the other things I talked about 
 while
 interfacing with another language .

 for example how do you correctly expose a given D data 
 structure
 to a scripting language of your choice ?


 Wouldn't that depend on the scripting language? But you can 
 control layout of structs and might be able to print out other 
 information you need using reflection, so you might be able to 
 create the stubs you need using traits and properties?

 The gc accepts foreign root pointers, but the gc itself might 
 be too slow. So you would rather use v8s gc and  nogc in D?


Indeed you can't, but when you have to deal with structures that
you don't know nothing about before hand, and you have to inspect
the properties of something that I'm passing to you, usually at
this point some kind of protocol kicks in.

Protocols are basically the way to "standardize" or describe
something that can be implemented in many different way; I know
that if you start from the ABI or the compiled object you are
basically going nowhere because you are focusing on
implementation specific details and limitations.
That's what I was expecting, something like a protocol .
Aug 25 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 25 August 2014 at 19:26:59 UTC, Brost wrote:

 It's really strange the fact that no one seems to feel about 
 this
 the same way I do, I mean you are happy with a C based bridge
 between 2 languages, you are ok at throwing away all the good
 stuff that both languages ( your X and your Y ) are offering ?


Because you want magic and we are pragmatical programmers :P

Any more advanced bridge will be totally different for each new 
language and at the same time can be implemented on top of C 
based ABI (exactly what LuaD, PyD & Co do). More than that, there 
is usually no single "true" way of doing it. All this combined 
clearly moves cross-language bridges to library responsibility.
Aug 25 2014
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 25 August 2014 at 19:26:59 UTC, Brost wrote:

 It's really strange the fact that no one seems to feel about 
 this
 the same way I do, I mean you are happy with a C based bridge
 between 2 languages, you are ok at throwing away all the good
 stuff that both languages ( your X and your Y ) are offering ?

 Thank you for posting various scripts and libraries but this is
 not my point, I think that the game changer will be some kind of
 support for this extensions and foreign languages built right
 into the language itself.


So, which language(s) do you have in mind in particular? Whether 
it's practicable surely depends on the type of language you want 
to bind to.

Take Ruby as an example: Ruby is a scripting language that cannot 
be compiled to native code (though it can be JITted). This means 
that there is no ABI, which would be required if you want to link 
against the language. It would theoretically be possible to 
somehow link against the interpreter, but that would require 
cooperation from it, i.e. it would have to expose an interface to 
do that. But there isn't only one implementation of Ruby: There 
is MRI (the most widespread one), JRuby (running on the JVM), 
Rubinius (with LLVM as a JIT backend), and probably others. All 
of them have different interfaces. There is however a FFI 
(foreign function interface) that is used to call native 
extensions from Ruby. This would be the only realistically usable 
interface between D and Ruby. And guess which ABI it uses...

Direct interfacing is more practicable for compiled languages. 
And indeed, there is already surprisingly good support for 
linking against C++, which is currently being extended even more. 
There's also Objective-C, for which there's this proposal [1]. I 
believe Jacob Carlborg even has a proof-of-concept 
implementation. But beyond these languages (and C, of course), 
there aren't even many natively compiled languages in widespread 
use. Pascal? The newer ones like Go and Rust? I guess for those 
there just isn't a great need.

[1] http://wiki.dlang.org/DIP43
Aug 25 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 25/08/14 22:34, "Marc Schütz" <schuetzm gmx.net>" wrote:


 Take Ruby as an example: Ruby is a scripting language that cannot be
 compiled to native code (though it can be JITted).


Both MacRuby [1] and RubyMotion [2] supports ahead of time compilation. 
I'm pretty sure Apple doesn't allow anything else than compiled 
languages on iOS. Except for HTML and JavaScript.

[1] http://macruby.org/
[2] http://www.rubymotion.com/

-- 
/Jacob Carlborg
Aug 25 2014
next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 26 August 2014 at 06:44:52 UTC, Jacob Carlborg wrote:

 Both MacRuby [1] and RubyMotion [2] supports ahead of time 
 compilation. I'm pretty sure Apple doesn't allow anything else 
 than compiled languages on iOS. Except for HTML and JavaScript.


I think Apple changed this policy in 2010, but IIRC they don't 
allow execution of downloaded scripts outside a webview.
Aug 26 2014
parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 26/08/2014 10:59, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" a écrit :

 On Tuesday, 26 August 2014 at 06:44:52 UTC, Jacob Carlborg wrote:

 Both MacRuby [1] and RubyMotion [2] supports ahead of time
 compilation. I'm pretty sure Apple doesn't allow anything else than
 compiled languages on iOS. Except for HTML and JavaScript.


 I think Apple changed this policy in 2010, but IIRC they don't allow
 execution of downloaded scripts outside a webview.


JIT is always no authorized.
Sep 03 2014
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 3 September 2014 at 23:04:13 UTC, Xavier Bigand 
wrote:

 JIT is always no authorized.


Not accurate. You are allowed to use Nitro.
Sep 03 2014
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 26 August 2014 at 06:44:52 UTC, Jacob Carlborg wrote:

 On 25/08/14 22:34, "Marc Schütz" <schuetzm gmx.net>" wrote:


 Take Ruby as an example: Ruby is a scripting language that 
 cannot be
 compiled to native code (though it can be JITted).


 Both MacRuby [1] and RubyMotion [2] supports ahead of time 
 compilation. I'm pretty sure Apple doesn't allow anything else 
 than compiled languages on iOS. Except for HTML and JavaScript.

 [1] http://macruby.org/
 [2] http://www.rubymotion.com/


I've heard about them, but I don't know any details. Anyway, the 
way Ruby works, they either need to restrict the language, 
include a full interpreter/jitter, or at most they could only 
translate it to a very abstract level. Like for an expression 
like `a + b` generating a call to a helper function like 
`send(_var_a, SYMBOL(":+"), _var_b);`, and hoping that the 
compiler backend can eliminate this in some cases.

But this doesn't really change my point: it's just another two 
ABIs, and their probably not meant to be public interfaces, to 
let the implementers keep some freedom for future developments.
Aug 26 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 26/08/14 13:40, "Marc Schütz" <schuetzm gmx.net>" wrote:


 I've heard about them, but I don't know any details. Anyway, the way
 Ruby works, they either need to restrict the language, include a full
 interpreter/jitter, or at most they could only translate it to a very
 abstract level. Like for an expression like `a + b` generating a call to
 a helper function like `send(_var_a, SYMBOL(":+"), _var_b);`, and hoping
 that the compiler backend can eliminate this in some cases.


It's implemented on top of Objective-C.

Yeah, it needs to be able to interpret or JIT, otherwise a lot of meta 
programming and things like "eval" won't work. Although, I'm not sure 
about RubyMotion. I don't think you can use regular Ruby gems out of the 
box with RubyMotion.

-- 
/Jacob Carlborg
Aug 26 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 26 August 2014 at 14:46:56 UTC, Jacob Carlborg wrote:

 On 26/08/14 13:40, "Marc Schütz" <schuetzm gmx.net>" wrote:


 I've heard about them, but I don't know any details. Anyway, 
 the way
 Ruby works, they either need to restrict the language, include 
 a full
 interpreter/jitter, or at most they could only translate it to 
 a very
 abstract level. Like for an expression like `a + b` generating 
 a call to
 a helper function like `send(_var_a, SYMBOL(":+"), _var_b);`, 
 and hoping
 that the compiler backend can eliminate this in some cases.


 It's implemented on top of Objective-C.


This makes sense, Objective-C can cope with classes that change 
at runtime. Singleton methods are probably more challenging, 
though...
Aug 26 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 25 August 2014 at 19:26:59 UTC, Brost wrote:

 Protocols are basically the way to "standardize" or describe
 something that can be implemented in many different way; I know
 that if you start from the ABI or the compiled object you are
 basically going nowhere because you are focusing on
 implementation specific details and limitations.
 That's what I was expecting, something like a protocol .


Ok, but formally a protocol is just a structured technique for 
exchange between entities.

So the D protocol is the means to obtain typeinfo at compile 
time. Then you can translate the "D typeinfo protocol" into 
another protocol of your own choice?

It seems to me that all you need is a list of types you want to 
export. From this you should be able to write your own protocol 
by annotating members you want to export with user defined 
attributes?

http://dlang.org/expression.html#typeidexpression


http://dlang.org/traits.html#allMembers
http://dlang.org/traits.html#getMember
http://dlang.org/traits.html#getAttributes
…

Seems to be a workable solution to me.

Ola.
Aug 25 2014
prev sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 25/08/2014 09:18, Brost a écrit :

 On Monday, 25 August 2014 at 06:44:25 UTC, Jacob Carlborg wrote:

 On 25/08/14 08:18, Brost wrote:


 Is D different in this regard ?


 In addition to being ABI compatible with C, D is also compatible with
 C++ [1]. D recently got support for C++ templates and namespaces.
 Hopefully it will get support for Objective-C as well, in the not too
 far away future.

 [1] http://dlang.org/cpp_interface.html


 thanks, but unfortunately neither of those is a scripting
 language, I can't see any facility in the standard D library or
 in the language that can help me retain the type safety, the
 memory management, and all the other things I talked about while
 interfacing with another language .


You just can't introduce type safety to a scripting language without 
modifying the VM implementation, you may search if there is some fork of 
the library doing that.



 for example how do you correctly expose a given D data structure
 to a scripting language of your choice ?


In DQuick we expose D object to the Lua VM by using the Lua C library.
Its API allow us to expose D structures or classes directly.

If the API of the VM of your scripting language is written in C, it's 
almost the same as if you write a C++ application.
Sep 03 2014