digitalmars.D - Idea to make D great again!
- PiSquared (186/186) Oct 06 2019 D has the ability to embed most languages in to it through
- Chris (3/7) Oct 08 2019 Would you have time to put that on Github with a few examples?
- PiSquared (14/22) Oct 09 2019 I don't have the code in an organized state yet to make it
- Laeeth Isharc (84/258) Oct 09 2019 Yes, and for example in C# and F# I think Roslyn could take you a
- Chris (4/8) Jan 08 2020 [snip]
D has the ability to embed most languages in to it through
scripting. There is matplotlibD already, I have modified the code
to be able to run arbitrary scripts and also work with Lua. Any
language can be used through the method of:
1. Writing a D string representing the target language's program
to a file or stdin of the interpreter.
2. Before executing above, first modify the input to replace
special tokens with D data. This allows one to insert D data in
to the target program. This allows one to use D to generate the
data which may be much faster and can be used in other ways.
[See below for an example]
These methods should work with all languages since it simply
emulates coding manually. It becomes a powerful feature because
one and leverage the target language as if one was writing
directly in the languages.
More importantly, Visual D and Visual Studio allow one to load
many target languages in to a D project and get almost full
benefit such as intelligence(debugging does not work but I
imagine it could with some work).
Furthermore, with some work, interopt can be made to work really
well by transferring data between the programs using a client
server protocol which can be done through files or global
memory(depending on the languages level of OS access).
With the proper design 1. D can be used along side and as a host
for most languages. The most common will work fine. With a bit of
work it should be work extremely well. This allows D to be
leverage and piggy back on these languages and will bring users
of those languages to D. D can be used for what it is good at and
the holes it has can be filled by some other language. 2. The
design is not specific to D as most languages have this ability,
but with D it generally can be quite easy.
The only downside is that of "marshaling" the data, this can be
reduced significantly by not using text to transfer data but by
having appropriate data marshalers that can work in memory and
relate data between D and the target language correctly.
Further more, the target language needs some capacity too send
data back to the D program.
I have created a server and a client that do this. It is used for
writing latex code in D. This allows me to also use python's
scientific plotting libraries along with tikz by using a common D
library. Tikz is slow, D+mlab is fast.
The latex code, for example, can send data back to the server
using a client proxy and this can trigger different results when
the latex file is being compiled by latex.
It is all quite simple but it allows me to debug D programs in D
without issue as all the plumbing on the target language is
relatively hidden.
I would like you guys to consider taking this proposal seriously:
Create a proper D design that allows one to leverage any target
language in a unified way. Make it robust, powerful, feature rich.
It will bring many people to use D if done properly. With a
proper IDE D could act as a centerpiece to use many languages
together.
One could use a gui in lua, a plotting library in python, and
latex dox generation.
All done rather seamless.
here is a lua target, bare bones with no fancy features and
essentially ripped from matplotlibD(which essentially does the
same with python, of which I have a version that is essentially
the same as below):
module LuaD;
import std.stdio;
enum DDataToken = `" D$`;
alias immutable bool LuaBool;
alias immutable (void*) LuaNone;
LuaBool False = false;
LuaBool True = true;
LuaNone None = null;
// A header to use for common lua files
string luaHeader = `
`;
string luaFooter = ``;
string d2lua(T)(T v)
{
int x = 3242;
import std.conv, std.traits, std.array, std.algorithm,
std.format: format;
static if (is(T : LuaNone))
return "None";
else static if (is(T : bool))
return v ? "True" : "False";
else static if (is(T : string))
return format("\"%s\"", v);
else
{
static if (isArray!T)
return v.map!(a=>to!string(a)~",").join;
return format("%s", v);
}
}
string lua_path = `C:\Lua\lua.exe`;
// executes lua on the program
void lua(string program)
{
import std.process: environment, pipeProcess, wait, Redirect;
if (!lua_path)
lua_path = environment.get("LuaD", "lua");
auto pipes = pipeProcess(lua_path, Redirect.stdin |
Redirect.stderr);
pipes.stdin.writeln(program);
pipes.stdin.writeln("exit()");
pipes.stdin.close();
auto result = wait(pipes.pid);
if (result != 0) {
string error;
foreach (line; pipes.stderr.byLine)
error ~= line ~ "\n";
throw new Exception("\n\nERROR occurred in Lua:\n" ~
error);
}
}
// converts all " $..." strings to D variables by converting
auto luaD(bool ret = false)(string program)
{
if (program.length < 5) return "";
import std.string, std.range, std.algorithm;
string r;
for(int i = 0; i < program.length; i++)
{
if (i < program.length - 5)
{
if (program[i..i+DDataToken.length] == DDataToken)
{
i += DDataToken.length;
string id;
while(i < program.length && program[i] != '"')
id ~= program[i++];
r ~= `"~d2lua(`~id~`)~"`;
continue;
}
}
if (program[i] == '"') r ~= "\\";
r ~= program[i];
}
static if (ret) { return `"`~r~`"`; }
return `lua("`~r~`");`;
}
Then a simple lua program `test.lua`:
print(" D$TestString")
one calls this program in D:
auto TestString = "This is a D string";
// Will automatically substitute TestString
mixin(luaD(import(test.lua));
and the lua program will be executed with the D data.
I have used such things for computing data in D and then plotting
them using some of pythons plotting libraries(matplotlib, mlab,
mayavi, etc).
It is not as optimal since large datasets can cause problems
since they can take up so much textual space but this can be
alleviated by writing them to binary files, or ideally, using
memory transfers.
With a good robust design D can be the centerpiece of language
hosting bringing a very large number of users. Python, for some
odd reason, has a lot of scientific plotting libraries but, as
most people know, is not very fast. Imagine using D for the
algorithms and python for the plotting. That is precisely what I
have done and it works well(debugging errors is the hardest since
no IDE but that could change). In visual D I get full syntax
coloring, intellisense, and documentation for python and lua.
languages being usable with this method. With a good "marshaling"
library it one would could have limited performance degradation.
For apps that do not have performance issues, say typical gui's
one can use a language with good gui frameworks and D for the
business end. With D's meta capabilities one could probably
export most of the bridging automatically in to the gui.
Done well it could be a boon to D and bring in many new people
and $$$. As far as I'm aware, no language or framework exists to
do this well and easily yet it is not a difficult problem.
Also it lets a person become "multi-lingual" in real time. I know
quite a few languages but rarely use them because I typically use
only one at a time unless required. But being able to
essentially choose the best language for the job would be nice.
Diet templates sort of approach the idea the same but for html.
The above approach could allow embedding D code in to other
languages too by having "code strings".
By having a universal design it could be done quite effectively.
I've already been able to do far more in D by leveraging python
than I could do otherwise... but by leveraging I mean having D
automatically transfer in to the python program rather than
having to do complicated coding.
My code is not robust as it is proof of concept and meets my
limited needs. I'd like to see a more robust framework for doing
this... I'd do it myself but I simply do not have the time to do
it all and make it as good as it should be.
Oct 06 2019
On Monday, 7 October 2019 at 00:56:51 UTC, PiSquared wrote:My code is not robust as it is proof of concept and meets my limited needs. I'd like to see a more robust framework for doing this... I'd do it myself but I simply do not have the time to do it all and make it as good as it should be.Would you have time to put that on Github with a few examples? Maybe someone could pick it up from there.
Oct 08 2019
PiSquared <PiSquared gmail.com> On Tuesday, 8 October 2019 at 10:02:08 UTC, Chris wrote:On Monday, 7 October 2019 at 00:56:51 UTC, PiSquared wrote:I don't have the code in an organized state yet to make it valuable. What I posted is pretty much what I have(I just have the python version and some server/client stuff but is part of something separate and not finished). I might get around to it at some point but it will be slow. You should be able to take that code, and on windows at least, get it to work pretty easily. My main goal would be to be able to optimize memory transfer and enable some type of bi-directional interaction that is very simple but fast and robust. I just haven't though about it enough. I don't seem much interest in it since you are the only one who responded so I guess the D community doesn't really care about such things.My code is not robust as it is proof of concept and meets my limited needs. I'd like to see a more robust framework for doing this... I'd do it myself but I simply do not have the time to do it all and make it as good as it should be.Would you have time to put that on Github with a few examples? Maybe someone could pick it up from there.
Oct 09 2019
On Monday, 7 October 2019 at 00:56:51 UTC, PiSquared wrote:
D has the ability to embed most languages in to it through
scripting. There is matplotlibD already, I have modified the
code to be able to run arbitrary scripts and also work with
Lua. Any language can be used through the method of:
1. Writing a D string representing the target language's
program to a file or stdin of the interpreter.
2. Before executing above, first modify the input to replace
special tokens with D data. This allows one to insert D data in
to the target program. This allows one to use D to generate the
data which may be much faster and can be used in other ways.
[See below for an example]
These methods should work with all languages since it simply
emulates coding manually. It becomes a powerful feature because
one and leverage the target language as if one was writing
directly in the languages.
More importantly, Visual D and Visual Studio allow one to load
many target languages in to a D project and get almost full
benefit such as intelligence(debugging does not work but I
imagine it could with some work).
Furthermore, with some work, interopt can be made to work
really well by transferring data between the programs using a
client server protocol which can be done through files or
global memory(depending on the languages level of OS access).
With the proper design 1. D can be used along side and as a
host for most languages. The most common will work fine. With a
bit of work it should be work extremely well. This allows D to
be leverage and piggy back on these languages and will bring
users of those languages to D. D can be used for what it is
good at and the holes it has can be filled by some other
language. 2. The design is not specific to D as most languages
have this ability, but with D it generally can be quite easy.
The only downside is that of "marshaling" the data, this can be
reduced significantly by not using text to transfer data but by
having appropriate data marshalers that can work in memory and
relate data between D and the target language correctly.
Further more, the target language needs some capacity too send
data back to the D program.
I have created a server and a client that do this. It is used
for writing latex code in D. This allows me to also use
python's scientific plotting libraries along with tikz by using
a common D library. Tikz is slow, D+mlab is fast.
The latex code, for example, can send data back to the server
using a client proxy and this can trigger different results
when the latex file is being compiled by latex.
It is all quite simple but it allows me to debug D programs in
D without issue as all the plumbing on the target language is
relatively hidden.
I would like you guys to consider taking this proposal
seriously: Create a proper D design that allows one to leverage
any target language in a unified way. Make it robust, powerful,
feature rich.
It will bring many people to use D if done properly. With a
proper IDE D could act as a centerpiece to use many languages
together.
One could use a gui in lua, a plotting library in python, and
latex dox generation.
All done rather seamless.
here is a lua target, bare bones with no fancy features and
essentially ripped from matplotlibD(which essentially does the
same with python, of which I have a version that is essentially
the same as below):
module LuaD;
import std.stdio;
enum DDataToken = `" D$`;
alias immutable bool LuaBool;
alias immutable (void*) LuaNone;
LuaBool False = false;
LuaBool True = true;
LuaNone None = null;
// A header to use for common lua files
string luaHeader = `
`;
string luaFooter = ``;
string d2lua(T)(T v)
{
int x = 3242;
import std.conv, std.traits, std.array, std.algorithm,
std.format: format;
static if (is(T : LuaNone))
return "None";
else static if (is(T : bool))
return v ? "True" : "False";
else static if (is(T : string))
return format("\"%s\"", v);
else
{
static if (isArray!T)
return v.map!(a=>to!string(a)~",").join;
return format("%s", v);
}
}
string lua_path = `C:\Lua\lua.exe`;
// executes lua on the program
void lua(string program)
{
import std.process: environment, pipeProcess, wait,
Redirect;
if (!lua_path)
lua_path = environment.get("LuaD", "lua");
auto pipes = pipeProcess(lua_path, Redirect.stdin |
Redirect.stderr);
pipes.stdin.writeln(program);
pipes.stdin.writeln("exit()");
pipes.stdin.close();
auto result = wait(pipes.pid);
if (result != 0) {
string error;
foreach (line; pipes.stderr.byLine)
error ~= line ~ "\n";
throw new Exception("\n\nERROR occurred in Lua:\n" ~
error);
}
}
// converts all " $..." strings to D variables by converting
auto luaD(bool ret = false)(string program)
{
if (program.length < 5) return "";
import std.string, std.range, std.algorithm;
string r;
for(int i = 0; i < program.length; i++)
{
if (i < program.length - 5)
{
if (program[i..i+DDataToken.length] == DDataToken)
{
i += DDataToken.length;
string id;
while(i < program.length && program[i] != '"')
id ~= program[i++];
r ~= `"~d2lua(`~id~`)~"`;
continue;
}
}
if (program[i] == '"') r ~= "\\";
r ~= program[i];
}
static if (ret) { return `"`~r~`"`; }
return `lua("`~r~`");`;
}
Then a simple lua program `test.lua`:
print(" D$TestString")
one calls this program in D:
auto TestString = "This is a D string";
// Will automatically substitute TestString
mixin(luaD(import(test.lua));
and the lua program will be executed with the D data.
I have used such things for computing data in D and then
plotting them using some of pythons plotting
libraries(matplotlib, mlab, mayavi, etc).
It is not as optimal since large datasets can cause problems
since they can take up so much textual space but this can be
alleviated by writing them to binary files, or ideally, using
memory transfers.
With a good robust design D can be the centerpiece of language
hosting bringing a very large number of users. Python, for some
odd reason, has a lot of scientific plotting libraries but, as
most people know, is not very fast. Imagine using D for the
algorithms and python for the plotting. That is precisely what
I have done and it works well(debugging errors is the hardest
since no IDE but that could change). In visual D I get full
syntax coloring, intellisense, and documentation for python and
lua.
languages being usable with this method. With a good
"marshaling" library it one would could have limited
performance degradation. For apps that do not have performance
issues, say typical gui's one can use a language with good gui
frameworks and D for the business end. With D's meta
capabilities one could probably export most of the bridging
automatically in to the gui.
Done well it could be a boon to D and bring in many new people
and $$$. As far as I'm aware, no language or framework exists
to do this well and easily yet it is not a difficult problem.
long way since you can compile dotnet code at D runtime.
As is well known, C++ is an interpreted language where you can
instantiate templates at runtime.
I'm joking, but it can be if you want it to be and CERN do this
at considerable scale with Cling, which allows you to instantiate
templates at runtime and introspect, create instance of classes
and call them. Some interesting early results in calling cling
from D, but I was waiting till things were a bit more polished to
talk about them. The next stage might be to run cling on clang
to help with dpp development. Libclang looks amazing at first
sight but it is missing a lot.
I think there are two reasons why interoperability matters. One
is external libraries but another is that when you are exploring
using D then it's often a bit challenging if you have C++ code
and need to use it or implement part of a new feature in D. If
replacing working code you don't ideally want to do a complete
rewrite but replace small pieces at a time and have working code
all along the way. That might not be easy in many cases.
I think RoI is often a less important barrier than limits to
upfront pain. If you use D you are already an unusual sort of
person because the discomfort is front loaded and how do you know
it will be worth it to persevere? Atila downloaded the compiler
and back then it segfaulted when he tried to compile hello world!
Most reasonable people would have given up then.
But the dollar return on being reasonable is time varying and for
some people at some times it's not necessarily a bad thing to
hire people who are a bit unreasonable. Perhaps all progress
depends on people not being reasonable.
So if it becomes easier to use libraries and internal code
written in other languages then it's easy to make small
beginnings. Lots of large projects begin as little beginnings
and so making it easy to start work has value for the ecosystem I
think.
I agree that sometimes you don't need to do things that are that
clever to be quite productive. But one probably can do a bit
better than what you suggest even if for any individual it's more
than good enough.
It's almost easy to use python, C, Lua,R libraries from D but not
quite. Pyd is showing its age right now and LuaD isn't that well
maintained. It's not super easy to find embedr and it's not
trivially easy to get up and running.
I think lots of that is documentation, maintenance,small
improvements and blog posts.
Dpp is pretty solid a lot of the time. On Linux anyway but on
Windows not yet, judging by a conversation I had yesterday. I
asked Adam Ruppe to look into that and I think in time it will
just work for what it works for. It's a shame it doesn't
dependably yet at least handle Linux headers without choking. I
think that's an SAoC project if I recall. We could do a bit more
to programmatically just try wrapping every header we can get our
hands on and then try to programmatically analyse the results.
Ultimately there is a limit with macros, maybe with using
libclang and maybe how far you can get with generating strings in
D rather than AST to AST transformation. The latter for C isn't
a trivial project - DoD spent millions on C2Rust though on the
other hand they are not going to be able to do things on the
cheap. C to D ought to be quite doable though. Our focus has
been more on the cpp side for now but in time could do a bit more
with C.
With PyD under autowrap there is a project called PyNIH. Steady
progress with focus first on Python calling D but that helps with
the reverse.
do so via Mono also. It's not difficult work technically but
just a bit of a grind.
We did some very basic work on embedding Julia in D. That also
isn't difficult but time is limited.
The basic idea is to use Symmetry Integration Language as a
substrate where you can write plugins in any sensible language
and call it from any sensible language for us. We have to solve
our own problems first but in the process of doing so probably it
will help others.
One more thing - work Sebastian Koppe did shows a proof of
concept of D compiling to web assembly and using Javascript
libraries. He will generate the wrapping code from Typescript
bindings in time.
From work I did before I don't think embedding Java would be so
difficult but we just don't use Java that much.
So I think the biggest constraint right now is the ability to
others for other people to submit improvements. But in time I
think interoperability will just keep getting better.
Oct 09 2019
On Monday, 7 October 2019 at 00:56:51 UTC, PiSquared wrote:D has the ability to embed most languages in to it through scripting. There is matplotlibD already, I have modified the code to be able to run arbitrary scripts and also work with Lua. Any language can be used through the method of:[snip] Maybe something along these lines (jni.d): https://forum.dlang.org/post/cqdqhrporufpzkwjwkrw forum.dlang.org
Jan 08 2020