www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Is it possible to use DMD as a library to compile strings at runtime?

reply Saurabh Das <saurabh.das gmail.com> writes:
I see that DUB has DMD as a library package, but I was not able 
to understand how to use it.

Is it possible to use DMD as a library within a D program to 
compile a string to machine code and run the compiled code at 
runtime?

Thanks,
Saurabh
Jan 31 2020
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Friday, 31 January 2020 at 11:19:37 UTC, Saurabh Das wrote:

 I see that DUB has DMD as a library package, but I was not able 
 to understand how to use it.

 Is it possible to use DMD as a library within a D program to 
 compile a string to machine code and run the compiled code at 
 runtime?

 Thanks,
 Saurabh


Fundamentally DMD as a library is a front-end. Jitting is to the 
backend side.
You'll be able to lex and parse the source to get an AST, to 
perform the semantic passes on this AST and that's all.

Then to run this code you would need to make an AST visitor that 
will generate the binary code to execute. Even using a 
specialized library with jitting abilities, such as LLVM-d [1] or 
libfirm-d [2], this would be *quite* a journey.

[1] https://github.com/MoritzMaxeiner/llvm-d
[2] https://gitlab.com/basile.b/libfirm-d
Jan 31 2020
next sibling parent Saurabh Das <saurabh.das gmail.com> writes:
On Friday, 31 January 2020 at 14:25:30 UTC, Basile B. wrote:

 On Friday, 31 January 2020 at 11:19:37 UTC, Saurabh Das wrote:

 [...]


 Fundamentally DMD as a library is a front-end. Jitting is to 
 the backend side.
 You'll be able to lex and parse the source to get an AST, to 
 perform the semantic passes on this AST and that's all.

 Then to run this code you would need to make an AST visitor 
 that will generate the binary code to execute. Even using a 
 specialized library with jitting abilities, such as LLVM-d [1] 
 or libfirm-d [2], this would be *quite* a journey.

 [1] https://github.com/MoritzMaxeiner/llvm-d
 [2] https://gitlab.com/basile.b/libfirm-d


Thank you. That's enough to get me started tinkering!

Saurabh
Jan 31 2020
prev sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Friday, 31 January 2020 at 14:25:30 UTC, Basile B. wrote:

 On Friday, 31 January 2020 at 11:19:37 UTC, Saurabh Das wrote:

 I see that DUB has DMD as a library package, but I was not 
 able to understand how to use it.

 Is it possible to use DMD as a library within a D program to 
 compile a string to machine code and run the compiled code at 
 runtime?

 Thanks,
 Saurabh


 Fundamentally DMD as a library is a front-end. Jitting is to 
 the backend side.
 You'll be able to lex and parse the source to get an AST, to 
 perform the semantic passes on this AST and that's all.

 Then to run this code you would need to make an AST visitor 
 that will generate the binary code to execute. Even using a 
 specialized library with jitting abilities, such as LLVM-d [1] 
 or libfirm-d [2], this would be *quite* a journey.

 [1] https://github.com/MoritzMaxeiner/llvm-d
 [2] https://gitlab.com/basile.b/libfirm-d


about [1] (llvm) I've made a better binding this weekend:

https://gitlab.com/basile.b/llvmd-d

Seriouly I cant believe that at some point in the past I 
translated by hand. dstep can handle big C libraries.
Feb 10 2020
parent Basile B. <b2.temp gmx.com> writes:
On Monday, 10 February 2020 at 12:31:03 UTC, Basile B. wrote:

 On Friday, 31 January 2020 at 14:25:30 UTC, Basile B. wrote:

 [...]


 about [1] (llvm) I've made a better binding this weekend:

 https://gitlab.com/basile.b/llvmd-d

 Seriouly I cant believe that at some point in the past I 
 translated by hand. dstep can handle big C libraries.


ah ah ah, I meant

https://gitlab.com/basile.b/llvm-d
Feb 10 2020
prev sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Friday, 31 January 2020 at 11:19:37 UTC, Saurabh Das wrote:

 I see that DUB has DMD as a library package, but I was not able 
 to understand how to use it.

 Is it possible to use DMD as a library within a D program to 
 compile a string to machine code and run the compiled code at 
 runtime?

 Thanks,
 Saurabh


Another approach:
- include the dmd compiler package with your application
- within your app call the compiler executable and compile the 
source code to a dll / so
- call the dll / so function

Maybe you will get some trouble with AV software with this 
approach...

Kind regards
Andre
Feb 01 2020
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Feb 01, 2020 at 08:01:34PM +0000, Andre Pany via Digitalmars-d-learn
wrote:
[...]

 Another approach:
 - include the dmd compiler package with your application
 - within your app call the compiler executable and compile the source
 code to a dll / so
 - call the dll / so function


[...]

I've actually done this before in an equation grapher program: the user
inputs an equation, the program generates D code to compute the
equation, then runs dmd to compile it into a shared library, and opens
the shared library and looks up the symbol to execute the compiled code.
Dmd is fast enough that this actually works fairly well. When the input
to dmd is small, it's so fast you don't even notice it.


T

-- 
An imaginary friend squared is a real enemy.
Feb 01 2020
parent reply Saurabh Das <saurabh.das gmail.com> writes:
On Saturday, 1 February 2020 at 20:37:03 UTC, H. S. Teoh wrote:

 On Sat, Feb 01, 2020 at 08:01:34PM +0000, Andre Pany via 
 Digitalmars-d-learn wrote: [...]

 Another approach:
 - include the dmd compiler package with your application
 - within your app call the compiler executable and compile the 
 source
 code to a dll / so
 - call the dll / so function


 [...]

 I've actually done this before in an equation grapher program: 
 the user inputs an equation, the program generates D code to 
 compute the equation, then runs dmd to compile it into a shared 
 library, and opens the shared library and looks up the symbol 
 to execute the compiled code. Dmd is fast enough that this 
 actually works fairly well. When the input to dmd is small, 
 it's so fast you don't even notice it.


 T


This approach seems more tractable at present. Would you have any 
example code lying around for this?

Saurabh
Feb 01 2020
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Feb 02, 2020 at 03:16:46AM +0000, Saurabh Das via Digitalmars-d-learn
wrote:

 On Saturday, 1 February 2020 at 20:37:03 UTC, H. S. Teoh wrote:


[...]

 I've actually done this before in an equation grapher program: the
 user inputs an equation, the program generates D code to compute the
 equation, then runs dmd to compile it into a shared library, and
 opens the shared library and looks up the symbol to execute the
 compiled code.  Dmd is fast enough that this actually works fairly
 well. When the input to dmd is small, it's so fast you don't even
 notice it.




[...]

 This approach seems more tractable at present. Would you have any
 example code lying around for this?


[...]

It's very simple. Let's say you have your code in some string called
'code'. Since dmd nowadays can take stdin as input (specify "-" as input
filename), all you have to do is to assemble your dmd command and use
std.process's awesome API to run it:

	/*
	 * Step 1: Compile the code
	 */
	string code = ...;
	auto cmd = [
		"/usr/bin/dmd", // or wherever your dmd is
		"-O",	// or whatever other flags you need
		"-fPIC", "-shared", // this is important
		"-of" ~ soFilename, // specify output filename
		"-"	// read from stdin
	]

	// This part is a bit involved because we have to spawn the
	// compiler as a child process then write our code string into
	// its stdin.
	// Alternatively, just write your code into a temporary file and
	// pass the filename to dmd, then you can just use
	// std.process.execute() which has a much simpler API.
        import std.process : pipeProcess, Redirect, wait;
        auto pipes = pipeProcess(cmd, Redirect.stdin | Redirect.stdout |
                                      Redirect.stderrToStdout);

	// Send code to compiler
        pipes.stdin.write(code);
        pipes.stdin.flush();
        pipes.stdin.close();

	// Read compiler output (optional)
        auto app = appender!string();
        enum chunkSize = 4096;
        pipes.stdout.byChunk(chunkSize)
                    .copy(app);

        // Wait for compiler to finish
        auto status = wait(pipes.pid);
        auto output = app.data;
        if (status != 0)
            throw new Exception("Failed to compile code:\n" ~ output);

	/*
	 * Step 2: Load the compiled library.
	 */
	// This is for Posix; replace with Windows equivalent if you're
	// on Windows.
	auto libhandle = dlopen(soFilename.toStringz, RTLD_LAZY | RTLD_LOCAL);
	if (libhandle is null) ... /* handle error here */

	// Look up entry point by symbol.
	string entryPoint = ...; /* symbol of library entry point */
	alias FunType = int function(string); // your function signature here
	auto funptr = cast(FunType) dlsym(libhandle, entryPoint.toStringz);

	/*
	 * Step 3: Use the compiled code.
	 */

	// Call the compiled function with whatever arguments.
	int result = funptr("my input");

	...

	// Cleanup once you're done with the library.
	dlclose(libhandle);
	std.file.remove(soFilename);


T

-- 
First Rule of History: History doesn't repeat itself -- historians merely
repeat each other.
Feb 01 2020
parent Saurabh Das <saurabh.das gmail.com> writes:
On Sunday, 2 February 2020 at 06:03:01 UTC, H. S. Teoh wrote:

 On Sun, Feb 02, 2020 at 03:16:46AM +0000, Saurabh Das via 
 Digitalmars-d-learn wrote:

 [...]


 [...]

 [...]




 [...]

 [...]


 [...]

 It's very simple. Let's say you have your code in some string 
 called 'code'. Since dmd nowadays can take stdin as input 
 (specify "-" as input filename), all you have to do is to 
 assemble your dmd command and use std.process's awesome API to 
 run it:

 [...]


Thanks. Will try this out.

Saurabh
Feb 03 2020