www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DMD front-end can be used as a library with Dub

reply Alexander Breckel <example localhost.com> writes:
I just tried to use the DMD frontend as a Dub package in my own 
project to parse and analyze D files and it ... just worked. The 
dub.json for dmd is fairly small and doesn't require any changes 
to the code. This might be common knowledge, but I was completely 
unprepared for this :)

Please note: This is "only" the dmd frontend (lexer, parser, 
semantic passes, CTFE). Code generation will be more complicated.

The following dub pre-generation hooks were necessary:

- run src/idgen.d to generate src/id.d
- create verstr.h and SYSCONFDIR.imp
- create and link source/ddmd/ to src/

The last point makes ddmd modules reside in its correct package. 
I'm using a symbolic link for this, which is the only reason this 
approach is currently limited to Linux. In the long run, I think 
the cleanest way would be to place all ddmd files in src/ddmd/ 
and just leave mars.d and things like idgen.d in the main src/ 
directory.

For demonstration purposes, here is a dub package to play around 
with:
http://code.dlang.org/packages/ddmd-experimental

Here is the dub.json that was used:
https://github.com/aneas/ddmd-experimental/blob/master/dub.json

And here is an example program using this API, have fun!
(run with $ chmod +x file.d; ./file.d)
---
#!/usr/bin/env dub
/+ dub.sdl:
name "ddmd-example"
dependency "ddmd-experimental" version="~>2.71.2-b2dub"
+/

import std.path;
import std.stdio;
import std.string;
import ddmd.builtin;
import ddmd.dmodule;
import ddmd.dclass;
import ddmd.dsymbol;
import ddmd.expression;
import ddmd.func;
import ddmd.globals;
import ddmd.id;
import ddmd.identifier;
import ddmd.mtype;
import ddmd.visitor;

void main(string[] args) {
	if(args.length != 2) {
		writeln("prints top-level function and class declarations");
		writeln("usage: ", args[0].baseName, " d-filepath");
		return;
	}

	// Initialization
	global._init();
	global.params.isLinux = true;
	Type._init();
	Id.initialize();
	Module._init();
	Expression._init();
	builtin_init();

	// Read and parse specified module
	auto id = Identifier.idPool(args[1].baseName.stripExtension);
	auto m = new Module(args[1].toStringz, id, false, false);
	m.read(Loc());
	m.parse();

	// Output
	m.accept(new class Visitor {
		extern(C++):
		alias visit = Visitor.visit;
		override void visit(Module m) {
			foreach(i; 0 .. m.members.dim)
				(*m.members)[i].accept(this);
		}
		override void visit(Dsymbol s) {
		}
		override void visit(FuncDeclaration fd) {
			writeln("function ", fd.ident.toString);
		}
		override void visit(ClassDeclaration cd) {
			writeln("class ", cd.ident.toString);
		}
	});
}
Aug 29 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel 
wrote:
 I just tried to use the DMD frontend as a Dub package in my own 
 project to parse and analyze D files and it ... just worked. 
 The dub.json for dmd is fairly small and doesn't require any 
 changes to the code. This might be common knowledge, but I was 
 completely unprepared for this :)

 Please note: This is "only" the dmd frontend (lexer, parser, 
 semantic passes, CTFE). Code generation will be more 
 complicated.
Thx, that's interesting from an IDE developer perspective. However since DMD is a "single shot" program, using it as library could introduce memory problems. Also string handling is C style, unlike libdparse, which is truly written in D. However libdparse does nothing more than producing the AST, which becomes problematic with "static if" expressions, mixins, etc.
Aug 29 2016
parent Alexander Breckel <example localhost.com> writes:
On Monday, 29 August 2016 at 11:27:44 UTC, Basile B. wrote:
 On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel 
 wrote:
 I just tried to use the DMD frontend as a Dub package in my 
 own project to parse and analyze D files and it ... just 
 worked. The dub.json for dmd is fairly small and doesn't 
 require any changes to the code. This might be common 
 knowledge, but I was completely unprepared for this :)

 Please note: This is "only" the dmd frontend (lexer, parser, 
 semantic passes, CTFE). Code generation will be more 
 complicated.
Thx, that's interesting from an IDE developer perspective. However since DMD is a "single shot" program, using it as library could introduce memory problems. Also string handling is C style, unlike libdparse, which is truly written in D. However libdparse does nothing more than producing the AST, which becomes problematic with "static if" expressions, mixins, etc.
Right. With these limitations, it might also be useful for writing static analysis tools, coding guidelines checkers, ... If someone wants to write a tool that tells me which functions and methods are not reachable from main(), or could be declared pure nothrow nogc, i would be more than happy to use it :)
Aug 29 2016
prev sibling next sibling parent reply Seb <seb wilzba.ch> writes:
On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel 
wrote:
 I just tried to use the DMD frontend as a Dub package in my own 
 project to parse and analyze D files and it ... just worked. 
 The dub.json for dmd is fairly small and doesn't require any 
 changes to the code. This might be common knowledge, but I was 
 completely unprepared for this :)

 [...]
Nice :) How about submitting this as PR? Then you get updates for free and more people can work on tweaking the DUB file.
Aug 29 2016
parent reply Alexander Breckel <example localhost.com> writes:
On Monday, 29 August 2016 at 11:31:58 UTC, Seb wrote:
 On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel 
 wrote:
 I just tried to use the DMD frontend as a Dub package in my 
 own project to parse and analyze D files and it ... just 
 worked. The dub.json for dmd is fairly small and doesn't 
 require any changes to the code. This might be common 
 knowledge, but I was completely unprepared for this :)

 [...]
Nice :) How about submitting this as PR? Then you get updates for free and more people can work on tweaking the DUB file.
I'm afraid this is not yet ready for upstream. Dmd would first have to move all ddmd files into src/ddmd/ to have a clean package namespace. And someone smarter than me should probably adjust the dub.json to somehow emphasize that the library contains only the front-end.
Aug 29 2016
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 29/08/2016 11:54 PM, Alexander Breckel wrote:
 On Monday, 29 August 2016 at 11:31:58 UTC, Seb wrote:
 On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:
 I just tried to use the DMD frontend as a Dub package in my own
 project to parse and analyze D files and it ... just worked. The
 dub.json for dmd is fairly small and doesn't require any changes to
 the code. This might be common knowledge, but I was completely
 unprepared for this :)

 [...]
Nice :) How about submitting this as PR? Then you get updates for free and more people can work on tweaking the DUB file.
I'm afraid this is not yet ready for upstream. Dmd would first have to move all ddmd files into src/ddmd/ to have a clean package namespace. And someone smarter than me should probably adjust the dub.json to somehow emphasize that the library contains only the front-end.
With the help of sourceFiles you should be fine. You will also want sourcePaths to be set to [""] to ensure it doesn't have anything e.g. src in it. I would recommend you have dub pregenerate some import files and add them via importPaths.
Aug 29 2016
prev sibling next sibling parent Rory McGuire via Digitalmars-d <digitalmars-d puremagic.com> writes:
I didn't see your announcement, but... AWESOME!!

This could be the basis for some really good tutorials on making compiler
backends etc...

We need more little teaser examples like the one you posted in the
beginning of this thread.

PS: I did already check the code out on github because I watch
code.dlang.org (a lot).

Thanks!



On Mon, Aug 29, 2016 at 12:42 PM, Alexander Breckel via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 I just tried to use the DMD frontend as a Dub package in my own project to
 parse and analyze D files and it ... just worked. The dub.json for dmd is
 fairly small and doesn't require any changes to the code. This might be
 common knowledge, but I was completely unprepared for this :)

 Please note: This is "only" the dmd frontend (lexer, parser, semantic
 passes, CTFE). Code generation will be more complicated.

 The following dub pre-generation hooks were necessary:

 - run src/idgen.d to generate src/id.d
 - create verstr.h and SYSCONFDIR.imp
 - create and link source/ddmd/ to src/

 The last point makes ddmd modules reside in its correct package. I'm using
 a symbolic link for this, which is the only reason this approach is
 currently limited to Linux. In the long run, I think the cleanest way would
 be to place all ddmd files in src/ddmd/ and just leave mars.d and things
 like idgen.d in the main src/ directory.

 For demonstration purposes, here is a dub package to play around with:
 http://code.dlang.org/packages/ddmd-experimental

 Here is the dub.json that was used:
 https://github.com/aneas/ddmd-experimental/blob/master/dub.json

 And here is an example program using this API, have fun!
 (run with $ chmod +x file.d; ./file.d)
 ---
 #!/usr/bin/env dub
 /+ dub.sdl:
 name "ddmd-example"
 dependency "ddmd-experimental" version="~>2.71.2-b2dub"
 +/

 import std.path;
 import std.stdio;
 import std.string;
 import ddmd.builtin;
 import ddmd.dmodule;
 import ddmd.dclass;
 import ddmd.dsymbol;
 import ddmd.expression;
 import ddmd.func;
 import ddmd.globals;
 import ddmd.id;
 import ddmd.identifier;
 import ddmd.mtype;
 import ddmd.visitor;

 void main(string[] args) {
         if(args.length != 2) {
                 writeln("prints top-level function and class
 declarations");
                 writeln("usage: ", args[0].baseName, " d-filepath");
                 return;
         }

         // Initialization
         global._init();
         global.params.isLinux = true;
         Type._init();
         Id.initialize();
         Module._init();
         Expression._init();
         builtin_init();

         // Read and parse specified module
         auto id = Identifier.idPool(args[1].baseName.stripExtension);
         auto m = new Module(args[1].toStringz, id, false, false);
         m.read(Loc());
         m.parse();

         // Output
         m.accept(new class Visitor {
                 extern(C++):
                 alias visit = Visitor.visit;
                 override void visit(Module m) {
                         foreach(i; 0 .. m.members.dim)
                                 (*m.members)[i].accept(this);
                 }
                 override void visit(Dsymbol s) {
                 }
                 override void visit(FuncDeclaration fd) {
                         writeln("function ", fd.ident.toString);
                 }
                 override void visit(ClassDeclaration cd) {
                         writeln("class ", cd.ident.toString);
                 }
         });
 }
Sep 01 2016
prev sibling next sibling parent reply Cauterite <cauterite gmail.com> writes:
On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel 
wrote:

Because of the poor memory management in the compiler, I included 
a modified GC-stub when I compiled the frontend as a library, so 
that it can be used in long-running processes:

https://gist.github.com/Cauterite/4eb74347aea040f5d371fb49054e1819

You can call gc_annihilate(true) to delete the entire heap.

Obviously you need to keep the library in a separate DLL with its 
own runtime, and carefully avoid holding references to GC memory 
across annihilations.

This technique is working pretty smoothly for me so far.

Also to compile it as a DLL you either have to remove main() from 
mars.d or play games with the C runtime: 
https://gist.github.com/Cauterite/b190e62891c773703d0de3a1d99df362
Sep 01 2016
parent reply Rory McGuire via Digitalmars-d <digitalmars-d puremagic.com> writes:
Thanks, for the GC stub, that will be great for playing with whether or not
a little dmd app crashes after gc_annihilate(true).

Did I understand that right?

R


On Thu, Sep 1, 2016 at 6:16 PM, Cauterite via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:


 Because of the poor memory management in the compiler, I included a
 modified GC-stub when I compiled the frontend as a library, so that it can
 be used in long-running processes:

 https://gist.github.com/Cauterite/4eb74347aea040f5d371fb49054e1819

 You can call gc_annihilate(true) to delete the entire heap.

 Obviously you need to keep the library in a separate DLL with its own
 runtime, and carefully avoid holding references to GC memory across
 annihilations.

 This technique is working pretty smoothly for me so far.

 Also to compile it as a DLL you either have to remove main() from mars.d
 or play games with the C runtime: https://gist.github.com/Cauter
 ite/b190e62891c773703d0de3a1d99df362
Sep 01 2016
parent Chris Wright <dhasenan gmail.com> writes:
On Thu, 01 Sep 2016 18:42:13 +0200, Rory McGuire via Digitalmars-d wrote:

 Thanks, for the GC stub, that will be great for playing with whether or
 not a little dmd app crashes after gc_annihilate(true).
 
 Did I understand that right?
It sounds like this there's a special function `gc_annihilate` that immediately frees all memory that libdmd has allocated. It doesn't do any reference tracking or mark/sweep. libdmd disables the collector as dmdfe does. You need to keep dmd in its own DLL. Otherwise, symbols between dmd and your application will be deduplicated or something and either you get a linker error or an arbitrary implementation is selected. Another safe way of working is to use the GC as usual, but before you call into libdmd, you call GC.disable, and you wait until you've finished using the data structures it returns before calling GC.enable.
Sep 01 2016
prev sibling parent Mathias Lang <mathias.lang sociomantic.com> writes:
On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel 
wrote:
 Please note: This is "only" the dmd frontend (lexer, parser, 
 semantic passes, CTFE). Code generation will be more 
 complicated.
Interesting. Did you have inline ASM in your code ? From my experience it required the backend to work, and couldn't be mocked, because else some flow analysis would go wrong (e.g. it would complain about missing `return`).
Sep 19 2016