digitalmars.D - DMD front-end can be used as a library with Dub
- Alexander Breckel (81/81) Aug 29 2016 I just tried to use the DMD frontend as a Dub package in my own
- Basile B. (8/16) Aug 29 2016 Thx, that's interesting from an IDE developer perspective.
- Alexander Breckel (6/24) Aug 29 2016 Right. With these limitations, it might also be useful for
- Seb (6/12) Aug 29 2016 Nice :)
- Alexander Breckel (6/19) Aug 29 2016 I'm afraid this is not yet ready for upstream. Dmd would first
- rikki cattermole (6/24) Aug 29 2016 With the help of sourceFiles you should be fine.
- Rory McGuire via Digitalmars-d (10/89) Sep 01 2016 I didn't see your announcement, but... AWESOME!!
- Cauterite (14/14) Sep 01 2016 On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel
- Rory McGuire via Digitalmars-d (6/21) Sep 01 2016 Thanks, for the GC stub, that will be great for playing with whether or ...
- Chris Wright (11/15) Sep 01 2016 It sounds like this there's a special function `gc_annihilate` that
- Mathias Lang (6/9) Sep 19 2016 Interesting. Did you have inline ASM in your code ? From my
I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :) Please note: This is "only" the dmd frontend (lexer, parser, semantic passes, CTFE). Code generation will be more complicated. The following dub pre-generation hooks were necessary: - run src/idgen.d to generate src/id.d - create verstr.h and SYSCONFDIR.imp - create and link source/ddmd/ to src/ The last point makes ddmd modules reside in its correct package. I'm using a symbolic link for this, which is the only reason this approach is currently limited to Linux. In the long run, I think the cleanest way would be to place all ddmd files in src/ddmd/ and just leave mars.d and things like idgen.d in the main src/ directory. For demonstration purposes, here is a dub package to play around with: http://code.dlang.org/packages/ddmd-experimental Here is the dub.json that was used: https://github.com/aneas/ddmd-experimental/blob/master/dub.json And here is an example program using this API, have fun! (run with $ chmod +x file.d; ./file.d) --- /+ dub.sdl: name "ddmd-example" dependency "ddmd-experimental" version="~>2.71.2-b2dub" +/ import std.path; import std.stdio; import std.string; import ddmd.builtin; import ddmd.dmodule; import ddmd.dclass; import ddmd.dsymbol; import ddmd.expression; import ddmd.func; import ddmd.globals; import ddmd.id; import ddmd.identifier; import ddmd.mtype; import ddmd.visitor; void main(string[] args) { if(args.length != 2) { writeln("prints top-level function and class declarations"); writeln("usage: ", args[0].baseName, " d-filepath"); return; } // Initialization global._init(); global.params.isLinux = true; Type._init(); Id.initialize(); Module._init(); Expression._init(); builtin_init(); // Read and parse specified module auto id = Identifier.idPool(args[1].baseName.stripExtension); auto m = new Module(args[1].toStringz, id, false, false); m.read(Loc()); m.parse(); // Output m.accept(new class Visitor { extern(C++): alias visit = Visitor.visit; override void visit(Module m) { foreach(i; 0 .. m.members.dim) (*m.members)[i].accept(this); } override void visit(Dsymbol s) { } override void visit(FuncDeclaration fd) { writeln("function ", fd.ident.toString); } override void visit(ClassDeclaration cd) { writeln("class ", cd.ident.toString); } }); }
Aug 29 2016
On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :) Please note: This is "only" the dmd frontend (lexer, parser, semantic passes, CTFE). Code generation will be more complicated.Thx, that's interesting from an IDE developer perspective. However since DMD is a "single shot" program, using it as library could introduce memory problems. Also string handling is C style, unlike libdparse, which is truly written in D. However libdparse does nothing more than producing the AST, which becomes problematic with "static if" expressions, mixins, etc.
Aug 29 2016
Alexander Breckel <example localhost.com> On Monday, 29 August 2016 at 11:27:44 UTC, Basile B. wrote:On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:Right. With these limitations, it might also be useful for writing static analysis tools, coding guidelines checkers, ... If someone wants to write a tool that tells me which functions and methods are not reachable from main(), or could be declared pure nothrow nogc, i would be more than happy to use it :)I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :) Please note: This is "only" the dmd frontend (lexer, parser, semantic passes, CTFE). Code generation will be more complicated.Thx, that's interesting from an IDE developer perspective. However since DMD is a "single shot" program, using it as library could introduce memory problems. Also string handling is C style, unlike libdparse, which is truly written in D. However libdparse does nothing more than producing the AST, which becomes problematic with "static if" expressions, mixins, etc.
Aug 29 2016
On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :) [...]Nice :) How about submitting this as PR? Then you get updates for free and more people can work on tweaking the DUB file.
Aug 29 2016
On Monday, 29 August 2016 at 11:31:58 UTC, Seb wrote:On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:I'm afraid this is not yet ready for upstream. Dmd would first have to move all ddmd files into src/ddmd/ to have a clean package namespace. And someone smarter than me should probably adjust the dub.json to somehow emphasize that the library contains only the front-end.I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :) [...]Nice :) How about submitting this as PR? Then you get updates for free and more people can work on tweaking the DUB file.
Aug 29 2016
On 29/08/2016 11:54 PM, Alexander Breckel wrote:On Monday, 29 August 2016 at 11:31:58 UTC, Seb wrote:With the help of sourceFiles you should be fine. You will also want sourcePaths to be set to [""] to ensure it doesn't have anything e.g. src in it. I would recommend you have dub pregenerate some import files and add them via importPaths.On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:I'm afraid this is not yet ready for upstream. Dmd would first have to move all ddmd files into src/ddmd/ to have a clean package namespace. And someone smarter than me should probably adjust the dub.json to somehow emphasize that the library contains only the front-end.I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :) [...]Nice :) How about submitting this as PR? Then you get updates for free and more people can work on tweaking the DUB file.
Aug 29 2016
I didn't see your announcement, but... AWESOME!! This could be the basis for some really good tutorials on making compiler backends etc... We need more little teaser examples like the one you posted in the beginning of this thread. PS: I did already check the code out on github because I watch code.dlang.org (a lot). Thanks! On Mon, Aug 29, 2016 at 12:42 PM, Alexander Breckel via Digitalmars-d < digitalmars-d puremagic.com> wrote:I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :) Please note: This is "only" the dmd frontend (lexer, parser, semantic passes, CTFE). Code generation will be more complicated. The following dub pre-generation hooks were necessary: - run src/idgen.d to generate src/id.d - create verstr.h and SYSCONFDIR.imp - create and link source/ddmd/ to src/ The last point makes ddmd modules reside in its correct package. I'm using a symbolic link for this, which is the only reason this approach is currently limited to Linux. In the long run, I think the cleanest way would be to place all ddmd files in src/ddmd/ and just leave mars.d and things like idgen.d in the main src/ directory. For demonstration purposes, here is a dub package to play around with: http://code.dlang.org/packages/ddmd-experimental Here is the dub.json that was used: https://github.com/aneas/ddmd-experimental/blob/master/dub.json And here is an example program using this API, have fun! (run with $ chmod +x file.d; ./file.d) --- /+ dub.sdl: name "ddmd-example" dependency "ddmd-experimental" version="~>2.71.2-b2dub" +/ import std.path; import std.stdio; import std.string; import ddmd.builtin; import ddmd.dmodule; import ddmd.dclass; import ddmd.dsymbol; import ddmd.expression; import ddmd.func; import ddmd.globals; import ddmd.id; import ddmd.identifier; import ddmd.mtype; import ddmd.visitor; void main(string[] args) { if(args.length != 2) { writeln("prints top-level function and class declarations"); writeln("usage: ", args[0].baseName, " d-filepath"); return; } // Initialization global._init(); global.params.isLinux = true; Type._init(); Id.initialize(); Module._init(); Expression._init(); builtin_init(); // Read and parse specified module auto id = Identifier.idPool(args[1].baseName.stripExtension); auto m = new Module(args[1].toStringz, id, false, false); m.read(Loc()); m.parse(); // Output m.accept(new class Visitor { extern(C++): alias visit = Visitor.visit; override void visit(Module m) { foreach(i; 0 .. m.members.dim) (*m.members)[i].accept(this); } override void visit(Dsymbol s) { } override void visit(FuncDeclaration fd) { writeln("function ", fd.ident.toString); } override void visit(ClassDeclaration cd) { writeln("class ", cd.ident.toString); } }); }
Sep 01 2016
On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:Because of the poor memory management in the compiler, I included a modified GC-stub when I compiled the frontend as a library, so that it can be used in long-running processes: https://gist.github.com/Cauterite/4eb74347aea040f5d371fb49054e1819 You can call gc_annihilate(true) to delete the entire heap. Obviously you need to keep the library in a separate DLL with its own runtime, and carefully avoid holding references to GC memory across annihilations. This technique is working pretty smoothly for me so far. Also to compile it as a DLL you either have to remove main() from mars.d or play games with the C runtime: https://gist.github.com/Cauterite/b190e62891c773703d0de3a1d99df362
Sep 01 2016
Thanks, for the GC stub, that will be great for playing with whether or not a little dmd app crashes after gc_annihilate(true). Did I understand that right? R On Thu, Sep 1, 2016 at 6:16 PM, Cauterite via Digitalmars-d < digitalmars-d puremagic.com> wrote:On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:Because of the poor memory management in the compiler, I included a modified GC-stub when I compiled the frontend as a library, so that it can be used in long-running processes: https://gist.github.com/Cauterite/4eb74347aea040f5d371fb49054e1819 You can call gc_annihilate(true) to delete the entire heap. Obviously you need to keep the library in a separate DLL with its own runtime, and carefully avoid holding references to GC memory across annihilations. This technique is working pretty smoothly for me so far. Also to compile it as a DLL you either have to remove main() from mars.d or play games with the C runtime: https://gist.github.com/Cauter ite/b190e62891c773703d0de3a1d99df362
Sep 01 2016
On Thu, 01 Sep 2016 18:42:13 +0200, Rory McGuire via Digitalmars-d wrote:Thanks, for the GC stub, that will be great for playing with whether or not a little dmd app crashes after gc_annihilate(true). Did I understand that right?It sounds like this there's a special function `gc_annihilate` that immediately frees all memory that libdmd has allocated. It doesn't do any reference tracking or mark/sweep. libdmd disables the collector as dmdfe does. You need to keep dmd in its own DLL. Otherwise, symbols between dmd and your application will be deduplicated or something and either you get a linker error or an arbitrary implementation is selected. Another safe way of working is to use the GC as usual, but before you call into libdmd, you call GC.disable, and you wait until you've finished using the data structures it returns before calling GC.enable.
Sep 01 2016
On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:Please note: This is "only" the dmd frontend (lexer, parser, semantic passes, CTFE). Code generation will be more complicated.Interesting. Did you have inline ASM in your code ? From my experience it required the backend to work, and couldn't be mocked, because else some flow analysis would go wrong (e.g. it would complain about missing `return`).
Sep 19 2016