www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Offering Access to Parser

reply cy <dlang verge.info.tm> writes:
I know macros are an unpopular subject in D, but I think there's 
a way to get mostly equivalent capabilities to macros, without 
changing the language at all, simply by standardizing access to 
the compiler's own D language parser.

There are already 10 separate parsers for D here:
http://wiki.dlang.org/Lexers_Parsers

All these parsers are pretty much written from scratch (one even 
forked from dmd itself) and they all do pretty much the same 
thing: take a string of code, and turn it into an AST. Turning 
that AST back into valid D code is itself a rather difficult and 
error prone process, but if you could, then you could simply do 
something like:

import some.parser;
string foo(string bar) = 
unparse(IfDeclaration(parse(bar),parse(q{writeln("yay")}),parse(q{writeln("boo")}));

void main() {
   mixin foo!"true";
   mixin foo!"false";
}

D isn't a particularly hard to parse language, so it's not too 
hard to implement these parsers, but they're definitely 
reinventing the wheel, and there's no assurance that future 
language features won't break them. But within each D compiler 
(I'm pretty sure?) there is already a parser, that produces (and 
manipulates) an AST. For DMD, the parser is written in D itself! 
Using it would (in theory) be as simple as copying all the ddmd 
modules into your source directory and importing ddmd.parse. 
Eliminating the need to maintain all those third party parsers is 
as simple as having the compiler itself provide the parser as a 
module.

There is additionally a huge advantage to code being able to use 
the compiler's D parser, that third party parsers cannot match. 
The D compiler actually uses the AST generated by its parser. 
With the other parsers, you have to somehow figure out how to 
"unparse" their AST into an opaque string of D code, before you 
can supply that new code as a mixin, but if you used the 
compiler's own parser, it could accept an AST as a mixin omitting 
the "parse" step it normally does on mixins.

So the above would end up more like:

import std.parser;
AST foo(string bar) = IfDeclaration(parse(bar), 
parse(q{writeln("yay")}), parse(q{writeln("boo")}));


void main() {
   mixin foo!"true";
   mixin foo!"false";
}

omitting the complicated and error prone "unparse()" step 
entirely, not to mention removing the degradation in compilation 
speed if you have to unparse, and then re-parse an AST.

I'm not sure if it would be possible to have the parser available 
at runtime, but it seems like you could compile it into the 
program just like any other module.
Feb 09 2016
next sibling parent John Carter <john.carter taitradio.com> writes:
On Tuesday, 9 February 2016 at 22:14:25 UTC, cy wrote:


 There are already 10 separate parsers for D here:
 http://wiki.dlang.org/Lexers_Parsers


A couple of dead links on that page...

* https://code.google.com/archive/p/dil/
* 
https://github.com/DDT-IDE/DDT/tree/master/org.dsource.ddt.dtool/src/dtool/parser
* 
https://github.com/Hackerpilot/libdparse/blob/master/src/std/d/parser.d
* 
https://github.com/Hackerpilot/libdparse/blob/master/src/std/d/lexer.d

Anyone have some updates on those links?
Feb 09 2016
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-02-09 23:14, cy wrote:


 So the above would end up more like:

 import std.parser;
 AST foo(string bar) = IfDeclaration(parse(bar),
 parse(q{writeln("yay")}), parse(q{writeln("boo")}));


 void main() {
    mixin foo!"true";
    mixin foo!"false";
 }


That's very close to AST macros. The only difference is the need to 
parse the input, "bar" in this case.

-- 
/Jacob Carlborg
Feb 10 2016
parent reply cy <dlang verge.info.tm> writes:
On Wednesday, 10 February 2016 at 12:24:49 UTC, Jacob Carlborg 
wrote:

 That's very close to AST macros. The only difference is the 
 need to parse the input, "bar" in this case.


Well yes, that's my point. AST macros don't need any new special 
syntax or language support, just access to the compiler's syntax 
parser. Then you can use normal mixins to get the code you want.

I dunno. I don't really care that much. It was just an idea I had 
that turned into way too many words.
Feb 10 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-02-11 05:22, cy wrote:


 Well yes, that's my point. AST macros don't need any new special syntax
 or language support, just access to the compiler's syntax parser. Then
 you can use normal mixins to get the code you want.


Well, language support _is_ required. Perhaps not new syntax though.

In my macro DIP [1] basically the only difference is that one use the 
"macro" keyword when declaring a function, then "mixin" is not necessary 
on the call site. Plus parsing the input is not necessary, that's 
handled automatically. That's all, the rest is just AST manipulation 
using CTFE.

[1] http://wiki.dlang.org/DIP50

-- 
/Jacob Carlborg
Feb 11 2016