www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Parser generator?

reply Wouter Verhelst <wouter grep.be> writes:
Hi folks,

Does someone know of a parser generator for D?

If it doesn't exist, I can write a parser by hand, but having it
generated (at least for my initial permutation) seems like a better
idea...

Thanks,

-- 
The volume of a pizza of thickness a and radius z can be described by
the following formula:

pi zz a
Jul 04 2012
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, July 04, 2012 14:53:02 Wouter Verhelst wrote:
 Hi folks,
 
 Does someone know of a parser generator for D?
 
 If it doesn't exist, I can write a parser by hand, but having it
 generated (at least for my initial permutation) seems like a better
 idea...
 
 Thanks,

https://github.com/PhilippeSigaud/Pegged https://github.com/PhilippeSigaud/Pegged/wiki - Jonathan M Davis
Jul 04 2012
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 07/04/2012 11:41 PM, Jonathan M Davis wrote:
 On Wednesday, July 04, 2012 15:32:16 Wouter Verhelst wrote:
 Jonathan M Davis<jmdavisProg gmx.com>  writes:
 On Wednesday, July 04, 2012 14:53:02 Wouter Verhelst wrote:
 Hi folks,

 Does someone know of a parser generator for D?

 If it doesn't exist, I can write a parser by hand, but having it
 generated (at least for my initial permutation) seems like a better
 idea...

 Thanks,

https://github.com/PhilippeSigaud/Pegged https://github.com/PhilippeSigaud/Pegged/wiki

Whoa. That's so perfect, it makes me drool.

Yeah. It's pretty cool. It really shows up D's metaprogramming capabilities. The one thing to watch out for though is that such metaprogramming tends to eat up a lot of memory when compiling at this point (primarily because the compiler doesn't manage memory very well at this point - it's approach is very simplistic). So, that may or may not cause you problems. It should be fixed eventually, but it does sometimes cause problems with this sort of thing. std.regex has similar issues. - Jonathan M Davis

https://github.com/PhilippeSigaud/Pegged/wiki/Grammars-as-D-Modules
Jul 04 2012
prev sibling next sibling parent Wouter Verhelst <wouter grep.be> writes:
Jonathan M Davis <jmdavisProg gmx.com> writes:

 On Wednesday, July 04, 2012 14:53:02 Wouter Verhelst wrote:
 Hi folks,
 
 Does someone know of a parser generator for D?
 
 If it doesn't exist, I can write a parser by hand, but having it
 generated (at least for my initial permutation) seems like a better
 idea...
 
 Thanks,

https://github.com/PhilippeSigaud/Pegged https://github.com/PhilippeSigaud/Pegged/wiki

Whoa. That's so perfect, it makes me drool. Thanks. -- The volume of a pizza of thickness a and radius z can be described by the following formula: pi zz a
Jul 04 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, July 04, 2012 15:32:16 Wouter Verhelst wrote:
 Jonathan M Davis <jmdavisProg gmx.com> writes:
 On Wednesday, July 04, 2012 14:53:02 Wouter Verhelst wrote:
 Hi folks,
 
 Does someone know of a parser generator for D?
 
 If it doesn't exist, I can write a parser by hand, but having it
 generated (at least for my initial permutation) seems like a better
 idea...
 
 Thanks,

https://github.com/PhilippeSigaud/Pegged https://github.com/PhilippeSigaud/Pegged/wiki

Whoa. That's so perfect, it makes me drool.

Yeah. It's pretty cool. It really shows up D's metaprogramming capabilities. The one thing to watch out for though is that such metaprogramming tends to eat up a lot of memory when compiling at this point (primarily because the compiler doesn't manage memory very well at this point - it's approach is very simplistic). So, that may or may not cause you problems. It should be fixed eventually, but it does sometimes cause problems with this sort of thing. std.regex has similar issues. - Jonathan M Davis
Jul 04 2012
prev sibling next sibling parent Mirko Pilger <pilger cymotec.de> writes:
 Does someone know of a parser generator for D?

http://www.complang.org/ragel/ http://www.semitwist.com/goldie/
Jul 04 2012
prev sibling next sibling parent Jerome BENOIT <g6299304p rezozer.net> writes:
Apparently Flex/Bison can be used: see last answer in

http://www.digitalmars.com/d/archives/digitalmars/D/31679.html

hth,
Jerome

On 04/07/12 22:53, Wouter Verhelst wrote:
 Hi folks,

 Does someone know of a parser generator for D?

 If it doesn't exist, I can write a parser by hand, but having it
 generated (at least for my initial permutation) seems like a better
 idea...

 Thanks,

Jul 04 2012
prev sibling next sibling parent Wouter Verhelst <wouter grep.be> writes:
Jonathan M Davis <jmdavisProg gmx.com> writes:

 On Wednesday, July 04, 2012 15:32:16 Wouter Verhelst wrote:
 Jonathan M Davis <jmdavisProg gmx.com> writes:
 On Wednesday, July 04, 2012 14:53:02 Wouter Verhelst wrote:
 Hi folks,
 
 Does someone know of a parser generator for D?
 
 If it doesn't exist, I can write a parser by hand, but having it
 generated (at least for my initial permutation) seems like a better
 idea...
 
 Thanks,

https://github.com/PhilippeSigaud/Pegged https://github.com/PhilippeSigaud/Pegged/wiki

Whoa. That's so perfect, it makes me drool.

Yeah. It's pretty cool. It really shows up D's metaprogramming capabilities.

It's a bit hell to debug it, though. But I finally managed to get a working parser out of it. Except now I can't use it, for some reason. Code: void buildGraph(Output o) { void parseToGraph(ParseTree p) { writeln(p.ruleName); } parseToGraph(o.parseTree); } void parseconfigs() { Output o = ENI.parse(readText("/tmp/ifaces_data")); buildGraph(o); } produces the following error message: dmd -gc -w -unittest -I../../Pegged -c -ofmain.o main.d ipcfg/parser.d(45): Error: struct pegged.peg.Output(TParseTree) if (isParseTree!(TParseTree)) is used as a type make: *** [main.o] Error 1 Help?
 The one thing to watch out for though is that such metaprogramming tends to 
 eat up a lot of memory when compiling at this point (primarily because the 
 compiler doesn't manage memory very well at this point - it's approach is very 
 simplistic).

I'll say. My parser does more than just identifiers and comments, but it's still not that complex. And yet I manage 1GiB+. Whoa. But that's still acceptable.
 So, that may or may not cause you problems.

Not really. What does cause problems, though, is that it won't compile with gdc: gdmd -release -I../../Pegged -c -ofmain.o main.d /home/wouter/code/d/Pegged/pegged/grammar.d:128: Error: template pegged.grammar.PEGGED!(ParseTree).PEGGED.parse(ParseLevel pl = ParseLevel.parsing) parse(ParseLevel pl = ParseLevel.parsing) matches more than one template declaration, /home/wouter/code/d/Pegged/pegged/grammar.d(111):parse(ParseLevel pl = ParseLevel.parsing) and /home/wouter/code/d/Pegged/pegged/grammar.d(126):parse(ParseLevel pl = ParseLevel.parsing) make: *** [main.o] Error 1 GDC 4.6, though, so I'll wait until Iain uploads 4.7 to Debian; if that doesn't fix it, I'll file bugs where appropriate.
 It should be fixed 
 eventually, but it does sometimes cause problems with this sort of thing. 
 std.regex has similar issues.

Gotcha. -- The volume of a pizza of thickness a and radius z can be described by the following formula: pi zz a
Jul 05 2012
prev sibling next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
--f46d043bd6fa9eeeee04c4231dc3
Content-Type: text/plain; charset=UTF-8

 Yeah. It's pretty cool. It really shows up D's metaprogramming
 capabilities.

It's a bit hell to debug it, though. But I finally managed to get a working parser out of it.

I received your suggestion of a full debug mode explaining what rules where activated and wich did not.
 Except now I can't use it, for some reason. Code:

 void buildGraph(Output o) {
         void parseToGraph(ParseTree p) {
                 writeln(p.ruleName);
         }

         parseToGraph(o.parseTree);
 }

 void parseconfigs() {
         Output o = ENI.parse(readText("/tmp/ifaces_data"));
         buildGraph(o);
 }

 produces the following error message:

 dmd  -gc -w -unittest -I../../Pegged -c -ofmain.o main.d
 ipcfg/parser.d(45): Error: struct pegged.peg.Output(TParseTree) if

 make: *** [main.o] Error 1

 Help?

Roman recently templated the parse tree to allow multiple outputs and we didn't update the docs, sorry. That makes Pegged output something other than `Output`. One consequence is hat semantic actions should be templates to function on different kinds of parse trees. Try:
 void buildGraph(O)(O o) {
         void parseToGraph(ParseTree p) {
                 writeln(p.ruleName);
         }

         parseToGraph(o.parseTree);
 }

 void parseconfigs() {
         auto o = ENI.parse(readText("/tmp/ifaces_data"));
         buildGraph(o);
 }

What do parseToGraph and buildGraph do?
 What does cause problems, though, is that it won't compile with gdc:

 gdmd -release  -I../../Pegged -c -ofmain.o main.d
 /home/wouter/code/d/Pegged/pegged/grammar.d:128: Error: template

ParseLevel.parsing) parse(ParseLevel pl = ParseLevel.parsing) matches more than one template declaration, /home/wouter/code/d/Pegged/pegged/grammar.d(111):parse(ParseLevel pl = ParseLevel.parsing) and /home/wouter/code/d/Pegged/pegged/grammar.d(126):parse(ParseLevel pl = ParseLevel.parsing)
 make: *** [main.o] Error 1

 GDC 4.6, though, so I'll wait until Iain uploads 4.7 to Debian; if that
 doesn't fix it, I'll file bugs where appropriate.

I find it strange that I don't get that error with DMD. Philippe --f46d043bd6fa9eeeee04c4231dc3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <p>&gt; &gt; Yeah. It&#39;s pretty cool. It really shows up D&#39;s metapro= gramming<br> &gt; &gt; capabilities.<br> &gt;<br> &gt; It&#39;s a bit hell to debug it, though. But I finally managed to get = a<br> &gt; working parser out of it.</p> <p>I received your suggestion of a full debug mode explaining what rules wh= ere activated and wich did not.</p> <p>&gt;<br> &gt; Except now I can&#39;t use it, for some reason. Code:<br> &gt;<br> &gt; void buildGraph(Output o) {<br> &gt; =C2=A0 =C2=A0 =C2=A0 =C2=A0 void parseToGraph(ParseTree p) {<br> &gt; =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 writeln(p.rule= Name);<br> &gt; =C2=A0 =C2=A0 =C2=A0 =C2=A0 }<br> &gt;<br> &gt; =C2=A0 =C2=A0 =C2=A0 =C2=A0 parseToGraph(o.parseTree);<br> &gt; }<br> &gt;<br> &gt; void parseconfigs() {<br> &gt; =C2=A0 =C2=A0 =C2=A0 =C2=A0 Output o =3D ENI.parse(readText(&quot;/tmp= /ifaces_data&quot;));<br> &gt; =C2=A0 =C2=A0 =C2=A0 =C2=A0 buildGraph(o);<br> &gt; }<br> &gt;<br> &gt; produces the following error message:<br> &gt;<br> &gt; dmd =C2=A0-gc -w -unittest -I../../Pegged -c -ofmain.o main.d<br> &gt; ipcfg/parser.d(45): Error: struct pegged.peg.Output(TParseTree) if (is= ParseTree!(TParseTree)) is used as a type<br> &gt; make: *** [main.o] Error 1<br> &gt;<br> &gt; Help?</p> <p>Roman recently templated the parse tree to allow multiple outputs and we= didn&#39;t update the docs, sorry. That makes Pegged output something othe= r than `Output`. One consequence is hat semantic actions should be template= s to function on different kinds of parse trees.</p> <p>Try:</p> <p>&gt; void buildGraph(O)(O o) {<br> &gt; =C2=A0 =C2=A0 =C2=A0 =C2=A0 void parseToGraph(ParseTree p) {<br> &gt; =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 writeln(p.rule= Name);<br> &gt; =C2=A0 =C2=A0 =C2=A0 =C2=A0 }<br> &gt;<br> &gt; =C2=A0 =C2=A0 =C2=A0 =C2=A0 parseToGraph(o.parseTree);<br> &gt; }<br> &gt;<br> &gt; void parseconfigs() {<br> &gt; =C2=A0 =C2=A0 =C2=A0 =C2=A0 auto o =3D ENI.parse(readText(&quot;/tmp/i= faces_data&quot;));<br> &gt; =C2=A0 =C2=A0 =C2=A0 =C2=A0 buildGraph(o);<br> &gt; }</p> <p>What do parseToGraph and buildGraph do?<br></p> <p>&gt;<br> &gt; What does cause problems, though, is that it won&#39;t compile with gd= c:<br> &gt;<br> &gt; gdmd -release =C2=A0-I../../Pegged -c -ofmain.o main.d<br> &gt; /home/wouter/code/d/Pegged/pegged/grammar.d:128: Error: template pegge= d.grammar.PEGGED!(ParseTree).PEGGED.parse(ParseLevel pl =3D ParseLevel.pars= ing) parse(ParseLevel pl =3D ParseLevel.parsing) matches more than one temp= late declaration, /home/wouter/code/d/Pegged/pegged/grammar.d(111):parse(Pa= rseLevel pl =3D ParseLevel.parsing) and /home/wouter/code/d/Pegged/pegged/g= rammar.d(126):parse(ParseLevel pl =3D ParseLevel.parsing)<br> &gt; make: *** [main.o] Error 1<br> &gt;<br> &gt; GDC 4.6, though, so I&#39;ll wait until Iain uploads 4.7 to Debian; if= that<br> &gt; doesn&#39;t fix it, I&#39;ll file bugs where appropriate.</p> <p>I find it strange that I don&#39;t get that error with DMD. </p> <p>Philippe<br> </p> --f46d043bd6fa9eeeee04c4231dc3--
Jul 05 2012
prev sibling next sibling parent "Roman D. Boiko" <rb d-coding.com> writes:
On Friday, 6 July 2012 at 06:10:28 UTC, Philippe Sigaud wrote:
 Roman recently templated the parse tree to allow multiple 
 outputs and we didn't update the docs, sorry.

that my needs have been met so well.
Jul 06 2012
prev sibling next sibling parent "Roman D. Boiko" <rb d-coding.com> writes:
On Friday, 6 July 2012 at 06:10:28 UTC, Philippe Sigaud wrote:
 void buildGraph(Output o) {
         void parseToGraph(ParseTree p) {
                 writeln(p.ruleName);
         }

         parseToGraph(o.parseTree);
 }


buildGraph unwraps parseTree from output, and parseToGraph writes a line with rule name of a parse tree. ;)))
Jul 06 2012
prev sibling parent Wouter Verhelst <wouter grep.be> writes:
Philippe Sigaud <philippe.sigaud gmail.com> writes:

 Yeah. It's pretty cool. It really shows up D's metaprogramming
 capabilities.

It's a bit hell to debug it, though. But I finally managed to get a working parser out of it.

I received your suggestion of a full debug mode explaining what rules where activated and wich did not.

Good :-)
 Except now I can't use it, for some reason. Code:

 void buildGraph(Output o) {
         void parseToGraph(ParseTree p) {
                 writeln(p.ruleName);
         }

         parseToGraph(o.parseTree);
 }

 void parseconfigs() {
         Output o = ENI.parse(readText("/tmp/ifaces_data"));
         buildGraph(o);
 }

 produces the following error message:

 dmd  -gc -w -unittest -I../../Pegged -c -ofmain.o main.d
 ipcfg/parser.d(45): Error: struct pegged.peg.Output(TParseTree) if

 make: *** [main.o] Error 1

 Help?

Roman recently templated the parse tree to allow multiple outputs and we didn't update the docs, sorry. That makes Pegged output something other than `Output`. One consequence is hat semantic actions should be templates to function on different kinds of parse trees. Try:
 void buildGraph(O)(O o) {


Ah yes, that explains. Thanks.
         void parseToGraph(ParseTree p) {
                 writeln(p.ruleName);
         }

         parseToGraph(o.parseTree);
 }

 void parseconfigs() {
         auto o = ENI.parse(readText("/tmp/ifaces_data"));
         buildGraph(o);
 }

What do parseToGraph and buildGraph do?

Nothing more than that, _currently_. The plan is for them to eventually build data structures in memory, but I'm not there yet.
 What does cause problems, though, is that it won't compile with gdc:

 gdmd -release  -I../../Pegged -c -ofmain.o main.d
 /home/wouter/code/d/Pegged/pegged/grammar.d:128: Error: template

ParseLevel.parsing) parse(ParseLevel pl = ParseLevel.parsing) matches more than one template declaration, /home/wouter/code/d/Pegged/pegged/grammar.d (111):parse(ParseLevel pl = ParseLevel.parsing) and /home/wouter/code/d/Pegged/ pegged/grammar.d(126):parse(ParseLevel pl = ParseLevel.parsing)
 make: *** [main.o] Error 1

 GDC 4.6, though, so I'll wait until Iain uploads 4.7 to Debian; if that
 doesn't fix it, I'll file bugs where appropriate.

I find it strange that I don't get that error with DMD.

Agreed. -- The volume of a pizza of thickness a and radius z can be described by the following formula: pi zz a
Jul 06 2012