www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - ES6 Parser in Dlang

reply James Lu <jamtlu gmail.com> writes:
I want to implement an ES6 (JavaScript) parser in Dlang so I can 
implement a transpiler. Because it's a transpiler, I need to be 
able to visit and rewrite AST nodes, then generate code from the 
AST. Ideally, the parser generator supports streaming the input 
into the AST visitor out to the output.

What parser generator library should I use for this purpose?
Feb 27
next sibling parent Dejan Lekic <dejan.lekic gmail.com> writes:
On Thursday, 27 February 2020 at 14:06:19 UTC, James Lu wrote:
 I want to implement an ES6 (JavaScript) parser in Dlang so I 
 can implement a transpiler. Because it's a transpiler, I need 
 to be able to visit and rewrite AST nodes, then generate code 
 from the AST. Ideally, the parser generator supports streaming 
 the input into the AST visitor out to the output.

 What parser generator library should I use for this purpose?
Have a look at similar work: https://github.com/higgsjs/Higgs
Feb 27
prev sibling next sibling parent JN <666total wp.pl> writes:
On Thursday, 27 February 2020 at 14:06:19 UTC, James Lu wrote:
 I want to implement an ES6 (JavaScript) parser in Dlang so I 
 can implement a transpiler. Because it's a transpiler, I need 
 to be able to visit and rewrite AST nodes, then generate code 
 from the AST. Ideally, the parser generator supports streaming 
 the input into the AST visitor out to the output.

 What parser generator library should I use for this purpose?
Check this out: http://code.dlang.org/packages/es6-grammar looks like a good starting point.
Feb 27
prev sibling next sibling parent =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
Am 27.02.2020 um 15:06 schrieb James Lu:
 I want to implement an ES6 (JavaScript) parser in Dlang so I can 
 implement a transpiler. Because it's a transpiler, I need to be able to 
 visit and rewrite AST nodes, then generate code from the AST. Ideally, 
 the parser generator supports streaming the input into the AST visitor 
 out to the output.
 
 What parser generator library should I use for this purpose?
There is also DMDScript, which is more or less ECMAScript v3. It should pretty straight forward to get that updated to v5, as there haven't many parser related changes (property getters/setters is an exception). I was already thinking of whether it makes sense to invest that work, because that would enable running the TypeScript compiler through it. v6 has a few more parser related changes, but still nothing really bad. The DMDScript parser is in https://github.com/DigitalMars/DMDScript/blob/master/engine/source/dmdscript/parse.d Higgs is much closer to ES6, though.
Feb 27
prev sibling next sibling parent reply Meta <jared771 gmail.com> writes:
On Thursday, 27 February 2020 at 14:06:19 UTC, James Lu wrote:
 I want to implement an ES6 (JavaScript) parser in Dlang so I 
 can implement a transpiler. Because it's a transpiler, I need 
 to be able to visit and rewrite AST nodes, then generate code 
 from the AST. Ideally, the parser generator supports streaming 
 the input into the AST visitor out to the output.

 What parser generator library should I use for this purpose?
There's also Pegged: https://code.dlang.org/packages/pegged
Feb 27
parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 27 February 2020 at 16:49:19 UTC, Meta wrote:
 On Thursday, 27 February 2020 at 14:06:19 UTC, James Lu wrote:
 I want to implement an ES6 (JavaScript) parser in Dlang so I 
 can implement a transpiler. Because it's a transpiler, I need 
 to be able to visit and rewrite AST nodes, then generate code 
 from the AST. Ideally, the parser generator supports streaming 
 the input into the AST visitor out to the output.

 What parser generator library should I use for this purpose?
There's also Pegged: https://code.dlang.org/packages/pegged
http://code.dlang.org/packages/es6-grammar is based on Pegged, btw.
Feb 27
prev sibling next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 27 February 2020 at 14:06:19 UTC, James Lu wrote:
 I want to implement an ES6 (JavaScript) parser in Dlang so I 
 can implement a transpiler. Because it's a transpiler, I need 
 to be able to visit and rewrite AST nodes, then generate code 
 from the AST. Ideally, the parser generator supports streaming 
 the input into the AST visitor out to the output.

 What parser generator library should I use for this purpose?
I would write my own. Maybe with some tooling to take care of the boring stuff.
Feb 27
prev sibling next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 27 February 2020 at 14:06:19 UTC, James Lu wrote:
 I want to implement an ES6 (JavaScript) parser in Dlang so I 
 can implement a transpiler. Because it's a transpiler, I need 
 to be able to visit and rewrite AST nodes, then generate code 
 from the AST. Ideally, the parser generator supports streaming 
 the input into the AST visitor out to the output.

 What parser generator library should I use for this purpose?
I dunno. I wrote the es6-grammar library that uses pegged. Then I decided to hand write mine own. https://github.com/skoppe/jazr It is a pretty complete and fast JS minifier. It has lots of passes and minifies code pretty well: https://github.com/skoppe/jazr/blob/master/source/es6/minifier.d#L286 In general it is about 30 times faster than uglify-js (and uses 3-4 times less memory). It parses JS code at about 60 Mb/s. The minifier was a bit slower at 20 Mb/s if I remember correctly. I fuzzed the lexer and parser, so there are little to no spurious crashes on weird input. Use it as you please, or just as an indication of the effort involved.
Feb 27
next sibling parent aberba <karabutaworld gmail.com> writes:
On Thursday, 27 February 2020 at 22:22:59 UTC, Sebastiaan Koppe 
wrote:
 On Thursday, 27 February 2020 at 14:06:19 UTC, James Lu wrote:
 [...]
I dunno. I wrote the es6-grammar library that uses pegged. Then I decided to hand write mine own. https://github.com/skoppe/jazr It is a pretty complete and fast JS minifier. It has lots of passes and minifies code pretty well: https://github.com/skoppe/jazr/blob/master/source/es6/minifier.d#L286 In general it is about 30 times faster than uglify-js (and uses 3-4 times less memory). It parses JS code at about 60 Mb/s. The minifier was a bit slower at 20 Mb/s if I remember correctly. I fuzzed the lexer and parser, so there are little to no spurious crashes on weird input. Use it as you please, or just as an indication of the effort involved.
Nice!!
Feb 27
prev sibling next sibling parent reply drug <drug2004 bk.ru> writes:
28.02.2020 01:22, Sebastiaan Koppe пишет:
 On Thursday, 27 February 2020 at 14:06:19 UTC, James Lu wrote:
 I want to implement an ES6 (JavaScript) parser in Dlang so I can 
 implement a transpiler. Because it's a transpiler, I need to be able 
 to visit and rewrite AST nodes, then generate code from the AST. 
 Ideally, the parser generator supports streaming the input into the 
 AST visitor out to the output.

 What parser generator library should I use for this purpose?
I dunno. I wrote the es6-grammar library that uses pegged. Then I decided to hand write mine own.
What was the reason? Pegged parsing errors?
 
 https://github.com/skoppe/jazr
 
 It is a pretty complete and fast JS minifier. It has lots of passes and 
 minifies code pretty well:
 
 https://github.com/skoppe/jazr/blob/master/source/es6/minifier.d#L286
 
 In general it is about 30 times faster than uglify-js (and uses 3-4 
 times less memory). It parses JS code at about 60 Mb/s. The minifier was 
 a bit slower at 20 Mb/s if I remember correctly.
 
 I fuzzed the lexer and parser, so there are little to no spurious 
 crashes on weird input.
 
 Use it as you please, or just as an indication of the effort involved.
Feb 27
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Friday, 28 February 2020 at 05:24:16 UTC, drug wrote:
 28.02.2020 01:22, Sebastiaan Koppe пишет:
 On Thursday, 27 February 2020 at 14:06:19 UTC, James Lu wrote:
 I want to implement an ES6 (JavaScript) parser in Dlang so I 
 can implement a transpiler. Because it's a transpiler, I need 
 to be able to visit and rewrite AST nodes, then generate code 
 from the AST. Ideally, the parser generator supports 
 streaming the input into the AST visitor out to the output.

 What parser generator library should I use for this purpose?
I dunno. I wrote the es6-grammar library that uses pegged. Then I decided to hand write mine own.
What was the reason? Pegged parsing errors?
No, it was mainly speed and memory usage. Pegged is pretty convenient for small grammar or small input files, but es6-grammar contains about 500 grammar rules and I was running it on Mb's of javascript. Incidentally I improved pegged's parsing errors in https://github.com/PhilippeSigaud/Pegged/pull/269 you now get a much more complete parsetree on errors. Bastiaan, however, found some memory usage problems but I hope to tackle them soonish.
Feb 27
parent reply drug <drug2004 bk.ru> writes:
On 2/28/20 10:30 AM, Sebastiaan Koppe wrote:
 
 No, it was mainly speed and memory usage.
 
 Pegged is pretty convenient for small grammar or small input files, but 
 es6-grammar contains about 500 grammar rules and I was running it on 
 Mb's of javascript.
 
 Incidentally I improved pegged's parsing errors in 
 https://github.com/PhilippeSigaud/Pegged/pull/269 you now get a much 
 more complete parsetree on errors. Bastiaan, however, found some memory 
 usage problems but I hope to tackle them soonish.
Yes, I've seen your PR so I asked this question) Nice work! I'm curious can Pegged be improved to be usable in industry and become standard parser generator in D land de-facto.
Feb 27
parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Friday, 28 February 2020 at 07:53:22 UTC, drug wrote:
 Yes, I've seen your PR so I asked this question) Nice work!
thx
 I'm curious can Pegged be improved to be usable in industry and 
 become standard parser generator in D land de-facto.
It is already used in industry. Rather successfully I would say. Making and keeping something a de-facto standard requires a lot of work spread over years, I don't see that happening.
Feb 28
prev sibling parent James Lu <jamtlu gmail.com> writes:
On Thursday, 27 February 2020 at 22:22:59 UTC, Sebastiaan Koppe 
wrote:
 I dunno. I wrote the es6-grammar library that uses pegged. Then 
 I decided to hand write mine own.

 https://github.com/skoppe/jazr

 It is a pretty complete and fast JS minifier. It has lots of 
 passes and minifies code pretty well:

 https://github.com/skoppe/jazr/blob/master/source/es6/minifier.d#L286

 In general it is about 30 times faster than uglify-js (and uses 
 3-4 times less memory). It parses JS code at about 60 Mb/s. The 
 minifier was a bit slower at 20 Mb/s if I remember correctly.

 I fuzzed the lexer and parser, so there are little to no 
 spurious crashes on weird input.

 Use it as you please, or just as an indication of the effort 
 involved.
Thanks. I will use this to create wrappers that ensure JS closures can be serialized.
Mar 02
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/27/2020 6:06 AM, James Lu wrote:
 What parser generator library should I use for this purpose?
https://github.com/DigitalMars/DMDScript
Feb 27