www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - tree-sitter: parser generator tool and an incremental parsing library

reply James Blachly <james.blachly gmail.com> writes:
https://tree-sitter.github.io/tree-sitter/
https://github.com/tree-sitter/tree-sitter
https://news.ycombinator.com/item?id=26225298

Will be keeping an eye on this. If it gains traction*, it would be nice 
(and possibly important) to have a Dlang parser. Haven't looked in depth 
at the grammar yet -- appears to be context free grammar.

*Supposedly Neovim will be using this going forward
Feb 22 2021
next sibling parent Dukc <ajieskola gmail.com> writes:
On Monday, 22 February 2021 at 15:56:39 UTC, James Blachly wrote:

 https://tree-sitter.github.io/tree-sitter/
 https://github.com/tree-sitter/tree-sitter
 https://news.ycombinator.com/item?id=26225298
 HN)

 Will be keeping an eye on this. If it gains traction*, it would 
 be nice (and possibly important) to have a Dlang parser.


I wonder how it compares to Pegged.
Feb 22 2021
prev sibling next sibling parent Laurent =?UTF-8?B?VHLDqWd1aWVy?= <laurent.treguier.sink gmail.com> writes:
On Monday, 22 February 2021 at 15:56:39 UTC, James Blachly wrote:

 https://tree-sitter.github.io/tree-sitter/
 https://github.com/tree-sitter/tree-sitter
 https://news.ycombinator.com/item?id=26225298
 HN)

 Will be keeping an eye on this. If it gains traction*, it would 
 be nice (and possibly important) to have a Dlang parser. 
 Haven't looked in depth at the grammar yet -- appears to be 
 context free grammar.

 *Supposedly Neovim will be using this going forward


I tried making a tree-sitter grammar for D a long while ago, it 
needed some custom C code for stuff like WYSIWYG strings and some 
other things I think. Another issue was that D's official grammar 
not quite always reflecting the compiler's actual behavior.
Feb 24 2021
prev sibling next sibling parent Mike Brown <mikey.be gmail.com> writes:
On Monday, 22 February 2021 at 15:56:39 UTC, James Blachly wrote:

 https://tree-sitter.github.io/tree-sitter/
 https://github.com/tree-sitter/tree-sitter
 https://news.ycombinator.com/item?id=26225298
 HN)

 Will be keeping an eye on this. If it gains traction*, it would 
 be nice (and possibly important) to have a Dlang parser. 
 Haven't looked in depth at the grammar yet -- appears to be 
 context free grammar.

 *Supposedly Neovim will be using this going forward


tree-sitter is a GLR parser. This makes sense for a general 
purpose text editor like Atom and Neovim.

It can handle more grammars
It can handle grammars that require fixups in the parser better 
(e.g. C++)
It can potentially give better syntax error reporting

But does come at a cost, and these benefits might not apply to an 
editor that handles just one grammar (Like a D IDE)
Feb 24 2021
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2021-02-22 16:56, James Blachly wrote:

 https://tree-sitter.github.io/tree-sitter/
 https://github.com/tree-sitter/tree-sitter
 https://news.ycombinator.com/item?id=26225298
 
 Will be keeping an eye on this. If it gains traction*, it would be nice 
 (and possibly important) to have a Dlang parser. Haven't looked in depth 
 at the grammar yet -- appears to be context free grammar.
 
 *Supposedly Neovim will be using this going forward


I wonder how the text editor/IDE integration is supposed to work. To 
build the parser you need Node.js and a C compiler. Is the author of the 
grammar supposed to pre-compile everything and it's loaded as a dynamic 
library? That would defiantly make it less flexible that the current 
approach TextMate grammars. TextMate grammars are very easy to modify on 
the fly and the editor will immediately be able to use the new grammar. 
No building, no compilation and no loading of dynamic libraries. But I 
do know how limiting TextMate grammars are.

I guess in theory the editor could ship with Node.js and a C compiler 
and compile on the fly to have the same flexibility as TextMate grammars.

BTW, I'm wondering if it needs to be Node.js or if any JavaScript engine 
would work. At least macOS already ships with a JavaScript engine.

-- 
/Jacob Carlborg
Feb 26 2021
next sibling parent Mike Brown <mikey.be gmail.com> writes:
On Friday, 26 February 2021 at 20:06:46 UTC, Jacob Carlborg wrote:

 On 2021-02-22 16:56, James Blachly wrote:

 https://tree-sitter.github.io/tree-sitter/
 https://github.com/tree-sitter/tree-sitter
 https://news.ycombinator.com/item?id=26225298
 HN)
 
 Will be keeping an eye on this. If it gains traction*, it 
 would be nice (and possibly important) to have a Dlang parser. 
 Haven't looked in depth at the grammar yet -- appears to be 
 context free grammar.
 
 *Supposedly Neovim will be using this going forward


 I wonder how the text editor/IDE integration is supposed to 
 work. To build the parser you need Node.js and a C compiler. Is 
 the author of the grammar supposed to pre-compile everything 
 and it's loaded as a dynamic library? That would defiantly make 
 it less flexible that the current approach TextMate grammars. 
 TextMate grammars are very easy to modify on the fly and the 
 editor will immediately be able to use the new grammar. No 
 building, no compilation and no loading of dynamic libraries. 
 But I do know how limiting TextMate grammars are.

 I guess in theory the editor could ship with Node.js and a C 
 compiler and compile on the fly to have the same flexibility as 
 TextMate grammars.

 BTW, I'm wondering if it needs to be Node.js or if any 
 JavaScript engine would work. At least macOS already ships with 
 a JavaScript engine.


tree-sitter is written in C, and all parsers are written in C. 
Atom is the primary editor using tree-sitter, that uses Node.js 
(/Electron).

I also forgot to mention the other major benefit/goal to 
tree-sitter. It is an incremental parser, so the parse tree is 
updated at every keystoke. (this is mentioned in this threads 
title).

Is this discussion for parsing in general, or for the D 
frontend/IDEs?
Feb 27 2021
prev sibling parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Friday, 26 February 2021 at 20:06:46 UTC, Jacob Carlborg wrote:

 I wonder how the text editor/IDE integration is supposed to 
 work. To build the parser you need Node.js and a C compiler. Is 
 the author of the grammar supposed to pre-compile everything 
 and it's loaded as a dynamic library?


Yes.


 That would defiantly make it less flexible that the current 
 approach TextMate grammars. TextMate grammars are very easy to 
 modify on the fly and the editor will immediately be able to 
 use the new grammar. No building, no compilation and no loading 
 of dynamic libraries. But I do know how limiting TextMate 
 grammars are.


Yes, there is definitely an edit-compile-test cycle.

The D grammar I am working on currently takes about 15 seconds to 
compile.


 I guess in theory the editor could ship with Node.js and a C 
 compiler and compile on the fly to have the same flexibility as 
 TextMate grammars.


The great majority of users won't need to edit the grammar, so I 
think this is not an important goal to aim for. Parsing speed 
would probably be better appreciated than the ability to edit the 
grammar on the fly.


 BTW, I'm wondering if it needs to be Node.js or if any 
 JavaScript engine would work. At least macOS already ships with 
 a JavaScript engine.


I believe the tool chain is built on node/npm.

Grammar authors seem to keep the generated C files in git, so 
running the JavaScript part isn't needed to get the usable .so 
file.

For the future, I understand that the plan is to target 
WebAssembly instead. WebAssembly binaries are portable and it 
executes sufficiently quickly.
Jun 14 2021