www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - tree-sitter: parser generator tool and an incremental parsing library

reply James Blachly <james.blachly gmail.com> writes:
https://tree-sitter.github.io/tree-sitter/
https://github.com/tree-sitter/tree-sitter
https://news.ycombinator.com/item?id=26225298

Will be keeping an eye on this. If it gains traction*, it would be nice 
(and possibly important) to have a Dlang parser. Haven't looked in depth 
at the grammar yet -- appears to be context free grammar.

*Supposedly Neovim will be using this going forward
Feb 22
next sibling parent Dukc <ajieskola gmail.com> writes:
On Monday, 22 February 2021 at 15:56:39 UTC, James Blachly wrote:
 https://tree-sitter.github.io/tree-sitter/
 https://github.com/tree-sitter/tree-sitter
 https://news.ycombinator.com/item?id=26225298
 HN)

 Will be keeping an eye on this. If it gains traction*, it would 
 be nice (and possibly important) to have a Dlang parser.
I wonder how it compares to Pegged.
Feb 22
prev sibling next sibling parent Laurent =?UTF-8?B?VHLDqWd1aWVy?= <laurent.treguier.sink gmail.com> writes:
On Monday, 22 February 2021 at 15:56:39 UTC, James Blachly wrote:
 https://tree-sitter.github.io/tree-sitter/
 https://github.com/tree-sitter/tree-sitter
 https://news.ycombinator.com/item?id=26225298
 HN)

 Will be keeping an eye on this. If it gains traction*, it would 
 be nice (and possibly important) to have a Dlang parser. 
 Haven't looked in depth at the grammar yet -- appears to be 
 context free grammar.

 *Supposedly Neovim will be using this going forward
I tried making a tree-sitter grammar for D a long while ago, it needed some custom C code for stuff like WYSIWYG strings and some other things I think. Another issue was that D's official grammar not quite always reflecting the compiler's actual behavior.
Feb 24
prev sibling next sibling parent Mike Brown <mikey.be gmail.com> writes:
On Monday, 22 February 2021 at 15:56:39 UTC, James Blachly wrote:
 https://tree-sitter.github.io/tree-sitter/
 https://github.com/tree-sitter/tree-sitter
 https://news.ycombinator.com/item?id=26225298
 HN)

 Will be keeping an eye on this. If it gains traction*, it would 
 be nice (and possibly important) to have a Dlang parser. 
 Haven't looked in depth at the grammar yet -- appears to be 
 context free grammar.

 *Supposedly Neovim will be using this going forward
tree-sitter is a GLR parser. This makes sense for a general purpose text editor like Atom and Neovim. It can handle more grammars It can handle grammars that require fixups in the parser better (e.g. C++) It can potentially give better syntax error reporting But does come at a cost, and these benefits might not apply to an editor that handles just one grammar (Like a D IDE)
Feb 24
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2021-02-22 16:56, James Blachly wrote:
 https://tree-sitter.github.io/tree-sitter/
 https://github.com/tree-sitter/tree-sitter
 https://news.ycombinator.com/item?id=26225298
 
 Will be keeping an eye on this. If it gains traction*, it would be nice 
 (and possibly important) to have a Dlang parser. Haven't looked in depth 
 at the grammar yet -- appears to be context free grammar.
 
 *Supposedly Neovim will be using this going forward
I wonder how the text editor/IDE integration is supposed to work. To build the parser you need Node.js and a C compiler. Is the author of the grammar supposed to pre-compile everything and it's loaded as a dynamic library? That would defiantly make it less flexible that the current approach TextMate grammars. TextMate grammars are very easy to modify on the fly and the editor will immediately be able to use the new grammar. No building, no compilation and no loading of dynamic libraries. But I do know how limiting TextMate grammars are. I guess in theory the editor could ship with Node.js and a C compiler and compile on the fly to have the same flexibility as TextMate grammars. BTW, I'm wondering if it needs to be Node.js or if any JavaScript engine would work. At least macOS already ships with a JavaScript engine. -- /Jacob Carlborg
Feb 26
next sibling parent Mike Brown <mikey.be gmail.com> writes:
On Friday, 26 February 2021 at 20:06:46 UTC, Jacob Carlborg wrote:
 On 2021-02-22 16:56, James Blachly wrote:
 https://tree-sitter.github.io/tree-sitter/
 https://github.com/tree-sitter/tree-sitter
 https://news.ycombinator.com/item?id=26225298
 HN)
 
 Will be keeping an eye on this. If it gains traction*, it 
 would be nice (and possibly important) to have a Dlang parser. 
 Haven't looked in depth at the grammar yet -- appears to be 
 context free grammar.
 
 *Supposedly Neovim will be using this going forward
I wonder how the text editor/IDE integration is supposed to work. To build the parser you need Node.js and a C compiler. Is the author of the grammar supposed to pre-compile everything and it's loaded as a dynamic library? That would defiantly make it less flexible that the current approach TextMate grammars. TextMate grammars are very easy to modify on the fly and the editor will immediately be able to use the new grammar. No building, no compilation and no loading of dynamic libraries. But I do know how limiting TextMate grammars are. I guess in theory the editor could ship with Node.js and a C compiler and compile on the fly to have the same flexibility as TextMate grammars. BTW, I'm wondering if it needs to be Node.js or if any JavaScript engine would work. At least macOS already ships with a JavaScript engine.
tree-sitter is written in C, and all parsers are written in C. Atom is the primary editor using tree-sitter, that uses Node.js (/Electron). I also forgot to mention the other major benefit/goal to tree-sitter. It is an incremental parser, so the parse tree is updated at every keystoke. (this is mentioned in this threads title). Is this discussion for parsing in general, or for the D frontend/IDEs?
Feb 27
prev sibling parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Friday, 26 February 2021 at 20:06:46 UTC, Jacob Carlborg wrote:
 I wonder how the text editor/IDE integration is supposed to 
 work. To build the parser you need Node.js and a C compiler. Is 
 the author of the grammar supposed to pre-compile everything 
 and it's loaded as a dynamic library?
Yes.
 That would defiantly make it less flexible that the current 
 approach TextMate grammars. TextMate grammars are very easy to 
 modify on the fly and the editor will immediately be able to 
 use the new grammar. No building, no compilation and no loading 
 of dynamic libraries. But I do know how limiting TextMate 
 grammars are.
Yes, there is definitely an edit-compile-test cycle. The D grammar I am working on currently takes about 15 seconds to compile.
 I guess in theory the editor could ship with Node.js and a C 
 compiler and compile on the fly to have the same flexibility as 
 TextMate grammars.
The great majority of users won't need to edit the grammar, so I think this is not an important goal to aim for. Parsing speed would probably be better appreciated than the ability to edit the grammar on the fly.
 BTW, I'm wondering if it needs to be Node.js or if any 
 JavaScript engine would work. At least macOS already ships with 
 a JavaScript engine.
I believe the tool chain is built on node/npm. Grammar authors seem to keep the generated C files in git, so running the JavaScript part isn't needed to get the usable .so file. For the future, I understand that the plan is to target WebAssembly instead. WebAssembly binaries are portable and it executes sufficiently quickly.
Jun 14