www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - A new Tree-Sitter Grammar for D

reply Garrett D'Amore <garrett damore.org> writes:
I'm happy to announce that I've created what I believe is a 
complete, or at least very nearly so, Tree-Sitter grammar for D.

You can find it at https://github.com/gdamore/tree-sitter-d

Tree-Sitter is a tool that enables dynamic AST generation for a 
variety of purposes, and is becoming quite popular with many 
editor projects.

I've tested this grammar with as many different sources as I can 
find, including the test cases for the DMD compiler itself, as 
well as various other community sources and proprietary sources.

It does not include support for preview syntaxes for bit fields 
or shortened function bodies, but I believe it should cover just 
about every other case.  I've been using this with the Helix 
editor, along with the Serve-D language server, with some success.

Included in my repository are queries for highlighting, injection 
(really just comments), and text objects (so you can navigate 
across major structures if your editor supports it.). I have not 
yet implemented indent queries.

This work includes a test suite that has a lot of test cases, but 
of course is probably still far from complete.

For folks that care, out of 1067 test cases in the DMD compiler, 
this parses successfully all but 5.  The five that do not parse 
are ones that contain errors in uninstantiated templates, a 
problem with #line directives involving multi-line comments (you 
should never encounter this!) and preview syntax support already 
mentioned.

This grammar is slightly more strict than the officially posted 
grammar, as some constructs which are flagged only at semantic 
analysis are caught at parse time in my grammar.  (Notably comma 
expressions are not legal in constructs where they would be 
evaluated as a single value -- DMD generates a compilation error 
at semantic analysis time whereas my grammar simply rejects them 
as legal syntax.  This was done to reduce the overall size of the 
generated parser as reduce the number of conflicts that would 
have resolution.)

I believe this grammar may be the complete and accurate machine 
readable grammar outside of the DMD compiler itself.  Certainly 
this has fixes to numerous defects found in both libdparse and in 
the official grammar, although both those projects were extremely 
useful as foundations to build upon.  It is my hope that others 
will find this useful.

I do welcome contributions of all forms -- whether bug reports, 
additional test cases, or grammar fixes or corrections.  I am 
quite new to both Tree Sitter and to D, so it's entirely possible 
that I've missed something or misunderstood something!

I will probably see if this can be adopted into either the Tree 
Sitter or DLang community projects -- I'm not sure which is the 
better location.  If you have thoughts please don't hesitate to 
let me know.  I'm quite sure that the grammar itself could 
probably benefit from some further optimization, and I welcome 
advice or contributions!
Oct 16 2022
next sibling parent Tejas <notrealemail gmail.com> writes:
On Monday, 17 October 2022 at 05:21:10 UTC, Garrett D'Amore wrote:

 I will probably see if this can be adopted into either the Tree 
 Sitter or DLang community projects -- I'm not sure which is the 
 better location.  If you have thoughts please don't hesitate to 
 let me know.  I'm quite sure that the grammar itself could 
 probably benefit from some further optimization, and I welcome 
 advice or contributions!


Please try to put it in tree-sitter official, since that will 
allow other editors(like the one I use) to automatically provide 
it as an option.


Thanks for this wonderful project, really sad that the 
cybershadow one is basically stalled, but I sure hope that this 
replacement will fill in the gap wonderfully!!
Oct 16 2022
prev sibling next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Monday, 17 October 2022 at 05:21:10 UTC, Garrett D'Amore wrote:

 I'm happy to announce that I've created what I believe is a 
 complete, or at least very nearly so, Tree-Sitter grammar for D.

 You can find it at https://github.com/gdamore/tree-sitter-d


Congratulations!

Linking to a response in another thread you pinged - perspective 
from the existing project: 
https://github.com/CyberShadow/tree-sitter-d/issues/3#issuecomment-1280343366
Oct 16 2022
parent user1234 <user1234 12.de> writes:
On Monday, 17 October 2022 at 06:24:08 UTC, Vladimir Panteleev 
wrote:

 On Monday, 17 October 2022 at 05:21:10 UTC, Garrett D'Amore 
 wrote:

 I'm happy to announce that I've created what I believe is a 
 complete, or at least very nearly so, Tree-Sitter grammar for 
 D.

 You can find it at https://github.com/gdamore/tree-sitter-d


 Congratulations!

 Linking to a response in another thread you pinged - 
 perspective from the existing project: 
 https://github.com/CyberShadow/tree-sitter-d/issues/3#issuecomment-1280343366


As mentioned in your answer, there could be a problem of 
maintenance why not making the project an official dlang.org 
project ? I mean "right now", even if unfinished.
Oct 17 2022
prev sibling next sibling parent rassoc <rassoc posteo.de> writes:
On 10/17/22 07:21, Garrett D'Amore via Digitalmars-d-announce wrote:

 I'm happy to announce that I've created what I believe is a complete, or at
least very nearly so, Tree-Sitter grammar for D.
 


Woot, thank you for your work on that front!
Oct 17 2022
prev sibling next sibling parent reply Jack Applegame <japplegame gmail.com> writes:
On Monday, 17 October 2022 at 05:21:10 UTC, Garrett D'Amore wrote:

 I'm happy to announce that I've created what I believe is a 
 complete, or at least very nearly so, Tree-Sitter grammar for D.

 You can find it at https://github.com/gdamore/tree-sitter-d


What do you think? If I don't want to #StandWithUkraine, should I 
stay away from your protestware?
Oct 27 2022
parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Thursday, 27 October 2022 at 07:37:41 UTC, Jack Applegame 
wrote:

 On Monday, 17 October 2022 at 05:21:10 UTC, Garrett D'Amore 
 wrote:

 I'm happy to announce that I've created what I believe is a 
 complete, or at least very nearly so, Tree-Sitter grammar for 
 D.

 You can find it at https://github.com/gdamore/tree-sitter-d


 What do you think? If I don't want to #StandWithUkraine, should 
 I stay away from your protestware?


You should remember this is not a political platform and just use 
the code if you find it useful.
Oct 27 2022
prev sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Monday, 17 October 2022 at 05:21:10 UTC, Garrett D'Amore wrote:

 I'm happy to announce that I've created what I believe is a 
 complete, or at least very nearly so, Tree-Sitter grammar for D.

 [...]


Nice project! Good work
Oct 27 2022