www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - dmdtags 1.0.0: an accurate tag generator for D source code

reply Paul Backus <snarwin gmail.com> writes:
`dmdtags` is a tags file generator for D source code that uses 
the DMD compiler frontend for accurate parsing.

This release supports 100%-accurate parsing of arbitrary D code 
(tested on DMD and Phobos sources), as well as the most 
commonly-used command line options, `-R`, `-o`, and `-a`. The 
generated tags file has been tested for compatibility with Vim 
and is compliant with the [POSIX standard for `ctags`][posix], so 
any editor with `ctags` support should be able to use it.

[posix]: 
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/ctags.html



A _tags file_ is a lightweight plain-text index of the symbols in 
a project. Editors that support tags files, such as Vim and 
Emacs, can use this index to help with things like project 
navigation and tab completion.

A _tags file generator_ is, as you might expect, a program that 
reads source code and generates a tags file with entries for the 
symbols in that code.



[`universal-ctags`][uctags], the current most-popular and 
best-maintained tags file generator, claims support for many 
programming languages, including D. However, its D parser is not 
well-maintained, and it often excludes large numbers of symbols 
from its output due to parsing failures.

Because `dmdtags` uses the DMD frontend for parsing, its results 
will always be accurate and up-to-date. For pure D projects, it 
can be used as a replacement for `universal-ctags`. For 
mixed-language projects, it can be used together with other tag 
generators with the `--append` option.

[uctags]: https://ctags.io



* On Github: https://github.com/pbackus/dmdtags
* On Dub: https://code.dlang.org/packages/dmdtags
Aug 27 2021
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Friday, 27 August 2021 at 21:38:58 UTC, Paul Backus wrote:

 Editors that support tags files, such as Vim and Emacs, can use 
 this index to help with things like project navigation and tab 
 completion.


Cool! Now I'll have to look if I can make this work with Visual 
Studio Code, since code-d which uses Dsymbol tends to be 
unreliable, and this looks ideal for Phobos / Druntime symbol 
completion.
Aug 27 2021
parent reply WebFreak001 <d.forum webfreak.org> writes:
On Friday, 27 August 2021 at 22:01:59 UTC, Dennis wrote:

 On Friday, 27 August 2021 at 21:38:58 UTC, Paul Backus wrote:

 Editors that support tags files, such as Vim and Emacs, can 
 use this index to help with things like project navigation and 
 tab completion.


 Cool! Now I'll have to look if I can make this work with Visual 
 Studio Code, since code-d which uses Dsymbol tends to be 
 unreliable, and this looks ideal for Phobos / Druntime symbol 
 completion.


actually it uses D-Scanner and should be a fairly easy drop-in 
replacement.

I'm just worried about how the memory usage will grow with this, 
considering dmd never frees. Maybe I should make it run as 
external tool instead of a library so the OS cleans up, but for 
that get a performance penalty especially on Windows.
Aug 27 2021
next sibling parent Paul Backus <snarwin gmail.com> writes:
On Friday, 27 August 2021 at 22:45:15 UTC, WebFreak001 wrote:

 I'm just worried about how the memory usage will grow with 
 this, considering dmd never frees. Maybe I should make it run 
 as external tool instead of a library so the OS cleans up, but 
 for that get a performance penalty especially on Windows.


`dmdtags` does not run in the background, so any memory it uses 
will be freed at process exit, as soon as it has finished writing 
the tags file. And because it does not do any semantic analysis, 
only parsing, it does not use much memory while running in the 
first place.

According to `/usr/bin/time`, on my personal machine, using 
`dmdtags` to generate a tags file for all of Phobos takes 
slightly less than 1 second and has a peak resident set size of 
around 100MB.
Aug 27 2021
prev sibling parent evilrat <evilrat666 gmail.com> writes:
On Friday, 27 August 2021 at 22:45:15 UTC, WebFreak001 wrote:

 I'm just worried about how the memory usage will grow with 
 this, considering dmd never frees.


Still not sure if it *should* be used in same process, but DMD 
definitely has -lowmem switch for turning on GC, and for DMD as a 
library there is initDMD() and deinitializeDMD() pair.

Haven't tested it myself wrt. to memory usage, but it is used by 
DMD tests.
Another concern would be code analysis time, even with skipping 
semantic passes every run is a new invocation - DMD simply wasn't 
designed to be a language server where it could just iteratively 
update compilation database.

IIRC VisualD with DMD frontend enabled has noticable long pauses 
on showing completion (from 100ms to few seconds on larger 
projects), which renders it unusable for me. Could be that same 
reason, no idea.
Aug 27 2021
prev sibling next sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 8/27/21 2:38 PM, Paul Backus wrote:


 `dmdtags` is a tags file generator for D source code that uses the DMD
 compiler frontend for accurate parsing.


Thanks! I had stopped using tags files with Emacs for the reasons you 
give. With this, I will be more efficient. :)

And, I will be a happy puppy once Vladimir Panteleev finishes his D mode 
for Emacs, which he is writing from scratch.

Ali
Aug 27 2021
prev sibling next sibling parent =?UTF-8?B?0JLQuNGC0LDQu9C40Lkg0KTQsNC0?= =?UTF-8?B?0LXQtdCy?= writes:
On Friday, 27 August 2021 at 21:38:58 UTC, Paul Backus wrote:

 `dmdtags` is a tags file generator for D source code that uses 
 the DMD compiler frontend for accurate parsing.

 This release supports 100%-accurate parsing of arbitrary D code 
 (tested on DMD and Phobos sources), as well as the most 
 commonly-used command line options, `-R`, `-o`, and `-a`. The 
 generated tags file has been tested for compatibility with Vim 
 and is compliant with the [POSIX standard for `ctags`][posix], 
 so any editor with `ctags` support should be able to use it.

 [posix]: 
 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/ctags.html



 A _tags file_ is a lightweight plain-text index of the symbols 
 in a project. Editors that support tags files, such as Vim and 
 Emacs, can use this index to help with things like project 
 navigation and tab completion.

 A _tags file generator_ is, as you might expect, a program that 
 reads source code and generates a tags file with entries for 
 the symbols in that code.



 [`universal-ctags`][uctags], the current most-popular and 
 best-maintained tags file generator, claims support for many 
 programming languages, including D. However, its D parser is 
 not well-maintained, and it often excludes large numbers of 
 symbols from its output due to parsing failures.

 Because `dmdtags` uses the DMD frontend for parsing, its 
 results will always be accurate and up-to-date. For pure D 
 projects, it can be used as a replacement for 
 `universal-ctags`. For mixed-language projects, it can be used 
 together with other tag generators with the `--append` option.

 [uctags]: https://ctags.io



 * On Github: https://github.com/pbackus/dmdtags
 * On Dub: https://code.dlang.org/packages/dmdtags



```

!_TAG_FILE_SORTED	1	/0=unsorted, 1=sorted, 2=foldcase/
App	./source/ui/app.d	13;"	T
App	./source/ui/app.d	15;"	a
App	./source/ui/app.d	19;"	T
App	./source/ui/app.d	21;"	s
__ctor	./source/ui/app.d	27;"	f
__ctor	./source/ui/app.d	30;"	f
_result	./source/ui/app.d	23;"	v
app	./source/ui/app.d	1;"	M
defaultEventLoop	./source/ui/app.d	48;"	f
defaultInitFunc	./source/ui/app.d	62;"	T
defaultInitFunc	./source/ui/app.d	62;"	f
window	./source/ui/app.d	26;"	v
```

Perfecto!

i can use it in Sublime Text editor...
Need to thinks, how it will be implemented...
Aug 27 2021
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Aug 27, 2021 at 09:38:58PM +0000, Paul Backus via
Digitalmars-d-announce wrote:

 `dmdtags` is a tags file generator for D source code that uses the DMD
 compiler frontend for accurate parsing.
 
 This release supports 100%-accurate parsing of arbitrary D code
 (tested on DMD and Phobos sources), as well as the most commonly-used
 command line options, `-R`, `-o`, and `-a`. The generated tags file
 has been tested for compatibility with Vim and is compliant with the
 [POSIX standard for `ctags`][posix], so any editor with `ctags`
 support should be able to use it.


This is AWESOME!!!  Thanks a ton for this... I'll definitely be using
this in the near future!


[...]

 [`universal-ctags`][uctags], the current most-popular and
 best-maintained tags file generator, claims support for many
 programming languages, including D. However, its D parser is not
 well-maintained, and it often excludes large numbers of symbols from
 its output due to parsing failures.
 
 Because `dmdtags` uses the DMD frontend for parsing, its results will
 always be accurate and up-to-date. For pure D projects, it can be used
 as a replacement for `universal-ctags`. For mixed-language projects,
 it can be used together with other tag generators with the `--append`
 option.


Is there any hope of merging this back to upstream ctags?

Regardless, this is awesome.


T

-- 
Long, long ago, the ancient Chinese invented a device that lets them see
through walls. It was called the "window".
Aug 31 2021
prev sibling parent Chris Piker <chris hoopjump.com> writes:
On Friday, 27 August 2021 at 21:38:58 UTC, Paul Backus wrote:

 `dmdtags` is a tags file generator for D source code that uses 
 the DMD compiler frontend for accurate parsing.


...


 * On Github: https://github.com/pbackus/dmdtags
 * On Dub: https://code.dlang.org/packages/dmdtags


Wow, what perfect timing!  I just switched sublime text only 4
days ago.  Time to read some docs and figure out how to set this
up.

Thanks a lot!
Sep 03 2021