www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Removing libdparse dependency from DLS

reply Cristian Creteanu <cristian.creteanu pm.me> writes:
I am working on DLS (https://github.com/d-language-server/dls), 
trying to change the implementation to use the compiler as a 
library. The reason behind this is to make DLS be always up to 
date with the compiler so that it won't provide any obsolete 
information.

At the moment, libdparse 
(https://github.com/dlang-community/libdparse) is used instead 
and I want to get rid of this dependency. However, as DLS doesn't 
really do much by itself, since it relies on other tools, such as 
dcd (https://github.com/dlang-community/DCD), dscanner 
(https://github.com/dlang-community/D-Scanner), dfmt 
(https://github.com/dlang-community/dfmt), to do its work. 
Unfortunately, these dependencies also use libdparse, so removing 
it from DLS wouldn't be as easy as just adding the dmd logic to 
the DLS code where libdparse is used.

The cleanest way to solve this would be to replace libdparse with 
dmd in dcd and dscanner, as well. Would this be a good idea? 
Should I approach this differently?
Feb 20 2020
next sibling parent Eugene Wissner <belka caraus.de> writes:
On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian Creteanu 
wrote:

 ?


https://forum.dlang.org/post/wrwbcujfccjjnnlhwgak forum.dlang.org
Feb 20 2020
prev sibling next sibling parent reply Laurent =?UTF-8?B?VHLDqWd1aWVy?= <laurent.treguier.sink gmail.com> writes:
On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian Creteanu 
wrote:

 I am working on DLS (https://github.com/d-language-server/dls), 
 trying to change the implementation to use the compiler as a 
 library. The reason behind this is to make DLS be always up to 
 date with the compiler so that it won't provide any obsolete 
 information.

 At the moment, libdparse 
 (https://github.com/dlang-community/libdparse) is used instead 
 and I want to get rid of this dependency. However, as DLS 
 doesn't really do much by itself, since it relies on other 
 tools, such as dcd (https://github.com/dlang-community/DCD), 
 dscanner (https://github.com/dlang-community/D-Scanner), dfmt 
 (https://github.com/dlang-community/dfmt), to do its work. 
 Unfortunately, these dependencies also use libdparse, so 
 removing it from DLS wouldn't be as easy as just adding the dmd 
 logic to the DLS code where libdparse is used.

 The cleanest way to solve this would be to replace libdparse 
 with dmd in dcd and dscanner, as well. Would this be a good 
 idea? Should I approach this differently?


As DLS' developer, I've looked into doing this myself a while 
back, close to a year ago now.
At that point I wasn't sure if it was really a good idea; even if 
DMD has technically been usable as a library for quite some time, 
in practice it wasn't as straightforward as I had hoped.

 From what I recall, the necessary changes to DMD would be:
- Removing any remaining global state. I don't know what the 
status of global state in DMD is nowadays. Although this might 
not be "necessary", it's always good to know that we can safely 
repeat calls to some APIs and always get the same output, without 
interference from some internal variables.
- Being able to fully control the standard input and output. When 
I tried to use DMD, there were some places where it was 
impossible to prevent DMD from using stdout IIRC. This is really 
critical, as standard input/output is the way language servers 
are used 99% of the time, and thus they need 110% control over 
stdin/stdout.
- Being able to free all the memory DMD allocates. From what I've 
read here, DMD doesn't free memory to get a higher compilation 
speed. However, language servers are long-running background 
programs, and any kind of memory leak is extremely bad in this 
situation. Since the apparition of the `-lowmem` flag, maybe this 
has improved.

Now, I haven't looked at all that since something like March of 
last year, so things might be better today (and they probably 
are, i think I recall Walter not being fond of DMD's global state 
either). I've gotten my hands busy with other projects, so 
development on DLS has basically halted and I don't expect to get 
back on it very soon.
My original plan wasn't to convert DCD/D-Scanner/DFMT, but to 
rewrite all functionality inside DLS from scratch using DMD. On 
the other hand, converting those tools would certainly benefit 
the whole community.

Rainers, VisualD's maintainer, has recently released a version of 
VisualD that can use DMD as a library; maybe its source code 
could provide you with some useful insight.

Good luck converting all those tools to use DMD, this is quite 
some work. If you do follow up on converting DLS, don't hesitate 
to open up issues in its repo if you have questions about how it 
works, I'll try my best to remember what I did and why!
Feb 20 2020
parent Jacob Carlborg <doob me.com> writes:
On 2020-02-20 21:53, Laurent Tréguier wrote:


  From what I recall, the necessary changes to DMD would be:
 - Removing any remaining global state. I don't know what the status of 
 global state in DMD is nowadays. Although this might not be "necessary", 
 it's always good to know that we can safely repeat calls to some APIs 
 and always get the same output, without interference from some internal 
 variables.


Still an issue. There are some APIs to reset the state to it's original 
state of the globals but they're not complete.


 - Being able to fully control the standard input and output. When I 
 tried to use DMD, there were some places where it was impossible to 
 prevent DMD from using stdout IIRC. This is really critical, as standard 
 input/output is the way language servers are used 99% of the time, and 
 thus they need 110% control over stdin/stdout.


Still an issue. There are other issue like it's not possible to 
configure the compiler to read everything from memory, for example. 
Regular imports and import expressions will be read from disk.

-- 
/Jacob Carlborg
Feb 21 2020
prev sibling next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian Creteanu 
wrote:

 [...]


I'll only comment this parapgraph since I don't use DLS:


 The cleanest way to solve this would be to replace libdparse 
 with dmd in dcd and dscanner, as well. Would this be a good 
 idea? Should I approach this differently?


You should rather start new tools. dsymbol (so the library used 
by dcd) has a lot of problems related to import and out-of-order 
declarations, templates or even expressions are not handled at 
all (BTW dparse doesn't decompose unaries at all !).

Replacing dparse by DMD-FE in the tools, from my POV, would be 
like puting a Rolls-Royce turbo reactor in an old Renault 4L car. 
The way the tools are made is too bent to dparse.
Feb 20 2020
parent reply RazvanN <razvan.nitu1305 gmail.com> writes:
On Friday, 21 February 2020 at 02:27:37 UTC, Basile B. wrote:

 On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian 
 Creteanu wrote:

 [...]


 I'll only comment this parapgraph since I don't use DLS:


 The cleanest way to solve this would be to replace libdparse 
 with dmd in dcd and dscanner, as well. Would this be a good 
 idea? Should I approach this differently?


 You should rather start new tools. dsymbol (so the library used 
 by dcd) has a lot of problems related to import and 
 out-of-order declarations, templates or even expressions are 
 not handled at all (BTW dparse doesn't decompose unaries at all 
 !).

 Replacing dparse by DMD-FE in the tools, from my POV, would be 
 like puting a Rolls-Royce turbo reactor in an old Renault 4L 
 car. The way the tools are made is too bent to dparse.


First, let me give you some context on this project. Cristi is a 
bachelor student that is doing his thesis with me and Edi.

The primary objective of the project is twofold: make DLs use dmd 
as library but at the same time improve the library interface and 
do the necessary compiler modifications so that dmd-fe becomes 
usable. Actually, integrating with DLS is just a dry run to see 
what kind of interface does the dmd-fe need to be usable in a 
real life project.

However, seeing how DLS uses all the other tools that rely on 
libdparse we thought that maybe it would be a better strategy to 
try and update those other tools.

Any suggestions are welcome.
Feb 20 2020
next sibling parent Basile . <b2.temp gmx.com> writes:
On Friday, 21 February 2020 at 07:57:48 UTC, RazvanN wrote:

 On Friday, 21 February 2020 at 02:27:37 UTC, Basile B. wrote:

 [...]


 First, let me give you some context on this project. Cristi is 
 a bachelor student that is doing his thesis with me and Edi.

 The primary objective of the project is twofold: make DLs use 
 dmd as library but at the same time improve the library 
 interface and do the necessary compiler modifications so that 
 dmd-fe becomes usable. Actually, integrating with DLS is just a 
 dry run to see what kind of interface does the dmd-fe need to 
 be usable in a real life project.

 However, seeing how DLS uses all the other tools that rely on 
 libdparse we thought that maybe it would be a better strategy 
 to try and update those other tools.

 Any suggestions are welcome.


Yes so I've clarified what I said a few minutes ago, see 
https://forum.dlang.org/post/bnmhguoawnljqkduwmcq forum.dlang.org. It's not
only about dparse, it's also about dsymbol (the library, not the dmd module).
Feb 21 2020
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2020-02-21 08:57, RazvanN wrote:


 First, let me give you some context on this project. Cristi is a 
 bachelor student that is doing his thesis with me and Edi.
 
 The primary objective of the project is twofold: make DLs use dmd as 
 library but at the same time improve the library interface and do the 
 necessary compiler modifications so that dmd-fe becomes usable. 
 Actually, integrating with DLS is just a dry run to see what kind of 
 interface does the dmd-fe need to be usable in a real life project.
 
 However, seeing how DLS uses all the other tools that rely on libdparse 
 we thought that maybe it would be a better strategy to try and update 
 those other tools.
 
 Any suggestions are welcome.


I would start at the bottom, not the top. That is, start by working on 
one of the tools, the one(s) at the lowest level, that DLS depends on.

I would probably start with the simplest one, that might be dfmt. But 
for, I think, that DMD needs to support end locations for tokens and 
possibly AST nodes as well.

-- 
/Jacob Carlborg
Feb 21 2020
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2020-02-21 13:53, Jacob Carlborg wrote:


 I would probably start with the simplest one, that might be dfmt. But 
 for, I think, that DMD needs to support end locations for tokens and 
 possibly AST nodes as well.


Oh, there are some AST nodes that don't contain any location at all.

-- 
/Jacob Carlborg
Feb 21 2020
prev sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 22/02/2020 1:53 AM, Jacob Carlborg wrote:

 On 2020-02-21 08:57, RazvanN wrote:
 

 First, let me give you some context on this project. Cristi is a 
 bachelor student that is doing his thesis with me and Edi.

 The primary objective of the project is twofold: make DLs use dmd as 
 library but at the same time improve the library interface and do the 
 necessary compiler modifications so that dmd-fe becomes usable. 
 Actually, integrating with DLS is just a dry run to see what kind of 
 interface does the dmd-fe need to be usable in a real life project.

 However, seeing how DLS uses all the other tools that rely on 
 libdparse we thought that maybe it would be a better strategy to try 
 and update those other tools.

 Any suggestions are welcome.


 
 I would start at the bottom, not the top. That is, start by working on 
 one of the tools, the one(s) at the lowest level, that DLS depends on.
 
 I would probably start with the simplest one, that might be dfmt. But 
 for, I think, that DMD needs to support end locations for tokens and 
 possibly AST nodes as well.


Dfix is < 1k LOC, easier to get into but may not teach as much (hence I 
suggest DCD as a starting point).

https://github.com/dlang-community/dfix/blob/master/src/dfix.d
Feb 21 2020
prev sibling next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
I and at least one other have discussed this in the past (I am one of 
the "owners" of dlang-community  rikkimax so feel free to ping).

DCD needs rewriting to use dmd-fe.

Keep or improve feature parity while stripping out how it does everything.

The other tools are in the same position, ignore DLS most IDE's don't 
use it in the D community for now and as you said it has dependencies on 
the other programs anyway.
Feb 20 2020
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Friday, 21 February 2020 at 03:42:37 UTC, rikki cattermole 
wrote:

 I and at least one other have discussed this in the past (I am 
 one of the "owners" of dlang-community  rikkimax so feel free 
 to ping).

 DCD needs rewriting to use dmd-fe.

 Keep or improve feature parity while stripping out how it does 
 everything.

 The other tools are in the same position, ignore DLS most IDE's 
 don't use it in the D community for now and as you said it has 
 dependencies on the other programs anyway.


If you drop dparse and dsymbol then DCD is an empty shell that 
only handles the requests and their serialization. Why keeping 
this ? That are other ways of implementing an IPC than sockets, 
that were never done in DCD (shared memory, standard IO).
Feb 20 2020
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 21/02/2020 6:21 PM, Basile B. wrote:

 On Friday, 21 February 2020 at 03:42:37 UTC, rikki cattermole wrote:

 I and at least one other have discussed this in the past (I am one of 
 the "owners" of dlang-community  rikkimax so feel free to ping).

 DCD needs rewriting to use dmd-fe.

 Keep or improve feature parity while stripping out how it does 
 everything.

 The other tools are in the same position, ignore DLS most IDE's don't 
 use it in the D community for now and as you said it has dependencies 
 on the other programs anyway.


 
 If you drop dparse and dsymbol then DCD is an empty shell that only 
 handles the requests and their serialization. Why keeping this ? That 
 are other ways of implementing an IPC than sockets, that were never done 
 in DCD (shared memory, standard IO).


Lets not break existing tooling just because we can.

New IO methods can be added later.
Feb 20 2020
parent Basile . <b2.temp gmx.com> writes:
On Friday, 21 February 2020 at 05:37:53 UTC, rikki cattermole 
wrote:

 On 21/02/2020 6:21 PM, Basile B. wrote:

 If you drop dparse and dsymbol then DCD is an empty shell that 
 only handles the requests and their serialization. Why keeping 
 this ? That are other ways of implementing an IPC than 
 sockets, that were never done in DCD (shared memory, standard 
 IO).


 Lets not break existing tooling just because we can.

 New IO methods can be added later.


Let's say but the most important point I would say then is to 
stop using dsymbol.
That was the principal idea raised by my first answer. for DCD 
both dparse and dsymbol must be droped, otherwise it'll end up 
with the same limitation as the actual ones.
Feb 20 2020
prev sibling parent reply Laurent =?UTF-8?B?VHLDqWd1aWVy?= <laurent.treguier.sink gmail.com> writes:
On Friday, 21 February 2020 at 03:42:37 UTC, rikki cattermole 
wrote:

 ignore DLS most IDE's don't use it


I'll chime in a bit to disagree on that. I don't know where 
you're getting this idea from, but DLS can be used with VSCode, 
Visual Studio, Sublime text, Atom, Vim/Neovim, Emacs, and 
Intellij IDEA. It's certainly not as popular as code-d and 
VisualD, but it's certainly usable with a whole range of editors 
and IDE's
Feb 20 2020
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 21/02/2020 8:24 PM, Laurent Tréguier wrote:

 On Friday, 21 February 2020 at 03:42:37 UTC, rikki cattermole wrote:

 ignore DLS most IDE's don't use it


 
 I'll chime in a bit to disagree on that. I don't know where you're 
 getting this idea from, but DLS can be used with VSCode, Visual Studio, 
 Sublime text, Atom, Vim/Neovim, Emacs, and Intellij IDEA. It's certainly 
 not as popular as code-d and VisualD, but it's certainly usable with a 
 whole range of editors and IDE's


For the more used IDE's the community from what I have observed use the 
dedicated plugins for the IDE's to integrate into external resources 
like DCD.

I was not commenting on if an IDE supports it, but instead on what the 
priority is.

Keeping in mind that this is in context on rewriting DLS which depends 
on these very same resources.
Feb 20 2020
next sibling parent Laurent =?UTF-8?B?VHLDqWd1aWVy?= <laurent.treguier.sink gmail.com> writes:
On Friday, 21 February 2020 at 07:34:39 UTC, rikki cattermole 
wrote:

 For the more used IDE's the community from what I have observed 
 use the dedicated plugins for the IDE's to integrate into 
 external resources like DCD.

 I was not commenting on if an IDE supports it, but instead on 
 what the priority is.

 Keeping in mind that this is in context on rewriting DLS which 
 depends on these very same resources.


My mistake, I misinterpreted what you said indeed.
Feb 20 2020
prev sibling parent drug <drug2004 bk.ru> writes:
On 2/21/20 10:34 AM, rikki cattermole wrote:

 For the more used IDE's the community from what I have observed use the 
 dedicated plugins for the IDE's to integrate into external resources 
 like DCD.
 
 I was not commenting on if an IDE supports it, but instead on what the 
 priority is.
 
 Keeping in mind that this is in context on rewriting DLS which depends 
 on these very same resources.


Well, for example me prefer DLS over code-d for long time
Feb 21 2020
prev sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Friday, 21 February 2020 at 07:24:10 UTC, Laurent Tréguier 
wrote:

 On Friday, 21 February 2020 at 03:42:37 UTC, rikki cattermole 
 wrote:

 ignore DLS most IDE's don't use it


 I'll chime in a bit to disagree on that. I don't know where 
 you're getting this idea from, but DLS can be used with VSCode, 
 Visual Studio, Sublime text, Atom, Vim/Neovim, Emacs, and 
 Intellij IDEA. It's certainly not as popular as code-d and 
 VisualD, but it's certainly usable with a whole range of 
 editors and IDE's


+1

I use it with VSCode and I'm very satisfied about it.

  Some period ago, it was the only VSCode plugin with a "just 
work" approach, while vs-code was plagued by instability, also if 
right now it works pretty well also. I especially love the idea 
of glueing all the different tools together.
Feb 21 2020
prev sibling parent Mathias Lang <pro.mathias.lang gmail.com> writes:
On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian Creteanu 
wrote:

 The cleanest way to solve this would be to replace libdparse 
 with dmd in dcd and dscanner, as well. Would this be a good 
 idea? Should I approach this differently?


Not familiar with DLS / DCD source code, but quite familiar with 
DMD.
Things have definitely gotten better since the first time I 
attempted this, but it's definitely non-trivial, depending on 
what you are trying to do.

I can confirm that the global state and the memory allocation are 
still issues.
I don't know about standard output, has some PRs have been made 
to use a delegate (https://github.com/dlang/dmd/pull/10072 and 
https://github.com/dlang/dmd/pull/10711), so things might be 
better on that front.

Two other large issues:
- DMD is not built to be able to handle edit cycles, only 
complete & valid source files.
   When working on an IDE, you'll have to deal with incomplete 
source code (e.g. typing `format("`) and you want to be able to 
provide completion for parameter, return value, overload sets, 
etc... Doing so would require large refactorings in DMD which are 
likely to raise some eyebrows.
- DMD semantic analysis is destructive. Many things are rewritten 
and some links are lost, and the rewrite might not be valid D 
code. One example, any `alias this` is just rewritten as a direct 
call, but bypassing visibility checks on the target, so writing 
the same code would produce an error.
  Some expressions are rewritten as ComaExpression and use the 
return of that expression. This is also very evident with 
`alias`, as they are resolved as early as possible. There are 
quite a few places in DMD that work around this issue (e.g. 
`immutable(char)[]` is special-cased to always show as `string` 
in error message, and the POSIX C++ mangler uses a mix of pre and 
post-semantic AST to work around that issue as soon as templates 
are involved).
Feb 20 2020