www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Removing libdparse dependency from DLS

reply Cristian Creteanu <cristian.creteanu pm.me> writes:
I am working on DLS (https://github.com/d-language-server/dls), 
trying to change the implementation to use the compiler as a 
library. The reason behind this is to make DLS be always up to 
date with the compiler so that it won't provide any obsolete 
information.

At the moment, libdparse 
(https://github.com/dlang-community/libdparse) is used instead 
and I want to get rid of this dependency. However, as DLS doesn't 
really do much by itself, since it relies on other tools, such as 
dcd (https://github.com/dlang-community/DCD), dscanner 
(https://github.com/dlang-community/D-Scanner), dfmt 
(https://github.com/dlang-community/dfmt), to do its work. 
Unfortunately, these dependencies also use libdparse, so removing 
it from DLS wouldn't be as easy as just adding the dmd logic to 
the DLS code where libdparse is used.

The cleanest way to solve this would be to replace libdparse with 
dmd in dcd and dscanner, as well. Would this be a good idea? 
Should I approach this differently?
Feb 20
next sibling parent Eugene Wissner <belka caraus.de> writes:
On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian Creteanu 
wrote:
 ?
https://forum.dlang.org/post/wrwbcujfccjjnnlhwgak forum.dlang.org
Feb 20
prev sibling next sibling parent reply Laurent =?UTF-8?B?VHLDqWd1aWVy?= <laurent.treguier.sink gmail.com> writes:
On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian Creteanu 
wrote:
 I am working on DLS (https://github.com/d-language-server/dls), 
 trying to change the implementation to use the compiler as a 
 library. The reason behind this is to make DLS be always up to 
 date with the compiler so that it won't provide any obsolete 
 information.

 At the moment, libdparse 
 (https://github.com/dlang-community/libdparse) is used instead 
 and I want to get rid of this dependency. However, as DLS 
 doesn't really do much by itself, since it relies on other 
 tools, such as dcd (https://github.com/dlang-community/DCD), 
 dscanner (https://github.com/dlang-community/D-Scanner), dfmt 
 (https://github.com/dlang-community/dfmt), to do its work. 
 Unfortunately, these dependencies also use libdparse, so 
 removing it from DLS wouldn't be as easy as just adding the dmd 
 logic to the DLS code where libdparse is used.

 The cleanest way to solve this would be to replace libdparse 
 with dmd in dcd and dscanner, as well. Would this be a good 
 idea? Should I approach this differently?
As DLS' developer, I've looked into doing this myself a while back, close to a year ago now. At that point I wasn't sure if it was really a good idea; even if DMD has technically been usable as a library for quite some time, in practice it wasn't as straightforward as I had hoped. From what I recall, the necessary changes to DMD would be: - Removing any remaining global state. I don't know what the status of global state in DMD is nowadays. Although this might not be "necessary", it's always good to know that we can safely repeat calls to some APIs and always get the same output, without interference from some internal variables. - Being able to fully control the standard input and output. When I tried to use DMD, there were some places where it was impossible to prevent DMD from using stdout IIRC. This is really critical, as standard input/output is the way language servers are used 99% of the time, and thus they need 110% control over stdin/stdout. - Being able to free all the memory DMD allocates. From what I've read here, DMD doesn't free memory to get a higher compilation speed. However, language servers are long-running background programs, and any kind of memory leak is extremely bad in this situation. Since the apparition of the `-lowmem` flag, maybe this has improved. Now, I haven't looked at all that since something like March of last year, so things might be better today (and they probably are, i think I recall Walter not being fond of DMD's global state either). I've gotten my hands busy with other projects, so development on DLS has basically halted and I don't expect to get back on it very soon. My original plan wasn't to convert DCD/D-Scanner/DFMT, but to rewrite all functionality inside DLS from scratch using DMD. On the other hand, converting those tools would certainly benefit the whole community. Rainers, VisualD's maintainer, has recently released a version of VisualD that can use DMD as a library; maybe its source code could provide you with some useful insight. Good luck converting all those tools to use DMD, this is quite some work. If you do follow up on converting DLS, don't hesitate to open up issues in its repo if you have questions about how it works, I'll try my best to remember what I did and why!
Feb 20
parent Jacob Carlborg <doob me.com> writes:
On 2020-02-20 21:53, Laurent Tréguier wrote:

  From what I recall, the necessary changes to DMD would be:
 - Removing any remaining global state. I don't know what the status of 
 global state in DMD is nowadays. Although this might not be "necessary", 
 it's always good to know that we can safely repeat calls to some APIs 
 and always get the same output, without interference from some internal 
 variables.
Still an issue. There are some APIs to reset the state to it's original state of the globals but they're not complete.
 - Being able to fully control the standard input and output. When I 
 tried to use DMD, there were some places where it was impossible to 
 prevent DMD from using stdout IIRC. This is really critical, as standard 
 input/output is the way language servers are used 99% of the time, and 
 thus they need 110% control over stdin/stdout.
Still an issue. There are other issue like it's not possible to configure the compiler to read everything from memory, for example. Regular imports and import expressions will be read from disk. -- /Jacob Carlborg
Feb 21
prev sibling next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian Creteanu 
wrote:
 [...]
I'll only comment this parapgraph since I don't use DLS:
 The cleanest way to solve this would be to replace libdparse 
 with dmd in dcd and dscanner, as well. Would this be a good 
 idea? Should I approach this differently?
You should rather start new tools. dsymbol (so the library used by dcd) has a lot of problems related to import and out-of-order declarations, templates or even expressions are not handled at all (BTW dparse doesn't decompose unaries at all !). Replacing dparse by DMD-FE in the tools, from my POV, would be like puting a Rolls-Royce turbo reactor in an old Renault 4L car. The way the tools are made is too bent to dparse.
Feb 20
parent reply RazvanN <razvan.nitu1305 gmail.com> writes:
On Friday, 21 February 2020 at 02:27:37 UTC, Basile B. wrote:
 On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian 
 Creteanu wrote:
 [...]
I'll only comment this parapgraph since I don't use DLS:
 The cleanest way to solve this would be to replace libdparse 
 with dmd in dcd and dscanner, as well. Would this be a good 
 idea? Should I approach this differently?
You should rather start new tools. dsymbol (so the library used by dcd) has a lot of problems related to import and out-of-order declarations, templates or even expressions are not handled at all (BTW dparse doesn't decompose unaries at all !). Replacing dparse by DMD-FE in the tools, from my POV, would be like puting a Rolls-Royce turbo reactor in an old Renault 4L car. The way the tools are made is too bent to dparse.
First, let me give you some context on this project. Cristi is a bachelor student that is doing his thesis with me and Edi. The primary objective of the project is twofold: make DLs use dmd as library but at the same time improve the library interface and do the necessary compiler modifications so that dmd-fe becomes usable. Actually, integrating with DLS is just a dry run to see what kind of interface does the dmd-fe need to be usable in a real life project. However, seeing how DLS uses all the other tools that rely on libdparse we thought that maybe it would be a better strategy to try and update those other tools. Any suggestions are welcome.
Feb 20
next sibling parent Basile . <b2.temp gmx.com> writes:
On Friday, 21 February 2020 at 07:57:48 UTC, RazvanN wrote:
 On Friday, 21 February 2020 at 02:27:37 UTC, Basile B. wrote:
 [...]
First, let me give you some context on this project. Cristi is a bachelor student that is doing his thesis with me and Edi. The primary objective of the project is twofold: make DLs use dmd as library but at the same time improve the library interface and do the necessary compiler modifications so that dmd-fe becomes usable. Actually, integrating with DLS is just a dry run to see what kind of interface does the dmd-fe need to be usable in a real life project. However, seeing how DLS uses all the other tools that rely on libdparse we thought that maybe it would be a better strategy to try and update those other tools. Any suggestions are welcome.
Yes so I've clarified what I said a few minutes ago, see https://forum.dlang.org/post/bnmhguoawnljqkduwmcq forum.dlang.org. It's not only about dparse, it's also about dsymbol (the library, not the dmd module).
Feb 21
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2020-02-21 08:57, RazvanN wrote:

 First, let me give you some context on this project. Cristi is a 
 bachelor student that is doing his thesis with me and Edi.
 
 The primary objective of the project is twofold: make DLs use dmd as 
 library but at the same time improve the library interface and do the 
 necessary compiler modifications so that dmd-fe becomes usable. 
 Actually, integrating with DLS is just a dry run to see what kind of 
 interface does the dmd-fe need to be usable in a real life project.
 
 However, seeing how DLS uses all the other tools that rely on libdparse 
 we thought that maybe it would be a better strategy to try and update 
 those other tools.
 
 Any suggestions are welcome.
I would start at the bottom, not the top. That is, start by working on one of the tools, the one(s) at the lowest level, that DLS depends on. I would probably start with the simplest one, that might be dfmt. But for, I think, that DMD needs to support end locations for tokens and possibly AST nodes as well. -- /Jacob Carlborg
Feb 21
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2020-02-21 13:53, Jacob Carlborg wrote:

 I would probably start with the simplest one, that might be dfmt. But 
 for, I think, that DMD needs to support end locations for tokens and 
 possibly AST nodes as well.
Oh, there are some AST nodes that don't contain any location at all. -- /Jacob Carlborg
Feb 21
prev sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 22/02/2020 1:53 AM, Jacob Carlborg wrote:
 On 2020-02-21 08:57, RazvanN wrote:
 
 First, let me give you some context on this project. Cristi is a 
 bachelor student that is doing his thesis with me and Edi.

 The primary objective of the project is twofold: make DLs use dmd as 
 library but at the same time improve the library interface and do the 
 necessary compiler modifications so that dmd-fe becomes usable. 
 Actually, integrating with DLS is just a dry run to see what kind of 
 interface does the dmd-fe need to be usable in a real life project.

 However, seeing how DLS uses all the other tools that rely on 
 libdparse we thought that maybe it would be a better strategy to try 
 and update those other tools.

 Any suggestions are welcome.
I would start at the bottom, not the top. That is, start by working on one of the tools, the one(s) at the lowest level, that DLS depends on. I would probably start with the simplest one, that might be dfmt. But for, I think, that DMD needs to support end locations for tokens and possibly AST nodes as well.
Dfix is < 1k LOC, easier to get into but may not teach as much (hence I suggest DCD as a starting point). https://github.com/dlang-community/dfix/blob/master/src/dfix.d
Feb 21
prev sibling next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
I and at least one other have discussed this in the past (I am one of 
the "owners" of dlang-community  rikkimax so feel free to ping).

DCD needs rewriting to use dmd-fe.

Keep or improve feature parity while stripping out how it does everything.

The other tools are in the same position, ignore DLS most IDE's don't 
use it in the D community for now and as you said it has dependencies on 
the other programs anyway.
Feb 20
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Friday, 21 February 2020 at 03:42:37 UTC, rikki cattermole 
wrote:
 I and at least one other have discussed this in the past (I am 
 one of the "owners" of dlang-community  rikkimax so feel free 
 to ping).

 DCD needs rewriting to use dmd-fe.

 Keep or improve feature parity while stripping out how it does 
 everything.

 The other tools are in the same position, ignore DLS most IDE's 
 don't use it in the D community for now and as you said it has 
 dependencies on the other programs anyway.
If you drop dparse and dsymbol then DCD is an empty shell that only handles the requests and their serialization. Why keeping this ? That are other ways of implementing an IPC than sockets, that were never done in DCD (shared memory, standard IO).
Feb 20
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 21/02/2020 6:21 PM, Basile B. wrote:
 On Friday, 21 February 2020 at 03:42:37 UTC, rikki cattermole wrote:
 I and at least one other have discussed this in the past (I am one of 
 the "owners" of dlang-community  rikkimax so feel free to ping).

 DCD needs rewriting to use dmd-fe.

 Keep or improve feature parity while stripping out how it does 
 everything.

 The other tools are in the same position, ignore DLS most IDE's don't 
 use it in the D community for now and as you said it has dependencies 
 on the other programs anyway.
If you drop dparse and dsymbol then DCD is an empty shell that only handles the requests and their serialization. Why keeping this ? That are other ways of implementing an IPC than sockets, that were never done in DCD (shared memory, standard IO).
Lets not break existing tooling just because we can. New IO methods can be added later.
Feb 20
parent Basile . <b2.temp gmx.com> writes:
On Friday, 21 February 2020 at 05:37:53 UTC, rikki cattermole 
wrote:
 On 21/02/2020 6:21 PM, Basile B. wrote:
 If you drop dparse and dsymbol then DCD is an empty shell that 
 only handles the requests and their serialization. Why keeping 
 this ? That are other ways of implementing an IPC than 
 sockets, that were never done in DCD (shared memory, standard 
 IO).
Lets not break existing tooling just because we can. New IO methods can be added later.
Let's say but the most important point I would say then is to stop using dsymbol. That was the principal idea raised by my first answer. for DCD both dparse and dsymbol must be droped, otherwise it'll end up with the same limitation as the actual ones.
Feb 20
prev sibling parent reply Laurent =?UTF-8?B?VHLDqWd1aWVy?= <laurent.treguier.sink gmail.com> writes:
On Friday, 21 February 2020 at 03:42:37 UTC, rikki cattermole 
wrote:
 ignore DLS most IDE's don't use it
I'll chime in a bit to disagree on that. I don't know where you're getting this idea from, but DLS can be used with VSCode, Visual Studio, Sublime text, Atom, Vim/Neovim, Emacs, and Intellij IDEA. It's certainly not as popular as code-d and VisualD, but it's certainly usable with a whole range of editors and IDE's
Feb 20
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 21/02/2020 8:24 PM, Laurent Tréguier wrote:
 On Friday, 21 February 2020 at 03:42:37 UTC, rikki cattermole wrote:
 ignore DLS most IDE's don't use it
I'll chime in a bit to disagree on that. I don't know where you're getting this idea from, but DLS can be used with VSCode, Visual Studio, Sublime text, Atom, Vim/Neovim, Emacs, and Intellij IDEA. It's certainly not as popular as code-d and VisualD, but it's certainly usable with a whole range of editors and IDE's
For the more used IDE's the community from what I have observed use the dedicated plugins for the IDE's to integrate into external resources like DCD. I was not commenting on if an IDE supports it, but instead on what the priority is. Keeping in mind that this is in context on rewriting DLS which depends on these very same resources.
Feb 20
next sibling parent Laurent =?UTF-8?B?VHLDqWd1aWVy?= <laurent.treguier.sink gmail.com> writes:
On Friday, 21 February 2020 at 07:34:39 UTC, rikki cattermole 
wrote:
 For the more used IDE's the community from what I have observed 
 use the dedicated plugins for the IDE's to integrate into 
 external resources like DCD.

 I was not commenting on if an IDE supports it, but instead on 
 what the priority is.

 Keeping in mind that this is in context on rewriting DLS which 
 depends on these very same resources.
My mistake, I misinterpreted what you said indeed.
Feb 20
prev sibling parent drug <drug2004 bk.ru> writes:
On 2/21/20 10:34 AM, rikki cattermole wrote:
 For the more used IDE's the community from what I have observed use the 
 dedicated plugins for the IDE's to integrate into external resources 
 like DCD.
 
 I was not commenting on if an IDE supports it, but instead on what the 
 priority is.
 
 Keeping in mind that this is in context on rewriting DLS which depends 
 on these very same resources.
Well, for example me prefer DLS over code-d for long time
Feb 21
prev sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Friday, 21 February 2020 at 07:24:10 UTC, Laurent Tréguier 
wrote:
 On Friday, 21 February 2020 at 03:42:37 UTC, rikki cattermole 
 wrote:
 ignore DLS most IDE's don't use it
I'll chime in a bit to disagree on that. I don't know where you're getting this idea from, but DLS can be used with VSCode, Visual Studio, Sublime text, Atom, Vim/Neovim, Emacs, and Intellij IDEA. It's certainly not as popular as code-d and VisualD, but it's certainly usable with a whole range of editors and IDE's
+1 I use it with VSCode and I'm very satisfied about it. Some period ago, it was the only VSCode plugin with a "just work" approach, while vs-code was plagued by instability, also if right now it works pretty well also. I especially love the idea of glueing all the different tools together.
Feb 21
prev sibling parent Mathias Lang <pro.mathias.lang gmail.com> writes:
On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian Creteanu 
wrote:
 The cleanest way to solve this would be to replace libdparse 
 with dmd in dcd and dscanner, as well. Would this be a good 
 idea? Should I approach this differently?
Not familiar with DLS / DCD source code, but quite familiar with DMD. Things have definitely gotten better since the first time I attempted this, but it's definitely non-trivial, depending on what you are trying to do. I can confirm that the global state and the memory allocation are still issues. I don't know about standard output, has some PRs have been made to use a delegate (https://github.com/dlang/dmd/pull/10072 and https://github.com/dlang/dmd/pull/10711), so things might be better on that front. Two other large issues: - DMD is not built to be able to handle edit cycles, only complete & valid source files. When working on an IDE, you'll have to deal with incomplete source code (e.g. typing `format("`) and you want to be able to provide completion for parameter, return value, overload sets, etc... Doing so would require large refactorings in DMD which are likely to raise some eyebrows. - DMD semantic analysis is destructive. Many things are rewritten and some links are lost, and the rewrite might not be valid D code. One example, any `alias this` is just rewritten as a direct call, but bypassing visibility checks on the target, so writing the same code would produce an error. Some expressions are rewritten as ComaExpression and use the return of that expression. This is also very evident with `alias`, as they are resolved as early as possible. There are quite a few places in DMD that work around this issue (e.g. `immutable(char)[]` is special-cased to always show as `string` in error message, and the POSIX C++ mangler uses a mix of pre and post-semantic AST to work around that issue as soon as templates are involved).
Feb 20