www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.ide - RFC: Descent's Action Plan

reply Ary Borenszweig <ary esperanto.org.ar> writes:
Just to make my mind clear about how to go on with Descent, and to 
explain a conclusion I've arrived while making it, I wrote a small article:

http://www.dsource.org/projects/descent/wiki/ActionPlan

It talks about why Descent doesn't work as expected most of the time, 
what a solution for that is, and how languages with compile-time 
generation of symbols make writing an IDE harder (for me, of course).

You don't need to understand every bit of it in order to understand the 
idea, but I'll gladly explain antyhing that isn't clear (it'll help me, 
and possible future developers).

Any comment or suggestion (or even English corrections! :)) is very 
appreciated.
Mar 26 2008
next sibling parent reply Tim Burrell <tim timburrell.net> writes:
Informative article Ary!

A couple thoughts come to mind:

 If a module A depends on B, which in turns depends on B, semantic analysis
will not be done correctly.

I assume you meant if module A depends on B, which in turn depends on A?
 So if Descent wants to follow JDT's approach, it can't do semantic analysis
only on the needed bits. It must fully do it for involved modules, and just
then, begin symbol lookup, type resolution, etc. Even if selective imports are
used, the selected symbols may be compile-time generated. So performance, in
comparison with JDT, should be relatively bad.

I'm curious, is performance really a big issue? I'm thinking that the analysis step only happens when a file is saved? So the parsing / analysis step isn't run that often correct? Personally I'd be willing to take a pause every so often if it meant robust, and correct auto-completion. Maybe this could even be done in the background on a separate thread unbeknownst to the user, with the results of the updated syntax tree being available when the thread finishes processing? At any rate, it was interesting to hear some of the problems you encountered while writing Descent. Keep up the good work, it's a great project!
Mar 27 2008
parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
Tim Burrell wrote:
 Informative article Ary!
 
 A couple thoughts come to mind:
 
 If a module A depends on B, which in turns depends on B, semantic 
 analysis will not be done correctly.

I assume you meant if module A depends on B, which in turn depends on A?

Yes, I've corrected this. Thanks.
 
 So if Descent wants to follow JDT's approach, it can't do semantic 
 analysis only on the needed bits. It must fully do it for involved 
 modules, and just then, begin symbol lookup, type resolution, etc. 
 Even if selective imports are used, the selected symbols may be 
 compile-time generated. So performance, in comparison with JDT, should 
 be relatively bad.

I'm curious, is performance really a big issue? I'm thinking that the analysis step only happens when a file is saved? So the parsing / analysis step isn't run that often correct? Personally I'd be willing to take a pause every so often if it meant robust, and correct auto-completion.

Actually, semantic analysis is performed each time you stop typing for a while, or when you request autocompletion. It is also performed when doing "go to definition". This is working like that in JDT, and that's why it's also like that in Descent. If it takes a few milliseconds, than it doesn't matter running it often because you will not notice it. I guess performance won't be that bad because the lexer/parser stage is already done, and also semantic analysis is only needed for "surface" declarations (not in function bodies, for example, although if a function needs to be compile-time evaluated...). Has any one done a performance analysis comparing lexing/parsing/semantic analysis times?
 
 Maybe this could even be done in the background on a separate thread 
 unbeknownst to the user, with the results of the updated syntax tree 
 being available when the thread finishes processing?
 
 At any rate, it was interesting to hear some of the problems you 
 encountered while writing Descent.  Keep up the good work, it's a great 
 project!

Mar 27 2008
parent reply Jason House <jason.james.house gmail.com> writes:
Ary Borenszweig wrote:

 Tim Burrell wrote:
 If a module A depends on B, which in turns depends on B, semantic
 analysis will not be done correctly.

I assume you meant if module A depends on B, which in turn depends on A?

Yes, I've corrected this. Thanks.

I'm pretty certain that the D standard says that A depending on B, depending on A is illegal... This resolves issues with translation unit ordering such as variable initialization and static construction. It also makes ordering of unit tests logical. I think A depending on B depending on A should be flagged as an error.
Mar 27 2008
parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
Jason House escribió:
 Ary Borenszweig wrote:
 
 Tim Burrell wrote:
 If a module A depends on B, which in turns depends on B, semantic
 analysis will not be done correctly.



I'm pretty certain that the D standard says that A depending on B, depending on A is illegal... This resolves issues with translation unit ordering such as variable initialization and static construction. It also makes ordering of unit tests logical. I think A depending on B depending on A should be flagged as an error.

I meant something like this: --- module one; import two; class Parent { Child child; } --- module two; import one; class Child { Parent parent; } --- DMD correctly compiles this, while in Descent the problem I mentioned appears. It can be solved with the current approach, but it involves writing semantic code besides the one provided in DMD. Luckily, today in the bus I figured out a very clean and nice way to program this, which shouldn't take a lot of time, without writing any additional semantic code. :) The only unknown thing, yet, is how performance will be in this new approach... Since you mentioned order of compilation: can a module, compiled to an obj file, expose different interfaces (that it, top level classes, functions, variables, etc.) when compiled in different orders? Well, I've just came up with this example (anyone else have a more interesting one?), in which compilation order matters: --- module one; import two; mixin("class Parent { Child child; }"); --- module two; import one; mixin("class Child { }"); --- $ dmd one.d two.d -c (compiles ok) $ dmd two.d one.d -c Error: identifier 'Child' is not defined Error: Child is used as a type one.d: variable one.Parent.child voids have no value When I came up with it, I thought I understood why the error was happening. But... I actually don't. Is this a bug?
Mar 27 2008
parent Christopher Wright <dhasenan gmail.com> writes:
Ary Borenszweig wrote:
 DMD correctly compiles this, while in Descent the problem I mentioned 
 appears. It can be solved with the current approach, but it involves 
 writing semantic code besides the one provided in DMD. Luckily, today in 
 the bus I figured out a very clean and nice way to program this, which 
 shouldn't take a lot of time, without writing any additional semantic 
 code. :)

The easy way is just to catenate the modules internally and do analysis on that. Circular dependencies within a module are already handled, reducing the problem to an already solved one.
 Error: identifier 'Child' is not defined
 Error: Child is used as a type
 one.d: variable one.Parent.child voids have no value
 
 When I came up with it, I thought I understood why the error was 
 happening. But... I actually don't. Is this a bug?

Yes.
Mar 27 2008
prev sibling next sibling parent Jason House <jason.james.house gmail.com> writes:
Ary Borenszweig wrote:

 Just to make my mind clear about how to go on with Descent, and to
 explain a conclusion I've arrived while making it, I wrote a small
 article:
 
 http://www.dsource.org/projects/descent/wiki/ActionPlan
 
 It talks about why Descent doesn't work as expected most of the time,
 what a solution for that is, and how languages with compile-time
 generation of symbols make writing an IDE harder (for me, of course).
 
 You don't need to understand every bit of it in order to understand the
 idea, but I'll gladly explain antyhing that isn't clear (it'll help me,
 and possible future developers).
 
 Any comment or suggestion (or even English corrections! :)) is very
 appreciated.

I may be underestimating the compile-time capability of D, but what if top-level structure including basic information on the template conditions, static if's, and version/debug statements required for a particular piece? This would allow two levels of auto-completion. The top-level data can say under which conditions particular items are available. Follow-up analysis can say which are applicable for the current situation. Knowing what other options exist with different versions may actually help the programmer decide on how to structure their current code to take advantage? PS: Keep up the good work on Descent. I look forward to seeing this stuff in action :)
Mar 27 2008
prev sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Ary Borenszweig wrote:
 Just to make my mind clear about how to go on with Descent, and to 
 explain a conclusion I've arrived while making it, I wrote a small article:
 
 http://www.dsource.org/projects/descent/wiki/ActionPlan
 
 It talks about why Descent doesn't work as expected most of the time, 
 what a solution for that is, and how languages with compile-time 
 generation of symbols make writing an IDE harder (for me, of course).
 
 You don't need to understand every bit of it in order to understand the 
 idea, but I'll gladly explain antyhing that isn't clear (it'll help me, 
 and possible future developers).
 
 Any comment or suggestion (or even English corrections! :)) is very 
 appreciated.

I think I more or less understood the problems you mention, so I want to offer some general toughts. I think that trying to offer 100% correct completion and semantic analysis may not be the right way to go. I say this because doing might make the IDE too slow and memory-bloated, and make it's code much more complex, whereas having 99% correctness might make the IDE much more simple, and not make that much of a difference to the programmer. I say this based on the experiences of the lead CDT developer (see relevant post here: http://cdtdoug.blogspot.com/2007/03/ego-less-development.html) Now, since D is simpler than C++, it might be easier and worthwhile to achieve that goal of 100% accurateness in the IDE, but given the manpower available for the Eclipse D IDE project, it might not be worth it yet. So I personally wouldn't mind much an IDE was not able to process compile-time generate symbols and definitions, there are other things I'd rather have first. But that's my personal opinion. :) -- Bruno Medeiros - Software Developer, MSc. in CS/E graduate http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Apr 27 2008
parent Ary Borenszweig <ary esperanto.org.ar> writes:
Bruno Medeiros escribió:
 Ary Borenszweig wrote:
 Just to make my mind clear about how to go on with Descent, and to 
 explain a conclusion I've arrived while making it, I wrote a small 
 article:

 http://www.dsource.org/projects/descent/wiki/ActionPlan

 It talks about why Descent doesn't work as expected most of the time, 
 what a solution for that is, and how languages with compile-time 
 generation of symbols make writing an IDE harder (for me, of course).

 You don't need to understand every bit of it in order to understand 
 the idea, but I'll gladly explain antyhing that isn't clear (it'll 
 help me, and possible future developers).

 Any comment or suggestion (or even English corrections! :)) is very 
 appreciated.

I think I more or less understood the problems you mention, so I want to offer some general toughts. I think that trying to offer 100% correct completion and semantic analysis may not be the right way to go. I say this because doing might make the IDE too slow and memory-bloated, and make it's code much more complex, whereas having 99% correctness might make the IDE much more simple, and not make that much of a difference to the programmer. I say this based on the experiences of the lead CDT developer (see relevant post here: http://cdtdoug.blogspot.com/2007/03/ego-less-development.html) Now, since D is simpler than C++, it might be easier and worthwhile to achieve that goal of 100% accurateness in the IDE, but given the manpower available for the Eclipse D IDE project, it might not be worth it yet. So I personally wouldn't mind much an IDE was not able to process compile-time generate symbols and definitions, there are other things I'd rather have first. But that's my personal opinion. :)

Actually... We already implemented it, and made some optimizations to DMD's semantic code (for some symbols semantic is never run, if not needed). It offers 100% correctness, and it works pretty fast. Say, about 300~500 milliseconds for large files (for example, import dfl.all) from the moment you request autocompletion until the moment the IDE offers them to you, so for normal-small files it should be fast. I'd rather wait 100 milliseconds more, than rely on my memory or check the source code (both much slower than 100 milliseconds). We hope to release soon, and then we'll see if it's true based on user experience. At least I tried it inside some of Tang's, Phobo's and DFL's code, and it works quite well. The problem with not offering the correct autocompletion proposals, is that you confuse the user. She would start thinking "Wait, I'm sure this class has this method, why isn't the IDE showing it to me? Maybe I'm wrong... let's check the source code". And then, you are back to nothing, because without an IDE, checking the source code or the documentation is the only option, besides memory.
Apr 27 2008