www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Retrieving the Scope of a statement

reply Cristian Creteanu <cristian.creteanu pm.me> writes:
Hi!

I am trying to implement autocomplete in DCD using DMD as a 
library. After parsing & the semantic analysis of a given source 
file, given a location (line & column number), I want to retrieve 
the Scope* that the statement at the given line is part of so 
that I can get the symbol table.

I noticed that every Scope has a pointer to its parent Scope 
(enclosing), but as far as I could see there's no way to access 
inner scopes with the information that there is currently in the 
Scope struct. Would it be ok to add a list of children scopes 
inside the struct? So, during semantic analysis, whenever the 
push method of the Scope structure is called I could also push 
the newly created scope inside this list of children?

What I tried to do without being able to access children directly 
was to create a visitor which goes through the entire AST until 
it reaches the statement at the given location. Once I've found 
this statement, a solution would be to find a Dsymbol inside of 
it and get its _scope, but for many of these symbols (for 
instance, local variables of functions) this field is null. I had 
a look at visit(VarDeclaration) in dsymbolsem.d and the _scope 
would be set for the variable declaration symbol if its parent 
was an aggregate declaration, but why isn't it set for variable 
declarations local to functions?

Is there any easier way to go about this?
Apr 20 2020
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Monday, 20 April 2020 at 16:56:51 UTC, Cristian Creteanu wrote:

 Hi!

 I am trying to implement autocomplete in DCD using DMD as a 
 library. After parsing & the semantic analysis of a given 
 source file, given a location (line & column number), I want to 
 retrieve the Scope* that the statement at the given line is 
 part of so that I can get the symbol table.

 I noticed that every Scope has a pointer to its parent Scope 
 (enclosing), but as far as I could see there's no way to access 
 inner scopes with the information that there is currently in 
 the Scope struct. Would it be ok to add a list of children 
 scopes inside the struct?


No I think this is a wrong trail. You could be able to do that 
without modifying the compiler.


 So, during semantic analysis, whenever the push method of the 
 Scope structure is called I could also push the newly created 
 scope inside this list of children?

 What I tried to do without being able to access children 
 directly was to create a visitor which goes through the entire 
 AST until it reaches the statement at the given location.


Yes I think you can do that.


 Once I've found this statement, a solution would be to find a 
 Dsymbol inside of it and get its _scope, but for many of these 
 symbols (for instance, local variables of functions) this field 
 is null. I had a look at visit(VarDeclaration) in dsymbolsem.d 
 and the _scope would be set for the variable declaration symbol 
 if its parent was an aggregate declaration, but why isn't it 
 set for variable declarations local to functions?

 Is there any easier way to go about this?


For functions you can try to find the `Scope` when 
`ScopeStatement`s are visited because they have `.loc` (the 
begining) and `.endloc` information. So if the cursor loc is 
within the two values, the scope `sc` of the visitor 
(StatementSemanticVisitor.sc) gives you the available symbols at 
the cursor pos.

Now technically there might be a problem because I dont see how 
you can plug the code to make this in `StatementSemanticVisitor`. 
This would require a kind of signal/callback/event system. Since 
the `Scope` is not attached to an `AstNode` you CANT use a custom 
visitor.
Apr 20 2020
parent reply Basile B. <b2.temp gmx.com> writes:
On Tuesday, 21 April 2020 at 01:52:49 UTC, Basile B. wrote:

 On Monday, 20 April 2020 at 16:56:51 UTC, Cristian Creteanu 
 wrote:

 Hi!

 I am trying to implement autocomplete in DCD using DMD as a 
 library.
 [...]


 [...]
 to an `AstNode` you CANT use a custom visitor.


Sorry maybe I've been too opiniated.

in DMD, because of the lazy compilation model, visitors usually 
don't process a whole module so:

- the scope for a statement is the scope that's passed to the 
`StatementSemanticVisitor` class created when 
`statementSemantic()` is called for this statement.
- the scope for an expression is the scope that's passed to the 
`ExpressionSemanticVisitor` class created when 
`expressionSemantic()` is called for this expression.
- the scope for a type is the scope that's passed to the 
`TypeSemanticVisitor` class created when `typeSemantic()` is 
called for this type.

 From your original message I have the impression that you want to 
retrieve a scope by a node, and you noticed that this is not 
always possible. The time where a scope is always linked to a 
node is when its semantic is run, and it's a class member. So 
maybe (and only maybe) there's something to do with that fact but 
as said at first glance I don't see how, without a kind of 
callback system.
Apr 20 2020
parent reply Basile B. <b2.temp gmx.com> writes:
On Tuesday, 21 April 2020 at 06:58:47 UTC, Basile B. wrote:

 On Tuesday, 21 April 2020 at 01:52:49 UTC, Basile B. wrote:

 On Monday, 20 April 2020 at 16:56:51 UTC, Cristian Creteanu 
 wrote:

 Hi!

 I am trying to implement autocomplete in DCD using DMD as a 
 library.
 [...]


 [...]
 to an `AstNode` you CANT use a custom visitor.


 Sorry maybe I've been too opiniated.

 in DMD, because of the lazy compilation model, visitors usually 
 don't process a whole module so:

 - the scope for a statement is the scope that's passed to the 
 `StatementSemanticVisitor` class created when 
 `statementSemantic()` is called for this statement.
 - the scope for an expression is the scope that's passed to the 
 `ExpressionSemanticVisitor` class created when 
 `expressionSemantic()` is called for this expression.
 - the scope for a type is the scope that's passed to the 
 `TypeSemanticVisitor` class created when `typeSemantic()` is 
 called for this type.

 From your original message I have the impression that you want 
 to retrieve a scope by a node, and you noticed that this is not 
 always possible. The time where a scope is always linked to a 
 node is when its semantic is run, and it's a class member. So 
 maybe (and only maybe) there's something to do with that fact 
 but as said at first glance I don't see how, without a kind of 
 callback system.


Maybe that adding this in the compiler globals:

   alias OnStatementSemanticStart = void function(Statement, 
Scope*);
   alias OnStatementSemanticDone = void function(Statement, 
Scope*);
   alias OnExpressionSemanticStart = void function(Expression, 
Scope*);
   alias OnExpressionSemanticDone = void function(Expression, 
Scope*);

   OnStatementSemanticStart onStatementSemanticStart;
   OnStatementSemanticDone onStatementSemanticDone;
   OnExpressionSemanticStart onExpressionSemanticStart;
   OnExpressionSemanticDone onExpressionSemanticDone;

and add the handlers in statementSemantic() and 
expressionSemantic()

could help to build a useful enough hash map but without 
significant memory impact on the compiler. Actually just the 
"Done" events would be usefull because of the many lowering done 
during semantics.
Apr 21 2020
parent reply Cristian Creteanu <cristian.creteanu pm.me> writes:
On Tuesday, 21 April 2020 at 07:45:23 UTC, Basile B. wrote:

   alias OnStatementSemanticStart = void function(Statement, 
 Scope*);
   alias OnStatementSemanticDone = void function(Statement, 
 Scope*);
   alias OnExpressionSemanticStart = void function(Expression, 
 Scope*);
   alias OnExpressionSemanticDone = void function(Expression, 
 Scope*);

   OnStatementSemanticStart onStatementSemanticStart;
   OnStatementSemanticDone onStatementSemanticDone;
   OnExpressionSemanticStart onExpressionSemanticStart;
   OnExpressionSemanticDone onExpressionSemanticDone;

 and add the handlers in statementSemantic() and 
 expressionSemantic()



Thank you so much! It worked :D
May 02 2020
parent reply Basile B. <b2.temp gmx.com> writes:
On Saturday, 2 May 2020 at 21:36:00 UTC, Cristian Creteanu wrote:

 On Tuesday, 21 April 2020 at 07:45:23 UTC, Basile B. wrote:

   alias OnStatementSemanticStart = void function(Statement, 
 Scope*);
   alias OnStatementSemanticDone = void function(Statement, 
 Scope*);
   alias OnExpressionSemanticStart = void function(Expression, 
 Scope*);
   alias OnExpressionSemanticDone = void function(Expression, 
 Scope*);

   OnStatementSemanticStart onStatementSemanticStart;
   OnStatementSemanticDone onStatementSemanticDone;
   OnExpressionSemanticStart onExpressionSemanticStart;
   OnExpressionSemanticDone onExpressionSemanticDone;

 and add the handlers in statementSemantic() and 
 expressionSemantic()



 Thank you so much! It worked :D


I've seen the PR [1], so it's ok, you manage to get something to 
work with this ?
If so in DMD I would prefer a `version(callback_API)` because at 
some point more callbacks might be required, added, but not 
necessarily tied to the dmd library version.

[1] https://github.com/dlang/dmd/pull/11092
May 02 2020
parent Cristian Creteanu <cristian.creteanu pm.me> writes:
On Sunday, 3 May 2020 at 03:10:34 UTC, Basile B. wrote:

 I've seen the PR [1], so it's ok, you manage to get something 
 to work with this ?
 [1] https://github.com/dlang/dmd/pull/11092


Yes, I did manage to get results out of this approach.


 If so in DMD I would prefer a `version(callback_API)` because 
 at some point more callbacks might be required, added, but not 
 necessarily tied to the dmd library version.


You're right, I will change the version name in the PR.
May 03 2020
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On Monday, 20 April 2020 at 16:56:51 UTC, Cristian Creteanu wrote:

 Hi!

 I am trying to implement autocomplete in DCD using DMD as a 
 library. After parsing & the semantic analysis of a given 
 source file, given a location (line & column number), I want to 
 retrieve the Scope* that the statement at the given line is 
 part of so that I can get the symbol table.


In addition to what Basile mentioned, you can have a look at 
VisualD [1], which now uses DMD for semantic analysis and 
intellisense.

[1] https://github.com/rainers/visuald/blob/master/vdc/semantic.d

--
/Jacob Carlborg
Apr 23 2020
parent reply Cristian Creteanu <cristian.creteanu pm.me> writes:
On Thursday, 23 April 2020 at 09:26:22 UTC, Jacob Carlborg wrote:

 In addition to what Basile mentioned, you can have a look at 
 VisualD [1], which now uses DMD for semantic analysis and 
 intellisense.

 [1] 
 https://github.com/rainers/visuald/blob/master/vdc/semantic.d

 --
 /Jacob Carlborg


It seems that VisualD uses scopes defined internally rather than 
the ones used by DMD
May 02 2020
parent Cristian Creteanu <cristian.creteanu pm.me> writes:
On Saturday, 2 May 2020 at 21:40:28 UTC, Cristian Creteanu wrote:

 On Thursday, 23 April 2020 at 09:26:22 UTC, Jacob Carlborg 
 wrote:

 In addition to what Basile mentioned, you can have a look at 
 VisualD [1], which now uses DMD for semantic analysis and 
 intellisense.

 [1] 
 https://github.com/rainers/visuald/blob/master/vdc/semantic.d

 --
 /Jacob Carlborg


 It seems that VisualD uses scopes defined internally rather 
 than the ones used by DMD


My bad, VisualD does use ScopeDsymbol instead, so I should've 
taken a closer look. Thanks for the suggestion, I will pay more 
attention to VisualD in the future.
May 03 2020