www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - order of static constructor execution

reply Walter Bright <newshound1 digitalmars.com> writes:
Currently, it is performed as a strictly "depth-first" traversal of the 
graph defined by the import statements. As we've been discussing here, 
this works great until one has circular imports, meaning the depth-first 
graph has a loop in it.

The current behavior on detecting a loop is to quit with an error message.

The problems are:

1. The cycles are not easily gotten rid of when they are the result of 
template mixins.

2. Trying to analyze the static constructors to see what the 
dependencies actually are is fraught with unsolvable problems.


So, I propose the following:

1. Attempt the depth-first traversal of the static constructors.

2. If a loop is detected, rather than issuing an error message, simply 
arbitrarily pick one order and continue constructing.


The mitigating rationale is that modules that import each other are 
presumably written by the same person or team, and so that person is in 
the best place to explicitly control dependencies themselves.


I'm not happy with this solution, but it seems to be the best compromise 
I can come up with.

What do you think?
Mar 11 2010
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:

 1. Attempt the depth-first traversal of the static constructors.
 2. If a loop is detected, rather than issuing an error message, simply 
 arbitrarily pick one order and continue constructing.



Some alternative possibilities:
- Issue a warning too?
- Or allow that only if the code is compiled with "-allowloops"?
- Or accept one specific loop only if the programmer somewhere states
explicitly that this loop is OK, and issue an error otherwise?

Bye,
bearophile
Mar 11 2010
prev sibling next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 03/11/2010 08:42 PM, Walter Bright wrote:

 2. Trying to analyze the static constructors to see what the
 dependencies actually are is fraught with unsolvable problems.



could you elucidate on this point? Is it definitely impossible to get a 
hold of a complete dag, even if at runtime?
Mar 11 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Ellery Newcomer wrote:

 On 03/11/2010 08:42 PM, Walter Bright wrote:

 2. Trying to analyze the static constructors to see what the
 dependencies actually are is fraught with unsolvable problems.



 could you elucidate on this point? Is it definitely impossible to get a 
 hold of a complete dag, even if at runtime?




Suppose the static constructor calls foo(), the implementation of which 
is hidden to the compiler.
Mar 11 2010
parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 03/11/2010 10:15 PM, Walter Bright wrote:

 Ellery Newcomer wrote:

 On 03/11/2010 08:42 PM, Walter Bright wrote:

 2. Trying to analyze the static constructors to see what the
 dependencies actually are is fraught with unsolvable problems.



 could you elucidate on this point? Is it definitely impossible to get
 a hold of a complete dag, even if at runtime?




 Suppose the static constructor calls foo(), the implementation of which
 is hidden to the compiler.



Oh, you mean the true dependencies. I was just going to say that 
printing out the cycle that causes bombout would be better than nothing.
Mar 11 2010
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 03/11/2010 08:42 PM, Walter Bright wrote:

 Currently, it is performed as a strictly "depth-first" traversal of the
 graph defined by the import statements. As we've been discussing here,
 this works great until one has circular imports, meaning the depth-first
 graph has a loop in it.

 The current behavior on detecting a loop is to quit with an error message.

 The problems are:

 1. The cycles are not easily gotten rid of when they are the result of
 template mixins.

 2. Trying to analyze the static constructors to see what the
 dependencies actually are is fraught with unsolvable problems.


 So, I propose the following:

 1. Attempt the depth-first traversal of the static constructors.

 2. If a loop is detected, rather than issuing an error message, simply
 arbitrarily pick one order and continue constructing.


 The mitigating rationale is that modules that import each other are
 presumably written by the same person or team, and so that person is in
 the best place to explicitly control dependencies themselves.


 I'm not happy with this solution, but it seems to be the best compromise
 I can come up with.

 What do you think?



It looks like a step backwards to me.

Andrei
Mar 11 2010
parent Max Samukha <spambox d-coding.com> writes:
Andrei Alexandrescu wrote:


 
 It looks like a step backwards to me.
 
 Andrei



What solution do you propose for problem 1?
Mar 12 2010
prev sibling next sibling parent reply BCS <none anon.com> writes:
Hello Walter,


 2. If a loop is detected, rather than issuing an error message, simply
 arbitrarily pick one order and continue constructing.



How about a way to explicitly cut edges in the graph (tagging imports with 
"pragma(nodep)" or " nodep" for instance)? That has the same end effect but 
for only a little more work, removes any non-determinism and allows for easy 
control of how things are resolved.

-- 
... <IXOYE><
Mar 11 2010
parent reply Don <nospam nospam.com> writes:
BCS wrote:

 Hello Walter,
 

 2. If a loop is detected, rather than issuing an error message, simply
 arbitrarily pick one order and continue constructing.



 How about a way to explicitly cut edges in the graph (tagging imports 
 with "pragma(nodep)" or " nodep" for instance)? That has the same end 
 effect but for only a little more work, removes any non-determinism and 
 allows for easy control of how things are resolved.



I agree. Seems to me that if a circular import exists, it's really 
something that the programmer needs to think about, and so it's 
reasonable for it to be explicit.

A really harsh solution would be:

pragma(nodependency, somepackage.somemodule);

The compiler could check that somepackage.somemodule actually defines a 
static constructor. And it could even check that a circular import 
situation actually exists. This would force the pragma to be maintained 
correctly.
Mar 12 2010
parent reply Don <nospam nospam.com> writes:
Fawzi Mohamed wrote:

 
 On 12-mar-10, at 09:59, Don wrote:
 

 BCS wrote:

 Hello Walter,

 2. If a loop is detected, rather than issuing an error message, simply
 arbitrarily pick one order and continue constructing.



 with "pragma(nodep)" or " nodep" for instance)? That has the same end 
 effect but for only a little more work, removes any non-determinism 
 and allows for easy control of how things are resolved.



 I agree. Seems to me that if a circular import exists, it's really 
 something that the programmer needs to think about, and so it's 
 reasonable for it to be explicit.

 A really harsh solution would be:

 pragma(nodependency, somepackage.somemodule);

 The compiler could check that somepackage.somemodule actually defines 
 a static constructor. And it could even check that a circular import 
 situation actually exists. This would force the pragma to be 
 maintained correctly.



 I think that the main problem comes from static initializers that are 
 mixed in and by themselves have no circular dependency. 



If that's true, then perhaps a more restricted solution is better.
Eg, use Walter's proposal ONLY if the static initializer is mixed in.
Mar 12 2010
next sibling parent reply BCS <none anon.com> writes:
Hello Don,


 If that's true, then perhaps a more restricted solution is better. Eg,
 use Walter's proposal ONLY if the static initializer is mixed in.



Random thought: Aside from making a new corner case, why not make static 
constructors in mixins only depend on imports from where they are defined 
rather than where they are used?
 
-- 
... <IXOYE><
Mar 12 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
BCS wrote:

 Random thought: Aside from making a new corner case, why not make static 
 constructors in mixins only depend on imports from where they are 
 defined rather than where they are used?




Template mixins are intended to behave like macros - they are 
instantiated in the context of where they are used, not where they are 
defined.

Regular templates are instantiated in the context of where they are 
defined, not used.
Mar 12 2010
parent reply Max Samukha <spambox d-coding.com> writes:
On 12.03.2010 23:03, Walter Bright wrote:

 BCS wrote:

 Random thought: Aside from making a new corner case, why not make
 static constructors in mixins only depend on imports from where they
 are defined rather than where they are used?




 Template mixins are intended to behave like macros - they are
 instantiated in the context of where they are used, not where they are
 defined.

 Regular templates are instantiated in the context of where they are
 defined, not used.



But then:

module a;

static this() {} // 1

template Foo(int i)
{
     static this() // 2
     {
     }
}
----

module b;
import a;

static this() // 3
{
}

alias Foo!(1) foo;
----

Currently the construction order is 1, 3, 2 instead of the expected 1, 
2, 3. In other words, the constructor in the template instance (2) 
should be combined with constructor 1, not 3.

If static constructors in templates ran during the initialization of the 
module where they are defined, the problem with mixed-in code could be 
solved like this:

module a;

template StaticCtor(alias ctor)
{
     static this()
     {
         ctor();
     }
}

mixin template Foo(int i)
{
     void construct() { writeln("Constructing for ", i); }
     alias StaticCtor!(construct) ctor;
}

----
module b;

import a;
import c;

mixin Foo!(1);

----
module c;

import a;
import b;

mixin Foo!(2);

So, though Foo is mixed into modules 'b' and 'c', its instances would be 
constructed in module 'a' via the regular StaticCtor template.

Still a hack but the code at least makes it quite explicit for 
maintainers to stay alert.
Mar 12 2010
parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-03-12 19:10:10 -0500, Max Samukha <spambox d-coding.com> said:


 If static constructors in templates ran during the initialization of 
 the module where they are defined, the problem with mixed-in code could 
 be solved like this:



But templates can take function and types as argument. If the 'static 
this' of a template calls something through one of its argument, the 
module this function or type resides in needs to be initialized first.

So you can't always call a template's static this at the same time as 
the module the template is defined in.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Mar 12 2010
parent Max Samukha <spambox d-coding.com> writes:
On 13.03.2010 2:27, Michel Fortin wrote:

 On 2010-03-12 19:10:10 -0500, Max Samukha <spambox d-coding.com> said:


 If static constructors in templates ran during the initialization of
 the module where they are defined, the problem with mixed-in code
 could be solved like this:



 But templates can take function and types as argument. If the 'static
 this' of a template calls something through one of its argument, the
 module this function or type resides in needs to be initialized first.

 So you can't always call a template's static this at the same time as
 the module the template is defined in.



Yes that is the point and purpose.

The advantage is that the programmer is in control of when the static 
constructors in templated code are run - if you want the constructor to 
run in the context of the imported module use a regular template, 
otherwise, mixin.

I'd definitely preferred it over the current semantics. Others may disagree.
Mar 13 2010
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Don wrote:

 If that's true, then perhaps a more restricted solution is better.
 Eg, use Walter's proposal ONLY if the static initializer is mixed in.



I should note that all static constructors in a module are combined into 
one function. So, the mixin static initializers are combined with any 
other static initializers.
Mar 12 2010
parent reply BCS <none anon.com> writes:
Hello Walter,


 Don wrote:
 

 If that's true, then perhaps a more restricted solution is better.
 Eg, use Walter's proposal ONLY if the static initializer is mixed in.
 



 into one function. So, the mixin static initializers are combined with
 any other static initializers.
 



Does it have to run that way or could you lump them into several pieces and 
have them call each other as needed?

-- 
... <IXOYE><
Mar 12 2010
parent Walter Bright <newshound1 digitalmars.com> writes:
BCS wrote:

 Hello Walter,
 

 Don wrote:


 If that's true, then perhaps a more restricted solution is better.
 Eg, use Walter's proposal ONLY if the static initializer is mixed in.



 into one function. So, the mixin static initializers are combined with
 any other static initializers.



 Does it have to run that way or could you lump them into several pieces 
 and have them call each other as needed?
 



Right now, they are defined to run in lexical order. Regrouping them in 
arbitrary orders will mess up intra-module ordering dependencies.
Mar 12 2010
prev sibling next sibling parent "Bob Jones" <me not.com> writes:
"Walter Bright" <newshound1 digitalmars.com> wrote in message 
news:hnc9n2$2tkj$1 digitalmars.com...

 Currently, it is performed as a strictly "depth-first" traversal of the 
 graph defined by the import statements. As we've been discussing here, 
 this works great until one has circular imports, meaning the depth-first 
 graph has a loop in it.

 The current behavior on detecting a loop is to quit with an error message.

 The problems are:

 1. The cycles are not easily gotten rid of when they are the result of 
 template mixins.

 2. Trying to analyze the static constructors to see what the dependencies 
 actually are is fraught with unsolvable problems.



Surely there must be some low hanging fruit that can be exploited? Say a 
module has no static constructors then cant that be considered a break in 
the dependency cycle?
Mar 11 2010
prev sibling next sibling parent Fawzi Mohamed <fawzi gmx.ch> writes:
On 12-mar-10, at 09:59, Don wrote:


 BCS wrote:

 Hello Walter,

 2. If a loop is detected, rather than issuing an error message,  
 simply
 arbitrarily pick one order and continue constructing.



 imports with "pragma(nodep)" or " nodep" for instance)? That has  
 the same end effect but for only a little more work, removes any  
 non-determinism and allows for easy control of how things are  
 resolved.



 I agree. Seems to me that if a circular import exists, it's really  
 something that the programmer needs to think about, and so it's  
 reasonable for it to be explicit.

 A really harsh solution would be:

 pragma(nodependency, somepackage.somemodule);

 The compiler could check that somepackage.somemodule actually  
 defines a static constructor. And it could even check that a  
 circular import situation actually exists. This would force the  
 pragma to be maintained correctly.



I think that the main problem comes from static initializers that are  
mixed in and by themselves have no circular dependency. So annotations  
have to be at the static initializer level, not at the module level.

One way to solve this would be to add
 dependOnly(module1,...) static this(){ }

then each module would have 2 dependencies:

* indirectDeps
   imported modules + all dependOnly modules

* directDeps
   look at all static initializers in the module:
   - plain static this() -> add all imported modules as dependency
   - annotated static this -> add all dependOnly dependencies

* allDeps
   the list of module that have to be initialized before that module  
and is build from the previous ones as:
   direct deps + all indirect deps of those modules

more explicitly

allDeps(a){
   deps=[]
   foreach(m in a.directDeps){
     if (m in deps) continue;
     deps~=m;
     addDeps2(m,deps);
   }
   return deps;
}

addDeps2(a, ref deps){
   if (a in deps) return;
   foreach(m in a.indirectDeps){
     if (m in deps) continue;
     deps~=m;
     addDeps2(m,deps);
   }
}

then you can sort the modules that have static initializers (and only  
those) using:

compare(a,b){
   if (a in b.allDeps){
    if (b in a.allDeps){
      error("non comparable, circular dep between ",a,b);
    }
    return 1;
   }
   if (b in a.allDeps){
     return -1;
   }
   return 0;
}

this is more work, but would be perfectly defined.

conflicting ordering would still be disallowed, but circularly  
dependent modules can have initializers if all their initializers  
depend only on modules that are not circularly dependent.

Fawzi
Mar 12 2010
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 3/12/10 03:42, Walter Bright wrote:

 Currently, it is performed as a strictly "depth-first" traversal of the
 graph defined by the import statements. As we've been discussing here,
 this works great until one has circular imports, meaning the depth-first
 graph has a loop in it.

 The current behavior on detecting a loop is to quit with an error message.

 The problems are:

 1. The cycles are not easily gotten rid of when they are the result of
 template mixins.

 2. Trying to analyze the static constructors to see what the
 dependencies actually are is fraught with unsolvable problems.


 So, I propose the following:

 1. Attempt the depth-first traversal of the static constructors.

 2. If a loop is detected, rather than issuing an error message, simply
 arbitrarily pick one order and continue constructing.


 The mitigating rationale is that modules that import each other are
 presumably written by the same person or team, and so that person is in
 the best place to explicitly control dependencies themselves.


 I'm not happy with this solution, but it seems to be the best compromise
 I can come up with.

 What do you think?



Yes please. As other have suggested a warning might be a good idea.
Mar 12 2010
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-03-11 21:42:47 -0500, Walter Bright <newshound1 digitalmars.com> said:


 Currently, it is performed as a strictly "depth-first" traversal of the 
 graph defined by the import statements. As we've been discussing here, 
 this works great until one has circular imports, meaning the 
 depth-first graph has a loop in it.
 
 The current behavior on detecting a loop is to quit with an error message.
 
 The problems are:
 
 1. The cycles are not easily gotten rid of when they are the result of 
 template mixins.
 
 2. Trying to analyze the static constructors to see what the 
 dependencies actually are is fraught with unsolvable problems.
 
 
 So, I propose the following:
 
 1. Attempt the depth-first traversal of the static constructors.
 
 2. If a loop is detected, rather than issuing an error message, simply 
 arbitrarily pick one order and continue constructing.
 
 
 The mitigating rationale is that modules that import each other are 
 presumably written by the same person or team, and so that person is in 
 the best place to explicitly control dependencies themselves.
 
 
 I'm not happy with this solution, but it seems to be the best 
 compromise I can come up with.
 
 What do you think?



I think it'd be better if it was explicit. Perhaps better would be an 
opt-in using an attribute on the static constructor. For instance:

module a;
import b;

int a;

static this() {
	a = b;
}


module b;
import a;

int b;

 cyclehead static this() {
	b = 10;
}


Here, module 'a' initialization depends on 'b'. Module 'b' 
initialization does not depend on a or anything, so you can make it 
 cyclehead.  cyclehead means that this module, when part of a cycle, 
can be initialized before the others. If all 'static this' in a module 
are  cyclehead, then the module gets the cyclehead flag and is 
initialized first in the cycle.

This is better than just picking one module at random to break the 
cycle. If no module have the cyclehead flag, then having a cycle is an 
error. The programmer will have to consciously choose one module to 
break the cycle.

Most mixins static constructors will probably have to be  cyclehead.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Mar 12 2010
parent Fawzi Mohamed <fmohamed mac.com> writes:
On 2010-03-12 15:05:25 +0100, Michel Fortin <michel.fortin michelf.com> said:


 [...]
 Most mixins static constructors will probably have to be  cyclehead.



Here is an example of a reasonable use case to clarify why I think that 
my proposal, or something like it is the correct solution.
Often the static initializers like those in module b or c would be 
mixed in (for example to give serialization support, as for example the 
class name in template classes is not unique).

{{{
module a;
// imports modules but does not create a circular dep

class Register {
  void register(...){...}
  ...
}

Register globalRegister;
static this(){
  globalRegister=new Register();
}

-------
module b;
import c;
import a;

class Bla {...}
 dependOnly(a) static this {
  globalRegister.register(...);
}
----
module c;
import b;
import a;

class Bla2 {...}
 dependOnly(a) static this {
  globalRegister.register(...);
}
}}}

My proposal would give the correct initialization order, and would 
still detect invalid cases (assuming the programmers does not lie about 
the dependOnly).

Fawzi
Mar 12 2010
prev sibling next sibling parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 12/03/10 02:42, Walter Bright wrote:

 Currently, it is performed as a strictly "depth-first" traversal of the
 graph defined by the import statements. As we've been discussing here,
 this works great until one has circular imports, meaning the depth-first
 graph has a loop in it.

 The current behavior on detecting a loop is to quit with an error message.

 The problems are:

 1. The cycles are not easily gotten rid of when they are the result of
 template mixins.

 2. Trying to analyze the static constructors to see what the
 dependencies actually are is fraught with unsolvable problems.


 So, I propose the following:

 1. Attempt the depth-first traversal of the static constructors.

 2. If a loop is detected, rather than issuing an error message, simply
 arbitrarily pick one order and continue constructing.


 The mitigating rationale is that modules that import each other are
 presumably written by the same person or team, and so that person is in
 the best place to explicitly control dependencies themselves.


 I'm not happy with this solution, but it seems to be the best compromise
 I can come up with.

 What do you think?



I don't know the full situation, or even if this will help, but maybe 
something like the following could help?

a.d:
----
module a;
import b;

Foo fooA;

static this()
{
    fooA = bar();
}
----
b.d:
----
module b;
import a;
class Foo { }

Foo bar()
{
   return new Foo;
}

Foo fooB;

pure static this()
{
    fooB = bar();
}
----

Here the pure notates that the static constructor doesn't not depend on 
any other modules (it's pure at module scope rather than function 
scope). If a module's static ctor is pure it doesn't matter what order 
it is executed in, so can be decided arbitrarily by the compiler. Doing 
this means you can have cyclic dependencies both with static 
constructors, without worrying about cyclic dependencies. It does 
however mean that you wouldn't be able to use functions from module C, 
it's better than the current situation though.
Mar 12 2010
parent Robert Clipsham <robert octarineparrot.com> writes:
On 12/03/10 16:44, Robert Clipsham wrote:

 I don't know the full situation, or even if this will help, but maybe
 something like the following could help?

 a.d:
 ----
 module a;
 import b;

 Foo fooA;

 static this()
 {
 fooA = bar();
 }
 ----
 b.d:
 ----
 module b;
 import a;
 class Foo { }

 Foo bar()
 {
 return new Foo;
 }

 Foo fooB;

 pure static this()
 {
 fooB = bar();
 }
 ----

 Here the pure notates that the static constructor doesn't not depend on
 any other modules (it's pure at module scope rather than function
 scope). If a module's static ctor is pure it doesn't matter what order
 it is executed in, so can be decided arbitrarily by the compiler. Doing
 this means you can have cyclic dependencies both with static
 constructors, without worrying about cyclic dependencies. It does
 however mean that you wouldn't be able to use functions from module C,
 it's better than the current situation though.



Another possible expansion of this to make it more flexible, but then 
pure might not be the correct keyword to use... The ctor could allow 
dependencies on other modules, as long as the other modules did not have 
cyclic dependencies/only had pure ctor's also. This would allow for 
something more complex like:

a.d:
----
module a;
import b;

Foo fooA;

static this()
{
    fooA = bar();
}
----
b.d:
----
module b;
import a;
import c;
class Foo : C { }

Foo bar()
{
   return new Foo;
}

Foo fooB;

pure static this()
{
    // This depends on module c, but c's does not have cyclic
    // dependencies, and its ctor is pure so it is safe to allow it
    fooB = bar();
}
----
c.d:
----
module c;
class C
{
}

C myC;

pure static this()
{
   myC = new C;
}
----
Mar 12 2010
prev sibling next sibling parent Leandro Lucarella <llucax gmail.com> writes:
BCS, el 12 de marzo a las 04:43 me escribiste:

 Hello Walter,
 

2. If a loop is detected, rather than issuing an error message, simply
arbitrarily pick one order and continue constructing.



 How about a way to explicitly cut edges in the graph (tagging
 imports with "pragma(nodep)" or " nodep" for instance)? That has the
 same end effect but for only a little more work, removes any
 non-determinism and allows for easy control of how things are
 resolved.



Yei! Weak imports (like weak references =P)

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
Mar 14 2010
prev sibling parent Russell Lewis <webmaster villagersonline.com> writes:
Walter Bright wrote:

 So, I propose the following:
 
 1. Attempt the depth-first traversal of the static constructors.
 
 2. If a loop is detected, rather than issuing an error message, simply 
 arbitrarily pick one order and continue constructing.



I'm of the camp that "if it's broken, the programmer needs to fix it." 
Why not just include this as an optional flag on the import statement?

import __dependency__ foo.bar.baz;

The __dependency__ flag means that if there exists a loop which involves 
both this module and foo.bar.baz, then foo.bar.baz should be initialized 
first.  (Contradictory flags would be an error.)

Russ
Mar 17 2010