www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - order of static constructor execution

reply Walter Bright <newshound1 digitalmars.com> writes:
Currently, it is performed as a strictly "depth-first" traversal of the 
graph defined by the import statements. As we've been discussing here, 
this works great until one has circular imports, meaning the depth-first 
graph has a loop in it.

The current behavior on detecting a loop is to quit with an error message.

The problems are:

1. The cycles are not easily gotten rid of when they are the result of 
template mixins.

2. Trying to analyze the static constructors to see what the 
dependencies actually are is fraught with unsolvable problems.


So, I propose the following:

1. Attempt the depth-first traversal of the static constructors.

2. If a loop is detected, rather than issuing an error message, simply 
arbitrarily pick one order and continue constructing.


The mitigating rationale is that modules that import each other are 
presumably written by the same person or team, and so that person is in 
the best place to explicitly control dependencies themselves.


I'm not happy with this solution, but it seems to be the best compromise 
I can come up with.

What do you think?
Mar 11 2010
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:
 1. Attempt the depth-first traversal of the static constructors.
 2. If a loop is detected, rather than issuing an error message, simply 
 arbitrarily pick one order and continue constructing.

Some alternative possibilities: - Issue a warning too? - Or allow that only if the code is compiled with "-allowloops"? - Or accept one specific loop only if the programmer somewhere states explicitly that this loop is OK, and issue an error otherwise? Bye, bearophile
Mar 11 2010
prev sibling next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 03/11/2010 08:42 PM, Walter Bright wrote:
 2. Trying to analyze the static constructors to see what the
 dependencies actually are is fraught with unsolvable problems.

could you elucidate on this point? Is it definitely impossible to get a hold of a complete dag, even if at runtime?
Mar 11 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Ellery Newcomer wrote:
 On 03/11/2010 08:42 PM, Walter Bright wrote:
 2. Trying to analyze the static constructors to see what the
 dependencies actually are is fraught with unsolvable problems.

could you elucidate on this point? Is it definitely impossible to get a hold of a complete dag, even if at runtime?

Suppose the static constructor calls foo(), the implementation of which is hidden to the compiler.
Mar 11 2010
parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 03/11/2010 10:15 PM, Walter Bright wrote:
 Ellery Newcomer wrote:
 On 03/11/2010 08:42 PM, Walter Bright wrote:
 2. Trying to analyze the static constructors to see what the
 dependencies actually are is fraught with unsolvable problems.

could you elucidate on this point? Is it definitely impossible to get a hold of a complete dag, even if at runtime?

Suppose the static constructor calls foo(), the implementation of which is hidden to the compiler.

Oh, you mean the true dependencies. I was just going to say that printing out the cycle that causes bombout would be better than nothing.
Mar 11 2010
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 03/11/2010 08:42 PM, Walter Bright wrote:
 Currently, it is performed as a strictly "depth-first" traversal of the
 graph defined by the import statements. As we've been discussing here,
 this works great until one has circular imports, meaning the depth-first
 graph has a loop in it.

 The current behavior on detecting a loop is to quit with an error message.

 The problems are:

 1. The cycles are not easily gotten rid of when they are the result of
 template mixins.

 2. Trying to analyze the static constructors to see what the
 dependencies actually are is fraught with unsolvable problems.


 So, I propose the following:

 1. Attempt the depth-first traversal of the static constructors.

 2. If a loop is detected, rather than issuing an error message, simply
 arbitrarily pick one order and continue constructing.


 The mitigating rationale is that modules that import each other are
 presumably written by the same person or team, and so that person is in
 the best place to explicitly control dependencies themselves.


 I'm not happy with this solution, but it seems to be the best compromise
 I can come up with.

 What do you think?

It looks like a step backwards to me. Andrei
Mar 11 2010
parent Max Samukha <spambox d-coding.com> writes:
Andrei Alexandrescu wrote:

 
 It looks like a step backwards to me.
 
 Andrei

What solution do you propose for problem 1?
Mar 12 2010
prev sibling next sibling parent reply BCS <none anon.com> writes:
Hello Walter,

 2. If a loop is detected, rather than issuing an error message, simply
 arbitrarily pick one order and continue constructing.

How about a way to explicitly cut edges in the graph (tagging imports with "pragma(nodep)" or " nodep" for instance)? That has the same end effect but for only a little more work, removes any non-determinism and allows for easy control of how things are resolved. -- ... <IXOYE><
Mar 11 2010
parent reply Don <nospam nospam.com> writes:
BCS wrote:
 Hello Walter,
 
 2. If a loop is detected, rather than issuing an error message, simply
 arbitrarily pick one order and continue constructing.

How about a way to explicitly cut edges in the graph (tagging imports with "pragma(nodep)" or " nodep" for instance)? That has the same end effect but for only a little more work, removes any non-determinism and allows for easy control of how things are resolved.

I agree. Seems to me that if a circular import exists, it's really something that the programmer needs to think about, and so it's reasonable for it to be explicit. A really harsh solution would be: pragma(nodependency, somepackage.somemodule); The compiler could check that somepackage.somemodule actually defines a static constructor. And it could even check that a circular import situation actually exists. This would force the pragma to be maintained correctly.
Mar 12 2010
parent reply Don <nospam nospam.com> writes:
Fawzi Mohamed wrote:
 
 On 12-mar-10, at 09:59, Don wrote:
 
 BCS wrote:
 Hello Walter,
 2. If a loop is detected, rather than issuing an error message, simply
 arbitrarily pick one order and continue constructing.

with "pragma(nodep)" or " nodep" for instance)? That has the same end effect but for only a little more work, removes any non-determinism and allows for easy control of how things are resolved.

I agree. Seems to me that if a circular import exists, it's really something that the programmer needs to think about, and so it's reasonable for it to be explicit. A really harsh solution would be: pragma(nodependency, somepackage.somemodule); The compiler could check that somepackage.somemodule actually defines a static constructor. And it could even check that a circular import situation actually exists. This would force the pragma to be maintained correctly.

I think that the main problem comes from static initializers that are mixed in and by themselves have no circular dependency.

If that's true, then perhaps a more restricted solution is better. Eg, use Walter's proposal ONLY if the static initializer is mixed in.
Mar 12 2010
next sibling parent reply BCS <none anon.com> writes:
Hello Don,

 If that's true, then perhaps a more restricted solution is better. Eg,
 use Walter's proposal ONLY if the static initializer is mixed in.

Random thought: Aside from making a new corner case, why not make static constructors in mixins only depend on imports from where they are defined rather than where they are used? -- ... <IXOYE><
Mar 12 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
BCS wrote:
 Random thought: Aside from making a new corner case, why not make static 
 constructors in mixins only depend on imports from where they are 
 defined rather than where they are used?

Template mixins are intended to behave like macros - they are instantiated in the context of where they are used, not where they are defined. Regular templates are instantiated in the context of where they are defined, not used.
Mar 12 2010
parent reply Max Samukha <spambox d-coding.com> writes:
On 12.03.2010 23:03, Walter Bright wrote:
 BCS wrote:
 Random thought: Aside from making a new corner case, why not make
 static constructors in mixins only depend on imports from where they
 are defined rather than where they are used?

Template mixins are intended to behave like macros - they are instantiated in the context of where they are used, not where they are defined. Regular templates are instantiated in the context of where they are defined, not used.

But then: module a; static this() {} // 1 template Foo(int i) { static this() // 2 { } } ---- module b; import a; static this() // 3 { } alias Foo!(1) foo; ---- Currently the construction order is 1, 3, 2 instead of the expected 1, 2, 3. In other words, the constructor in the template instance (2) should be combined with constructor 1, not 3. If static constructors in templates ran during the initialization of the module where they are defined, the problem with mixed-in code could be solved like this: module a; template StaticCtor(alias ctor) { static this() { ctor(); } } mixin template Foo(int i) { void construct() { writeln("Constructing for ", i); } alias StaticCtor!(construct) ctor; } ---- module b; import a; import c; mixin Foo!(1); ---- module c; import a; import b; mixin Foo!(2); So, though Foo is mixed into modules 'b' and 'c', its instances would be constructed in module 'a' via the regular StaticCtor template. Still a hack but the code at least makes it quite explicit for maintainers to stay alert.
Mar 12 2010
parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-03-12 19:10:10 -0500, Max Samukha <spambox d-coding.com> said:

 If static constructors in templates ran during the initialization of 
 the module where they are defined, the problem with mixed-in code could 
 be solved like this:

But templates can take function and types as argument. If the 'static this' of a template calls something through one of its argument, the module this function or type resides in needs to be initialized first. So you can't always call a template's static this at the same time as the module the template is defined in. -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Mar 12 2010
parent Max Samukha <spambox d-coding.com> writes:
On 13.03.2010 2:27, Michel Fortin wrote:
 On 2010-03-12 19:10:10 -0500, Max Samukha <spambox d-coding.com> said:

 If static constructors in templates ran during the initialization of
 the module where they are defined, the problem with mixed-in code
 could be solved like this:

But templates can take function and types as argument. If the 'static this' of a template calls something through one of its argument, the module this function or type resides in needs to be initialized first. So you can't always call a template's static this at the same time as the module the template is defined in.

Yes that is the point and purpose. The advantage is that the programmer is in control of when the static constructors in templated code are run - if you want the constructor to run in the context of the imported module use a regular template, otherwise, mixin. I'd definitely preferred it over the current semantics. Others may disagree.
Mar 13 2010
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Don wrote:
 If that's true, then perhaps a more restricted solution is better.
 Eg, use Walter's proposal ONLY if the static initializer is mixed in.

I should note that all static constructors in a module are combined into one function. So, the mixin static initializers are combined with any other static initializers.
Mar 12 2010
parent reply BCS <none anon.com> writes:
Hello Walter,

 Don wrote:
 
 If that's true, then perhaps a more restricted solution is better.
 Eg, use Walter's proposal ONLY if the static initializer is mixed in.
 

into one function. So, the mixin static initializers are combined with any other static initializers.

Does it have to run that way or could you lump them into several pieces and have them call each other as needed? -- ... <IXOYE><
Mar 12 2010
parent Walter Bright <newshound1 digitalmars.com> writes:
BCS wrote:
 Hello Walter,
 
 Don wrote:

 If that's true, then perhaps a more restricted solution is better.
 Eg, use Walter's proposal ONLY if the static initializer is mixed in.

into one function. So, the mixin static initializers are combined with any other static initializers.

Does it have to run that way or could you lump them into several pieces and have them call each other as needed?

Right now, they are defined to run in lexical order. Regrouping them in arbitrary orders will mess up intra-module ordering dependencies.
Mar 12 2010
prev sibling next sibling parent "Bob Jones" <me not.com> writes:
"Walter Bright" <newshound1 digitalmars.com> wrote in message 
news:hnc9n2$2tkj$1 digitalmars.com...
 Currently, it is performed as a strictly "depth-first" traversal of the 
 graph defined by the import statements. As we've been discussing here, 
 this works great until one has circular imports, meaning the depth-first 
 graph has a loop in it.

 The current behavior on detecting a loop is to quit with an error message.

 The problems are:

 1. The cycles are not easily gotten rid of when they are the result of 
 template mixins.

 2. Trying to analyze the static constructors to see what the dependencies 
 actually are is fraught with unsolvable problems.

Surely there must be some low hanging fruit that can be exploited? Say a module has no static constructors then cant that be considered a break in the dependency cycle?
Mar 11 2010
prev sibling next sibling parent Fawzi Mohamed <fawzi gmx.ch> writes:
On 12-mar-10, at 09:59, Don wrote:

 BCS wrote:
 Hello Walter,
 2. If a loop is detected, rather than issuing an error message,  
 simply
 arbitrarily pick one order and continue constructing.

imports with "pragma(nodep)" or " nodep" for instance)? That has the same end effect but for only a little more work, removes any non-determinism and allows for easy control of how things are resolved.

I agree. Seems to me that if a circular import exists, it's really something that the programmer needs to think about, and so it's reasonable for it to be explicit. A really harsh solution would be: pragma(nodependency, somepackage.somemodule); The compiler could check that somepackage.somemodule actually defines a static constructor. And it could even check that a circular import situation actually exists. This would force the pragma to be maintained correctly.

I think that the main problem comes from static initializers that are mixed in and by themselves have no circular dependency. So annotations have to be at the static initializer level, not at the module level. One way to solve this would be to add dependOnly(module1,...) static this(){ } then each module would have 2 dependencies: * indirectDeps imported modules + all dependOnly modules * directDeps look at all static initializers in the module: - plain static this() -> add all imported modules as dependency - annotated static this -> add all dependOnly dependencies * allDeps the list of module that have to be initialized before that module and is build from the previous ones as: direct deps + all indirect deps of those modules more explicitly allDeps(a){ deps=[] foreach(m in a.directDeps){ if (m in deps) continue; deps~=m; addDeps2(m,deps); } return deps; } addDeps2(a, ref deps){ if (a in deps) return; foreach(m in a.indirectDeps){ if (m in deps) continue; deps~=m; addDeps2(m,deps); } } then you can sort the modules that have static initializers (and only those) using: compare(a,b){ if (a in b.allDeps){ if (b in a.allDeps){ error("non comparable, circular dep between ",a,b); } return 1; } if (b in a.allDeps){ return -1; } return 0; } this is more work, but would be perfectly defined. conflicting ordering would still be disallowed, but circularly dependent modules can have initializers if all their initializers depend only on modules that are not circularly dependent. Fawzi
Mar 12 2010
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 3/12/10 03:42, Walter Bright wrote:
 Currently, it is performed as a strictly "depth-first" traversal of the
 graph defined by the import statements. As we've been discussing here,
 this works great until one has circular imports, meaning the depth-first
 graph has a loop in it.

 The current behavior on detecting a loop is to quit with an error message.

 The problems are:

 1. The cycles are not easily gotten rid of when they are the result of
 template mixins.

 2. Trying to analyze the static constructors to see what the
 dependencies actually are is fraught with unsolvable problems.


 So, I propose the following:

 1. Attempt the depth-first traversal of the static constructors.

 2. If a loop is detected, rather than issuing an error message, simply
 arbitrarily pick one order and continue constructing.


 The mitigating rationale is that modules that import each other are
 presumably written by the same person or team, and so that person is in
 the best place to explicitly control dependencies themselves.


 I'm not happy with this solution, but it seems to be the best compromise
 I can come up with.

 What do you think?

Yes please. As other have suggested a warning might be a good idea.
Mar 12 2010
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-03-11 21:42:47 -0500, Walter Bright <newshound1 digitalmars.com> said:

 Currently, it is performed as a strictly "depth-first" traversal of the 
 graph defined by the import statements. As we've been discussing here, 
 this works great until one has circular imports, meaning the 
 depth-first graph has a loop in it.
 
 The current behavior on detecting a loop is to quit with an error message.
 
 The problems are:
 
 1. The cycles are not easily gotten rid of when they are the result of 
 template mixins.
 
 2. Trying to analyze the static constructors to see what the 
 dependencies actually are is fraught with unsolvable problems.
 
 
 So, I propose the following:
 
 1. Attempt the depth-first traversal of the static constructors.
 
 2. If a loop is detected, rather than issuing an error message, simply 
 arbitrarily pick one order and continue constructing.
 
 
 The mitigating rationale is that modules that import each other are 
 presumably written by the same person or team, and so that person is in 
 the best place to explicitly control dependencies themselves.
 
 
 I'm not happy with this solution, but it seems to be the best 
 compromise I can come up with.
 
 What do you think?

I think it'd be better if it was explicit. Perhaps better would be an opt-in using an attribute on the static constructor. For instance: module a; import b; int a; static this() { a = b; } module b; import a; int b; cyclehead static this() { b = 10; } Here, module 'a' initialization depends on 'b'. Module 'b' initialization does not depend on a or anything, so you can make it cyclehead. cyclehead means that this module, when part of a cycle, can be initialized before the others. If all 'static this' in a module are cyclehead, then the module gets the cyclehead flag and is initialized first in the cycle. This is better than just picking one module at random to break the cycle. If no module have the cyclehead flag, then having a cycle is an error. The programmer will have to consciously choose one module to break the cycle. Most mixins static constructors will probably have to be cyclehead. -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Mar 12 2010
parent Fawzi Mohamed <fmohamed mac.com> writes:
On 2010-03-12 15:05:25 +0100, Michel Fortin <michel.fortin michelf.com> said:

 [...]
 Most mixins static constructors will probably have to be  cyclehead.

Here is an example of a reasonable use case to clarify why I think that my proposal, or something like it is the correct solution. Often the static initializers like those in module b or c would be mixed in (for example to give serialization support, as for example the class name in template classes is not unique). {{{ module a; // imports modules but does not create a circular dep class Register { void register(...){...} ... } Register globalRegister; static this(){ globalRegister=new Register(); } ------- module b; import c; import a; class Bla {...} dependOnly(a) static this { globalRegister.register(...); } ---- module c; import b; import a; class Bla2 {...} dependOnly(a) static this { globalRegister.register(...); } }}} My proposal would give the correct initialization order, and would still detect invalid cases (assuming the programmers does not lie about the dependOnly). Fawzi
Mar 12 2010
prev sibling next sibling parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 12/03/10 02:42, Walter Bright wrote:
 Currently, it is performed as a strictly "depth-first" traversal of the
 graph defined by the import statements. As we've been discussing here,
 this works great until one has circular imports, meaning the depth-first
 graph has a loop in it.

 The current behavior on detecting a loop is to quit with an error message.

 The problems are:

 1. The cycles are not easily gotten rid of when they are the result of
 template mixins.

 2. Trying to analyze the static constructors to see what the
 dependencies actually are is fraught with unsolvable problems.


 So, I propose the following:

 1. Attempt the depth-first traversal of the static constructors.

 2. If a loop is detected, rather than issuing an error message, simply
 arbitrarily pick one order and continue constructing.


 The mitigating rationale is that modules that import each other are
 presumably written by the same person or team, and so that person is in
 the best place to explicitly control dependencies themselves.


 I'm not happy with this solution, but it seems to be the best compromise
 I can come up with.

 What do you think?

I don't know the full situation, or even if this will help, but maybe something like the following could help? a.d: ---- module a; import b; Foo fooA; static this() { fooA = bar(); } ---- b.d: ---- module b; import a; class Foo { } Foo bar() { return new Foo; } Foo fooB; pure static this() { fooB = bar(); } ---- Here the pure notates that the static constructor doesn't not depend on any other modules (it's pure at module scope rather than function scope). If a module's static ctor is pure it doesn't matter what order it is executed in, so can be decided arbitrarily by the compiler. Doing this means you can have cyclic dependencies both with static constructors, without worrying about cyclic dependencies. It does however mean that you wouldn't be able to use functions from module C, it's better than the current situation though.
Mar 12 2010
parent Robert Clipsham <robert octarineparrot.com> writes:
On 12/03/10 16:44, Robert Clipsham wrote:
 I don't know the full situation, or even if this will help, but maybe
 something like the following could help?

 a.d:
 ----
 module a;
 import b;

 Foo fooA;

 static this()
 {
 fooA = bar();
 }
 ----
 b.d:
 ----
 module b;
 import a;
 class Foo { }

 Foo bar()
 {
 return new Foo;
 }

 Foo fooB;

 pure static this()
 {
 fooB = bar();
 }
 ----

 Here the pure notates that the static constructor doesn't not depend on
 any other modules (it's pure at module scope rather than function
 scope). If a module's static ctor is pure it doesn't matter what order
 it is executed in, so can be decided arbitrarily by the compiler. Doing
 this means you can have cyclic dependencies both with static
 constructors, without worrying about cyclic dependencies. It does
 however mean that you wouldn't be able to use functions from module C,
 it's better than the current situation though.

Another possible expansion of this to make it more flexible, but then pure might not be the correct keyword to use... The ctor could allow dependencies on other modules, as long as the other modules did not have cyclic dependencies/only had pure ctor's also. This would allow for something more complex like: a.d: ---- module a; import b; Foo fooA; static this() { fooA = bar(); } ---- b.d: ---- module b; import a; import c; class Foo : C { } Foo bar() { return new Foo; } Foo fooB; pure static this() { // This depends on module c, but c's does not have cyclic // dependencies, and its ctor is pure so it is safe to allow it fooB = bar(); } ---- c.d: ---- module c; class C { } C myC; pure static this() { myC = new C; } ----
Mar 12 2010
prev sibling next sibling parent Leandro Lucarella <llucax gmail.com> writes:
BCS, el 12 de marzo a las 04:43 me escribiste:
 Hello Walter,
 
2. If a loop is detected, rather than issuing an error message, simply
arbitrarily pick one order and continue constructing.

How about a way to explicitly cut edges in the graph (tagging imports with "pragma(nodep)" or " nodep" for instance)? That has the same end effect but for only a little more work, removes any non-determinism and allows for easy control of how things are resolved.

Yei! Weak imports (like weak references =P) -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ----------------------------------------------------------------------
Mar 14 2010
prev sibling parent Russell Lewis <webmaster villagersonline.com> writes:
Walter Bright wrote:
 So, I propose the following:
 
 1. Attempt the depth-first traversal of the static constructors.
 
 2. If a loop is detected, rather than issuing an error message, simply 
 arbitrarily pick one order and continue constructing.

I'm of the camp that "if it's broken, the programmer needs to fix it." Why not just include this as an optional flag on the import statement? import __dependency__ foo.bar.baz; The __dependency__ flag means that if there exists a loop which involves both this module and foo.bar.baz, then foo.bar.baz should be initialized first. (Contradictory flags would be an error.) Russ
Mar 17 2010