www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Fixing cyclic import static construction problems

reply Walter Bright <newshound2 digitalmars.com> writes:
For discussion:

Cyclical Imports

Problem:

---- a.d ----
     module a;
     import b;
     static this () { ... }
---- b.d ----
     module b;
     import a;
     static this() { ... }
-------------

Static constructors for a module are only run after static constructors
for all its imports are run. Circular imports, such as the above, are
detected at run time and the program is aborted.

This can in general be solved by moving the static constructor(s) into
a third module, c.d, which does not import a or b. But, people find this
to be unnatural.

Proposed Solution:

Add a pragma,

     pragma(cyclic_imports);

This can appear anywhere in a module, and applies globally to that module.
It means that static constructors from imports that are not part of the 
cycle
are run first, and that the static constructor for this module may be 
run before
the static constructors of other modules that are part of the cycle.

If any static constructors in such a module with the pragma have
the  safe attribute, that is a compile time error.
Nov 28 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/28/12 9:34 PM, Walter Bright wrote:
 For discussion:

I'd say we better finish const, immutable, and shared first. Andrei
Nov 28 2012
next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei 
Alexandrescu wrote:
 I'd say we better finish const, immutable, and shared first.

If we put off every easy fix until all the hard fixes are done, it means we have longer wait times on everything....
Nov 28 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/28/12 10:34 PM, Adam D. Ruppe wrote:
 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei Alexandrescu wrote:
 I'd say we better finish const, immutable, and shared first.

If we put off every easy fix until all the hard fixes are done, it means we have longer wait times on everything....

On the other hand if we work on every easy and unimportant fix we miss big on the important stuff. http://c2.com/cgi/wiki?FourQuadrants Andrei
Nov 29 2012
prev sibling next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 I'd say we better finish const, immutable, and shared first.

There are few things left to implement for purity (they are listed in Bugzilla), but what's left to do for const and immutable? Bye, bearophile
Nov 28 2012
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/28/12 11:26 PM, bearophile wrote:
 Andrei Alexandrescu:

 I'd say we better finish const, immutable, and shared first.

There are few things left to implement for purity (they are listed in Bugzilla), but what's left to do for const and immutable?

Construction flow and copy conversion. Andrei
Nov 29 2012
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/29/12 12:07 AM, bearophile wrote:
 what's left to do for const and immutable?

I guess the answer is too much long, so please ignore the question.

Ars longa vita brevis est. Andrei
Nov 29 2012
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
 what's left to do for const and immutable?

I guess the answer is too much long, so please ignore the question. Bye, bearophile
Nov 28 2012
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, November 28, 2012 22:19:54 Andrei Alexandrescu wrote:
 On 11/28/12 9:34 PM, Walter Bright wrote:
 For discussion:

I'd say we better finish const, immutable, and shared first.

Both problems need to be addressed, and this one is probably easier. It also has a huge impact on std.benchmark, so I would have thought that you'd be more in favor of it. However, we do have a tendancy to bring up problems like this, discuss them for a while, and then let them be more or less forgotten for a while. It keeps happening with stuff like const ref / auto ref, shared, const and Object, const postblit constructors, trusted blocks, etc. And we need to actually get them sorted out. - Jonathan M Davis
Nov 28 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/29/12 12:25 AM, Jonathan M Davis wrote:
 On Wednesday, November 28, 2012 22:19:54 Andrei Alexandrescu wrote:
 On 11/28/12 9:34 PM, Walter Bright wrote:
 For discussion:

I'd say we better finish const, immutable, and shared first.

Both problems need to be addressed, and this one is probably easier. It also has a huge impact on std.benchmark, so I would have thought that you'd be more in favor of it.

My perception is that we need to tackle the major issues at this point: process and qualifiers (including shared).
 However, we do have a tendancy to bring up problems like this, discuss them
 for a while, and then let them be more or less forgotten for a while. It keeps
 happening with stuff like const ref / auto ref, shared, const and Object, const
 postblit constructors,  trusted blocks, etc. And we need to actually get them
 sorted out.

Enhancement requests are the right place for those. Now that we have forum.dlang.org we can link to discussions, too. Andrei
Nov 29 2012
prev sibling next sibling parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei 
Alexandrescu wrote:
 On 11/28/12 9:34 PM, Walter Bright wrote:
 For discussion:

I'd say we better finish const, immutable, and shared first. Andrei

+1 Fully agree. Cyclic imports are a minor nuisance that can be easily solvable with better code architecture. Turbo Pascal/Delphi is the only language that I know fully allows cyclic dependencies between modules. So this is not that important for most people. -- Paulo
Nov 29 2012
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
11/29/2012 4:17 PM, Jonathan M Davis пишет:
 On Thursday, November 29, 2012 12:39:19 Paulo Pinto wrote:
 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei

 Alexandrescu wrote:
 On 11/28/12 9:34 PM, Walter Bright wrote:
 For discussion:

I'd say we better finish const, immutable, and shared first. Andrei

+1 Fully agree. Cyclic imports are a minor nuisance that can be easily solvable with better code architecture. Turbo Pascal/Delphi is the only language that I know fully allows cyclic dependencies between modules. So this is not that important for most people.

Basic features in the language require static constructors (e.g. static variables frequently do), and some things just can't be done without them. Andrei's std.benchmark proposal actually doesn't work, because in order to do what it does, in needs to mixin in a static constructor and static destructor (to set up the benchmarking and record it at the end, I believe). That completely falls apart due to cyclical imports.

I do suspect it could be healed by inverting the control flow (or rather bringing it back). Basically there is this pattern: module a; ... mixin ScheduleForBenchmark; Same for module b, c, d... And then compiling all of this with -version benchmark should run the benchmarks. It already has another problem: there has to be D main stuffed in there somehow. It can't be in std.benchmark as it then will have to be included in phobos(.lib|.a) and it can't be magically turned on/off with -version switch either unless std.benchmark is passed directly to the compiler. The other solution would be: make a separate my_benchmark.d module that contains: import a,b,c,d...; void main(){ runBenchmarks!(a,b,c,d...)(); } And that's it. In any case I usually like having some separation between different sets of modules to benchmark so that I can start them in a 'clean room' scenario.
 It doesn't actually cause any
 true circular dependencies, but it's treated as such anyway. A _lot_ of us
 have run into this problem and a number of us consider it to be a huge design
 problem in the language. Personally, I'd put it towards the top of design
 mistakes at this point, and I'm very glad to see Walter actually finally being
 willing to do something about this. In the past when I've brought up similar
 solutions, he's been completely opposed to them.

 Yes, we have other major issues that need to be resolved, and those may very
 well be of a higher priority, but this is still a very import issue, and it's
 the sort of issue that can probably be implemented with minimal effort. The
 main thing is getting Walter to agree to it and how it will be done. Once a
 decision is made, it wouldn't surprise me if someone like Kenji were able to
 get it done very quickly.

 - Jonathan M Davis

Dmitry Olshansky
Nov 29 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/29/12 10:54 AM, Dmitry Olshansky wrote:
 Basically there is this pattern:

 module a;
 ...
 mixin ScheduleForBenchmark;

 Same for module b, c, d...

Correct.
 And then compiling all of this with -version benchmark should run the
 benchmarks.

 It already has another problem: there has to be D main stuffed in there
 somehow. It can't be in std.benchmark as it then will have to be
 included in phobos(.lib|.a) and it can't be magically turned on/off with
 -version switch either unless std.benchmark is passed directly to the
 compiler.

 The other solution would be: make a separate my_benchmark.d module that
 contains:
 import a,b,c,d...;

 void main(){
 runBenchmarks!(a,b,c,d...)();
 }

That's workable. I'm hoping, however, to make benchmarks as easy and as trivial to define as unittests. Andrei
Nov 29 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-11-29 16:58, Andrei Alexandrescu wrote:

 That's workable. I'm hoping, however, to make benchmarks as easy and as
 trivial to define as unittests.

Unit tests have the same problem. One need to create a module containing a main function and importing the modules one wants to benchmark/test. One idea would be to add more support for this in the compiler. Another idea, that might better and is easier to implement, is tool that: 1. Takes a couple of modules on the command line 2. Create a main module which imports the modules on the command line 3. Compile all the modules 4. Run the benchmarks or tests -- /Jacob Carlborg
Nov 29 2012
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-11-29 19:17, deadalnix wrote:

 That is understood, but Let me explain how I see things.

 We are here adding yet a new feature, however small. Nothing stabilize
 when adding new feature all the time.

 The annoyance exist, is real, but is not THAT bad, and 'm pretty sure
 the proposed solution is likely to backfire. Firstly because a module
 can have several constructors, and more can be added by mixins. Ensuring
 that constructor A will not create dependancy cycle error may silently
 break other constructor of the module.

 Considering the problem static constructor solve, I'd be happy to see
 some think out of the box. The way things work here may be broken. Many
 people consider that D runtime should evolve in a way that promote
 fibers and map them over system threads (this have many benefit : the
 program adapt autmatically to the provided hardware, yields can be
 introduced directly into the runtime or some libraries to hide IO
 latencies, and many other things), and static constructor/variables
 really get into the way (you would have to run all static constructors
 each time you start a new fiber or reset one to reuse it).

 I really wish we can focus on figuring out the uses cases we have for
 static constructor and start the reflection from theses sues case and
 not from the feature itself.

BTW, how does Java handle this? And C# if it has something similar. -- /Jacob Carlborg
Nov 29 2012
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 11/29/2012 10:45 PM, deadalnix wrote:
 On Thursday, 29 November 2012 at 21:43:30 UTC, Jonathan M Davis wrote:
 On Thursday, November 29, 2012 21:08:58 Jacob Carlborg wrote:
 BTW, how does Java handle this? And C# if it has something similar.

They just let you blow your foot off. All static variables can be directly initialized at runtime, so it's easy to use variables before they're actually initialized. I don't know how they decide what order to run static constructors in, but AFAIK, it never worries about circular dependencies. We're only running into this problem beacuse we're trying to provide higher safety and better guarantees with regards to when and how variables are initialized. - Jonathan M Davis

Java have static block to pre initialize stuff before it is used. But I'm not sure how they react in case of cyclic dependancies.

In Java and C# static constructors are run whenever the class is first referenced. This means that almost any statement in Java or C# may cause arbitrary side-effecting code to run. Circular dependencies are not considered at all. If two static constructors depend on each other's statically initialized parts, at least one of them will see uninitialized data.
Nov 29 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-11-29 22:43, Jonathan M Davis wrote:

 They just let you blow your foot off. All static variables can be directly
 initialized at runtime, so it's easy to use variables before they're actually
 initialized. I don't know how they decide what order to run static
 constructors in, but AFAIK, it never worries about circular dependencies.
 We're only running into this problem beacuse we're trying to provide higher
 safety and better guarantees with regards to when and how variables are
 initialized.

I see. -- /Jacob Carlborg
Nov 30 2012
prev sibling next sibling parent reply Manfred Nowak <svv1999 hotmail.com> writes:
Max Samukha wrote:

 there must not be circular dependency issues

The model shown does not force circular dependencies, because every receiver module only imports one module: `module reflect'. And I can not see, that this module imports any other module. Therefore `module relect' and its `import' statements are not responsible for any circular dependency issue. Where am I wrong? -manfred
Nov 29 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11/29/2012 09:30 PM, Manfred Nowak wrote:
 Max Samukha wrote:

 there must not be circular dependency issues

The model shown does not force circular dependencies, because every receiver module only imports one module: `module reflect'. And I can not see, that this module imports any other module. Therefore `module relect' and its `import' statements are not responsible for any circular dependency issue. Where am I wrong? -manfred

A precondition for (pseudo-) circular dependency issues of this kind are static constructors. The static constructors are what module reflect imposes in its users in order to operate.
Nov 29 2012
next sibling parent reply Manfred Nowak <svv1999 hotmail.com> writes:
Timon Gehr wrote:

 what module reflect imposes

Thank you. It seems to be true then, that no issue would be generated if the compiler assumes, that Walters pragma is existent in every module. -manfred
Nov 29 2012
parent Manfred Nowak <svv1999 hotmail.com> writes:
Jonathan M Davis wrote:

 Or do you mean something else by the suggestion that the
 existence of Walter's pragma can be assumed?

In http://forum.dlang.org/thread/k96hj2$2lus$1 digitalmars.com? page=4#post-k98o7j:24qb6:241:40digitalmars.com Walter pointed this out for a clique of imports. All but one members of a clique are to have that pragma anyway---and Walter claims, that there is no error introduced, when the remaining member has that pragma too. -manfred
Dec 02 2012
prev sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, November 30, 2012 03:10:57 Manfred Nowak wrote:
 Timon Gehr wrote:
 what module reflect imposes

Thank you. It seems to be true then, that no issue would be generated if the compiler assumes, that Walters pragma is existent in every module.

So, you're suggesting that the runtime should just not worry about circular dependencies between modules at all? What if you have something like module a; import b; immutable Foo foo; shared static this() { foo = new immutable(Foo)(bar); } module b; import a; immutable Bar bar; shared static this() { bar = new immutable(Bar)(foo); } That's a true circular dependency. It's a big problem if that's allowed. D rightly disallows it. The problem is that in the vast majority of cases, there is no such circular dependency, so the circular dependency detection disallows many valid use cases. Or do you mean something else by the suggestion that the existence of Walter's pragma can be assumed? I don't know what else you could mean other than suggesting that assuming that it's present in all modules which is the same as throwing away circular dependency detection entirely. - Jonathan M Davis
Nov 29 2012
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11/29/2012 01:17 PM, Jonathan M Davis wrote:
 In the past when I've brought up similar solutions, he's been completely
opposed to them.
 ...

It is not a solution, it is a workaround.
Nov 29 2012
parent Artur Skawina <art.08.09 gmail.com> writes:
On 11/29/12 23:34, Jonathan M Davis wrote:
 On Thursday, November 29, 2012 23:28:07 Timon Gehr wrote:
 On 11/29/2012 01:17 PM, Jonathan M Davis wrote:
 In the past when I've brought up similar solutions, he's been completely
 opposed to them. ...

It is not a solution, it is a workaround.

What do you mean? The runtime sees circular dependencies between modules even when there's no actual circular dependency between static constructors. We need to fix that. One way is to just make the runtime not care, which wouldn't be particularly safe. Another is to explicitly tell it that there are no such dependencies. I don't see how that's not a solution. And unless someone can come up with a way for the runtime to somehow determine on its own that there's no actual, circular dependency, I don't see how anything better could be done.

It's relatively easy for the /compiler/ to figure it out; it's just that implementing a simple user-provided flag requires the least amount of work. What Walter suggested can be tweaked to be sane (per-ctor flag) and will still be useful if/when the compiler becomes smarter (think lazily initted module fields). For the compiler to check if the value of every imported symbol accessed inside a mod-ctor can be evaluated at compile-time (if you encounter a case where this is not true it means there (potentially) is a true dependency and the ctors should be ordered) would require more work. artur
Nov 30 2012
prev sibling parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 11/29/2012 5:51 PM, Max Samukha wrote:
 On Thursday, 29 November 2012 at 15:18:11 UTC, Paulo Pinto wrote:
 Maybe you care to provide an example?

The general problem is constructing global data structures based on data introspected at compile-time. My specific problem is extending scarce runtime type information provided by the language with something usable for runtime reflection. With lots of detail omitted: module reflect; Meta[string] metas; mixin template Reflect(alias object) { static this() { auto m = meta!(object); metas[m.fullName] ~= m; } } module a; import reflect; struct S { } mixin Reflect!S; The meta-object for S is automatically made available at runtime through the global metas array. Note that we do not want to force the user to register the meta-object manually because then it would not be a "better architecture". The important (Andrei somehow thinks it is not) requirement is there must not be circular dependency issues for the users of the "reflect" module.

How about running your own set of "constructors" searching the module info array, searching for specific classes in the module that are added by a mixin: ---------------------------------- module register; RegisterBase[string] registry; void doRegister(string name, RegisterBase r) { registry[name] = r; } class RegisterBase { abstract void _register(); } template Register(string name) { enum string Register = " class Register : RegisterBase { override void _register() { doRegister(\"" ~ name ~ "\", this); } } "; } void registerAll() { foreach(m; ModuleInfo) { TypeInfo_Class[] clss = m.localClasses(); foreach(c; clss) { if(c.base is RegisterBase.classinfo) { if(auto reg = cast(RegisterBase) c.create()) { reg._register(); } } } } } ----------------------- module a; import register; mixin(Register!"a"); ----------------------- module main; import std.stdio; import register; void main() { registerAll(); foreach(a, o; registry) writeln(a, " ", o); } This might also work for the benchmark module.
Nov 30 2012
prev sibling next sibling parent "Max Samukha" <maxsamukha gmail.com> writes:
On Thursday, 29 November 2012 at 11:39:20 UTC, Paulo Pinto wrote:
 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei 
 Alexandrescu wrote:
 On 11/28/12 9:34 PM, Walter Bright wrote:
 For discussion:

I'd say we better finish const, immutable, and shared first. Andrei

+1 Fully agree. Cyclic imports are a minor nuisance that can be easily solvable with better code architecture.

Show me please how to solve that problem easily with acceptable results, would you?
 Turbo Pascal/Delphi is the only language that I know fully 
 allows cyclic dependencies between modules. So this is not that 
 important for most people.

 --
 Paulo

Nov 29 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, November 29, 2012 12:39:19 Paulo Pinto wrote:
 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei
 
 Alexandrescu wrote:
 On 11/28/12 9:34 PM, Walter Bright wrote:
 For discussion:

I'd say we better finish const, immutable, and shared first. Andrei

+1 Fully agree. Cyclic imports are a minor nuisance that can be easily solvable with better code architecture. Turbo Pascal/Delphi is the only language that I know fully allows cyclic dependencies between modules. So this is not that important for most people.

Basic features in the language require static constructors (e.g. static variables frequently do), and some things just can't be done without them. Andrei's std.benchmark proposal actually doesn't work, because in order to do what it does, in needs to mixin in a static constructor and static destructor (to set up the benchmarking and record it at the end, I believe). That completely falls apart due to cyclical imports. It doesn't actually cause any true circular dependencies, but it's treated as such anyway. A _lot_ of us have run into this problem and a number of us consider it to be a huge design problem in the language. Personally, I'd put it towards the top of design mistakes at this point, and I'm very glad to see Walter actually finally being willing to do something about this. In the past when I've brought up similar solutions, he's been completely opposed to them. Yes, we have other major issues that need to be resolved, and those may very well be of a higher priority, but this is still a very import issue, and it's the sort of issue that can probably be implemented with minimal effort. The main thing is getting Walter to agree to it and how it will be done. Once a decision is made, it wouldn't surprise me if someone like Kenji were able to get it done very quickly. - Jonathan M Davis
Nov 29 2012
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 29 November 2012 at 12:04:28 UTC, Max Samukha wrote:
 On Thursday, 29 November 2012 at 11:39:20 UTC, Paulo Pinto 
 wrote:
 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei 
 Alexandrescu wrote:
 On 11/28/12 9:34 PM, Walter Bright wrote:
 For discussion:

I'd say we better finish const, immutable, and shared first. Andrei

+1 Fully agree. Cyclic imports are a minor nuisance that can be easily solvable with better code architecture.

Show me please how to solve that problem easily with acceptable results, would you?

You just need to have a better architecture. In 20 years of software development experience I never found a case were this wasn't possible. Maybe you care to provide an example? -- Paulo
Nov 29 2012
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 29 November 2012 at 12:17:49 UTC, Jonathan M Davis 
wrote:
 On Thursday, November 29, 2012 12:39:19 Paulo Pinto wrote:
 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei
 
 Alexandrescu wrote:
 On 11/28/12 9:34 PM, Walter Bright wrote:
 For discussion:

I'd say we better finish const, immutable, and shared first. Andrei

+1 Fully agree. Cyclic imports are a minor nuisance that can be easily solvable with better code architecture. Turbo Pascal/Delphi is the only language that I know fully allows cyclic dependencies between modules. So this is not that important for most people.

Basic features in the language require static constructors (e.g. static variables frequently do), and some things just can't be done without them. Andrei's std.benchmark proposal actually doesn't work, because in order to do what it does, in needs to mixin in a static constructor and static destructor (to set up the benchmarking and record it at the end, I believe). That completely falls apart due to cyclical imports. It doesn't actually cause any true circular dependencies, but it's treated as such anyway. A _lot_ of us have run into this problem and a number of us consider it to be a huge design problem in the language. Personally, I'd put it towards the top of design mistakes at this point, and I'm very glad to see Walter actually finally being willing to do something about this. In the past when I've brought up similar solutions, he's been completely opposed to them. Yes, we have other major issues that need to be resolved, and those may very well be of a higher priority, but this is still a very import issue, and it's the sort of issue that can probably be implemented with minimal effort. The main thing is getting Walter to agree to it and how it will be done. Once a decision is made, it wouldn't surprise me if someone like Kenji were able to get it done very quickly. - Jonathan M Davis

Maybe I should keep my mouth shut, because actually I don't really use D besides some toy experiments, as there is no place for D in the type of work I currently do, either on the job or privately. I like however to point people to D as a possible C++ successor as way to do some language publicity, and think that these type of discussions might scare D newbies away in what concerns language stability, hence my post. -- Paulo
Nov 29 2012
prev sibling next sibling parent "Max Samukha" <maxsamukha gmail.com> writes:
On Thursday, 29 November 2012 at 15:18:11 UTC, Paulo Pinto wrote:
 On Thursday, 29 November 2012 at 12:04:28 UTC, Max Samukha 
 wrote:
 On Thursday, 29 November 2012 at 11:39:20 UTC, Paulo Pinto 
 wrote:
 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei 
 Alexandrescu wrote:
 On 11/28/12 9:34 PM, Walter Bright wrote:
 For discussion:

I'd say we better finish const, immutable, and shared first. Andrei

+1 Fully agree. Cyclic imports are a minor nuisance that can be easily solvable with better code architecture.

Show me please how to solve that problem easily with acceptable results, would you?

You just need to have a better architecture. In 20 years of software development experience I never found a case were this wasn't possible.

That's an argument from authority, sorry.
 Maybe you care to provide an example?

The general problem is constructing global data structures based on data introspected at compile-time. My specific problem is extending scarce runtime type information provided by the language with something usable for runtime reflection. With lots of detail omitted: module reflect; Meta[string] metas; mixin template Reflect(alias object) { static this() { auto m = meta!(object); metas[m.fullName] ~= m; } } module a; import reflect; struct S { } mixin Reflect!S; The meta-object for S is automatically made available at runtime through the global metas array. Note that we do not want to force the user to register the meta-object manually because then it would not be a "better architecture". The important (Andrei somehow thinks it is not) requirement is there must not be circular dependency issues for the users of the "reflect" module.
Nov 29 2012
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 29 November 2012 at 12:17:49 UTC, Jonathan M Davis 
wrote:
 Basic features in the language require static constructors 
 (e.g. static
 variables frequently do), and some things just can't be done 
 without them.
 Andrei's std.benchmark proposal actually doesn't work, because 
 in order to do
 what it does, in needs to mixin in a static constructor and 
 static destructor
 (to set up the benchmarking and record it at the end, I 
 believe). That
 completely falls apart due to cyclical imports. It doesn't 
 actually cause any
 true circular dependencies, but it's treated as such anyway. A 
 _lot_ of us
 have run into this problem and a number of us consider it to be 
 a huge design
 problem in the language. Personally, I'd put it towards the top 
 of design
 mistakes at this point, and I'm very glad to see Walter 
 actually finally being
 willing to do something about this. In the past when I've 
 brought up similar
 solutions, he's been completely opposed to them.

 Yes, we have other major issues that need to be resolved, and 
 those may very
 well be of a higher priority, but this is still a very import 
 issue, and it's
 the sort of issue that can probably be implemented with minimal 
 effort. The
 main thing is getting Walter to agree to it and how it will be 
 done. Once a
 decision is made, it wouldn't surprise me if someone like Kenji 
 were able to
 get it done very quickly.

 - Jonathan M Davis

That is understood, but Let me explain how I see things. We are here adding yet a new feature, however small. Nothing stabilize when adding new feature all the time. The annoyance exist, is real, but is not THAT bad, and 'm pretty sure the proposed solution is likely to backfire. Firstly because a module can have several constructors, and more can be added by mixins. Ensuring that constructor A will not create dependancy cycle error may silently break other constructor of the module. Considering the problem static constructor solve, I'd be happy to see some think out of the box. The way things work here may be broken. Many people consider that D runtime should evolve in a way that promote fibers and map them over system threads (this have many benefit : the program adapt autmatically to the provided hardware, yields can be introduced directly into the runtime or some libraries to hide IO latencies, and many other things), and static constructor/variables really get into the way (you would have to run all static constructors each time you start a new fiber or reset one to reuse it). I really wish we can focus on figuring out the uses cases we have for static constructor and start the reflection from theses sues case and not from the feature itself.
Nov 29 2012
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 29 November 2012 at 16:51:29 UTC, Max Samukha wrote:
 On Thursday, 29 November 2012 at 15:18:11 UTC, Paulo Pinto 
 wrote:
 On Thursday, 29 November 2012 at 12:04:28 UTC, Max Samukha 
 wrote:
 On Thursday, 29 November 2012 at 11:39:20 UTC, Paulo Pinto 
 wrote:
 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei 
 Alexandrescu wrote:
 On 11/28/12 9:34 PM, Walter Bright wrote:
 For discussion:

I'd say we better finish const, immutable, and shared first. Andrei

+1 Fully agree. Cyclic imports are a minor nuisance that can be easily solvable with better code architecture.

Show me please how to solve that problem easily with acceptable results, would you?

You just need to have a better architecture. In 20 years of software development experience I never found a case were this wasn't possible.

That's an argument from authority, sorry.
 Maybe you care to provide an example?

The general problem is constructing global data structures based on data introspected at compile-time. My specific problem is extending scarce runtime type information provided by the language with something usable for runtime reflection. With lots of detail omitted: module reflect; Meta[string] metas; mixin template Reflect(alias object) { static this() { auto m = meta!(object); metas[m.fullName] ~= m; } } module a; import reflect; struct S { } mixin Reflect!S; The meta-object for S is automatically made available at runtime through the global metas array. Note that we do not want to force the user to register the meta-object manually because then it would not be a "better architecture".

For me too much use of meta capabilities is not a better architecture, as quite often it leads to write only code. If you are alreading writing code on the client side for initialization, like your mixin definition, the client already needs to make the reflect module of its existence, so why not call an initialization function that avoids the static constructors issues altogether? -- Paulo
Nov 29 2012
prev sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 11/29/2012 02:50 PM, Andrei Alexandrescu wrote:
 On 11/28/12 11:26 PM, bearophile wrote:
 There are few things left to implement for purity (they are listed in
 Bugzilla), but what's left to do for const and immutable?

Construction flow and copy conversion.

... meaning e.g. being able to effectively .dup or .idup any class, struct, etc.?
Nov 30 2012
prev sibling next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 11/29/12 03:34, Walter Bright wrote:
 Proposed Solution:
 
 Add a pragma,
 
     pragma(cyclic_imports);
 
 This can appear anywhere in a module, and applies globally to that module.
 It means that static constructors from imports that are not part of the cycle
 are run first, and that the static constructor for this module may be run
before
 the static constructors of other modules that are part of the cycle.

Bad name. Something like "module_ctors_unordered" would be better (I don't like that one either, but this way it's at least clear what it does). The equivalent attribute version could be module mymodule unordered_ctors; which is better, but still has the problem that adding another module ctor can result is silent breakage. These things can be mixed in, so something as innocent looking as import some_lib; [...] mixin blah; can already be a bug. Hence: static this() unordered { /*whatever*/ } and then either enforce that all mod-ctors have this attribute (otherwise every ctor in that moduile gets treated as ordered), or split them into two sets (better, but larger change; backward compatible, just requires a new enough runtime to be used). Of course it could be also done as static this() pragma(unordered) { /*whatever*/ } but this wouldn't really be better, and would require language change (the fact that you cannot attach a pragma to /anything/ is already a problem, eg for gcc-specific attributes). " unordered" could even be inferred, but I'm not sure how often that would help in practice. At least w/o making module-level imports invisible inside module ctors (which would make the deps explicit). artur
Nov 28 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, November 29, 2012 07:36:41 Artur Skawina wrote:
 On 11/29/12 03:34, Walter Bright wrote:
 Proposed Solution:
 
 Add a pragma,
 
     pragma(cyclic_imports);
 
 This can appear anywhere in a module, and applies globally to that module.
 It means that static constructors from imports that are not part of the
 cycle are run first, and that the static constructor for this module may
 be run before the static constructors of other modules that are part of
 the cycle.

like that one either, but this way it's at least clear what it does).

no_cyclic_imports would probably be better given that you're trying to get around a cyclic import, but I don't see what's unclear about cyclic_imports given that that's exactly what the runtime complains about when this problem occurs.
 but still has the problem that adding another module ctor
 can result is silent breakage.

That's actually a really good argument IMHO for having to put the pragma or attribute on every single static constructor in a module. True, it may be a bit annoying, but it would avoid silent breakage. - Jonathan M Davis
Nov 28 2012
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-11-29 03:34, Walter Bright wrote:

 Add a pragma,

      pragma(cyclic_imports);

 This can appear anywhere in a module, and applies globally to that module.
 It means that static constructors from imports that are not part of the
 cycle
 are run first, and that the static constructor for this module may be
 run before
 the static constructors of other modules that are part of the cycle.

I would think that "cyclic_imports" sounds like the static constructors are part of the cycle. -- /Jacob Carlborg
Nov 28 2012
prev sibling next sibling parent "Max Samukha" <maxsamukha gmail.com> writes:
On Thursday, 29 November 2012 at 02:34:11 UTC, Walter Bright 
wrote:
 For discussion:

 Cyclical Imports

 Problem:

 ---- a.d ----
     module a;
     import b;
     static this () { ... }
 ---- b.d ----
     module b;
     import a;
     static this() { ... }
 -------------

 Static constructors for a module are only run after static 
 constructors
 for all its imports are run. Circular imports, such as the 
 above, are
 detected at run time and the program is aborted.

 This can in general be solved by moving the static 
 constructor(s) into
 a third module, c.d, which does not import a or b. But, people 
 find this
 to be unnatural.

It is natural. It just doesn't cover important use cases.
 Proposed Solution:

 Add a pragma,

     pragma(cyclic_imports);

 This can appear anywhere in a module, and applies globally to 
 that module.
 It means that static constructors from imports that are not 
 part of the cycle
 are run first, and that the static constructor for this module 
 may be run before
 the static constructors of other modules that are part of the 
 cycle.

 If any static constructors in such a module with the pragma have
 the  safe attribute, that is a compile time error.

Can we have that implemented in a branch and see how it goes?
Nov 29 2012
prev sibling next sibling parent reply Manfred Nowak <svv1999 hotmail.com> writes:
Walter Bright wrote:

 It means that [...] the static constructor for this
 module may be run before

This is sufficient only for a simple cycle without any branches or a trivial clique like the one shown. Please recall, that in a clique every member is connected to every other member---in this case by an import statement. It is already not sufficient for a simple clique consisting of three modules, because now orderinh would be given for the remaining two modules. And it is ambiguous if both of the modules in the example given are marked with that pragma. In general a topological sorting has to be specified for all strongly connected components of the graph of imports. This cannot be done within the module, because this would bind the module to that special strongly connected component in that special set of modules. Which in turn would destroy reusability of the module for other programming tasks. Therefore: marking with a pragma isn't a fix for the depicted problem. -manfred
Nov 29 2012
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/29/2012 11:58 PM, Manfred Nowak wrote:
 This is sufficient only for a simple cycle without any branches
 or a trivial clique like the one shown. Please recall, that in a
 clique every member is connected to every other member---in this
 case by an import statement.

 It is already not sufficient for a simple clique consisting of
 three modules, because now orderinh would be given for the
 remaining two modules.

If you have: a imports b,c b imports a,c c imports a,b then two of those will need the pragma, that is correct. I don't see an issue with that, because there are two cycles.
 And it is ambiguous if both of the modules in the example given
 are marked with that pragma.

If: a imports b b imports a and a has the pragma, then the order of construction is a,b. If both have the pragma, then the order is a,b or b,a, yes, it is ambiguous, but it is not an error. One gets picked arbitrarily.
 In general a topological sorting has to be specified for all
 strongly connected components of the graph of imports.

I believe that the pragma does that.
 This cannot be done within the module, because this would bind
 the module to that special strongly connected component in that
 special set of  modules. Which in turn would destroy reusability
 of the module for other programming tasks.

I just don't see the problem.
Nov 29 2012
parent Manfred Nowak <svv1999 hotmail.com> writes:
Walter Bright wrote:

 yes, it is ambiguous, but it is not an error.
 One gets picked arbitrarily.

I did not see that it would not be erroneous. But if it is true, then I do not see a sense in manually adding pragmas: they can be assumed to exist. -manfred
Nov 29 2012
prev sibling next sibling parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:k96hj2$2lus$1 digitalmars.com...
 For discussion:

 Cyclical Imports

 Problem:


 Proposed Solution:

 Add a pragma,

     pragma(cyclic_imports);

 This can appear anywhere in a module, and applies globally to that module.
 It means that static constructors from imports that are not part of the 
 cycle
 are run first, and that the static constructor for this module may be run 
 before
 the static constructors of other modules that are part of the cycle.

I don't think this is sufficient. Imagine a group of modules that really _do_ have a cyclic dependency, and a mixin that adds an independent static this. Ideally you'd be able to mark the mixed-in constructor as independent without tainting the whole module. So just make the pragma apply to declarations, you either mark specific functions (which can then be mixed in) or put `pragma(...):` at the top of your module and you get your behaviour.
Nov 29 2012
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/30/2012 12:09 AM, Daniel Murphy wrote:
 I don't think this is sufficient.  Imagine a group of modules that really
 _do_ have a cyclic dependency, and a mixin that adds an independent static
 this.  Ideally you'd be able to mark the mixed-in constructor as independent
 without tainting the whole module.

 So just make the pragma apply to declarations, you either mark specific
 functions (which can then be mixed in) or put `pragma(...):` at the top of
 your module and you get your behaviour.

It is possible for each static constructor to specify independently of the other static constructors which imports must be constructed first. But do we really want to go that far?
Nov 29 2012
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/30/2012 9:43 AM, Walter Bright wrote:
 It is possible for each static constructor to specify independently of
 the other static constructors which imports must be constructed first.
 But do we really want to go that far?

One way to do that might be to borrow syntax from classes: static this() : std.stdio, a, c { ... } and the this static constructor only requires that modules std.stdio, a, and c be constructed first. static this() : void { ... } means it has no dependencies on other imports. static this() { ... } has the current behavior (all imported modules must be constructed first).
Nov 29 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/30/12 9:09 AM, foobar wrote:
 On Thursday, 29 November 2012 at 23:02:17 UTC, Walter Bright wrote:
 On 11/30/2012 9:43 AM, Walter Bright wrote:
 It is possible for each static constructor to specify independently of
 the other static constructors which imports must be constructed first.
 But do we really want to go that far?

One way to do that might be to borrow syntax from classes: static this() : std.stdio, a, c


 Why not simplify?

Why not further simplify? static this() { // JUST AS BEFORE ... } There is no need to redundantly specify what modules are used because... well they are right there in the body of the static constructor. * no extra syntax * no change in the symbol visibility rules (why are symbols invisible by default in static cdtors?) * no change to the manual * no breakage of existing code (only code that was broken will be accepted) * no acceptance of actual circular dependencies go through compilation This would be purely an improvement to the implementation that would allow more correct programs to compile. It's a removal of limitation - the best kind of language change there ever is. We should have a "bootcamp" area with small compiler and library projects (such as this after we reach consensus). People who are interested in helping D, whether or not they've done it before, could find this area things that are well defined and will definitely be accepted if properly executed. Andrei
Nov 30 2012
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/29/12 5:43 PM, Walter Bright wrote:
 On 11/30/2012 12:09 AM, Daniel Murphy wrote:
 I don't think this is sufficient. Imagine a group of modules that really
 _do_ have a cyclic dependency, and a mixin that adds an independent
 static
 this. Ideally you'd be able to mark the mixed-in constructor as
 independent
 without tainting the whole module.

 So just make the pragma apply to declarations, you either mark specific
 functions (which can then be mixed in) or put `pragma(...):` at the
 top of
 your module and you get your behaviour.

It is possible for each static constructor to specify independently of the other static constructors which imports must be constructed first. But do we really want to go that far?

I think we either do it right or leave it as it is. It's not like there's no workaround so if we take a stand here we better have something compelling. Andrei
Nov 29 2012
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/30/2012 9:05 PM, Peter Alexander wrote:
 On Friday, 30 November 2012 at 01:07:57 UTC, Andrei Alexandrescu wrote:
 I think we either do it right or leave it as it is. It's not like
 there's no workaround so if we take a stand here we better have
 something compelling.

 Andrei

+1 FWIW, I think this proposal sounds like a massive hack. Not a fan.

Andrei has a point.
Nov 30 2012
prev sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, November 29, 2012 20:07:57 Andrei Alexandrescu wrote:
 I think we either do it right or leave it as it is. It's not like
 there's no workaround so if we take a stand here we better have
 something compelling.

I think that an attribute per static constructor indicating that it had no circular dependencies would solve the problem just fine (putting it on the module is problematic, because new static constructors which _do_ have actual circular dependencies could be added), but we certainly want to be sure of this before doing it. And if we want to focus on shared or whatever now because it's higher priority, that's fine, but we do need to move forward with outstanding issues like these, and I do think that really need to solve this particular problem rather than considering the workarounds to be okay. A situation where static constructors effectively must be shunned because of the problems that they cause is definitely problematic considering how many features require them (e.g. a const or immutable static variable). And that's what we have right now. - Jonathan M Davis
Nov 29 2012
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 30 November 2012 at 01:07:57 UTC, Andrei Alexandrescu 
wrote:
 I think we either do it right or leave it as it is. It's not 
 like there's no workaround so if we take a stand here we better 
 have something compelling.

 Andrei

Finally some sanity here !
Nov 29 2012
prev sibling next sibling parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Friday, 30 November 2012 at 01:07:57 UTC, Andrei Alexandrescu 
wrote:
 On 11/29/12 5:43 PM, Walter Bright wrote:
 On 11/30/2012 12:09 AM, Daniel Murphy wrote:
 I don't think this is sufficient. Imagine a group of modules 
 that really
 _do_ have a cyclic dependency, and a mixin that adds an 
 independent
 static
 this. Ideally you'd be able to mark the mixed-in constructor 
 as
 independent
 without tainting the whole module.

 So just make the pragma apply to declarations, you either 
 mark specific
 functions (which can then be mixed in) or put `pragma(...):` 
 at the
 top of
 your module and you get your behaviour.

It is possible for each static constructor to specify independently of the other static constructors which imports must be constructed first. But do we really want to go that far?

I think we either do it right or leave it as it is. It's not like there's no workaround so if we take a stand here we better have something compelling. Andrei

+1 FWIW, I think this proposal sounds like a massive hack. Not a fan.
Nov 30 2012
prev sibling next sibling parent "foobar" <foo bar.com> writes:
On Thursday, 29 November 2012 at 23:02:17 UTC, Walter Bright 
wrote:
 On 11/30/2012 9:43 AM, Walter Bright wrote:
 It is possible for each static constructor to specify 
 independently of
 the other static constructors which imports must be 
 constructed first.
 But do we really want to go that far?

One way to do that might be to borrow syntax from classes: static this() : std.stdio, a, c { ... } and the this static constructor only requires that modules std.stdio, a, and c be constructed first. static this() : void { ... } means it has no dependencies on other imports. static this() { ... } has the current behavior (all imported modules must be constructed first).

Why not simplify? static this() { import std.stdio, a, c; // existing syntax ... } static this() { // no imports -> no dependencies ... } The current behavior should just be dropped.
Nov 30 2012
prev sibling next sibling parent "Tove" <tove fransson.se> writes:
On Friday, 30 November 2012 at 14:09:48 UTC, foobar wrote:
 Why not simplify?

 static this()
 {
 import std.stdio, a, c; // existing syntax
    ...
 }

 static this()
 { // no imports -> no dependencies
    ...
 }

 The current behavior should just be dropped.

+2 Simple & Elegant.
Nov 30 2012
prev sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Fri, 30 Nov 2012 14:29:19 -0000, Tove <tove fransson.se> wrote:

 On Friday, 30 November 2012 at 14:09:48 UTC, foobar wrote:
 Why not simplify?

 static this()
 {
 import std.stdio, a, c; // existing syntax
    ...
 }

 static this()
 { // no imports -> no dependencies
    ...
 }

 The current behavior should just be dropped.

+2 Simple & Elegant.

-2 Confusing. What is the scope of the import? How does it interact with imports above/below the static this? R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
Nov 30 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, November 29, 2012 16:18:10 Paulo Pinto wrote:
 On Thursday, 29 November 2012 at 12:04:28 UTC, Max Samukha wrote:
 On Thursday, 29 November 2012 at 11:39:20 UTC, Paulo Pinto
 
 wrote:
 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei
 
 Alexandrescu wrote:
 On 11/28/12 9:34 PM, Walter Bright wrote:
 For discussion:

I'd say we better finish const, immutable, and shared first. Andrei

+1 Fully agree. Cyclic imports are a minor nuisance that can be easily solvable with better code architecture.

Show me please how to solve that problem easily with acceptable results, would you?

You just need to have a better architecture. In 20 years of software development experience I never found a case were this wasn't possible. Maybe you care to provide an example?

Considering that you need static constructors to initialize any static variables at runtime (and in the case of const and immutable variables, you _can't_ work around that by doing the initialization later - it _must_ be in the static constructor), and all it takes for the compiler to declare a circular import is to have two modules import each other - even indirectly - it becomes extremely easy to run into this problem. And if you're dealing with immutable static variables which are initialized at runtime, you can't fix it unless you can contort your modules so that they don't depend on each other, and with a lot of modules, that can become extremely difficult. We had to strip out all static constructors from std.datetime because of this, and in order to fix it, we had to do some nasty voodoo with casting in order to lazily initialize some variables which are supposed to be immutable. _None_ of that sort of thing should be necessary. As it stands, it's basically bad practice to use static constructors, because it's so easy to end up with circular dependencies, and they can involve modules which don't seem even vaguely related because of other stuff that they import and are often a royal pain to debug and fix, if you even can without seriously revamping your design. And for a library like Phobos which needs to avoid breaking backwards compatibility, redesigning things to avoid such circular dependencies isn't necessarily even possible, because those redesigns would break backwards compatibliity. We _need_ a solution for this. Whether now is the best time tackle it is another matter, but the current situation with static constructors is ridiculous. There are features which require them, but you have to avoid them or you're going to run into circular dependency issues very easily. This is a case of some great design decisions in D have leading to a very bad design, and it needs to be fixed. Fortunately, it shouldn't be all that hard to fix with an attribute of some kind. But prior to now, Walter wouldn't even consider it. I do think though that the idea of using only one pragma instead of an attribute/pragma per static constructor is a bad idea because of the silent breakage that it would cause, as has been pointed out in this thread. So, his proposed solution needs some tweaking. - Jonathan M Davis
Nov 29 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, November 29, 2012 21:08:58 Jacob Carlborg wrote:
 BTW, how does Java handle this? And C# if it has something similar.

They just let you blow your foot off. All static variables can be directly initialized at runtime, so it's easy to use variables before they're actually initialized. I don't know how they decide what order to run static constructors in, but AFAIK, it never worries about circular dependencies. We're only running into this problem beacuse we're trying to provide higher safety and better guarantees with regards to when and how variables are initialized. - Jonathan M Davis
Nov 29 2012
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 29 November 2012 at 21:43:30 UTC, Jonathan M Davis 
wrote:
 On Thursday, November 29, 2012 21:08:58 Jacob Carlborg wrote:
 BTW, how does Java handle this? And C# if it has something 
 similar.

They just let you blow your foot off. All static variables can be directly initialized at runtime, so it's easy to use variables before they're actually initialized. I don't know how they decide what order to run static constructors in, but AFAIK, it never worries about circular dependencies. We're only running into this problem beacuse we're trying to provide higher safety and better guarantees with regards to when and how variables are initialized. - Jonathan M Davis

Java have static block to pre initialize stuff before it is used. But I'm not sure how they react in case of cyclic dependancies.
Nov 29 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, November 29, 2012 23:28:07 Timon Gehr wrote:
 On 11/29/2012 01:17 PM, Jonathan M Davis wrote:
 In the past when I've brought up similar solutions, he's been completely
 opposed to them. ...

It is not a solution, it is a workaround.

What do you mean? The runtime sees circular dependencies between modules even when there's no actual circular dependency between static constructors. We need to fix that. One way is to just make the runtime not care, which wouldn't be particularly safe. Another is to explicitly tell it that there are no such dependencies. I don't see how that's not a solution. And unless someone can come up with a way for the runtime to somehow determine on its own that there's no actual, circular dependency, I don't see how anything better could be done. Granted, I think that having a single pragma for the whole module like Walter is suggesting (as opposed to marking static constructors individually) is a bad idea because of the silent breakage that it can cause, but the basic idea is solid. - Jonathan M Davis
Nov 29 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, November 30, 2012 10:27:31 Artur Skawina wrote:
 On 11/29/12 23:34, Jonathan M Davis wrote:
 On Thursday, November 29, 2012 23:28:07 Timon Gehr wrote:
 On 11/29/2012 01:17 PM, Jonathan M Davis wrote:
 In the past when I've brought up similar solutions, he's been completely
 opposed to them. ...

It is not a solution, it is a workaround.

What do you mean? The runtime sees circular dependencies between modules even when there's no actual circular dependency between static constructors. We need to fix that. One way is to just make the runtime not care, which wouldn't be particularly safe. Another is to explicitly tell it that there are no such dependencies. I don't see how that's not a solution. And unless someone can come up with a way for the runtime to somehow determine on its own that there's no actual, circular dependency, I don't see how anything better could be done.

It's relatively easy for the /compiler/ to figure it out; it's just that implementing a simple user-provided flag requires the least amount of work. What Walter suggested can be tweaked to be sane (per-ctor flag) and will still be useful if/when the compiler becomes smarter (think lazily initted module fields). For the compiler to check if the value of every imported symbol accessed inside a mod-ctor can be evaluated at compile-time (if you encounter a case where this is not true it means there (potentially) is a true dependency and the ctors should be ordered) would require more work.

It can't be evaluated at compile time because of .di files. The compiler doesn't necessarily have all of the source to work with - including the static constructors - and if it doesn't have that, it can't do it. If anything figures this out automatically, it has to be the runtime. If we can do that, great, but it means that it'll have actually have to look at individual symbols rather than just at the module level like it's doing now, and right now, we have nothing even close to that. Regardless, while the compiler may be able to provide additional information to the runtime, it's still the runtime that needs to figure this out and not the compiler. - Jonathan M Davis
Nov 30 2012
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 30 November 2012 at 20:40:23 UTC, Jonathan M Davis 
wrote:
 It can't be evaluated at compile time because of .di files. The 
 compiler
 doesn't necessarily have all of the source to work with - 
 including the static
 constructors - and if it doesn't have that, it can't do it. If 
 anything figures
 this out automatically, it has to be the runtime. If we can do 
 that, great,
 but it means that it'll have actually have to look at 
 individual symbols
 rather than just at the module level like it's doing now, and 
 right now, we
 have nothing even close to that. Regardless, while the compiler 
 may be able
 to provide additional information to the runtime, it's still 
 the runtime that
 needs to figure this out and not the compiler.

I'd bet you can solve most cases anyway.
Nov 30 2012
prev sibling next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 11/30/12 21:40, Jonathan M Davis wrote:
 On Friday, November 30, 2012 10:27:31 Artur Skawina wrote:
 On 11/29/12 23:34, Jonathan M Davis wrote:
 On Thursday, November 29, 2012 23:28:07 Timon Gehr wrote:
 On 11/29/2012 01:17 PM, Jonathan M Davis wrote:
 In the past when I've brought up similar solutions, he's been completely
 opposed to them. ...

It is not a solution, it is a workaround.

What do you mean? The runtime sees circular dependencies between modules even when there's no actual circular dependency between static constructors. We need to fix that. One way is to just make the runtime not care, which wouldn't be particularly safe. Another is to explicitly tell it that there are no such dependencies. I don't see how that's not a solution. And unless someone can come up with a way for the runtime to somehow determine on its own that there's no actual, circular dependency, I don't see how anything better could be done.

It's relatively easy for the /compiler/ to figure it out; it's just that implementing a simple user-provided flag requires the least amount of work. What Walter suggested can be tweaked to be sane (per-ctor flag) and will still be useful if/when the compiler becomes smarter (think lazily initted module fields). For the compiler to check if the value of every imported symbol accessed inside a mod-ctor can be evaluated at compile-time (if you encounter a case where this is not true it means there (potentially) is a true dependency and the ctors should be ordered) would require more work.

It can't be evaluated at compile time because of .di files. The compiler

It can. The case where all definitions aren't available is of course a potential source of cycles and means that the ctors have to be run in order. There's no way around that, and it's where the attribute (nee pragma) helps, because it can be attached to the declaration, when the programmer thinks he knows better. Andrei's naive suggestion would only handle the trivial toy example case, but fail for much of real code, that needs to access statically known properties of other modules. D has enough features that work for five-liners, but are unusable for real work. The important points are that: 1) a ctor that does not access any symbol from another module is not treated as dependent on that module. 2) accessing just /types/ defined in another module works. No RT dep here. 3) reading known initialized constant data works. That's const/immutable/enum - again, those can never become a RT dep. 4) calling side-effect free code that does not depend on non-local state works. This is why an is-it-ctfeable check works - it will catch even the indirect deps. Yeah, it's conservative, but is has to be. A kind of 'pure-but-w/o-external-refs" thing would help further, but that can be added incrementally and may not even be necessary.
 have nothing even close to that. Regardless, while the compiler may be able
 to provide additional information to the runtime, it's still the runtime that
 needs to figure this out and not the compiler.

No. Would, at least, require going from per-module to per-symbol which is a *much* larger change than was proposed here. I don't even want to think about the (runtime) cost. artur
Dec 01 2012
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 1 December 2012 at 10:54:37 UTC, Artur Skawina wrote:
 3) reading known initialized constant data works. That's 
 const/immutable/enum -
    again, those can never become a RT dep.

const or immutable can be instanciated by another static ctor.
 4) calling side-effect free code that does not depend on 
 non-local state works.
    This is why an is-it-ctfeable check works - it will catch 
 even the indirect deps.
    Yeah, it's conservative, but is has to be. A kind of 
 'pure-but-w/o-external-refs"
    thing would help further, but that can be added 
 incrementally and may not even
    be necessary.

pure is probably enough. CTFEable is more restrictive.
 have nothing even close to that. Regardless, while the 
 compiler may be able
 to provide additional information to the runtime, it's still 
 the runtime that
 needs to figure this out and not the compiler.


That is an undemonstrated assertion (and I think it is false).
 No. Would, at least, require going from per-module to 
 per-symbol which is a *much*
 larger change than was proposed here. I don't even want to 
 think about the (runtime)
 cost.

This reasonning is based on the assertion above, so is meaningless until the assertion is proven to be true.
Dec 01 2012
prev sibling next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 12/01/12 12:29, deadalnix wrote:
 On Saturday, 1 December 2012 at 10:54:37 UTC, Artur Skawina wrote:
 3) reading known initialized constant data works. That's const/immutable/enum -
    again, those can never become a RT dep.

const or immutable can be instanciated by another static ctor.

The key words are "known" and "initialized". IOW const int a = 42; doesn't add a dep, but const int a; does. /Modifying/ initialized const data from a mod ctor is (and should be) illegal.
 4) calling side-effect free code that does not depend on non-local state works.
    This is why an is-it-ctfeable check works - it will catch even the indirect
deps.
    Yeah, it's conservative, but is has to be. A kind of
'pure-but-w/o-external-refs"
    thing would help further, but that can be added incrementally and may not
even
    be necessary.

pure is probably enough. CTFEable is more restrictive.

D's pure isn't enough, because it will allow accesses to external immutable data - which can be modified by a mod ctor. But like i said - this might not be a problem in practice, as it's just about the cross-module non-ctfeable, but truly pure function calls inside mod ctors - is this really something that occurs often enough?
 have nothing even close to that. Regardless, while the compiler may be able
 to provide additional information to the runtime, it's still the runtime that
 needs to figure this out and not the compiler.


That is an undemonstrated assertion (and I think it is false).
 No. Would, at least, require going from per-module to per-symbol which is a
*much*
 larger change than was proposed here. I don't even want to think about the
(runtime)
 cost.

This reasonning is based on the assertion above, so is meaningless until the assertion is proven to be true.

I'm not sure what assertion you think is false, but theoretically it /is/ possible to figure out the deps at runtime, even w/o any hints. Think valgrind. But it's a very bad idea, with a significant runtime cost (not as bad as valgrind since you can figure out some things statically, but determining the order alone would already be too slow for larger projects with many symbols and deps.) artur
Dec 01 2012
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 1 December 2012 at 12:08:10 UTC, Artur Skawina wrote:
 The key words are "known" and "initialized". IOW

    const int a = 42;

 doesn't add a dep, but

   const int a;

 does.

 /Modifying/ initialized const data from a mod ctor is (and 
 should be) illegal.

That make sense.
 D's pure isn't enough, because it will allow accesses to 
 external immutable
 data - which can be modified by a mod ctor. But like i said - 
 this might not be
 a problem in practice, as it's just about the cross-module 
 non-ctfeable, but
 truly pure function calls inside mod ctors - is this really 
 something that occurs
 often enough?

Ho yes, this make sense too. CTFEable is still too restrictive but pure isn't enough.
 I'm not sure what assertion you think is false, but 
 theoretically it /is/ possible
 to figure out the deps at runtime, even w/o any hints. Think 
 valgrind. But it's
 a very bad idea, with a significant runtime cost (not as bad as 
 valgrind since you
 can figure out some things statically, but determining the 
 order alone would already
 be too slow for larger projects with many symbols and deps.)

The assertion is that it must be done at runtime. It is obviously doable at runtime, but complicated. I think that most of it can be done at compile time.
Dec 01 2012
prev sibling next sibling parent "Jason House" <jason.james.house gmail.com> writes:
On Thursday, 29 November 2012 at 02:34:11 UTC, Walter Bright 
wrote:
 For discussion:

 Cyclical Imports

 Problem:

 ---- a.d ----
     module a;
     import b;
     static this () { ... }
 ---- b.d ----
     module b;
     import a;
     static this() { ... }
 -------------

 Static constructors for a module are only run after static 
 constructors
 for all its imports are run. Circular imports, such as the 
 above, are
 detected at run time and the program is aborted.

 This can in general be solved by moving the static 
 constructor(s) into
 a third module, c.d, which does not import a or b. But, people 
 find this
 to be unnatural.

 Proposed Solution:

 Add a pragma,

     pragma(cyclic_imports);

 This can appear anywhere in a module, and applies globally to 
 that module.
 It means that static constructors from imports that are not 
 part of the cycle
 are run first, and that the static constructor for this module 
 may be run before
 the static constructors of other modules that are part of the 
 cycle.

 If any static constructors in such a module with the pragma have
 the  safe attribute, that is a compile time error.

While more complex, would it be possible to declare which imported modules can be constructed after the current one? That allows catching any other unintended cyclic imports. For example: static this { pragma(cyclic_import, b); // code that does not depend on // static constructor(s) in module b }
Dec 01 2012
prev sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Thu, 29 Nov 2012 12:39:19 +0100
"Paulo Pinto" <pjmlp progtools.org> wrote:

 On Thursday, 29 November 2012 at 03:19:55 UTC, Andrei 
 Alexandrescu wrote:
 On 11/28/12 9:34 PM, Walter Bright wrote:
 For discussion:

I'd say we better finish const, immutable, and shared first. Andrei

+1 Fully agree. Cyclic imports are a minor nuisance that can be easily solvable with better code architecture.

Or just lazy initialization: Foo f = blah(); --> private Foo _f; private bool fInited=false; property Foo f() { if(!fInited) { _f = blah(); fInited = true; } return _f; } And that boilerplate's trivially wrapped up with a mixin.
Dec 07 2012