www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - Module init ordering bug - analysis and code example

reply pragma <pragma_member pathlink.com> writes:
Walter,
in learning how D handles module constructors, I came across the same bug that
others have stumbled across.  I did a little sleuthing to discover what was
causing this problem. ;)

My analysis (see below) showed that modules without constructors (static
this(){}) cannot be prioritized in _moduleCtor(), nor will their imports be
prioritized, since without such a constructor a module isnt't added to the
_moduleinfo_array.  Is this correct?

If so (and if you don't mind), I'd like to propose that dmd be changed to add
all modules (not just those with ctors) to the _moduleinfo_array.  This would
not only fix these particular init problems, but would also be very useful for
reflection purposes.

Many, many thanks,
Eric



[Example]
The expected init order was: mod4, mod3, mod2, mod1 as that is the order of
import and the order of dependency by function call.  Instead the modules are
run in a different order: mod2, mod1, mod4.  Inserting a module constructor into
mod3 forces the correct init order.

//// mod1.d ////
import std.stdio;
import std.moduleinit;
import mod2;

static this(){
writefln("mod1");
}

void main(){
writefln("\n--MODULE INFO--\n");

foreach(ModuleInfo info; _moduleinfo_array){
writefln("%s", info.name);
foreach(ModuleInfo imported; info.importedModules){
writefln("\timport %s",imported.name);
}
}
}


//// mod2.d ////
import std.stdio;
import mod3;

static this(){
writefln("mod2");
mod3.foo();
}


//// mod3.d ////
import std.stdio;
import mod4;

/* Uncomment to force correct ordering
static this(){
writefln("mod3");
}
*/

static void foo(){
writefln("\tmod3.foo()");
mod4.bar();
}


//// mod4.d ////
import std.stdio;

static this(){
writefln("mod4");
}

static void bar(){
writefln("\tmod4.bar()");
}


//// Output ////
mod2
mod3.foo()
mod4.bar()
mod1
mod4

--MODULE INFO--

mod1
import mod2
mod2
mod4
format
utf
string
adi
import string
gcbits
import string
thread

- Pragma
[[ EricAnderton at (static this{}) yahoo dot com ]]
Sep 02 2004
next sibling parent reply pragma <pragma_member pathlink.com> writes:
Update:
The problem gets worse with 'private import'.  Module ctor dependencies seem to
only be added if the 'private' declaration is dropped, regardless of the
presence of a module ctor.  

An revised version of my example (with private imports used) is attached.  The
output shows that mod1 no longer shows a dependency upon mod2.

mod1
mod2
mod3.foo()
mod4.bar()
mod4

--MODULE INFO--

mod1
mod2
mod4
format
utf
string
adi
import string
gcbits
import string
thread

-Pragma

In article <ch7s0p$mi8$1 digitaldaemon.com>, pragma says...
Walter,
in learning how D handles module constructors, I came across the same bug that
others have stumbled across.  I did a little sleuthing to discover what was
causing this problem. ;)

My analysis (see below) showed that modules without constructors (static
this(){}) cannot be prioritized in _moduleCtor(), nor will their imports be
prioritized, since without such a constructor a module isnt't added to the
_moduleinfo_array.  Is this correct?

If so (and if you don't mind), I'd like to propose that dmd be changed to add
all modules (not just those with ctors) to the _moduleinfo_array.  This would
not only fix these particular init problems, but would also be very useful for
reflection purposes.

Many, many thanks,
Eric



[Example]
The expected init order was: mod4, mod3, mod2, mod1 as that is the order of
import and the order of dependency by function call.  Instead the modules are
run in a different order: mod2, mod1, mod4.  Inserting a module constructor into
mod3 forces the correct init order.

//// mod1.d ////
import std.stdio;
import std.moduleinit;
import mod2;

static this(){
writefln("mod1");
}

void main(){
writefln("\n--MODULE INFO--\n");

foreach(ModuleInfo info; _moduleinfo_array){
writefln("%s", info.name);
foreach(ModuleInfo imported; info.importedModules){
writefln("\timport %s",imported.name);
}
}
}


//// mod2.d ////
import std.stdio;
import mod3;

static this(){
writefln("mod2");
mod3.foo();
}


//// mod3.d ////
import std.stdio;
import mod4;

/* Uncomment to force correct ordering
static this(){
writefln("mod3");
}
*/

static void foo(){
writefln("\tmod3.foo()");
mod4.bar();
}


//// mod4.d ////
import std.stdio;

static this(){
writefln("mod4");
}

static void bar(){
writefln("\tmod4.bar()");
}


//// Output ////
mod2
mod3.foo()
mod4.bar()
mod1
mod4

--MODULE INFO--

mod1
import mod2
mod2
mod4
format
utf
string
adi
import string
gcbits
import string
thread

- Pragma
[[ EricAnderton at (static this{}) yahoo dot com ]]

Sep 02 2004
next sibling parent "antiAlias" <fu bar.com> writes:
Yep; I just posted you a reply on dsource.

The issue is with the importedModules[], which is supposed to house a list
of imported modules (filtered to include only those which require static
initialization) ~ per-module.

Unfortunately, if you add /any/ kind of attribute to an import statement, it
is excluded from importedModules[]. Entire chains of dependency are thus
lost, and static constructors are never executed. Anything project that (for
example) makes use "private import" is thus inviting doom.


"pragma" <pragma_member pathlink.com> wrote in message
news:ch84ft$r1b$1 digitaldaemon.com...
Update:
The problem gets worse with 'private import'.  Module ctor dependencies seem
to
only be added if the 'private' declaration is dropped, regardless of the
presence of a module ctor.

An revised version of my example (with private imports used) is attached.
The
output shows that mod1 no longer shows a dependency upon mod2.

mod1
mod2
mod3.foo()
mod4.bar()
mod4

--MODULE INFO--

mod1
mod2
mod4
format
utf
string
adi
import string
gcbits
import string
thread

-Pragma

In article <ch7s0p$mi8$1 digitaldaemon.com>, pragma says...
Walter,
in learning how D handles module constructors, I came across the same bug

others have stumbled across.  I did a little sleuthing to discover what was
causing this problem. ;)

My analysis (see below) showed that modules without constructors (static
this(){}) cannot be prioritized in _moduleCtor(), nor will their imports be
prioritized, since without such a constructor a module isnt't added to the
_moduleinfo_array.  Is this correct?

If so (and if you don't mind), I'd like to propose that dmd be changed to

all modules (not just those with ctors) to the _moduleinfo_array.  This

not only fix these particular init problems, but would also be very useful

reflection purposes.

Many, many thanks,
Eric



[Example]
The expected init order was: mod4, mod3, mod2, mod1 as that is the order of

import and the order of dependency by function call.  Instead the modules

run in a different order: mod2, mod1, mod4.  Inserting a module constructor

mod3 forces the correct init order.

//// mod1.d ////
import std.stdio;
import std.moduleinit;
import mod2;

static this(){
writefln("mod1");
}

void main(){
writefln("\n--MODULE INFO--\n");

foreach(ModuleInfo info; _moduleinfo_array){
writefln("%s", info.name);
foreach(ModuleInfo imported; info.importedModules){
writefln("\timport %s",imported.name);
}
}
}


//// mod2.d ////
import std.stdio;
import mod3;

static this(){
writefln("mod2");
mod3.foo();
}


//// mod3.d ////
import std.stdio;
import mod4;

/* Uncomment to force correct ordering
static this(){
writefln("mod3");
}
*/

static void foo(){
writefln("\tmod3.foo()");
mod4.bar();
}


//// mod4.d ////
import std.stdio;

static this(){
writefln("mod4");
}

static void bar(){
writefln("\tmod4.bar()");
}


//// Output ////
mod2
mod3.foo()
mod4.bar()
mod1
mod4

--MODULE INFO--

mod1
import mod2
mod2
mod4
format
utf
string
adi
import string
gcbits
import string
thread

- Pragma
[[ EricAnderton at (static this{}) yahoo dot com ]]

Sep 02 2004
prev sibling parent "Walter" <newshound digitalmars.com> writes:
"pragma" <pragma_member pathlink.com> wrote in message
news:ch84ft$r1b$1 digitaldaemon.com...
 Update:
 The problem gets worse with 'private import'.  Module ctor dependencies

 only be added if the 'private' declaration is dropped, regardless of the
 presence of a module ctor.

I'm glad you were able to track this down. Now I can fix it!
Sep 02 2004
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
D's only requirement for ordering module constructors between modules is
that the imported module constructors must happen first. They are not
ordered by the order of the import statements - or at least such order is an
artifact of the implementation.

You should therefore be able to control the order of module construction by
making sure that each module depending on another module constructor imports
that module.

"pragma" <pragma_member pathlink.com> wrote in message
news:ch7s0p$mi8$1 digitaldaemon.com...
 Walter,
 in learning how D handles module constructors, I came across the same bug

 others have stumbled across.  I did a little sleuthing to discover what

 causing this problem. ;)

 My analysis (see below) showed that modules without constructors (static
 this(){}) cannot be prioritized in _moduleCtor(), nor will their imports

 prioritized, since without such a constructor a module isnt't added to the
 _moduleinfo_array.  Is this correct?

 If so (and if you don't mind), I'd like to propose that dmd be changed to

 all modules (not just those with ctors) to the _moduleinfo_array.  This

 not only fix these particular init problems, but would also be very useful

 reflection purposes.

 Many, many thanks,
 Eric



 [Example]
 The expected init order was: mod4, mod3, mod2, mod1 as that is the order

 import and the order of dependency by function call.  Instead the modules

 run in a different order: mod2, mod1, mod4.  Inserting a module

 mod3 forces the correct init order.

 //// mod1.d ////
 import std.stdio;
 import std.moduleinit;
 import mod2;

 static this(){
 writefln("mod1");
 }

 void main(){
 writefln("\n--MODULE INFO--\n");

 foreach(ModuleInfo info; _moduleinfo_array){
 writefln("%s", info.name);
 foreach(ModuleInfo imported; info.importedModules){
 writefln("\timport %s",imported.name);
 }
 }
 }


 //// mod2.d ////
 import std.stdio;
 import mod3;

 static this(){
 writefln("mod2");
 mod3.foo();
 }


 //// mod3.d ////
 import std.stdio;
 import mod4;

 /* Uncomment to force correct ordering
 static this(){
 writefln("mod3");
 }
 */

 static void foo(){
 writefln("\tmod3.foo()");
 mod4.bar();
 }


 //// mod4.d ////
 import std.stdio;

 static this(){
 writefln("mod4");
 }

 static void bar(){
 writefln("\tmod4.bar()");
 }


 //// Output ////
 mod2
 mod3.foo()
 mod4.bar()
 mod1
 mod4

 --MODULE INFO--

 mod1
 import mod2
 mod2
 mod4
 format
 utf
 string
 adi
 import string
 gcbits
 import string
 thread

 - Pragma
 [[ EricAnderton at (static this{}) yahoo dot com ]]

Sep 02 2004
next sibling parent Ben Hinkle <bhinkle4 juno.com> writes:
So are you saying the supplied example isn't a bug? How about something like

mod1.m:
module mod1;
import mod2;
static this() { mod2.run(); }

mod2.m:
module mod2;
import mod3;
void run() { mod3.run(); }

mod3.m:
module mod3;
void* never_null;
static this() { never_null = new int; }
void run(){ printf("is never_null null? %p eek!\n",never_null); }

I would think this problem makes module initialization very easy to get
around. How about a default (trivial) module constructor gets generated for
a module if any of its imported modules have a constructor.

Walter wrote:

 D's only requirement for ordering module constructors between modules is
 that the imported module constructors must happen first. They are not
 ordered by the order of the import statements - or at least such order is
 an artifact of the implementation.
 
 You should therefore be able to control the order of module construction
 by making sure that each module depending on another module constructor
 imports that module.
 
 "pragma" <pragma_member pathlink.com> wrote in message
 news:ch7s0p$mi8$1 digitaldaemon.com...
 Walter,
 in learning how D handles module constructors, I came across the same bug

 others have stumbled across.  I did a little sleuthing to discover what

 causing this problem. ;)

 My analysis (see below) showed that modules without constructors (static
 this(){}) cannot be prioritized in _moduleCtor(), nor will their imports

 prioritized, since without such a constructor a module isnt't added to
 the
 _moduleinfo_array.  Is this correct?

 If so (and if you don't mind), I'd like to propose that dmd be changed to

 all modules (not just those with ctors) to the _moduleinfo_array.  This

 not only fix these particular init problems, but would also be very
 useful

 reflection purposes.

 Many, many thanks,
 Eric



 [Example]
 The expected init order was: mod4, mod3, mod2, mod1 as that is the order

 import and the order of dependency by function call.  Instead the modules

 run in a different order: mod2, mod1, mod4.  Inserting a module

 mod3 forces the correct init order.

 //// mod1.d ////
 import std.stdio;
 import std.moduleinit;
 import mod2;

 static this(){
 writefln("mod1");
 }

 void main(){
 writefln("\n--MODULE INFO--\n");

 foreach(ModuleInfo info; _moduleinfo_array){
 writefln("%s", info.name);
 foreach(ModuleInfo imported; info.importedModules){
 writefln("\timport %s",imported.name);
 }
 }
 }


 //// mod2.d ////
 import std.stdio;
 import mod3;

 static this(){
 writefln("mod2");
 mod3.foo();
 }


 //// mod3.d ////
 import std.stdio;
 import mod4;

 /* Uncomment to force correct ordering
 static this(){
 writefln("mod3");
 }
 */

 static void foo(){
 writefln("\tmod3.foo()");
 mod4.bar();
 }


 //// mod4.d ////
 import std.stdio;

 static this(){
 writefln("mod4");
 }

 static void bar(){
 writefln("\tmod4.bar()");
 }


 //// Output ////
 mod2
 mod3.foo()
 mod4.bar()
 mod1
 mod4

 --MODULE INFO--

 mod1
 import mod2
 mod2
 mod4
 format
 utf
 string
 adi
 import string
 gcbits
 import string
 thread

 - Pragma
 [[ EricAnderton at (static this{}) yahoo dot com ]]


Sep 02 2004
prev sibling parent reply pragma <pragma_member pathlink.com> writes:
In article <ch8c1i$tdd$1 digitaldaemon.com>, Walter says...
D's only requirement for ordering module constructors between modules is
that the imported module constructors must happen first. They are not
ordered by the order of the import statements - or at least such order is an
artifact of the implementation.

You should therefore be able to control the order of module construction by
making sure that each module depending on another module constructor imports
that module.

Gotcha. Thank you for confirming this, as that's pretty much where all my hacking lead me. (Actually it was pretty much in plain D for everyone in moduleinit.d, but that's another story) :) However, the order of operations really isn't so crucial since it visits all the dependencies recursively, and flags everything on the way out to keep from running things more than once. (its a good compact design!) We also now know what conditions dmd pays attention to (at least in 0.101) when adding nodes to this constructor list. As to the problem at hand: I feel that having dmd add all modules to the list (rather than just those with ctors) will be a huge improvement to the language. Thanks Walter! - Pragma [[ Eric Anderton at (_moduleCtor) yahoo dot com ]]
Sep 02 2004
parent reply "Walter" <newshound digitalmars.com> writes:
"pragma" <pragma_member pathlink.com> wrote in message
news:ch8fa0$ujm$1 digitaldaemon.com...
 As to the problem at hand: I feel that having dmd add all modules to the

 (rather than just those with ctors) will be a huge improvement to the

The problem with that is that then two modules could not import each other.
Sep 06 2004
parent reply pragma <pragma_member pathlink.com> writes:
In article <chjl73$259k$1 digitaldaemon.com>, Walter says...
"pragma" <pragma_member pathlink.com> wrote in message
news:ch8fa0$ujm$1 digitaldaemon.com...
 As to the problem at hand: I feel that having dmd add all modules to the

 (rather than just those with ctors) will be a huge improvement to the

The problem with that is that then two modules could not import each other.

Would this be only if both modules have static ctors, just one, or none at all? I would think that the latter two cases would never be a problem given the way moduleinit.d is already coded: it's already assuming that there are modules in the list w/o ctors as it is checking for that condition (lines 111 and 121). Just put modules on the list with null ctors/dtors if none are declared and it should work fine (IMO). And you already know the first case is already being handled quite nicely. :) Or have I missed the point completely and there is some other case that's causing a problem? - Pragma [[Eric Anderton at yahoo dot com]]
Sep 08 2004
parent reply "Walter" <newshound digitalmars.com> writes:
"pragma" <pragma_member pathlink.com> wrote in message
news:chn51k$r3a$1 digitaldaemon.com...
 In article <chjl73$259k$1 digitaldaemon.com>, Walter says...
"pragma" <pragma_member pathlink.com> wrote in message
news:ch8fa0$ujm$1 digitaldaemon.com...
 As to the problem at hand: I feel that having dmd add all modules to



list
 (rather than just those with ctors) will be a huge improvement to the

The problem with that is that then two modules could not import each


 Would this be only if both modules have static ctors, just one, or none at

 I would think that the latter two cases would never be a problem given the

 moduleinit.d is already coded: it's already assuming that there are

 the list w/o ctors as it is checking for that condition (lines 111 and

 Just put modules on the list with null ctors/dtors if none are declared

 should work fine (IMO).

 And you already know the first case is already being handled quite nicely.

 Or have I missed the point completely and there is some other case that's
 causing a problem?

The problem is if A imports B, and B imports A, whose initialization code gets run first? The imports imply a dependency, so we have "A requires that B's initializers run first" coupled with "B requires that A's initializers run first." There's no resolution. The only one I can think of is that a module with no initializers doesn't have a requirement that the imports get initialized first. And I think that is always correct - that if there *is* such a dependency in the code, then there's something else wrong.
Sep 09 2004
parent reply pragma <pragma_member pathlink.com> writes:
In article <chr7l5$v3p$1 digitaldaemon.com>, Walter says...
 Or have I missed the point completely and there is some other case that's
 causing a problem?

The problem is if A imports B, and B imports A, whose initialization code gets run first? The imports imply a dependency, so we have "A requires that B's initializers run first" coupled with "B requires that A's initializers run first." There's no resolution.

I see your point: the problem /is/ deeper than just running static ctors in order. It also gets worse when you throw additional modules into such an "import cycle".
The only one I can think of is that a
module with no initializers doesn't have a requirement that the imports get
initialized first. And I think that is always correct - that if there *is*
such a dependency in the code, then there's something else wrong.

You're talking about the relaxed init rules already in phobos, correct? _moduleCtor2() treats modules with and without ctors differently, in the manner you prescribe. I think there may be another way. Perhaps mixing the two cases would be best: if the imported module doesn't have a ctor then don't worry if its flagged as 'MIctorstart', but throw an error if it has a ctor and is flagged as such. That way you'd be able to detect init cycles regarless of the intermediate imported module types (modules with or w/o ctors). In addition, modules w/o ctors can cycle all they want. This is (slightly) different from the current behavior which is dependenent upon the current module type. Instead, it relies on the /imported/ module type. If I'm correct this would be the new behavior from this, and previously mentioned, changes: - 'import' and 'private import' now behave the same (new) - all modules are included in the init sequence (new) - imported modules w/o ctors can participate in cyclic imports. (somewhat new) - modules w/static ctors cannot participate in cyclic imports. (same) - (implied) cycles of mixed module types are not allowed (new) - Pragma [[ eric anderton at yahoo dot com ]]
Sep 10 2004
parent reply "Walter" <newshound digitalmars.com> writes:
"pragma" <pragma_member pathlink.com> wrote in message
news:chsfc1$1j7d$1 digitaldaemon.com...
 If I'm correct this would be the new behavior from this, and previously
 mentioned, changes:
 - 'import' and 'private import' now behave the same (new)
 - all modules are included in the init sequence  (new)
 - imported modules w/o ctors can participate in cyclic imports. (somewhat

 - modules w/static ctors cannot participate in cyclic imports. (same)
 - (implied) cycles of mixed module types are not allowed (new)

Ok, but *why* should a module that has no static ctors have a dependency on other module static ctors?
Sep 10 2004
parent pragma <pragma_member pathlink.com> writes:
In article <chtp3c$26mj$1 digitaldaemon.com>, Walter says...
Ok, but *why* should a module that has no static ctors have a dependency on
other module static ctors?

Easy. A module w/o any static ctors may have functions that are callable from *other* module ctors. It may have no initalization requirements (and rightly so since it doesn't have a static ctor), but it may depend on the behavior of additional modules that in turn rely on their own static ctors to run first. Here's a more concrete example. :) // api.d (just an interface, so there's no module ctor here) private import logger; private import ilogger; public static ILogger getDefaultLogger(){ return logger.defaultLogger; } // ilogger.d interface ILogger{ /*methods omitted*/} // logger.d private import ILogger; class Logger: ILogger{ /*methods omitted*/ } private Logger defaultLogger; static this(){ defaultLogger = new Logger(); // client.d (not in the library at all) import api.d import ilogger.d /** this is where it would break: getDefaultLogger() might return null **/ static this(){ api.getDefaultLogger().log("started"); } The above example yields unpredictable results because one cannot guarantee that logger.d will be initalized before client.d. (In fact, one cannot gaurantee that it *won't* initalize before client.d either) If api.d, a module w/o static init code, were included in the init dependency chain, then this would run predictably and as expected. - Pragma [[ Eric Anderton at yahoo dot com ]]
Sep 11 2004