www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - static init cycle detection problem

reply =?UTF-8?B?IsOYaXZpbmQi?= <oivind.loe gmail.com> writes:
I am struggeling to get around the cycle detection kicking in 
when I have static init in modules that depend on eachother.

I have seen some threads on 'fixes' for this, e.g. adding a 
 standalone property to the module or similar. Has there been any 
progress on this?

If not would it be possible in e.g. main() to get a list of all 
compiled-in modules, and then iterate over them and call an init 
function where it exists? As long as there is a way to list the 
name of the modules at compile-time, this should be pretty easy..?

-Øivind
Sep 19 2012
next sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On Wed, 19 Sep 2012 22:25:46 +0200, =C3=98ivind <oivind.loe gmail.com> w=
rote:

 I am struggeling to get around the cycle detection kicking in when I  =

 have static init in modules that depend on eachother.

 I have seen some threads on 'fixes' for this, e.g. adding a  standalon=

 property to the module or similar. Has there been any progress on this=

 If not would it be possible in e.g. main() to get a list of all  =

 compiled-in modules, and then iterate over them and call an init  =

 function where it exists? As long as there is a way to list the name o=

 the modules at compile-time, this should be pretty easy..?

There's no way to get that list at compile-time, because object files ma= y be added at link-time. However, D has a ModuleInfo object, which contain= s information on all modules in the program: import std.stdio; void main( ) { foreach( m; ModuleInfo ) { writeln( m.name ); } } For details on how this object works, have a look-see at src/druntime/src/object_.d in your DMD installation folder. I'm not sure what you're asking for is possible even given this object, but it's probably the closest you'll (easily) get. -- = Simen
Sep 19 2012
prev sibling next sibling parent =?UTF-8?B?IsOYaXZpbmQi?= <oivind.loe gmail.com> writes:
On Wednesday, 19 September 2012 at 20:56:17 UTC, Simen Kjaeraas 
wrote:
 On Wed, 19 Sep 2012 22:25:46 +0200, Øivind 
 <oivind.loe gmail.com> wrote:

 I am struggeling to get around the cycle detection kicking in 
 when I have static init in modules that depend on eachother.

 I have seen some threads on 'fixes' for this, e.g. adding a 
  standalone property to the module or similar. Has there been 
 any progress on this?

 If not would it be possible in e.g. main() to get a list of 
 all compiled-in modules, and then iterate over them and call 
 an init function where it exists? As long as there is a way to 
 list the name of the modules at compile-time, this should be 
 pretty easy..?

There's no way to get that list at compile-time, because object files may be added at link-time. However, D has a ModuleInfo object, which contains information on all modules in the program: import std.stdio; void main( ) { foreach( m; ModuleInfo ) { writeln( m.name ); } } For details on how this object works, have a look-see at src/druntime/src/object_.d in your DMD installation folder. I'm not sure what you're asking for is possible even given this object, but it's probably the closest you'll (easily) get.

If I define a function: void getMembers() { } in one of my modules, all modules including this get the xgetMembers property of the ModuleInfo struct set. It seems like I should be able to do something with this, but i then get the following assertion failure: ./dboss(_d_assertm+0x26) [0x8aa1da] ./dboss() [0x7c8555] ./dboss(boss.core.proc.ProcBase!(boss.core.boss.Boss, "boss").ProcBase boss.core.proc.ProcBase!(boss.core.boss.Boss, "boss").ProcBase.__ctor()+0x31) [0x797805] ./dboss(boss.core.boss.Boss boss.core.boss.Boss.__ctor(immutable(char)[][])+0x21) [0x791099] ./dboss(_Dmain+0x33) [0x7488fb] ./dboss(extern (C) int rt.dmain2.main(int, char**).void runMain()+0x1c) [0x8aaa70] ./dboss(extern (C) int rt.dmain2.main(int, char**).void tryExec(scope void delegate())+0x2a) [0x8aa3ea] ./dboss(extern (C) int rt.dmain2.main(int, char**).void runAll()+0x3b) [0x8aaab7] ./dboss(extern (C) int rt.dmain2.main(int, char**).void tryExec(scope void delegate())+0x2a) [0x8aa3ea] ./dboss(main+0xd1) [0x8aa375] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f1d0edcc76d] Removing the function from the module makes the assertion failure go away.. What is the cause of this? And what is the getMembers function used for? Another way of approaching this would be if I could feed a list of modules into DMD during compile time. In C++, i would be able to do this by passing a define to g++ on the command line when invoking it. Is it possible to do something similar with DMD? E.g. create a list of modules before invoking DMD, pass it to DMD, and use this at compile time to import and call my init function?
Sep 19 2012
prev sibling next sibling parent =?UTF-8?B?IsOYaXZpbmQi?= <oivind.loe gmail.com> writes:
 Another way of approaching this would be if I could feed a list 
 of modules into DMD during compile time. In C++, i would be 
 able to do this by passing a define to g++ on the command line 
 when invoking it. Is it possible to do something similar with 
 DMD? E.g. create a list of modules before invoking DMD, pass it 
 to DMD, and use this at compile time to import and call my init 
 function?

I can of course just generate a separate init.d file containing all of this info and then use that for static init of my modules. Think that should work :)
Sep 19 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, September 19, 2012 22:25:46 Øivind wrote:
 I am struggeling to get around the cycle detection kicking in
 when I have static init in modules that depend on eachother.
 
 I have seen some threads on 'fixes' for this, e.g. adding a
  standalone property to the module or similar. Has there been any
 progress on this?

The solution is to do something like what std.stdio does. Instead of having a static constructor, it has an extern(C) function which does the same, and then it has std/stdiobase.d which has a static constructor which calls that extern(C) function in std.stdio. As long as you don't actually have any inter- dependencies, you can make it work. Walter has refused suggestions which involve telling the compiler that there isn't actually dependency (via an attribute or whatnot) on the grounds that you're throwing away the compiler's checks, and it's too easy to screw up. In general, you can refactor things so that you don't have the circular dependency anymore, and if you can't, and the static constructors aren't actually forming a circular dependency, then you can pull the same trick that std.stdio does.
 If not would it be possible in e.g. main() to get a list of all
 compiled-in modules, and then iterate over them and call an init
 function where it exists? As long as there is a way to list the
 name of the modules at compile-time, this should be pretty easy..?

It will never be possible to determine the full list of modules at compile time due to C's linking model. For instance, you could link together modules which were built years apart. There's no way that they could know about each other. And since you could be using .di files, and the circular dependencies could actually be really indirect, the compiler can't possibly have all of the information that it needs to determine circular dependencies. A linker could do it but not as long as we use standard, C linkers. And that's not going to change. So, the situation sucks, but there _are_ workarounds. The main one of course is to simply limit how much your modules import each other so that the few places that you need static constructors don't import each other even indirectly, but regardless, it can be worked around as long there isn't a true circular dependency. - Jonathan M Davis
Sep 19 2012
prev sibling next sibling parent =?UTF-8?B?IsOYaXZpbmQi?= <oivind.loe gmail.com> writes:
On Thursday, 20 September 2012 at 00:23:33 UTC, Jonathan M Davis 
wrote:
 On Wednesday, September 19, 2012 22:25:46 Øivind wrote:
 I am struggeling to get around the cycle detection kicking in
 when I have static init in modules that depend on eachother.
 
 I have seen some threads on 'fixes' for this, e.g. adding a
  standalone property to the module or similar. Has there been 
 any
 progress on this?

The solution is to do something like what std.stdio does. Instead of having a static constructor, it has an extern(C) function which does the same, and then it has std/stdiobase.d which has a static constructor which calls that extern(C) function in std.stdio. As long as you don't actually have any inter- dependencies, you can make it work. Walter has refused suggestions which involve telling the compiler that there isn't actually dependency (via an attribute or whatnot) on the grounds that you're throwing away the compiler's checks, and it's too easy to screw up. In general, you can refactor things so that you don't have the circular dependency anymore, and if you can't, and the static constructors aren't actually forming a circular dependency, then you can pull the same trick that std.stdio does.
 If not would it be possible in e.g. main() to get a list of all
 compiled-in modules, and then iterate over them and call an 
 init
 function where it exists? As long as there is a way to list the
 name of the modules at compile-time, this should be pretty 
 easy..?

It will never be possible to determine the full list of modules at compile time due to C's linking model. For instance, you could link together modules which were built years apart. There's no way that they could know about each other. And since you could be using .di files, and the circular dependencies could actually be really indirect, the compiler can't possibly have all of the information that it needs to determine circular dependencies. A linker could do it but not as long as we use standard, C linkers. And that's not going to change. So, the situation sucks, but there _are_ workarounds. The main one of course is to simply limit how much your modules import each other so that the few places that you need static constructors don't import each other even indirectly, but regardless, it can be worked around as long there isn't a true circular dependency. - Jonathan M Davis

Thanks for the explination. The trick you talk about has worked for me before, but now I really need static init of modules that depend on eachother. Only solution I can think of is to list all modules in a script and generate d code from this to call init functions.. Then pass it to dmd along with the other code. Sad that we can't get this to work like it really should. D is a very flexible language in many ways, and it seems strange that it should not allow the stuff I want to do here.. -Øivind
Sep 20 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, September 20, 2012 20:49:32 Øivind wrote:
 Thanks for the explination. The trick you talk about has worked
 for me before, but now I really need static init of modules that
 depend on eachother. Only solution I can think of is to list all
 modules in a script and generate d code from this to call init
 functions.. Then pass it to dmd along with the other code.
 
 Sad that we can't get this to work like it really should. D is a
 very flexible language in many ways, and it seems strange that it
 should not allow the stuff I want to do here..

If you have a true circular dependency, then you're just plain screwed and need to redesign your code. That's fundamental and _can't_ be fixed. Off the top of my head, the only reason that you wouldn't be able to use the trick that std.stdio uses but not have a true circular dependency is if your initializing const or immutable variables in a static constructor. But then the solution is to refactor your code so that you don't have the circular dependency, even if it's just moving those variables into a separate module, which is generally quite feasible, even if it's not necessarily quite what you wanted. - Jonathan M Davis
Sep 20 2012
prev sibling next sibling parent Brad Roberts <braddr puremagic.com> writes:
On Thu, 20 Sep 2012, Jonathan M Davis wrote:

 On Thursday, September 20, 2012 20:49:32 ?ivind wrote:
 Thanks for the explination. The trick you talk about has worked
 for me before, but now I really need static init of modules that
 depend on eachother. Only solution I can think of is to list all
 modules in a script and generate d code from this to call init
 functions.. Then pass it to dmd along with the other code.
 
 Sad that we can't get this to work like it really should. D is a
 very flexible language in many ways, and it seems strange that it
 should not allow the stuff I want to do here..

If you have a true circular dependency, then you're just plain screwed and need to redesign your code. That's fundamental and _can't_ be fixed. Off the top of my head, the only reason that you wouldn't be able to use the trick that std.stdio uses but not have a true circular dependency is if your initializing const or immutable variables in a static constructor. But then the solution is to refactor your code so that you don't have the circular dependency, even if it's just moving those variables into a separate module, which is generally quite feasible, even if it's not necessarily quite what you wanted. - Jonathan M Davis

I'm not picking on Jonathan's response, just needed an entry point to the thread... Static initializers are overused in D, particularly phobos and druntime, imho. They're a particular problem when it comes to use in library code. Mixing static and dynamic linkage along with initialization == things break in wierd ways. I'd _much_ rather see them moved into explicit api calls or removed entirely where possible. Every static initializtion is unavoidable (by users of the library) overhead. My 2 cents, Brad
Sep 20 2012
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 19 Sep 2012 16:25:46 -0400, =C3=98ivind <oivind.loe gmail.com> w=
rote:

 I am struggeling to get around the cycle detection kicking in when I  =

 have static init in modules that depend on eachother.

 I have seen some threads on 'fixes' for this, e.g. adding a  standalon=

 property to the module or similar. Has there been any progress on this=

No. If two initializations truly don't depend on each other, you can = initialize in their own module. The issue is that you have to move *all* the initializations into their = = own module. The compiler doesn't keep track of complex dependencies, it= 's = all on a module-level. I redesigned the static cycle detection code a while back (it was broken= = (allowed certain cycles) and much less informative), and I can say it wa= s = a very difficult problem to solve.
 If not would it be possible in e.g. main() to get a list of all  =

 compiled-in modules, and then iterate over them and call an init  =

 function where it exists? As long as there is a way to list the name o=

 the modules at compile-time, this should be pretty easy..?

No, you have no idea at compile-time what other modules will be linked i= n. You need to do the cycle detection at link-time or run-time, and as long= = as we use standard C linkers, we cannot add that feature. -Steve
Sep 24 2012