www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Our template emission strategy is broken

reply David Nadlinger <code klickverbot.at> writes:
Hi all,

Kenji and Walter have been working on improving the template 
emission strategy during the last couple of releases, i.e. 
whether a template instance is emitted to a given object file or 
not. Nevertheless, I've been continually hearing complaints from 
various people with large D code bases (our commercial users and 
some of the more complex open source projects) that they have 
problems compiling their code doing anything else than an 
all-at-once build.

This of course detracts from what we like to cite as one of D's 
key advantages, compilation speed, because you cannot exploit the 
many cores of your build machine anymore, and the 
edit-compile-debug cycle is slowed down because you have to build 
the full application every time. But this turns into a severe 
problem as soon as your compilation does not fit into RAM 
anymore, which happens quite quickly when using D's advanced 
features – see for example Liran Zvibel's DConf talk, where he 
describes that they needed machines with more than 100 GB of RAM 
in order to build the Weka code base.

In any case, I hope you agree that fixing these kinds of issues 
that prevent D code from being compiled all is strategically 
important for us, since they are a deciding factor in driving 
widespread adoption. Sadly, the template problems didn't seem to 
receive a lot of attention beyond quick fixes recently, possibly 
because they don't occur as readily for smaller projects and if 
so, are easier to work around.

With all that out of my system, I'd like to point your attention 
to a particular template instantiation issue I managed to reduce 
recently when working with the folks at Weka. It seems like there 
is a fundamental oversight in the way the code tries to elide 
repeat code generation of instances – or, of course, I'm just 
missing something:

https://issues.dlang.org/show_bug.cgi?id=15318

This is one particular issue that makes writing "idiomatic D" 
(which is rather template-heavy) in large code bases hard, but 
likely not the only one. In fact, I already know about another 
issue where function-local imports cause semantic analysis not to 
be properly run on template instances, but I'm still struggling 
to find a minimal example for the problem. My hope would be that 
we can clean up this mess soon and replace the current ad-hoc 
patchwork with a more principled approach. In the process, we 
should also make sure that all of our assumptions [1] about the 
compilation process are clearly stated in the documentation, as 
that frequently leads to confusion among newcomers.

Best,
David



[1] That is: All imported modules must also be compiled into the 
executable; incremental compilation is only guaranteed to work if 
precisely the same subsets of modules are compiled every time.
Nov 11 2015
next sibling parent Paolo Invernizzi <paolo.invernizzi no.address> writes:
On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:
 Hi all,

 Nevertheless, I've been continually hearing complaints from 
 various people with large D code bases (our commercial users 
 and some of the more complex open source projects) that they 
 have problems compiling their code doing anything else than an 
 all-at-once build.
The same here, we are not able to compile two of our products without an all-at-once build. Actually we are plagued also by that: https://issues.dlang.org/show_bug.cgi?id=14517 That's a real pain... --- Paolo
Nov 11 2015
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 11 Nov 2015 13:08:06 +0000
schrieb David Nadlinger <code klickverbot.at>:

 Hi all,
=20
 Kenji and Walter have been working on improving the template=20
 emission strategy during the last couple of releases, i.e.=20
 whether a template instance is emitted to a given object file or=20
 not. Nevertheless, I've been continually hearing complaints from=20
 various people with large D code bases (our commercial users and=20
 some of the more complex open source projects) that they have=20
 problems compiling their code doing anything else than an=20
 all-at-once build.
=20
 This of course detracts from what we like to cite as one of D's=20
 key advantages, compilation speed, because you cannot exploit the=20
 many cores of your build machine anymore, and the=20
 edit-compile-debug cycle is slowed down because you have to build=20
 the full application every time. But this turns into a severe=20
 problem as soon as your compilation does not fit into RAM=20
 anymore, which happens quite quickly when using D's advanced=20
 features =E2=80=93=C2=A0see for example Liran Zvibel's DConf talk, where =
he=20
 describes that they needed machines with more than 100 GB of RAM=20
 in order to build the Weka code base.
=20
 In any case, I hope you agree that fixing these kinds of issues=20
 that prevent D code from being compiled all is strategically=20
 important for us, since they are a deciding factor in driving=20
 widespread adoption. Sadly, the template problems didn't seem to=20
 receive a lot of attention beyond quick fixes recently, possibly=20
 because they don't occur as readily for smaller projects and if=20
 so, are easier to work around.
=20
 With all that out of my system, I'd like to point your attention=20
 to a particular template instantiation issue I managed to reduce=20
 recently when working with the folks at Weka. It seems like there=20
 is a fundamental oversight in the way the code tries to elide=20
 repeat code generation of instances =E2=80=93 or, of course, I'm just=20
 missing something:
=20
 https://issues.dlang.org/show_bug.cgi?id=3D15318
=20
 This is one particular issue that makes writing "idiomatic D"=20
 (which is rather template-heavy) in large code bases hard, but=20
 likely not the only one. In fact, I already know about another=20
 issue where function-local imports cause semantic analysis not to=20
 be properly run on template instances, but I'm still struggling=20
 to find a minimal example for the problem. My hope would be that=20
 we can clean up this mess soon and replace the current ad-hoc=20
 patchwork with a more principled approach. In the process, we=20
 should also make sure that all of our assumptions [1] about the=20
 compilation process are clearly stated in the documentation, as=20
 that frequently leads to confusion among newcomers.
=20
 Best,
 David
=20
=20
=20
 [1] That is: All imported modules must also be compiled into the=20
 executable; incremental compilation is only guaranteed to work if=20
 precisely the same subsets of modules are compiled every time.
http://forum.dlang.org/thread/n1omke$1bh5$1 digitalmars.com
Nov 11 2015
parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 11 Nov 2015 14:44:39 +0100
schrieb Johannes Pfau <nospam example.com>:

 Am Wed, 11 Nov 2015 13:08:06 +0000
 schrieb David Nadlinger <code klickverbot.at>:
=20
 Hi all,
=20
 Kenji and Walter have been working on improving the template=20
 emission strategy during the last couple of releases, i.e.=20
 whether a template instance is emitted to a given object file or=20
 not. Nevertheless, I've been continually hearing complaints from=20
 various people with large D code bases (our commercial users and=20
 some of the more complex open source projects) that they have=20
 problems compiling their code doing anything else than an=20
 all-at-once build.
=20
 This of course detracts from what we like to cite as one of D's=20
 key advantages, compilation speed, because you cannot exploit the=20
 many cores of your build machine anymore, and the=20
 edit-compile-debug cycle is slowed down because you have to build=20
 the full application every time. But this turns into a severe=20
 problem as soon as your compilation does not fit into RAM=20
 anymore, which happens quite quickly when using D's advanced=20
 features =E2=80=93=C2=A0see for example Liran Zvibel's DConf talk, wher=
e he=20
 describes that they needed machines with more than 100 GB of RAM=20
 in order to build the Weka code base.
A quick workaround could be enabling the GC for DDMD. IIRC I read somewhere on github that the segfaulting code was actually rewritten now and the GC might just work.
=20
 In any case, I hope you agree that fixing these kinds of issues=20
 that prevent D code from being compiled all is strategically=20
 important for us, since they are a deciding factor in driving=20
 widespread adoption. Sadly, the template problems didn't seem to=20
 receive a lot of attention beyond quick fixes recently, possibly=20
 because they don't occur as readily for smaller projects and if=20
 so, are easier to work around.
=20
 With all that out of my system, I'd like to point your attention=20
 to a particular template instantiation issue I managed to reduce=20
 recently when working with the folks at Weka. It seems like there=20
 is a fundamental oversight in the way the code tries to elide=20
 repeat code generation of instances =E2=80=93 or, of course, I'm just=20
 missing something:
=20
 https://issues.dlang.org/show_bug.cgi?id=3D15318
=20
 This is one particular issue that makes writing "idiomatic D"=20
 (which is rather template-heavy) in large code bases hard, but=20
 likely not the only one. In fact, I already know about another=20
 issue where function-local imports cause semantic analysis not to=20
 be properly run on template instances, but I'm still struggling=20
 to find a minimal example for the problem. My hope would be that=20
 we can clean up this mess soon and replace the current ad-hoc=20
 patchwork with a more principled approach. In the process, we=20
 should also make sure that all of our assumptions [1] about the=20
 compilation process are clearly stated in the documentation, as=20
 that frequently leads to confusion among newcomers.
=20
 Best,
 David
=20
=20
=20
 [1] That is: All imported modules must also be compiled into the=20
 executable; incremental compilation is only guaranteed to work if=20
 precisely the same subsets of modules are compiled every time. =20
=20 =20 http://forum.dlang.org/thread/n1omke$1bh5$1 digitalmars.com =20
Sorry, sent the wrong message ;-) Ellery Newcomer recently reported a template emission bug where templates are emitted twice: http://forum.dlang.org/thread/n1omke$1bh5$1 digitalmars.com=20 I think we should really fix these issues, working separate compilation is very important.
Nov 11 2015
next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
wrote:
 I think we should really fix these issues, working separate 
 compilation is very important.
No, it's not. Separate compilation is C++-ishly slow by design (b/c all dependencies get imported over and over again, just like headers). We should endorse compiling small (rather independent) packages to static/shared libraries.
Nov 11 2015
next sibling parent reply Dicebot <public dicebot.lv> writes:
On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
wrote:
 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
 wrote:
 I think we should really fix these issues, working separate 
 compilation is very important.
No, it's not. Separate compilation is C++-ishly slow by design (b/c all dependencies get imported over and over again, just like headers). We should endorse compiling small (rather independent) packages to static/shared libraries.
Note that many of template emission issues that affect "plain" separate compilation are likely to affect package-based static libraties as well, thus in context of this thread distinction is not very important.
Nov 11 2015
parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 14:06:43 UTC, Dicebot wrote:
 Note that many of template emission issues that affect "plain" 
 separate compilation are likely to affect package-based static 
 libraties as well, thus in context of this thread distinction 
 is not very important.
Yes, you can get similar issues with package libraries, but only if the packages are mutually dependent on each other. Think of that for a moment, no package manager allows you to have cycles in your dependencies. The case where you do compile packages with circular dependencies seems to mostly come from artificially splitting packages to deal with OOM, and we better address that at it's root. In general you save many troubles with a library b/c you always link with all objects that were compiled (so it's only important that a template was emitted but not into which object).
Nov 11 2015
next sibling parent reply Dicebot <public dicebot.lv> writes:
On Wednesday, 11 November 2015 at 14:23:16 UTC, Martin Nowak 
wrote:
 On Wednesday, 11 November 2015 at 14:06:43 UTC, Dicebot wrote:
 Note that many of template emission issues that affect "plain" 
 separate compilation are likely to affect package-based static 
 libraties as well, thus in context of this thread distinction 
 is not very important.
Yes, you can get similar issues with package libraries, but only if the packages are mutually dependent on each other. Think of that for a moment, no package manager allows you to have cycles in your dependencies. The case where you do compile packages with circular dependencies seems to mostly come from artificially splitting packages to deal with OOM, and we better address that at it's root. In general you save many troubles with a library b/c you always link with all objects that were compiled (so it's only important that a template was emitted but not into which object).
I was referring to D packages, not dub packages.
Nov 11 2015
parent Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 14:26:28 UTC, Dicebot wrote:
 I was referring to D packages, not dub packages.
Not much of a difference, compiling something separately that has mutual dependencies is flawed but can be made working to some extend.
Nov 11 2015
prev sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 11 Nov 2015 14:23:15 +0000
schrieb Martin Nowak <code dawg.eu>:

 Think of that for a moment, no package manager allows you to
 have cycles in your dependencies.
There are package managers that allow packages to mutually depend on each other. Order is established by qualifying the dependency similar to how you declare an owning and a weak reference in a parent/child pointer relation. E.g. You must install DMD before druntime/Phobos, but DMD is only usable after druntime/Phobos has also been installed. -- Marco
Nov 16 2015
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, 16 November 2015 at 09:05:16 UTC, Marco Leise wrote:
 Am Wed, 11 Nov 2015 14:23:15 +0000
 schrieb Martin Nowak <code dawg.eu>:

 Think of that for a moment, no package manager allows you to 
 have cycles in your dependencies.
There are package managers that allow packages to mutually depend on each other. Order is established by qualifying the dependency similar to how you declare an owning and a weak reference in a parent/child pointer relation.
LOL. Yeah. I expect that 99+% of Linux boxes contain mutually dependent packages. Circular dependencies make setting up Gentoo that much more entertaining (e.g. IIRC, OpenSSL and Cyrus SASL depend on each other normally). At least some dependencies can be eliminated with binary packages, because they're only build dependencies, but not all of them are that way, unfortunately. - Jonathan M Davis
Nov 16 2015
prev sibling next sibling parent reply Daniel Kozak via Digitalmars-d <digitalmars-d puremagic.com> writes:
V Wed, 11 Nov 2015 13:56:50 +0000
Martin Nowak via Digitalmars-d <digitalmars-d puremagic.com> napsáno:

 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
 wrote:
 I think we should really fix these issues, working separate 
 compilation is very important.  
No, it's not. Separate compilation is C++-ishly slow by design (b/c all dependencies get imported over and over again, just like headers). We should endorse compiling small (rather independent) packages to static/shared libraries.
This isn't a solution it's a workaround. It seems to me like "It's not a bug it's a feature"
Nov 11 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/15 9:14 AM, Daniel Kozak via Digitalmars-d wrote:
 V Wed, 11 Nov 2015 13:56:50 +0000
 Martin Nowak via Digitalmars-d <digitalmars-d puremagic.com> napsáno:

 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau
 wrote:
 I think we should really fix these issues, working separate
 compilation is very important.
No, it's not. Separate compilation is C++-ishly slow by design (b/c all dependencies get imported over and over again, just like headers). We should endorse compiling small (rather independent) packages to static/shared libraries.
This isn't a solution it's a workaround. It seems to me like "It's not a bug it's a feature"
Rust, Haskell, and Go have similar approaches if that helps. -- Andrei
Nov 11 2015
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 11 Nov 2015 13:56:50 +0000
schrieb Martin Nowak <code dawg.eu>:

 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
 wrote:
 I think we should really fix these issues, working separate 
 compilation is very important.  
No, it's not. Separate compilation is C++-ishly slow by design (b/c all dependencies get imported over and over again, just like headers). We should endorse compiling small (rather independent) packages to static/shared libraries.
And where do you draw the line? Do we expect ARM users to have 4GB of RAM to compile phobos? Is 512MB enough? 64 MB? Even huge C++ codebases such as GCC compile with 128MB memory. I use docker hub to build docker images. One Dockerfile compiles one small D tool which depends on vibe.d for JSON parsing. Without separate compilation the build fails as docker hub kills the build process, out of memory...
Nov 11 2015
next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 14:15:54 UTC, Johannes Pfau 
wrote:
 And where do you draw the line? Do we expect ARM users to have 
 4GB of RAM to compile phobos? Is 512MB enough? 64 MB? Even huge 
 C++ codebases such as GCC compile with 128MB memory.
OK let's clarify the terminology first. separate compilation: -c single_source.d incremental compilation: -c source_a.d source_b.d (as well as -c -ofsingle.o source_a.d source_b.d) library compilation: -lib all_pkg_sources Separate compilation works fine (and predictable) but is slow (due to reparsing), you do need to link all objects from all your sources though. Choosing this strategy makes sense to mitigate OOM issues. Incremental compilation doesn't work b/c templates are easily emitted into different objects, causing all sorts of linker errors. It works somewhat when you delete all objects and recompile all of them. Library compilation works and recompiles a whole library whenever something changes. It's fast b/c modules/semantic can be reused by the compiler. Choose this strategy unless your running into OOM issues.
Nov 11 2015
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 11 Nov 2015 14:31:53 +0000
schrieb Martin Nowak <code dawg.eu>:

 On Wednesday, 11 November 2015 at 14:15:54 UTC, Johannes Pfau 
 wrote:
 And where do you draw the line? Do we expect ARM users to have 
 4GB of RAM to compile phobos? Is 512MB enough? 64 MB? Even huge 
 C++ codebases such as GCC compile with 128MB memory.  
OK let's clarify the terminology first. separate compilation: -c single_source.d incremental compilation: -c source_a.d source_b.d (as well as -c -ofsingle.o source_a.d source_b.d) library compilation: -lib all_pkg_sources Separate compilation works fine (and predictable) but is slow (due to reparsing), you do need to link all objects from all your sources though. Choosing this strategy makes sense to mitigate OOM issues.
OK, that makes sense. I thought there were some issues with separate compilation as well, but I'm not sure. Anyway, I meant working _separate_ compilation is important, incremental compilations is much less important for me. GDC doesn't even support multiobj style compilation. BTW: shouldn't incremental compilation with one single output object have exactly the same result as compiling into one static library? So I guess we already avoid most of these problems in gdc by simply not offering a multiobj mode ;-)
Nov 11 2015
parent David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 14:50:27 UTC, Johannes Pfau 
wrote:
 Anyway, I meant working _separate_ compilation is important, 
 incremental compilations is much less important for me. GDC 
 doesn't even support multiobj style compilation.
The particular issue I mentioned in the original post (that will hit GDC as well, by the way, if/as soon as you are using TemplateInstance::needsCodegen in your glue layer) has nothing to do with "multiobj" compilation.
 BTW: shouldn't incremental compilation with one single output 
 object have exactly the same result as compiling into one 
 static library?
Yes, and both are broken. — David
Nov 11 2015
prev sibling next sibling parent David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 14:31:54 UTC, Martin Nowak 
wrote:
 OK let's clarify the terminology first.

 separate compilation: -c single_source.d
 incremental compilation: -c source_a.d source_b.d (as well as 
 -c -ofsingle.o source_a.d source_b.d)
 library compilation: -lib all_pkg_sources

 Separate compilation works fine (and predictable) but is slow 
 (due to reparsing), you do need to link all objects from all 
 your sources though.
 Choosing this strategy makes sense to mitigate OOM issues.

 Incremental compilation doesn't work b/c templates are easily 
 emitted into different objects, causing all sorts of linker 
 errors. It works somewhat when you delete all objects and 
 recompile all of them.

 Library compilation works and recompiles a whole library 
 whenever something changes.
 It's fast b/c modules/semantic can be reused by the compiler.
 Choose this strategy unless your running into OOM issues.
That terminology seems a bit misleading for discussing the problem at hand. There is nothing incremental about compiling per-package with a single .o file, and it is in fact rather similar to what you call "library compilation". As noted on Bugzilla, the particular template instantiation bug that was plaguing Weka occurs with "-lib" just the same as it does with "-c -osingleobj.o". — David
Nov 11 2015
prev sibling parent Paolo Invernizzi <paolo.invernizzi no.address> writes:
On Wednesday, 11 November 2015 at 14:31:54 UTC, Martin Nowak 
wrote:
 On Wednesday, 11 November 2015 at 14:15:54 UTC, Johannes Pfau 
 wrote:

 Separate compilation works fine (and predictable) but is slow 
 (due to reparsing), you do need to link all objects from all 
 your sources though.
That's not the experience we have in my company: we are not able to compile and link project right now, using separate compilation. From dmd 2.066 up to 2.069. --- Paolo
Nov 11 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/15 9:15 AM, Johannes Pfau wrote:
 Am Wed, 11 Nov 2015 13:56:50 +0000
 schrieb Martin Nowak <code dawg.eu>:

 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau
 wrote:
 I think we should really fix these issues, working separate
 compilation is very important.
No, it's not. Separate compilation is C++-ishly slow by design (b/c all dependencies get imported over and over again, just like headers). We should endorse compiling small (rather independent) packages to static/shared libraries.
And where do you draw the line? Do we expect ARM users to have 4GB of RAM to compile phobos? Is 512MB enough? 64 MB? Even huge C++ codebases such as GCC compile with 128MB memory.
Large codebases are best organized to have more packages, not larger packages. Phobos takes about 600MB max to build. Andrei
Nov 11 2015
parent reply David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 15:07:15 UTC, Andrei 
Alexandrescu wrote:
 Phobos takes about 600MB max to build.
When was the last time you did "dmd -unittest -main $(find std -name '*.d')"? On current master, dmd takes just below 9 GiB to complete the build. Also, consider that many of the features in Phobos – apart from the underlying molasses consisting of std.traits and friends – are not really widely used internally. This is not surprising, considering that large parts of std.* are mostly a grab-bag of orthogonal stuff, and the unit tests are, as they should, fairly isolated. For example, how many Phobos modules actually use std.regex internally, let alone its compile-time variants (which we are quick to advertise)? What about range "pipelines" longer than the one or two stages you'd typically write in a unit test? Apologies for maybe being a bit too confrontational about this, but I don't think anybody is helped by dismissing entire classes of problems just because they tend not to be so apparent from a library writer's perspective. If you were to, say, visit the folks at Weka, you might be surprised to see how easy it is to do the latter without even realizing so. — David
Nov 11 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/2015 12:13 PM, David Nadlinger wrote:
 On Wednesday, 11 November 2015 at 15:07:15 UTC, Andrei Alexandrescu wrote:
 Phobos takes about 600MB max to build.
When was the last time you did "dmd -unittest -main $(find std -name '*.d')"? On current master, dmd takes just below 9 GiB to complete the build.
"Then don't do that". Seriously, I agree reducing memory consumed is necessary but there are always extreme cases to construct.
 Also, consider that many of the features in Phobos – apart from the
 underlying molasses consisting of std.traits and friends – are not
 really widely used internally. This is not surprising, considering that
 large parts of std.* are mostly a grab-bag of orthogonal stuff, and the
 unit tests are, as they should, fairly isolated. For example, how many
 Phobos modules actually use std.regex internally, let alone its
 compile-time variants (which we are quick to advertise)? What about
 range "pipelines" longer than the one or two stages you'd typically
 write in a unit test?
My perception is Phobos uses its own dog food intensively, which also leads to the criticisms that everything in Phobos depends on everything else. That said, clearly the library + unittests themselves are not a large project using the library.
 Apologies for maybe being a bit too confrontational about this,
There is agreement that a problem exists. No need to get overly carried over this; I recall last time you got really upset, you said something that was so incredibly offensive, Walter and I had to hold a pow-wow and decide to simply ignore it and just give you time to calm down. There's no need or time for all that. We're on the same boat.
 but I
 don't think anybody is helped by dismissing entire classes of problems
 just because they tend not to be so apparent from a library writer's
 perspective. If you were to, say, visit the folks at Weka, you might be
 surprised to see how easy it is to do the latter without even realizing so.
Agreed. Glad you're helping them out! Andrei
Nov 11 2015
parent David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 18:02:01 UTC, Andrei 
Alexandrescu wrote:
 On 11/11/2015 12:13 PM, David Nadlinger wrote:
 When was the last time you did "dmd -unittest -main $(find std 
 -name
 '*.d')"? On current master, dmd takes just below 9 GiB to 
 complete the
 build.
"Then don't do that". Seriously, I agree reducing memory consumed is necessary but there are always extreme cases to construct.
Sure. Let me point out, though, that this very thread is about a bug making that impossible. ;)
 There is agreement that a problem exists.
I'm glad that there is now.
 I recall last time you got really upset, you said something 
 that was so incredibly offensive, Walter and I had to hold a 
 pow-wow and decide to simply ignore it and just give you time 
 to calm down.
I've had to do the same quite a few times with either of you. What specific occasion are you thinking of? Maybe just call me out on it right there, right then next time (even privately, if you prefer so)?
 Agreed. Glad you're helping them out!
Well, part of the reason I wanted to make sure that this topic makes it near the center of our collective attention again is precisely that I won't be able to spend any more time (at least no significant amount) working with them anytime soon. They are doing some quite remarkable things with D, but unfortunately their computers are not quantum enough for me. — David
Nov 11 2015
prev sibling parent =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig rejectedsoftware.com> writes:
Am 11.11.2015 um 15:15 schrieb Johannes Pfau:
 I use docker hub to build docker images. One Dockerfile compiles one
 small D tool which depends on vibe.d for JSON parsing. Without
 separate compilation the build fails as docker hub kills the build
 process, out of memory...
Sorry for OT: the latest 0.7.27-alpha.1 of vibe.d is now separated into sub packages ("vibe-d:data" in particular contains only JSON/BSON and the serialization mechanics).
Nov 11 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/15 8:56 AM, Martin Nowak wrote:
 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau wrote:
 I think we should really fix these issues, working separate
 compilation is very important.
No, it's not. Separate compilation is C++-ishly slow by design (b/c all dependencies get imported over and over again, just like headers). We should endorse compiling small (rather independent) packages to static/shared libraries.
Yes, compiling package-at-a-time should be the mode endorsed by the core community. Liran Zvibel and I discussed this privately following his DConf 2015 talk, and I recall he mentioned the package-at-a-time solved essentially all of their build problems. It would be fantastic if somebody could write a wiki/dlang.org article about this with details on how to do it and measurements. One prime example is Phobos itself, for which conversion to package-at-a-time build (http://forum.dlang.org/thread/mkvpgn$2fg3$2 digitalmars.com) has led to dramatic improvements in memory consumed and build times (https://github.com/D-Programming-Language/phobos/pull/3379). Andrei
Nov 11 2015
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 11 November 2015 at 15:04:01 UTC, Andrei 
Alexandrescu wrote:
 Yes, compiling package-at-a-time should be the mode endorsed by 
 the core community. Liran Zvibel and I discussed this privately 
 following his DConf 2015 talk, and I recall he mentioned the 
 package-at-a-time solved essentially all of their build 
 problems.
They must have encountered more since then since their build has had a lot of difficulty. I tried to hack around the problems they had using the various tricks I've accumulated (btw you can hit several of these problems without necessarily having a huge codebase, though reproducing them in a reduced test is extremely difficult) and failed. They tried throwing more hardware at it and that didn't work either with newer compiler versions. David has been working on fixing the fundamental problems and surely, if he succeeded completely already, he wouldn't have made these posts. We, as a community, need to start believing people when they say they are having troubles instead of blaming the customer for doing it wrong.
Nov 11 2015
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/15 10:12 AM, Adam D. Ruppe wrote:
 We, as a community, need to start believing people when they say they
 are having troubles instead of blaming the customer for doing it wrong.
Total agreement with that! -- Andrei
Nov 11 2015
prev sibling parent Paolo Invernizzi <paolo.invernizzi no.address> writes:
On Wednesday, 11 November 2015 at 15:12:16 UTC, Adam D. Ruppe 
wrote:
 We, as a community, need to start believing people when they 
 say they are having troubles instead of blaming the customer 
 for doing it wrong.
+10_000 --- Paolo
Nov 11 2015
prev sibling next sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 15:04:01 UTC, Andrei 
Alexandrescu wrote:
 Yes, compiling package-at-a-time should be the mode endorsed by 
 the core community. Liran Zvibel and I discussed this privately 
 following his DConf 2015 talk, and I recall he mentioned the 
 package-at-a-time solved essentially all of their build 
 problems.
If you have a look at the bug report I linked in the original post, you'll find that this case occurs precisely with per-package compilation. Yes, there is a single cyclic inter-package dependency hidden somewhere deep inside the dependency graph, but getting rid of it after the fact would have either required re-structuring large parts of a ˜10^5 LOC code base, or would have again involved creating logical packages so large that the memory inefficiencies (CTFE is the biggest problem here) and run-time scaling issues (e.g. O(n^2) behavior in the overall number of templates) in the front-end become a problem again. Of course, many of the problems could have probably been avoided if there was an iron-clad rule that the module dependency graph must remain acyclic from the beginning of development (at least at the level of units of compilation). But how could they have known how bad it would get otherwise? I don't think this is reflected in our documentation anywhere, at least not in a prominent place. — David
Nov 11 2015
next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 17:19:31 UTC, David Nadlinger 
wrote:
 Yes, there is a single cyclic inter-package dependency hidden
 somewhere deep inside the dependency graph, but getting rid
 of it after the fact would  have either required re-structuring
 large parts of a ˜10^5 LOC code base, or would have again
 involved creating logical
Understood. FWIW it'd be more of a help to dig down and fix that particular bug sooner rather than going back to come up with a new template emission design. Cross-Posting my Bugzilla comment here. https://issues.dlang.org/show_bug.cgi?id=15318#c1 This seems to be a case where the second rule of this issue 14431 fix doesn't work. http://forum.dlang.org/post/mailman.697.1440962414.13986.digitalmars-d-bugs puremagic.com
 If a template is instantiated in non-root module, compiler 
 usually does not have to put it in object file. But if a 
 template is instantiated in both of root and non-root modules 
 which mutually import each other, it needs to placed in objfile.
=== In both of the compilations there is a root and a non-root module instantiating bar!5. Now unfortunately the compiler decides both times that the non-root module should do it. This problem would be solved by my proposal to define a global order for who is responsible to instantiate a template, by choosing the module with the lexicographically smaller module name. https://github.com/D-Programming-Language/dmd/pull/4384#discussion_r29910422 https://issues.dlang.org/show_bug.cgi?id=14431#c12 This would establish a stable order between B and C, and no matter how you compile them, B gets to instantiate the template. === An intermediate workaround for your problem is to use the -allinst switch, even though it slows down compilation a lot. You can also compile each module separately in which case both B and C get the instance (in general this is even slower than -allinst). ===
 packages so large that the memory inefficiencies (CTFE is the 
 biggest problem here) and run-time scaling issues (e.g. O(n^2) 
 behavior in the overall number of templates) in the front-end 
 become a problem again.
There shouldn't be anything quadratic left for template instantiation in the frontend.
Nov 11 2015
parent David Nadlinger <code klickverbot.at> writes:
On Thursday, 12 November 2015 at 02:13:04 UTC, Martin Nowak wrote:
 There shouldn't be anything quadratic left for template 
 instantiation in the frontend.
Maybe not for a single instantiation, but constructing the module member list is overall still quadratic in the number of members (which might be quite a lot if you heavily use templates, because all the instances get added first, whether they make it to codegen or not). See: https://issues.dlang.org/show_bug.cgi?id=15323. — David
Nov 12 2015
prev sibling next sibling parent James Hofmann <jhofmann 321f.net> writes:
On Wednesday, 11 November 2015 at 17:19:31 UTC, David Nadlinger 
wrote:
 Of course, many of the problems could have probably been 
 avoided if there was an iron-clad rule that the module 
 dependency graph must remain acyclic from the beginning of 
 development (at least at the level of units of compilation). 
 But how could they have known how bad it would get otherwise? I 
 don't think this is reflected in our documentation anywhere, at 
 least not in a prominent place.

  — David
There is some literature about whether this kind of rule, enforced at compile-time, can benefit software architecture in general, in the context of F# projects vs. similar C# projects. (Answer: probably yes - there's a relationship between cyclical dependencies and other accidental coupling; you can make some case for it both theoretically and in statistics) http://evelinag.com/blog/2014/06-09-comparing-dependency-networks/#.VkP6mL_eNYU Looking at what D does now, at least according to "The D Programming Language" (2010), it tries to allow any ordering but throws an exception at runtime when ambiguious cases are detected. So there's already some precedent to avoid cyclical dependency simply to avoid those errors. The coupling argument and the compile-times argument just add more urgency to it. Nim enforced the same dependency rule as F# the last time I looked, but I think that position was softening towards "optional" due to some community pressure. I don't think anything makes this architecture style actually impractical and it might help to have the compiler warn towards it - although the same kind of community pressure is going to arise if it did do so. After all, nobody likes to be told that they are writing Bad Code :)
Nov 11 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2015 9:19 AM, David Nadlinger wrote:
 Of course, many of the problems could have probably been avoided if there was
an
 iron-clad rule that the module dependency graph must remain acyclic from the
 beginning of development (at least at the level of units of compilation).
In hindsight, I would have made D enforce that (Go does).
 But how could they have known how bad it would get otherwise?
I hadn't anticipated that code would inevitably be constructed so that every module imports every other module.
Nov 12 2015
next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Thursday, 12 November 2015 at 22:50:08 UTC, Walter Bright 
wrote:
 On 11/11/2015 9:19 AM, David Nadlinger wrote:
 Of course, many of the problems could have probably been 
 avoided if there was an
 iron-clad rule that the module dependency graph must remain 
 acyclic from the
 beginning of development (at least at the level of units of 
 compilation).
In hindsight, I would have made D enforce that (Go does).
That's impossible because of the way we do mixin. You can have a template that do mixin without creating loop because then you pass it down identifiers that won't resolve. That has been a giant pain in various situations in project I work on.
Nov 12 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/12/2015 2:58 PM, deadalnix wrote:
 You can have a template that
 do mixin without creating loop because then you pass it down identifiers that
 won't resolve.
My brain seg faulted trying to parse that.
Nov 12 2015
parent reply deadalnix <deadalnix gmail.com> writes:
On Friday, 13 November 2015 at 00:58:57 UTC, Walter Bright wrote:
 On 11/12/2015 2:58 PM, deadalnix wrote:
 You can have a template that
 do mixin without creating loop because then you pass it down 
 identifiers that
 won't resolve.
My brain seg faulted trying to parse that.
module a; struct S(E) { import std.bitfield; mixin(bitfield!( bool, "flag", 1, T, "myT", SizeOf!T, uint, "", 32 - SizeOf!T - 1, )); } module b; enum MyEnum { Val1, Val2, Val3 } import a; alias UsefulStruct = S!MyEnum; // Boom, MyEnum is undefined Now you end up having to tie stupid node into your code, like adding all user of library X as import in library X. Which sucks, but is forced by the language design.
Nov 12 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/12/2015 5:14 PM, deadalnix wrote:
 module a;

 struct S(E) {
      import std.bitfield;
      mixin(bitfield!(
          bool, "flag", 1,
          T, "myT", SizeOf!T,
          uint, "", 32 - SizeOf!T - 1,
      ));
 }

 module b;

 enum MyEnum { Val1, Val2, Val3 }

 import a;
 alias UsefulStruct = S!MyEnum; // Boom, MyEnum is undefined

 Now you end up having to tie stupid node into your code, like adding all user
of
 library X as import in library X. Which sucks, but is forced by the language
 design.
The trouble there is that E is turned into MyEnum in the mixin string, and MyEnum would be unknown to module a. You can file an enhancement request on this in bugzilla if you like.
Nov 12 2015
parent reply deadalnix <deadalnix gmail.com> writes:
On Friday, 13 November 2015 at 04:03:23 UTC, Walter Bright wrote:
 On 11/12/2015 5:14 PM, deadalnix wrote:
 module a;

 struct S(E) {
      import std.bitfield;
      mixin(bitfield!(
          bool, "flag", 1,
          T, "myT", SizeOf!T,
          uint, "", 32 - SizeOf!T - 1,
      ));
 }

 module b;

 enum MyEnum { Val1, Val2, Val3 }

 import a;
 alias UsefulStruct = S!MyEnum; // Boom, MyEnum is undefined

 Now you end up having to tie stupid node into your code, like 
 adding all user of
 library X as import in library X. Which sucks, but is forced 
 by the language
 design.
The trouble there is that E is turned into MyEnum in the mixin string, and MyEnum would be unknown to module a. You can file an enhancement request on this in bugzilla if you like.
I know what the problem is. Hence you need to import a in b and b in a. There is not really anyway around this because this is intrinsically how mixin works.
Nov 12 2015
parent Dicebot <public dicebot.lv> writes:
 I know what the problem is. Hence you need to import a in b and 
 b in a. There is not really anyway around this because this is 
 intrinsically how mixin works.
It could be all much easier if resolving aliases was controllable in generic code, most importantly in regards of .stringof
Nov 13 2015
prev sibling next sibling parent reply rsw0x <anonymous anonymous.com> writes:
On Thursday, 12 November 2015 at 22:50:08 UTC, Walter Bright 
wrote:
 On 11/11/2015 9:19 AM, David Nadlinger wrote:
 Of course, many of the problems could have probably been 
 avoided if there was an
 iron-clad rule that the module dependency graph must remain 
 acyclic from the
 beginning of development (at least at the level of units of 
 compilation).
In hindsight, I would have made D enforce that (Go does).
Would there be any gain from adding an optional compiler warning for it?
Nov 12 2015
parent reply Jakob Ovrum <jakobovrum gmail.com> writes:
On Thursday, 12 November 2015 at 23:06:01 UTC, rsw0x wrote:
 On Thursday, 12 November 2015 at 22:50:08 UTC, Walter Bright 
 wrote:
 On 11/11/2015 9:19 AM, David Nadlinger wrote:
 Of course, many of the problems could have probably been 
 avoided if there was an
 iron-clad rule that the module dependency graph must remain 
 acyclic from the
 beginning of development (at least at the level of units of 
 compilation).
In hindsight, I would have made D enforce that (Go does).
Would there be any gain from adding an optional compiler warning for it?
It can be implemented as an external tool with the -deps compiler switch.
Nov 12 2015
parent reply Stefan <dl ng.rocks> writes:
On Thursday, 12 November 2015 at 23:08:57 UTC, Jakob Ovrum wrote:

 [...dependency check...]
 It can be implemented as an external tool with the -deps 
 compiler switch.
there is depend [1]. It warns about cycles and unintended dependencies if you specify the target dependencies. It helps us to maintain our commercial code base. src as well as unittest. [1] https://github.com/funkwerk/depend
Nov 13 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/13/2015 07:51 AM, Stefan wrote:
 On Thursday, 12 November 2015 at 23:08:57 UTC, Jakob Ovrum wrote:

 [...dependency check...]
 It can be implemented as an external tool with the -deps compiler switch.
there is depend [1]. It warns about cycles and unintended dependencies if you specify the target dependencies. It helps us to maintain our commercial code base. src as well as unittest. [1] https://github.com/funkwerk/depend
Hi Stefan, it was great to meet in Stuttgart! This is a simple but very useful tool, it ought to be made part of the standard distribution. I just created https://issues.dlang.org/show_bug.cgi?id=15337, in summary: 1. Relicensing the work under Boost 2. Make the tool easier to use, e.g. have it run dmd automatically if needed etc. 3. Integrate the tool within the tools/ repository and make it part of the standard dmd distribution. Good tooling has been a huge asset for Go - we could take a page from their book. Stefan, Dragos - do you guys think you could take this to completion? Thanks, Andrei
Nov 14 2015
prev sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 12 November 2015 at 22:50:08 UTC, Walter Bright 
wrote:
 I hadn't anticipated that code would inevitably be constructed 
 so that every module imports every other module.
One day, I was cleaning my house and found all kinds of the cat's stuff under the couch. I wondered: why does the cat love shoving her stuff down there? Then I realized the answer: she randomly bats it around the whole floor. If it lands out in the open, she knocks it somewhere else. When it randomly ends up under the couch though, she can no longer reach it, so that is where it stays. Eventually, everything will accumulate there. I guess this is just a long winded way of saying that if something is even a little useful and possible, it will end up being used until it reaches some point where we can't change it anymore...
Nov 12 2015
prev sibling parent David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 15:04:01 UTC, Andrei 
Alexandrescu wrote:
 Liran Zvibel […] mentioned the package-at-a-time solved 
 essentially all of their build problems.
They were stuck at 2.066 until very recently, though, because of issues very similar to the one discussed here. Right now, they are using LDC 2.068.2 with a couple of local workarounds for e.g. the issue from the original post that are not really general enough to go into upstream. The idea would be to return to an unpatched frontend as soon as possible, though. — David
Nov 11 2015
prev sibling next sibling parent David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
wrote:
 No, it's not. Separate compilation is C++-ishly slow by design 
 (b/c all dependencies get imported over and over again, just 
 like headers).
To a certain extent this is not true. As long as the amount repeated work is not too big, the ability to parallelize the actual semantic3/glue/codegen parts in a separate compilation setting still leads to a considerable speedup. All-at-once compilation forces you to throw away 31 of the 32 cores in your nice heavy duty development box. (Yes, in theory the different parts of the compilation process could be parallelized even for an all-at-once build, but by the time we reach the level of separation/well-definedness of the internal state required for this, all the by-package compilation problems would have probably disappeared rather automatically.) — David
Nov 11 2015
prev sibling next sibling parent reply rsw0x <anonymous anonymous.com> writes:
On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
wrote:
 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
 wrote:
 I think we should really fix these issues, working separate 
 compilation is very important.
No, it's not. Separate compilation is C++-ishly slow by design (b/c all dependencies get imported over and over again, just like headers). We should endorse compiling small (rather independent) packages to static/shared libraries.
This would be great if cross-module inlining actually worked. pragma(inline,true) doesn't even work across modules afaik
Nov 11 2015
parent deadalnix <deadalnix gmail.com> writes:
On Wednesday, 11 November 2015 at 18:06:22 UTC, rsw0x wrote:
 On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
 wrote:
 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
 wrote:
 I think we should really fix these issues, working separate 
 compilation is very important.
No, it's not. Separate compilation is C++-ishly slow by design (b/c all dependencies get imported over and over again, just like headers). We should endorse compiling small (rather independent) packages to static/shared libraries.
This would be great if cross-module inlining actually worked. pragma(inline,true) doesn't even work across modules afaik
It does if you compile by package with something else than DMD, because DMD does inlining in some very weird ways.
Nov 11 2015
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
wrote:
 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
 wrote:
 I think we should really fix these issues, working separate 
 compilation is very important.
No, it's not. Separate compilation is C++-ishly slow by design (b/c all dependencies get imported over and over again, just like headers). We should endorse compiling small (rather independent) packages to static/shared libraries.
But compiling by package isn't slow by design. Also, it depends on the modules you're compiling. If, like me, you have all your unit tests in separate files from the implementation, it's faster when only changing the test to recompile that one file and relink than compiling the whole program again. So... it depends. Atila
Nov 13 2015
prev sibling next sibling parent reply Robert burner Schadek <rburners gmail.com> writes:
On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
wrote:
 A quick workaround could be enabling the GC for DDMD. IIRC I 
 read somewhere on github that the segfaulting code was actually 
 rewritten now and the GC might just work.
I just run make -f posix.mak unittest -j10 on phobos after removing GC.disable() from ddmd. It worked just fine. Everything build everything passed the tests. (the time program told 1:41.25 for nogc and 1:49.80 with gc. This is with running all the tests.) Shouldn't we at least add a command line switch to dmd to activate the GC.
Nov 11 2015
next sibling parent Robert burner Schadek <rburners gmail.com> writes:
druntime also works with GC activated
Nov 11 2015
prev sibling next sibling parent rsw0x <anonymous anonymous.com> writes:
On Wednesday, 11 November 2015 at 23:44:04 UTC, Robert burner 
Schadek wrote:
 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
 wrote:
 A quick workaround could be enabling the GC for DDMD. IIRC I 
 read somewhere on github that the segfaulting code was 
 actually rewritten now and the GC might just work.
I just run make -f posix.mak unittest -j10 on phobos after removing GC.disable() from ddmd. It worked just fine. Everything build everything passed the tests. (the time program told 1:41.25 for nogc and 1:49.80 with gc. This is with running all the tests.) Shouldn't we at least add a command line switch to dmd to activate the GC.
if it works, shouldn't the runtime CLI(http://dlang.org/changelog/2.067.0.html#gc-options) just be used to manually deactivate it if needed?
Nov 11 2015
prev sibling parent reply Daniel Murphy <yebbliesnospam gmail.com> writes:
On 12/11/2015 10:44 AM, Robert burner Schadek wrote:
 I just run make -f posix.mak unittest -j10 on phobos after removing
 GC.disable() from ddmd. It worked just fine. Everything build everything
 passed the tests.

 (the time program told 1:41.25 for nogc and 1:49.80 with gc. This is
 with running all the tests.)

 Shouldn't we at least add a command line switch to dmd to activate the GC.
You also need to modify root/rmem.d to actually use the GC as the allocator.
Nov 12 2015
parent reply Robert burner Schadek <rburners gmail.com> writes:
On Friday, 13 November 2015 at 02:50:07 UTC, Daniel Murphy wrote:
 You also need to modify root/rmem.d to actually use the GC as 
 the allocator.
I should have known that it couldn't be that simple. Anyway, after doing so. Building druntime and phobos die with a segfault, but all dmd tests pass, except runnable/arrayop.d Furthermore, I had to create a function called: extern (C) void* allocmemory(size_t m_size) { return GC.malloc(m_size); }
Nov 13 2015
parent Daniel Murphy <yebbliesnospam gmail.com> writes:
On 13/11/2015 8:26 PM, Robert burner Schadek wrote:
 On Friday, 13 November 2015 at 02:50:07 UTC, Daniel Murphy wrote:
 You also need to modify root/rmem.d to actually use the GC as the
 allocator.
I should have known that it couldn't be that simple. Anyway, after doing so. Building druntime and phobos die with a segfault, but all dmd tests pass, except runnable/arrayop.d Furthermore, I had to create a function called: extern (C) void* allocmemory(size_t m_size) { return GC.malloc(m_size); }
Yep, that sounds about right. It's probably just some malloced memory being used to store GC pointers, somewhere in there.
Nov 13 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2015 5:47 AM, Johannes Pfau wrote:
 Ellery Newcomer recently reported a template emission bug where
 templates are emitted twice:
 http://forum.dlang.org/thread/n1omke$1bh5$1 digitalmars.com
Please file bug reports in bugzilla. Filing them in the n.g. means they will most likely get overlooked.
Nov 12 2015
parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 12 Nov 2015 15:15:24 -0800
schrieb Walter Bright <newshound2 digitalmars.com>:

 On 11/11/2015 5:47 AM, Johannes Pfau wrote:
 Ellery Newcomer recently reported a template emission bug where
 templates are emitted twice:
 http://forum.dlang.org/thread/n1omke$1bh5$1 digitalmars.com  
Please file bug reports in bugzilla. Filing them in the n.g. means they will most likely get overlooked.
Ellery filed a bug report and I've posted a reduced test case: https://issues.dlang.org/show_bug.cgi?id=15324
Nov 13 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2015 9:23 AM, Johannes Pfau wrote:
 Ellery filed a bug report and I've posted a reduced test case:

 https://issues.dlang.org/show_bug.cgi?id=15324
Thank you!
Nov 13 2015
parent deadalnix <deadalnix gmail.com> writes:
On Saturday, 14 November 2015 at 03:43:38 UTC, Walter Bright 
wrote:
 On 11/13/2015 9:23 AM, Johannes Pfau wrote:
 Ellery filed a bug report and I've posted a reduced test case:

 https://issues.dlang.org/show_bug.cgi?id=15324
Thank you!
https://issues.dlang.org/show_bug.cgi?id=15333 Also, thanks to whoever was able to reduce it more than I.
Nov 16 2015
prev sibling next sibling parent Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:

We're well aware of the template problems and FWIW the most 
important step is to simplify the current implementation (not the 
rules).
http://forum.dlang.org/post/55F07E92.7040104 dawg.eu
Once that is done and we have a clearer understanding of what the 
problems are, we could rediscuss the emission rules.

Ultimatively the decision to instantiate/emit as few templates as 
possible sped up compilation a lot but basically means -c with 
multiple objects will never work reliably.
Compiling separate objects is a C++-idiom anyhow and doesn't make 
too much sense for D. Instead separating your codebase into 
packages with little interdependencies and always recompiling 
those packages when something changes should be the preferred way.
Nov 11 2015
prev sibling next sibling parent Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:

Let's please make sure that we come out of this discussion with 
some concrete actions to work on. Ideally you'd fill our backlog 
with some stories, even if they are just research stories.
Nov 11 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/2015 08:08 AM, David Nadlinger wrote:
[snip]

Regarding the top-level issue. Walter and I agree it's an important 
problem, and also that plugging holes as they start leaking (which is 
what we've been doing so far) is not going to work well in the long haul.

A redesign of template instantiation is necessary, and Walter needs to 
be fully involved in it. However, please give it time. Walter is 
currently working full time on catching C++ exceptions from D code, and 
as we all know the best way of getting many things done is to do one 
thing at a given time and do it fully. It should take him at least two 
weeks' time to get there.

This is a good time to collect cases known to be troublesome and to 
discuss high-level approaches.


Thanks,

Andrei
Nov 11 2015
next sibling parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 November 2015 at 17:25, Andrei Alexandrescu via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 On 11/11/2015 08:08 AM, David Nadlinger wrote:
 [snip]

 Regarding the top-level issue. Walter and I agree it's an important
 problem, and also that plugging holes as they start leaking (which is what
 we've been doing so far) is not going to work well in the long haul.

 A redesign of template instantiation is necessary, and Walter needs to be
 fully involved in it. However, please give it time. Walter is currently
 working full time on catching C++ exceptions from D code, and as we all
 know the best way of getting many things done is to do one thing at a given
 time and do it fully. It should take him at least two weeks' time to get
 there.
Please no. I've done my fair share of investigating this in libunwind, and it just isn't worth it.
Nov 11 2015
next sibling parent David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 17:19:51 UTC, Iain Buclaw wrote:
 Please no.  I've done my fair share of investigating this in 
 libunwind, and it just isn't worth it.
It isn't worth it, or you couldn't figure out how to? ;P In other words, what I'm asking is: Yes, all the lifetime management depends crucially on the C++ standard library internals. But given that we do this anyway (name mangling, etc.), how would you decide that the effort is not worth it? — David
Nov 11 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/2015 12:19 PM, Iain Buclaw via Digitalmars-d wrote:
 On 11 November 2015 at 17:25, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:

     On 11/11/2015 08:08 AM, David Nadlinger wrote:
     [snip]

     Regarding the top-level issue. Walter and I agree it's an important
     problem, and also that plugging holes as they start leaking (which
     is what we've been doing so far) is not going to work well in the
     long haul.

     A redesign of template instantiation is necessary, and Walter needs
     to be fully involved in it. However, please give it time. Walter is
     currently working full time on catching C++ exceptions from D code,
     and as we all know the best way of getting many things done is to do
     one thing at a given time and do it fully. It should take him at
     least two weeks' time to get there.


 Please no.  I've done my fair share of investigating this in libunwind,
 and it just isn't worth it.
Could you please provide Walter more detail? What's not worth it - catching C++ exceptions from D code? -- Andrei
Nov 11 2015
parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 November 2015 at 19:02, Andrei Alexandrescu via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 On 11/11/2015 12:19 PM, Iain Buclaw via Digitalmars-d wrote:

 On 11 November 2015 at 17:25, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:

     On 11/11/2015 08:08 AM, David Nadlinger wrote:
     [snip]

     Regarding the top-level issue. Walter and I agree it's an important
     problem, and also that plugging holes as they start leaking (which
     is what we've been doing so far) is not going to work well in the
     long haul.

     A redesign of template instantiation is necessary, and Walter needs
     to be fully involved in it. However, please give it time. Walter is
     currently working full time on catching C++ exceptions from D code,
     and as we all know the best way of getting many things done is to do
     one thing at a given time and do it fully. It should take him at
     least two weeks' time to get there.


 Please no.  I've done my fair share of investigating this in libunwind,
 and it just isn't worth it.
Could you please provide Walter more detail? What's not worth it - catching C++ exceptions from D code? -- Andrei
The main problem for seamless support is having some way to generate the C++ typeinfo in D to allow work across boundaries. However there is nothing stopping catch-all style exceptions from working. Consider the following desired example (this works today with minimal library changes in gdc's libunwind library): ***************************** #include <iostream> int compute(int a, int b) { if (b == 0) throw "Division by 0 in C++!"; return a / b; } void cpp_main() { try { std::cout << compute(1, 0); } catch (...) { std::cout << "Unknown exception caught in C++, rethrowing..." << "\n"; throw; } } ***************************** Then put in bindings and main in D. ***************************** import std.stdio; extern(C++) void cpp_main(); void main() { try { cpp_main(); } catch { writeln("Unknown exception caught in D, exiting..."); } } ***************************** In our unwind library, we can put in a copy of the C++ unwind implementation: ***************************** static if (GNU_ARM_EABI_Unwinder) { const _Unwind_Exception_Class __gxx_exception_class = ['G', 'N', 'U', 'C', 'C', '+', '+', '\0']; } else { const _Unwind_Exception_Class __gxx_exception_class = 0x474e5543432b2b00UL; } // Structure of a C++ exception, represented as a C structure... // See unwind-cxx.h for the full definition. struct __cxa_exception { void *exceptionType; void function(void *) exceptionDestructor; void function() unexpectedHandler; void function() terminateHandler; __cxa_exception *nextException; int handlerCount; static if (GNU_ARM_EABI_Unwinder) { __cxa_exception* nextPropagatingException; int propagationCount; } else { int handlerSwitchValue; const ubyte *actionRecord; const ubyte *languageSpecificData; _Unwind_Ptr catchTemp; void *adjustedPtr; } _Unwind_Exception unwindHeader; } ***************************** Ignoring the main guts of gdc's unwind implementation (should be very similar in ldc and sdc by coincidence), let's instead focus on the exact part where we try to match the exception object, you'd have something like the following: ***************************** if (ar_filter > 0) { // Positive filter values are handlers. void *catch_type = get_ttype_entry (&info, ar_filter); if (!foreign_exception) { if (_d_isbaseof (xh.object.classinfo, cast(ClassInfo)catch_type)) saw_handler = true; } else { // == Case A == // Null catch type is a catch-all handler; we can catch foreign exceptions with this. if (catch_type is null) saw_handler = true; else if (ue_header.exception_class == __gxx_exception_class) { // == Case B == // Typeinfo are directly compared, which might not be correct if they aren't merged. void *except_typeinfo = ((cast(__cxa_exception *)(ue_header + 1)) - 1).exceptionType; if (catch_type == except_typeinfo) saw_handler = true; // == Case C == // Typeid are compared between catch type and some specially crafted C++ Exception class. if (cast(ClassInfo)catch_type is typeid(UnknownCPPException)) saw_handler = true; } } } ***************************** Now let's discuss the three cases: - Case A: A `null` catch type never happens in D code, because `catch { }` is always re-written to `catch(Throwable) { }` (this might have changed since 2.066, but I doubt it). So any kind of `catch(...)` equivalent in D would make this is a valid candidate for catching not just C++ exceptions, but also exceptions in any language that use libunwind for EH (Ada, Go, Java, etc...). Bonus, we don't need to know anything about C++ exception objects or typeinfo to achieve this. In the other cases, checking the exception class is simple enough, libunwind provides a field for that, and just requires to declare it somewhere on the D bindings side. - Case B: Explicitly try to match a C++ exception object. Copying the __cxa_exception struct from unwind-cxx.h, takes advantage of ABI compatibility - though we must take care that they are kept in sync (not that it ever changes in an incompatible way). Also from the C++ unwind library, is one small trick to get the C++ typeinfo pointer. Where this falls short is that ``catch_type == except_typeinfo`` will always be false unless we expose some way to generate C++ typeinfo in D. *Maybe* this could be achieved by generating an pointer to an `extern(C++)` symbol to the typeinfo - just need to get mangling right. But this is the ugliest thing we can possibly do. - Case C: An alternative to the catch all handler in case A, but instead test whether the catch we are inspecting matches some explicit object (derived from `Throwable` to be compatible with the existing compiler constraints). Again, this can be extended for other languages (UnknownAdaException, UnknownGoException, UnknownJavaException, etc...) - however, unlike case A, we need to copy the exception class signature of other languages into the D bindings. With the possible exception being in case B, there is no way to return the foreign language object from EH to the catch handler. Using case C and storing the caught exception object to a variable `catch (UnknownCPPException e)` will always result with the object being equal to `null`. ***************************** Coming back round full circle to the original example, I've implemented case C but instead using `Throwable` to make `catch { }` succeed in catching C++ exceptions. Compilation and Result: $ gdc dcode.d cxxcode.cc $ ./a.out Unknown exception caught in C++, rethrowing... Unknown exception caught in D, exiting... Even throwing in D, and catching in C++ works thanks to C++'s existing support for foreign language exceptions and `catch(...)`. So moving `compute` into D, we have the following. ***************************** extern(C++) int compute(int a, int b) { if (b == 0) { writeln("Exception thrown in D..."); throw new Exception("Division by 0 in D!"); } return a / b; } ***************************** Compilation and Result: $ gdc dcode.d cxxcode.cc $ ./a.out Exception thrown in D... Unknown exception caught in C++, rethrowing... Unknown exception caught in D, exiting... Andrei - Hope this answers your question. -- Regards Iain
Nov 11 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2015 4:16 PM, Iain Buclaw via Digitalmars-d wrote:
 The main problem for seamless support is having some way to generate the C++
 typeinfo in D to allow work across boundaries.
I suspect that is unnecessary. The whole point of D's C++ interface is to interface with C++ code. Defining C++ classes in D code that does not link to any C++ code that also defines those classes won't be supported. I.e. we can reply on the C++ code generating the necessary C++ typeinfo. The generated D code only has to link to it, i.e. generate an extern reference to the name of the typeinfo. Similarly, we never need to throw a C++ exception from D code. All we need to do is call a library function written in C++ that throws it for us. What D does need to do is support catching an std::exception*, or a class derived from std::exception. I believe we are on solid ground in not supporting any other C++ types being thrown. Nor do I believe we need to catch C++ objects with catch(...). Catching C++ objects will need an explicit catch(std.exception*). Over time, perhaps it may become apparent that we do need to support more kinds of C++ types being thrown. But it is not necessary to get this lead brick airborne, and we shouldn't waste time worrying about it.
Nov 11 2015
next sibling parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 Nov 2015 7:15 am, "Walter Bright via Digitalmars-d" <
digitalmars-d puremagic.com> wrote:
 On 11/11/2015 4:16 PM, Iain Buclaw via Digitalmars-d wrote:
 The main problem for seamless support is having some way to generate the
C++
 typeinfo in D to allow work across boundaries.
I suspect that is unnecessary. The whole point of D's C++ interface is to interface with C++ code.
Defining C++ classes in D code that does not link to any C++ code that also defines those classes won't be supported. I.e. we can reply on the C++ code generating the necessary C++ typeinfo. The generated D code only has to link to it, i.e. generate an extern reference to the name of the typeinfo.
 Similarly, we never need to throw a C++ exception from D code. All we
need to do is call a library function written in C++ that throws it for us.

Yes, never throw a C++ exception from D, even if we manage to intercept the
object.

 What D does need to do is support catching an std::exception*, or a class
derived from std::exception. I believe we are on solid ground in not supporting any other C++ types being thrown.
 Nor do I believe we need to catch C++ objects with catch(...). Catching
C++ objects will need an explicit catch(std.exception*).

Well, as demonstrated, that is the one thing that can be supported now for
free.  Either by allowing C++-style catch-all or using language specific
catch-all class matching.

 Over time, perhaps it may become apparent that we do need to support more
kinds of C++ types being thrown. But it is not necessary to get this lead brick airborne, and we shouldn't waste time worrying about it.

Any type could be caught, again I demonstrated this, but I would have more
confidence if g++ generated the typeinfo bindings for us using
pragma(mangle) to some D-friendly symbol.  Rather than us pushing in more
frail mangling support from our end.

--
Regards,
Iain
Nov 11 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2015 11:45 PM, Iain Buclaw via Digitalmars-d wrote:
  > Nor do I believe we need to catch C++ objects with catch(...). Catching C++
 objects will need an explicit catch(std.exception*).
  >

 Well, as demonstrated, that is the one thing that can be supported now for
 free.  Either by allowing C++-style catch-all or using language specific
 catch-all class matching.
I don't agree with implementing things just because we can. There has to be a good case for it. I cannot see a case for catching ints, strings, etc. My understanding is that this is more or less acknowledged by the C++ community, and std::exceptions are used.
 Any type could be caught, again I demonstrated this, but I would have more
 confidence if g++ generated the typeinfo bindings for us using pragma(mangle)
to
 some D-friendly symbol.  Rather than us pushing in more frail mangling support
 from our end.
Having dealt with mangling issues for decades, I'm not overly concerned about it.
Nov 12 2015
prev sibling next sibling parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 Nov 2015 7:15 am, "Walter Bright via Digitalmars-d" <
digitalmars-d puremagic.com> wrote:
 On 11/11/2015 4:16 PM, Iain Buclaw via Digitalmars-d wrote:
 The main problem for seamless support is having some way to generate the
C++
 typeinfo in D to allow work across boundaries.
I suspect that is unnecessary. The whole point of D's C++ interface is to interface with C++ code.
Defining C++ classes in D code that does not link to any C++ code that also defines those classes won't be supported. I.e. we can reply on the C++ code generating the necessary C++ typeinfo. The generated D code only has to link to it, i.e. generate an extern reference to the name of the typeinfo.

It is a small concern to me that C++ catch handlers work by calling some
support library code, If I recall correctly both at the start and end of
the handler.  These calls may change the state of C++'s language specific
unwind library, such as adjust any reference counters, mark the thrown
object as having been caught.

Maybe less of a concern now than last year.  But I do wonder what effect it
will incur on a long running program.

--
Regards,
Iain
Nov 11 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/12/2015 01:15 AM, Walter Bright wrote:
 What D does need to do is support catching an std::exception*, or a
 class derived from std::exception. I believe we are on solid ground in
 not supporting any other C++ types being thrown.

 Nor do I believe we need to catch C++ objects with catch(...). Catching
 C++ objects will need an explicit catch(std.exception*).
I notice you use a star - did you back off from the idea of defining C++'s std::exception as an extern(C++) class (as opposed to an extern(C++) struct)? -- Andrei
Nov 12 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/12/2015 5:38 AM, Andrei Alexandrescu wrote:
 I notice you use a star - did you back off from the idea of defining C++'s
 std::exception as an extern(C++) class (as opposed to an extern(C++) struct)?
--
I just wanted to emphasize receiving it by reference rather than by value.
Nov 12 2015
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Wednesday, 11 November 2015 at 17:19:51 UTC, Iain Buclaw wrote:
 Please no.  I've done my fair share of investigating this in 
 libunwind, and it just isn't worth it.
Ok I know Walter won't listen, and I wouldn't be as radical as Iain, but I concur. There are thing that can be done to improve C++ and D interrop with Exception. For instance, SDC makes sure that C++ exception unwind properly through D and vice versa, but don't allow to catch across language. There is a reason for that. To be able to catch C++ exceptions, you need to duplicate the whole RTTI mechanism from C++ + runtime support for it + hook oneself in the C++ runtime to signal catching and rethrowing. This is definitively not impossible, but the cost benefit ratio is such as I wouldn't prioritize this. There are definitively more way to have impact without summoning a Golem when it comes to C++ interrop.
Nov 11 2015
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/2015 07:45 PM, deadalnix wrote:
 This is definitively not impossible, but the cost benefit ratio is such
 as I wouldn't prioritize this.
The way we see it is the benefit is all C++ interop. You can't really do any meaningful interop whatsoever if you can't catch std::exception. -- Andrei
Nov 11 2015
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2015 4:45 PM, deadalnix wrote:
 Ok I know Walter won't listen,
Does anyone else hear a buzzing sound? Or is my hearing aid on the fritz again?
Nov 11 2015
prev sibling next sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 16:25:34 UTC, Andrei 
Alexandrescu wrote:
 Walter is currently working full time on catching C++ 
 exceptions from D code […]
It would have sure been interesting to know about this, but I couldn't find anything to that effect on the mailing lists or on Trello – where Martin would certainly wish it could be found. There are quite a few people around here who very familiar with what is required to make that happen (DWARF/libunwind based EH, etc…), and although I'm certainly not going to tell Walter how to do his job, it seems rather pointless for him to go through the process of rediscovering all this again on his own. — David
Nov 11 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/2015 12:25 PM, David Nadlinger wrote:
 On Wednesday, 11 November 2015 at 16:25:34 UTC, Andrei Alexandrescu wrote:
 Walter is currently working full time on catching C++ exceptions from
 D code […]
It would have sure been interesting to know about this, but I couldn't find anything to that effect on the mailing lists or on Trello – where Martin would certainly wish it could be found. There are quite a few people around here who very familiar with what is required to make that happen (DWARF/libunwind based EH, etc…), and although I'm certainly not going to tell Walter how to do his job, it seems rather pointless for him to go through the process of rediscovering all this again on his own.
That's great. We have a history of initiatives that nobody followed and endless debates without work getting done; also this particular issue has been up in the air and spoken about for a good while. So we figured if Walter doesn't sit down and do it, nobody will. I'll leave it up to him to reach out to you and/or this forum in this matter. Andrei
Nov 11 2015
parent reply David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 18:05:46 UTC, Andrei 
Alexandrescu wrote:
 That's great. We have a history of initiatives that nobody 
 followed and endless debates without work getting done; also 
 this particular issue has been up in the air and spoken about 
 for a good while. So we figured if Walter doesn't sit down and 
 do it, nobody will.
I couldn't agree more. What I tried to point out is merely that between Amaury, Kai, Martin, Iain, Dan, Joakim and me there is quite a bit of working knowledge regarding the internals of the various forms of the libunwind-based C++ exception handling mechanism, and I'm sure most of us would be happy to help out with any specific questions and design decisions (Walter of course being the authority on his own EH mechanism, which DMD will however have to transition away from on Win64 and Linux/… to transparently interface with C++). — David
Nov 11 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2015 12:07 PM, David Nadlinger wrote:
 On Wednesday, 11 November 2015 at 18:05:46 UTC, Andrei Alexandrescu wrote:
 That's great. We have a history of initiatives that nobody followed and
 endless debates without work getting done; also this particular issue has been
 up in the air and spoken about for a good while. So we figured if Walter
 doesn't sit down and do it, nobody will.
I couldn't agree more. What I tried to point out is merely that between Amaury, Kai, Martin, Iain, Dan, Joakim and me there is quite a bit of working knowledge regarding the internals of the various forms of the libunwind-based C++ exception handling mechanism, and I'm sure most of us would be happy to help out with any specific questions and design decisions (Walter of course being the authority on his own EH mechanism, which DMD will however have to transition away from on Win64 and Linux/… to transparently interface with C++). — David
I appreciate the offer. I'll start a new thread on it.
Nov 11 2015
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 16:25:34 UTC, Andrei 
Alexandrescu wrote:
 A redesign of template instantiation is necessary, and Walter 
 needs to be fully involved in it. However, please give it time.
I'd still opt to first refactor the existing code into something that's understandable before starting to redesign the whole topic. Maybe the design is already fine but buried by complex to understand control flow. Also just redesigning this on a blank sheet will easily miss important edge cases. And last but not least if we can't come up with a better design we'd still have a better codebase.
Nov 11 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/2015 06:06 PM, Martin Nowak wrote:
 On Wednesday, 11 November 2015 at 16:25:34 UTC, Andrei Alexandrescu wrote:
 A redesign of template instantiation is necessary, and Walter needs to
 be fully involved in it. However, please give it time.
I'd still opt to first refactor the existing code into something that's understandable before starting to redesign the whole topic. Maybe the design is already fine but buried by complex to understand control flow. Also just redesigning this on a blank sheet will easily miss important edge cases. And last but not least if we can't come up with a better design we'd still have a better codebase.
We have to come up with a better design. Not coming with a better design is not an option. Also there is ample evidence that the months-long process of patching things as problems are found does not work. -- Andrei
Nov 11 2015
prev sibling next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:
 Hi all,

 Kenji and Walter have been working on improving the template 
 emission strategy during the last couple of releases, i.e. 
 whether a template instance is emitted to a given object file 
 or not. Nevertheless, I've been continually hearing complaints 
 from various people with large D code bases (our commercial 
 users and some of the more complex open source projects) that 
 they have problems compiling their code doing anything else 
 than an all-at-once build.

 [...]
I've had problems compiling part of my code as a static library and linking to a file that used a template mixin that in turn did compile-time reflection. The mangling was different between the two and I had linker errors. It was weird and frustrating, and I had to go back to compiling all at once, which is slower. Atila
Nov 13 2015
prev sibling parent Tofu Ninja <emmons0 purdue.edu> writes:
On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:
 ...
Sorry I haven't read this whole thread so maybe this has been mentioned before or not. I am not even sure if its really related to this topic, but it seemed related to me so I figured I would just post it here rather than make a new thread. ModuleInfo.localClasses doesn't list template class instances. I just ran into this(like 5 min ago) and thought it was odd. I expected it too list template instances as well so I thought it might be a bug, not really sure. Thankfully I don't really need it to list template instances of classes so it not affecting my project, but I could see it being a problem for others.
Nov 16 2015