www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - What is the compilation model of D?

reply "David Piepgrass" <qwertie256 gmail.com> writes:
(Maybe this should be in D.learn but it's a somewhat advanced 
topic)

I would really like to understand how D compiles a program or 
library. I looked through TDPL and it doesn't seem to say 
anything about how compilation works.

- Does it compile all source files in a project at once?
- Does the compiler it have to re-parse all Phobos templates (in 
modules used by the program) whenever it starts?
- Is there any concept of an incremental build?
- Obviously, one can set up circular dependencies in which the 
compile-time meaning of some code in module A depends on the 
meaning of some code in module B, which in turn depends on the 
meaning of some other code in module A. Sometimes the D compiler 
can resolve the ultimate meaning, other times it cannot. I was 
pleased that the compiler successfully understood this:

// Y.d
import X;
struct StructY {
	int a = StructX().c;
	auto b() { return StructX().d(); }
}

// X.d
import Y;
struct StructX {
	int c = 3;
	auto d()
	{
		static if (StructY().a == 3 && StructY().a.sizeof == 3)
			return 3;
		else
			return "C";
	}
}

But what procedure does the compiler use to resolve the semantics 
of the code? Is there a specification anywhere? Does it have some 
limitations, such that there is code with an unambiguous meaning 
that a human could resolve but the compiler cannot?

- In light of the above (that the meaning of D code can be 
interdependent with other D code, plus the presence of mixins and 
all that), what are the limitations of __traits(allMembers...) 
and other compile-time reflection operations, and what kind of 
problems might a user expect to encounter?
Jul 24 2012
next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Wed, 25 Jul 2012 02:16:04 +0200
"David Piepgrass" <qwertie256 gmail.com> wrote:

 (Maybe this should be in D.learn but it's a somewhat advanced 
 topic)
 
 I would really like to understand how D compiles a program or 
 library. I looked through TDPL and it doesn't seem to say 
 anything about how compilation works.
 

The compilation model is very similar to C or C++, so that's a good starting point for understanding how D's works. Here's how it works: Whatever file *or files* you pass to DMD on the command line, *those* are the files it will compile and generate object files for. No more, no less. However, in the process, it will *also* parse and perform semantic analysis on any files that are directly or indirectly imported, but it won't actually generate any machine code or object files for them (it will find these files via the -Ipath command line switch you pass to DMD - this -I switch is like D's equivalent of Java's classpaths). This does mean that, unlike what's typically done in C/C++, it's generally much faster to pass all your files into DMD at once, instead of the typical C/C++ route of making separate calls to the compiler for each source file. After DMD generates the object files for all source files you give it, it will automatically send them to the linker (OPTLINK on windows, or gcc/ld on Posix) to be linked into an executable. That is, *unless* you give it either -c ("compile-only, do not link") or -lib ("generate library instead of object files"). That way, you can link manually if you wish. So typically, you pass DMD all the .d files in your program, and it'll compile them all, and pass them to the linker to be linked into an executable. But if you don't want to automatically link, you don't have to. If you want to compile them all separately, you can do so (though it'd be very slow - probably almost as slow as C++, but not quite). But that's just the DMD compiler itself. Instead of using DMD directly, there's a better modern trick that's generally preferred: RDMD. If you use rdmd to compile (instead of dmd), you *just* give it your *one* main source file (typically the one with your "main()" function). This file must be the *last* parameter passed to rdmd: $rdmd --build-only (any other flags) main.d Then, RDMD will figure out *all* of the source files needed (using the full compiler's frontend, so it never gets fooled into missing anything), and if any of them have been changed, it will automatically pass them *all* into DMD for you. This way, you don't have to manually keep track of all your files and pass them all into DMD youself. Just give RDMD your main file and that's it, you're golden. Side note: Another little trick with RDMD: Omit the --build-only and it will compile AND then run your program: $cat simpleecho.d import std.stdio; void main(string[] args) { writeln(args[1]); } $rdmd simpleecho.d "Anything after the .d file is passed to your app" {automatically compiles all sources if needed} Anything after the .d file is passed to your app $wheee!! command not found
 - Does it compile all source files in a project at once?

Answered this above. In short: It compiles whatever you give it (and processes, but doesn't compile, any needed imports). Unless you use RDMD in which case it automatically detects and compiles all your needed sources (unless none of them have changed).
 - Does the compiler it have to re-parse all Phobos templates (in 
 modules used by the program) whenever it starts?

Yes. (Unless you never import anything from in phobos...I think.) But it's very, very fast to parse. Lightning-speed if you compare it to C++. But it shouldn't run full semantic analysis on templates that are never actually used. (Unless they're used in a piece of dead code.)
 - Is there any concept of an incremental build?

Yes, but there's a few "gotcha"s: 1. D compiles so damn fast that it's not nearly as much of an issue as it is with C++ (which is notoriously ultra-slow compared to...everything, hence the monumental importance of C++'s incremental builds). 2. Historically, there can be problems with templates when incrementally compiling. DMD has been known to get confused about which object file it put an instantiated template into, which can lead to occasional linker errors. These errors can be fixed by doing a full rebuild (which is WAAAY faster than it would be with C++). I don't know whether or not this has been fixed. 3. Incremental building typically involves compiling files one-at-a-time. But with D, you get a HUGE boost in compilation speed by not compiling one-at-a-time. So if you have a huge, slow-to-compile codebase (for example, 15 seconds or so), and you change a handful of files, it may actually be much *faster* to do a full rebuild (since you're not re-analysing all the imports). Of course, you could probably get around that issue by passing all the changed files (and only the changed files) into DMD at once (instead of one-at-a-time), but I don't know whether typical build tools (like make) can realistically handle that.
 - Obviously, one can set up circular dependencies in which the 
 compile-time meaning of some code in module A depends on the 
 meaning of some code in module B, which in turn depends on the 
 meaning of some other code in module A. Sometimes the D compiler 
 can resolve the ultimate meaning, other times it cannot. I was 
 pleased that the compiler successfully understood this:
 
 // Y.d
 import X;
 struct StructY {
 	int a = StructX().c;
 	auto b() { return StructX().d(); }
 }
 
 // X.d
 import Y;
 struct StructX {
 	int c = 3;
 	auto d()
 	{
 		static if (StructY().a == 3 && StructY().a.sizeof ==
 3) return 3;
 		else
 			return "C";
 	}
 }
 
 But what procedure does the compiler use to resolve the semantics 
 of the code? Is there a specification anywhere? Does it have some 
 limitations, such that there is code with an unambiguous meaning 
 that a human could resolve but the compiler cannot?
 

It keeps diving deeper and deeper to find anything it can "start" with. One it finds that, it'll just build everything back up in whatever order is necessary. If it *truly is* a circular definition, and there isn't any place it can actually start with, then it issues an error. (If there's any cases where it doesn't work this way, they should be filed as bugs in the compiler.)
 - In light of the above (that the meaning of D code can be 
 interdependent with other D code, plus the presence of mixins and 
 all that), what are the limitations of __traits(allMembers...) 
 and other compile-time reflection operations, and what kind of 
 problems might a user expect to encounter?

Shouldn't really be an issue. Such things won't get evaluated until the types/identifiers involved are *fully* analyzed (or at least to the extent that they need to be analyzed). So the results of things like __traits(allMembers...) should *never* change during compilation, or when changing the order of files or imports (unless there's some compiler bug). Any situation that *would* result in any such ambiguity will get flagged as an error in your code. I would however, recommend avoiding static constructors and module constructors whenever you reasonably can. If you have a circular import (ie: module a imports b, which imports c, which imports a), then that's normally OK, *UNLESS* they all have static and/or module constructors. If they do, then the startup code D builds into your application won't know which needs to run first (and it doesn't analyze the actual code, it just assumes there *could* be an order-of-execution dependency), so you'll get a circular dependency error when you run your program. And the safest, easiest way to get rid of those errors is to eliminate one or more static/module constructors.
Jul 24 2012
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-07-25 04:00, Nick Sabalausky wrote:

 But that's just the DMD compiler itself. Instead of using DMD
 directly, there's a better modern trick that's generally preferred:
 RDMD.

 If you use rdmd to compile (instead of dmd), you *just* give it
 your *one* main source file (typically the one with your "main()"
 function). This file must be the *last* parameter passed to rdmd:

 $rdmd --build-only (any other flags) main.d

 Then, RDMD will figure out *all* of the source files needed (using
 the full compiler's frontend, so it never gets fooled into missing
 anything), and if any of them have been changed, it will automatically
 pass them *all* into DMD for you. This way, you don't have to
 manually keep track of all your files and pass them all into
 DMD youself. Just give RDMD your main file and that's it, you're golden.

RDMD is mostly useful for executables, not so much for libraries. For libraries you would need to pass _all_ of your project files directly to DMD (or find some other tool). It's perfectly fine to have a library which consists of two files with no interaction between them. Neither RDMD or the compiler can track that. -- /Jacob Carlborg
Jul 25 2012
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-07-25 17:35, David Piepgrass wrote:

 Plus, it isn't just build times that concern me. In C# I'm used to
 having an IDE that immediately understands what I have typed, giving me
 error messages and keeping metadata about the program up-to-date within
 2 seconds. I can edit a class definition in file A and get code
 completion for it in file B, 2 seconds later. I don't expect the IDE can
 ever do that if the compiler can't do a debug build in a similar timeframe.

That's not necessarily true. The C# and Java compilers in these IDEs are built to be able to handle incremental compilation at a very fine grained level. We're not talking recompiling just a single file, we're talking recompiling just a part of a single file. DMD and other D compiler are just not built to handle this. They don't handle incremental builds at all. There are various reason why it's more difficult to make an incremental build system with D. Most of the reason are due to meta programming (templates, CTFE, mixins and other things). -- /Jacob Carlborg
Jul 26 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-07-25 21:54, David Piepgrass wrote:
 Thanks for the very good description, Nick! So if I understand
 correctly, if

 1. I use an "auto" return value or suchlike in a module Y.d
 2. module X.d calls this function
 3. I call "dmd -c X.d" and "dmd -c Y.d" as separate steps

 Then the compiler will have to fully parse Y twice and fully analyze the
 Y function twice, although it generates object code for the function
 only once. Right? I wonder how smart it is about not analyzing things it
 does not need to analyze (e.g. when Y is a big module but X only calls
 one function from it - the compiler has to parse Y fully but it should
 avoid most of the semantic analysis.)

Yes, I think that's correct. But if you give the compiler all the source code at once it should only need to parse a given module only once. D doesn't use textual includes like C/C++ does, it just symbolically refers to other symbols (or something like that).
 What about templates? In C++ it is a problem that the compiler will
 instantiate templates repeatedly, say if I use vector<string> in 20
 source files, the compiler will generate and store 20 copies of
 vector<string> (plus 20 copies of basic_string<char>, too) in object files.

 1. So in D, if I compile the 20 sources separately, does the same thing
 happen (same collection template instantiated 20 times with all 20
 copies stored)?

If you compile them separately I think so, yes. How would it otherwise work, store some info between compile runs?
 2. If I compile the 20 sources all together, I guess the template would
 be instantiated just once, but then which .obj file does the
 instantiated template go in?

I think it only need to instantiate it once. If it does that or not, I don't know. About the object file, that is probably unspecified. Although if you compile with the -lib flag it will output the templates to all object files. This is one of the problems making it hard to create an incremental build system for D.
 I figure as CTFE is used more, especially when it is used to decide
 which template overloads are valid or how a mixin will behave, this will
 slow down the compiler more and more, thus making incremental builds
 more important. A typical example would be a compile-time
 parser-generator, or compiled regexes.

I think that's correct. I did some simple benchmarking comparing different uses of string mixins in Derelict. It turns out that it's a lot better to have few string mixins containing a lot of code then many string mixins containing very little code. I suspect other meta programming features (CTFE, templates, static if, mixins) could behave in a similar way.
 Plus, I've heard some people complaining that the compiler uses over 1
 GB RAM, and splitting up compilation into parts might help with that.

Yeah, I just run in to a compiler bug (not been able to create a simple test case) where it consumed around 3.5 GB of memory then just crashed after a while.
 BTW, I think I heard the compiler uses multithreading to speed up the
 build, is that right?

Yes, I'm pretty sure it reads all (many) the files in concurrently or in parallel. It probably can lex and parse in parallel as well, don't know if it does that though.
 Anyway, I can't even figure out how to enumerate the members of a module
 A; __traits(allMembers, A) causes "Error: import Y has no members".

Currently there's a bug which forces you to put the module in a package, try: module foo.A; __traits(allMembers, foo.A); -- /Jacob Carlborg
Jul 26 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, July 24, 2012 22:00:56 Nick Sabalausky wrote:
 But with D, you get a HUGE boost in compilation speed by
 not compiling one-at-a-time. So if you have a huge, slow-to-compile
 codebase (for example, 15 seconds or so),

I find it shocking that anyone would consider 15 seconds slow to compile for a large program. Yes, D's builds are lightning fast in general, and 15 seconds is probably a longer build, but calling 15 seconds "slow-to-compile" just about blows my mind. 15 seconds for a large program is _fast_. If anyone complains about a large program taking 15 seconds to build, then they're just plain spoiled or naive. I've dealt with _Java_ apps which took in the realm of 10 minutes to compile, let alone C++ apps which take _hours_ to compile. 15 seconds is a godsend. - Jonathan M Davis
Jul 24 2012
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Tue, 24 Jul 2012 20:35:27 -0700
Jonathan M Davis <jmdavisProg gmx.com> wrote:

 On Tuesday, July 24, 2012 22:00:56 Nick Sabalausky wrote:
 But with D, you get a HUGE boost in compilation speed by
 not compiling one-at-a-time. So if you have a huge, slow-to-compile
 codebase (for example, 15 seconds or so),

I find it shocking that anyone would consider 15 seconds slow to compile for a large program. Yes, D's builds are lightning fast in general, and 15 seconds is probably a longer build, but calling 15 seconds "slow-to-compile" just about blows my mind. 15 seconds for a large program is _fast_. If anyone complains about a large program taking 15 seconds to build, then they're just plain spoiled or naive. I've dealt with _Java_ apps which took in the realm of 10 minutes to compile, let alone C++ apps which take _hours_ to compile. 15 seconds is a godsend.

I just meant that I haven't heard of much D stuff that took much longer than that, so it's somewhat on the long end as far as D stuff goes. But I may be off-base. 'Course it depends a lot of the computer, too. I probably worded it weird.
Jul 24 2012
prev sibling next sibling parent Russel Winder <russel winder.org.uk> writes:
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Tue, 2012-07-24 at 20:35 -0700, Jonathan M Davis wrote:
[=E2=80=A6]
 I find it shocking that anyone would consider 15 seconds slow to compile =

 large program. Yes, D's builds are lightning fast in general, and 15 seco=

 is probably a longer build, but calling 15 seconds "slow-to-compile" just=

 about blows my mind. 15 seconds for a large program is _fast_. If anyone=

 complains about a large program taking 15 seconds to build, then they're =

 plain spoiled or naive. I've dealt with _Java_ apps which took in the rea=

 10 minutes to compile, let alone C++ apps which take _hours_ to compile. =

 seconds is a godsend.

A company I did some Python training for (they used Python for their integration and system testing, and a bit of unit testing) back in 2006 had a C++ product whose "from scratch" build time genuinely was 56 hours. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jul 25 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, July 25, 2012 08:54:24 Russel Winder wrote:
 On Tue, 2012-07-24 at 20:35 -0700, Jonathan M Davis wrote:
 [=E2=80=A6]
=20
 I find it shocking that anyone would consider 15 seconds slow to co=


 for a large program. Yes, D's builds are lightning fast in general,=


 15 seconds is probably a longer build, but calling 15 seconds
 "slow-to-compile" just about blows my mind. 15 seconds for a large
 program is _fast_. If anyone complains about a large program taking=


 seconds to build, then they're just plain spoiled or naive. I've de=


 with _Java_ apps which took in the realm of 10 minutes to compile, =


 alone C++ apps which take _hours_ to compile. 15 seconds is a godse=


=20
 A company I did some Python training for (they used Python for their
 integration and system testing, and a bit of unit testing) back in 20=

 had a C++ product whose "from scratch" build time genuinely was 56
 hours.

I've heard of overnight builds, and I've heard of _regression tests_ ru= nning=20 for over a week, but I've never heard of builds being over 2 days. Ouch= . It has got to have been possible to have a shorter build than that. Of = course,=20 if their code was bad enough that the build was that long, it may have = been=20 rather disgusting code to clean up. But then again, maybe they genuinel= y had a=20 legitimate reason for having the build take that long. I'd be very surp= rised=20 though. In any case, much as I like C++ (not as much as D, but I still like it = quite a=20 bit), its build times are undeniably horrible. - Jonathan M Davis
Jul 25 2012
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Wed, 25 Jul 2012 08:54:24 +0100
Russel Winder <russel winder.org.uk> wrote:

 On Tue, 2012-07-24 at 20:35 -0700, Jonathan M Davis wrote:
 [=E2=80=A6]
 I find it shocking that anyone would consider 15 seconds slow to
 compile for a large program. Yes, D's builds are lightning fast in
 general, and 15 seconds is probably a longer build, but calling 15
 seconds "slow-to-compile" just about blows my mind. 15 seconds for
 a large program is _fast_. If anyone complains about a large
 program taking 15 seconds to build, then they're just plain spoiled
 or naive. I've dealt with _Java_ apps which took in the realm of 10
 minutes to compile, let alone C++ apps which take _hours_ to
 compile. 15 seconds is a godsend.

A company I did some Python training for (they used Python for their integration and system testing, and a bit of unit testing) back in 2006 had a C++ product whose "from scratch" build time genuinely was 56 hours. =20

Yea, my understanding is that full-build times measured in days are (or used to be, don't know if they still are) also typical of high-budget C++-based videogames.
Jul 25 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/25/12, Jonathan M Davis <jmdavisProg gmx.com> wrote:
 I find it shocking that anyone would consider 15 seconds slow to compile for
 a large program.

It's not shocking if you're used to a fast edit-compile-run cycle which takes a few seconds and then starts to slow down considerably when you involve more and more templates. When I start working on a new D app it almost feels like programming in Python, the edit-compile-run cycle is really fast. But eventually the codebase grows, things slow down and I lose that "Python" feeling when it starts taking a dozen seconds to compile. It just breaks my concentration having to wait for something to finish. Hell I can't believe how outdated the compiler technology is. I can play incredibly realistic and interactive 3D games in real-time with practically no input lag, but I have to wait a dozen seconds for a tool to convert lines of text into object code? From a syntax perspective D has moved forward but from a compilation perspective it hasn't innovated at all.
Jul 25 2012
prev sibling next sibling parent "David Piepgrass" <qwertie256 gmail.com> writes:
 I find it shocking that anyone would consider 15 seconds slow 
 to compile for a
 large program. Yes, D's builds are lightning fast in general, 
 and 15 seconds
 is probably a longer build, but calling 15 seconds 
 "slow-to-compile" just
 about blows my mind. 15 seconds for a large program is _fast_. 
 If anyone
 complains about a large program taking 15 seconds to build, 
 then they're just
 plain spoiled or naive. I've dealt with _Java_ apps which took 
 in the realm of
 10 minutes to compile, let alone C++ apps which take _hours_ to 
 compile. 15
 seconds is a godsend.

I agree with Andrej, 15 seconds *is* slow for a edit-compile-run cycle, although it might be understandable when editing code that uses a lot of CTFE and static foreach and reinstantiates templates with a crapton of different arguments. I am neither spoiled nor naive to think it can be done in under 15 seconds. Fully rebuilding all my C# code takes less than 10 seconds (okay, not a big program, but several smaller programs). Plus, it isn't just build times that concern me. In C# I'm used to having an IDE that immediately understands what I have typed, giving me error messages and keeping metadata about the program up-to-date within 2 seconds. I can edit a class definition in file A and get code completion for it in file B, 2 seconds later. I don't expect the IDE can ever do that if the compiler can't do a debug build in a similar timeframe.
Jul 25 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, July 25, 2012 14:57:23 Andrej Mitrovic wrote:
 Hell I can't believe how outdated the compiler technology is. I can
 play incredibly realistic and interactive 3D games in real-time with
 practically no input lag, but I have to wait a dozen seconds for a
 tool to convert lines of text into object code? From a syntax
 perspective D has moved forward but from a compilation perspective it
 hasn't innovated at all.

And dmc and dmd are lightning fast in comparison to most compilers. I think that a lot of it comes down to the fact that optimizing code is _expensive_, and doing a lot of operations on an AST isn't necessarily all that cheap either. dmd is actually _lightning_ fast at processing text. That's not what's slow. It's everything which is after that which is. And for most compilers, the speed of the resultant code matters a lot more than the speed of compilation. Compare this to games which need to maintain a certain number of FPS. They optimize _everything_ towards that goal, which is why they achieve it. There's also no compiler equivalent of parallelizing optimizations to the AST or asm like games have parallelizing geometric computations and the like with GPUs. The priorities are completely different, what they're doing is very different, and what they have to work with is very different. As great as it would be if compilers were faster, it's an apples to oranges comparison. - Jonathan M Davis
Jul 25 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, July 25, 2012 17:35:09 David Piepgrass wrote:
 I find it shocking that anyone would consider 15 seconds slow
 to compile for a
 large program. Yes, D's builds are lightning fast in general,
 and 15 seconds
 is probably a longer build, but calling 15 seconds
 "slow-to-compile" just
 about blows my mind. 15 seconds for a large program is _fast_.
 If anyone
 complains about a large program taking 15 seconds to build,
 then they're just
 plain spoiled or naive. I've dealt with _Java_ apps which took
 in the realm of
 10 minutes to compile, let alone C++ apps which take _hours_ to
 compile. 15
 seconds is a godsend.

I agree with Andrej, 15 seconds *is* slow for a edit-compile-run cycle, although it might be understandable when editing code that uses a lot of CTFE and static foreach and reinstantiates templates with a crapton of different arguments. I am neither spoiled nor naive to think it can be done in under 15 seconds. Fully rebuilding all my C# code takes less than 10 seconds (okay, not a big program, but several smaller programs).

Sure, smaller programs should should build quickly, and having build times get slower as the program grows can definitely be a problem. I'm not about to argue with that. But having a _large_ application build in 15 seconds is arguably a luxory. Large applications just aren't the sort of thing that builds quickly. But that's the sort of project that's usually commercial (either that or a major open source one), and I don't think that D's been used in that domain a lot yet. While D compiles far faster than C++, the kind of application which takes hours to compile in C++ and the one that takes 10+ seconds in D are on a completely different level in terms of amount of source code and the level of complexity, even if D _would_ probably only take minutes on a similar project instead of hours. - Jonathan M Davis
Jul 25 2012
prev sibling next sibling parent "David Piepgrass" <qwertie256 gmail.com> writes:
Thanks for the very good description, Nick! So if I understand 
correctly, if

1. I use an "auto" return value or suchlike in a module Y.d
2. module X.d calls this function
3. I call "dmd -c X.d" and "dmd -c Y.d" as separate steps

Then the compiler will have to fully parse Y twice and fully 
analyze the Y function twice, although it generates object code 
for the function only once. Right? I wonder how smart it is about 
not analyzing things it does not need to analyze (e.g. when Y is 
a big module but X only calls one function from it - the compiler 
has to parse Y fully but it should avoid most of the semantic 
analysis.)

What about templates? In C++ it is a problem that the compiler 
will instantiate templates repeatedly, say if I use 
vector<string> in 20 source files, the compiler will generate and 
store 20 copies of vector<string> (plus 20 copies of 
basic_string<char>, too) in object files.

1. So in D, if I compile the 20 sources separately, does the same 
thing happen (same collection template instantiated 20 times with 
all 20 copies stored)?
2. If I compile the 20 sources all together, I guess the template 
would be instantiated just once, but then which .obj file does 
the instantiated template go in?

 $rdmd --build-only (any other flags) main.d

 Then, RDMD will figure out *all* of the source files needed 
 (using
 the full compiler's frontend, so it never gets fooled into 
 missing
 anything), and if any of them have been changed, it will 
 automatically
 pass them *all* into DMD for you. This way, you don't have to
 manually keep track of all your files and pass them all into
 DMD youself. Just give RDMD your main file and that's it, 
 you're golden.

 Side note: Another little trick with RDMD: Omit the 
 --build-only and it will compile AND then run your program:

 Yes. (Unless you never import anything from in phobos...I 
 think.) But
 it's very, very fast to parse. Lightning-speed if you compare 
 it to C++.

I don't even want to legitimize C++ compiler speed by comparing it to any other language ;)
 - Is there any concept of an incremental build?

Yes, but there's a few "gotcha"s: 1. D compiles so damn fast that it's not nearly as much of an issue as it is with C++ (which is notoriously ultra-slow compared to...everything, hence the monumental importance of C++'s incremental builds).

I figure as CTFE is used more, especially when it is used to decide which template overloads are valid or how a mixin will behave, this will slow down the compiler more and more, thus making incremental builds more important. A typical example would be a compile-time parser-generator, or compiled regexes. Plus, I've heard some people complaining that the compiler uses over 1 GB RAM, and splitting up compilation into parts might help with that. BTW, I think I heard the compiler uses multithreading to speed up the build, is that right?
 It keeps diving deeper and deeper to find anything it can 
 "start" with.
 One it finds that, it'll just build everything back up in 
 whatever
 order is necessary.

I hope someone can give more details about this.
 - In light of the above (that the meaning of D code can be 
 interdependent with other D code, plus the presence of mixins 
 and all that), what are the limitations of 
 __traits(allMembers...) and other compile-time reflection 
 operations, and what kind of problems might a user expect to 
 encounter?

Shouldn't really be an issue. Such things won't get evaluated until the types/identifiers involved are *fully* analyzed (or at least to the extent that they need to be analyzed). So the results of things like __traits(allMembers...) should *never* change during compilation, or when changing the order of files or imports (unless there's some compiler bug). Any situation that *would* result in any such ambiguity will get flagged as an error in your code.

Hmm. Well, I couldn't find an obvious example... for example, you are right, this doesn't work, although the compiler annoyingly doesn't give a reason: struct OhCrap { void a() {} // main.d(72): Error: error evaluating static if expression // (what error? syntax error? type error? c'mon...) static if ([ __traits(allMembers, OhCrap) ].length > 1) { auto b() { return 2; } } void c() {} } But won't this be a problem when it comes time to produce run-time reflection information? I mean, when module A asks to create run-time reflection information for all the functions and types in module A.... er, I naively thought the information would be created as a set of types and functions *in module A*, which would then change the set of allMembers of A. But, maybe it makes more sense to create that stuff in a different module (which A could then import??) Anyway, I can't even figure out how to enumerate the members of a module A; __traits(allMembers, A) causes "Error: import Y has no members". Aside: I first wrote the above code as follows: // Shouldn't this be in Phobos somewhere? bool contains(alias pred = "a == b", R, E)(R haystack, E needle) if (isInputRange!R && is(typeof(binaryFun!pred(haystack.front, needle)) : bool)) { return !(find!(pred, R, E)(haystack, needle).empty); } struct OhCrap { void a() {} static if ([ __traits(allMembers, OhCrap) ].contains("a")) { auto b() { return 2; } } void c() {} } But it causes a series of 204 error messages that I don't understand.
Jul 25 2012
prev sibling next sibling parent "Roman D. Boiko" <rb d-coding.com> writes:
On Wednesday, 25 July 2012 at 19:54:31 UTC, David Piepgrass wrote:
 It keeps diving deeper and deeper to find anything it can 
 "start" with.
 One it finds that, it'll just build everything back up in 
 whatever
 order is necessary.

I hope someone can give more details about this.

TDPL chapter 11 "Scaling Up".
Jul 25 2012
prev sibling next sibling parent "David Piepgrass" <qwertie256 gmail.com> writes:
 If you use rdmd to compile (instead of dmd), you *just* give it
 your *one* main source file (typically the one with your 
 "main()"
 function). This file must be the *last* parameter passed to 
 rdmd:

 $rdmd --build-only (any other flags) main.d

 Then, RDMD will figure out *all* of the source files needed 
 (using
 the full compiler's frontend, so it never gets fooled into 
 missing
 anything), and if any of them have been changed, it will 
 automatically
 pass them *all* into DMD for you. This way, you don't have to
 manually keep track of all your files and pass them all into
 DMD youself. Just give RDMD your main file and that's it, 
 you're golden.

I meant to ask, why would it recompile *all* of the source files if only one changed? Seems like it only should recompile the changed ones (but still compile them together as a unit.) Is it because of bugs (e.g. the template problem you mentioned)?
Jul 25 2012
prev sibling next sibling parent "David Piepgrass" <qwertie256 gmail.com> writes:
 I hope someone can give more details about this.

TDPL chapter 11 "Scaling Up".

That's where I was looking. As I said already, TDPL does not explain how compilation works, especially not anything about the low-level semantic analysis which has me most curious.
Jul 25 2012
prev sibling next sibling parent "Roman D. Boiko" <rb d-coding.com> writes:
On Wednesday, 25 July 2012 at 20:25:19 UTC, David Piepgrass wrote:
 I hope someone can give more details about this.

TDPL chapter 11 "Scaling Up".

That's where I was looking. As I said already, TDPL does not explain how compilation works, especially not anything about the low-level semantic analysis which has me most curious.

Strange, because it seems to me this chapter answers all your previous questions. What exact details are you interested in?
Jul 25 2012
prev sibling next sibling parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Wednesday, 25 July 2012 at 08:06:23 UTC, Nick Sabalausky wrote:
 Yea, my understanding is that full-build times measured in days 
 are (or
 used to be, don't know if they still are) also typical of 
 high-budget
 C++-based videogames.

You must be thinking of full data rebuilds, not code recompiles. There's no way a game could take over a day to compile and still produce an executable that would fit on a console. Several minutes is more typical. Maybe up to 30 minutes in bad cases.
Jul 25 2012
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Wed, 25 Jul 2012 23:20:04 +0200
"Peter Alexander" <peter.alexander.au gmail.com> wrote:

 On Wednesday, 25 July 2012 at 08:06:23 UTC, Nick Sabalausky wrote:
 Yea, my understanding is that full-build times measured in days 
 are (or
 used to be, don't know if they still are) also typical of 
 high-budget
 C++-based videogames.

You must be thinking of full data rebuilds, not code recompiles. There's no way a game could take over a day to compile and still produce an executable that would fit on a console. Several minutes is more typical. Maybe up to 30 minutes in bad cases.

Yea, you're probably right. I meant "full project", which almost certainly involves going through gigabytes of assets.
Jul 25 2012
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Wed, 25 Jul 2012 22:18:37 +0200
"David Piepgrass" <qwertie256 gmail.com> wrote:
 
 I meant to ask, why would it recompile *all* of the source files 
 if only one changed? Seems like it only should recompile the 
 changed ones (but still compile them together as a unit.) Is it 
 because of bugs (e.g. the template problem you mentioned)?

I'm not 100% certain, but, yes, I think it's a combination of that, and the fact that nobody's actually gone and tried to make that change to RDMD yet. AIUI, The original motivating purpose for RDMD was to be able to execute a D source file as if it were a script. So finding all relevant source files, passing them to DMD, etc, was all just necessary steps towards that end. Which turned out to also be useful in many cases for general project building.
Jul 25 2012
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Wed, 25 Jul 2012 21:54:29 +0200
"David Piepgrass" <qwertie256 gmail.com> wrote:

 Thanks for the very good description, Nick! So if I understand 
 correctly, if
 
 1. I use an "auto" return value or suchlike in a module Y.d
 2. module X.d calls this function
 3. I call "dmd -c X.d" and "dmd -c Y.d" as separate steps
 

See, now you're getting into some details that I'm not entirely familiar with ;)...
 Then the compiler will have to fully parse Y twice and fully 
 analyze the Y function twice, although it generates object code 
 for the function only once. Right?

That's my understanding of it, yes.
 I wonder how smart it is about 
 not analyzing things it does not need to analyze (e.g. when Y is 
 a big module but X only calls one function from it - the compiler 
 has to parse Y fully but it should avoid most of the semantic 
 analysis.)

I don't know how smart it is about that. If you have a template that never gets instantiated by *anything*, then I do know that semantic analysis won't get run on it since D's templates, like C++ templates (and unlike C#'s generics) can *only* be evaluated once they're instantiated. If, OTOH, you have a plain old function that never gets called, I'm guessing semantics probably still get run on it. Anything else: I dunno. :/
 
 What about templates? In C++ it is a problem that the compiler 
 will instantiate templates repeatedly, say if I use 
 vector<string> in 20 source files, the compiler will generate and 
 store 20 copies of vector<string> (plus 20 copies of 
 basic_string<char>, too) in object files.
 
 1. So in D, if I compile the 20 sources separately, does the same 
 thing happen (same collection template instantiated 20 times with 
 all 20 copies stored)?

Again, I'm not certain about this, other people would be able to answer better, but I *think* it works like this: If you pass all the files into DMD at once, then it'll only evaluate and generate code for vector<string> once. If you pass the files in as separate calls to DMD, then it's do semantic analysis on vector<string> twenty times, and I have no idea whether code will get generated one time or twenty times.
 2. If I compile the 20 sources all together, I guess the template 
 would be instantiated just once, but then which .obj file does 
 the instantiated template go in?
 

Unless things have been fixed since last I heared, this is actually the root of the problem with incremental compilation and templates. The compiler apparently makes some odd, or maybe inconsistent choices about what obj to stick the template into. I don't know the details of it though, just that in the past, people attempting to do incremental compilation have run into occasional linking issues that were traced back to problems in how DMD handles where to put instantiated templates.
 
 I don't even want to legitimize C++ compiler speed by comparing 
 it to any other language ;)
 

Fair enough :)
 - Is there any concept of an incremental build?

Yes, but there's a few "gotcha"s: 1. D compiles so damn fast that it's not nearly as much of an issue as it is with C++ (which is notoriously ultra-slow compared to...everything, hence the monumental importance of C++'s incremental builds).

I figure as CTFE is used more, especially when it is used to decide which template overloads are valid or how a mixin will behave, this will slow down the compiler more and more, thus making incremental builds more important. A typical example would be a compile-time parser-generator, or compiled regexes.

That's probably a fair assumption.
 Plus, I've heard some people complaining that the compiler uses 
 over 1 GB RAM, and splitting up compilation into parts might help 
 with that.
 

Yea, the problem is, DMD doesn't currently free any of the memory it takes, so mem usage just grows and grows. That's a known issue that needs to be taken care of at some point.
 BTW, I think I heard the compiler uses multithreading to speed up 
 the build, is that right?
 

Yes, it does. But someone else will have to explain how it actually uses multithreading, ie, what it multithreads, because I've got no clue ;) I think it's fairly coarse-grained, like on the module-level, but that's all I know.
 It keeps diving deeper and deeper to find anything it can 
 "start" with.
 One it finds that, it'll just build everything back up in 
 whatever
 order is necessary.

I hope someone can give more details about this.

I hope so too :)
Jul 25 2012
prev sibling next sibling parent Russel Winder <russel winder.org.uk> writes:
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Wed, 2012-07-25 at 01:03 -0700, Jonathan M Davis wrote:
[=E2=80=A6]
 I've heard of overnight builds, and I've heard of _regression tests_ runn=

 for over a week, but I've never heard of builds being over 2 days. Ouch.

Indeed the full test suite did take about a week to run. I think the core problem then was it was 2006, computers were slower, parallel compilation was not as well managed as multicore hadn't really taken hold, and they were doing the equivalent of trying -O2 and -O3 to see which space/time balance was best.
 It has got to have been possible to have a shorter build than that. Of co=

 if their code was bad enough that the build was that long, it may have be=

 rather disgusting code to clean up. But then again, maybe they genuinely =

 legitimate reason for having the build take that long. I'd be very surpri=

 though.

These were smart people, so my suspicion is very much that there was a necessary complexity. I think there was also an element of they were in the middle of a global refactoring. I suspect they have now had time to get stuff into a better state, but I do not know.
 In any case, much as I like C++ (not as much as D, but I still like it qu=

 bit), its build times are undeniably horrible.

Indeed, especially with -O2 or -O3. This is an area where VM + JIT can actually make things a lot better. Optimization happens on actually running code and is therefore focused on the "hot spot" rather than trying to optimize the entire code base. Java is doing this quite successfully, as is PyPy. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jul 26 2012
prev sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Thu, 26 Jul 2012 09:27:03 +0100
Russel Winder <russel winder.org.uk> wrote:

 On Wed, 2012-07-25 at 01:03 -0700, Jonathan M Davis wrote:
 
 In any case, much as I like C++ (not as much as D, but I still like
 it quite a bit), its build times are undeniably horrible.

Indeed, especially with -O2 or -O3. This is an area where VM + JIT can actually make things a lot better. Optimization happens on actually running code and is therefore focused on the "hot spot" rather than trying to optimize the entire code base. Java is doing this quite successfully, as is PyPy.

That's not something that actually necessitates a VM though. It's just that no native-compiled language (to my knowledge) has actually put something like that into its runtime yet.
Jul 26 2012