www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Compilation strategy

reply Russel Winder <russel winder.org.uk> writes:
A quick straw poll.  Do people prefer to have all sources compiled in a
single compiler call, or (more like C++) separate compilation of each
object followed by a link call.=20

Thanks.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Dec 15 2012
next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder 
wrote:
 Do people prefer to have all sources compiled in a
 single compiler call

I prefer the single call.
Dec 15 2012
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/15/12 11:55 AM, Russel Winder wrote:
 A quick straw poll.  Do people prefer to have all sources compiled in a
 single compiler call, or (more like C++) separate compilation of each
 object followed by a link call.

In phobos we use a single call for building the library. Then (at least on Posix) we use multiple calls for running unittests. Andrei
Dec 15 2012
parent "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 15 December 2012 at 17:02:08 UTC, Andrei 
Alexandrescu wrote:
 In phobos we use a single call for building the library. Then 
 (at least on Posix) we use multiple calls for running unittests.

This highlights the problem with giving a single answer to the question: Building a large project in one call is often impractical. It only works for Phobos library builds because many templates don't even get instantiated. David
Dec 15 2012
prev sibling next sibling parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder 
wrote:
 A quick straw poll.  Do people prefer to have all sources 
 compiled in a
 single compiler call, or (more like C++) separate compilation 
 of each
 object followed by a link call.

Single compiler call is easier for small projects, but I worry about compile times for larger projects...
Dec 15 2012
next sibling parent reply "Jakob Bornecrantz" <wallbraker gmail.com> writes:
On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander 
wrote:
 On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder 
 wrote:
 A quick straw poll.  Do people prefer to have all sources 
 compiled in a
 single compiler call, or (more like C++) separate compilation 
 of each
 object followed by a link call.

Single compiler call is easier for small projects, but I worry about compile times for larger projects...

As evident by Phobos and my own project[1], for larger projects multiple concurrent calls is the only way to go. Rebuilding everything does take a bit, but a bit of thought behind the layout of the project and things work much faster when working on specific areas. Cheers, Jakob. [1] http://github.com/Charged/Miners
Dec 15 2012
parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Saturday, 15 December 2012 at 17:27:38 UTC, Jakob Bornecrantz 
wrote:
 On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander
 Single compiler call is easier for small projects, but I worry 
 about compile times for larger projects...

As evident by Phobos and my own project[1], for larger projects multiple concurrent calls is the only way to go. Rebuilding everything does take a bit, but a bit of thought behind the layout of the project and things work much faster when working on specific areas.

Phobos is only around 200kloc. I'm worrying about the really large projects (multi-million lines of code).
Dec 15 2012
prev sibling parent reply "RenatoUtsch" <renatoutsch gmail.com> writes:
On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander 
wrote:
 On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder 
 wrote:
 A quick straw poll.  Do people prefer to have all sources 
 compiled in a
 single compiler call, or (more like C++) separate compilation 
 of each
 object followed by a link call.

Single compiler call is easier for small projects, but I worry about compile times for larger projects...

Yes, I'm writing a build system for D (that will be pretty damn good, I think, it has some interesting new concepts), and compiling each source separately to an object, and then linking everything will allow easily to make the build parallel, dividing the sources to compile in various threads. Or the compiler already does that if I pass all source files in one call? -- Renato Utsch
Dec 15 2012
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Dec 15, 2012 at 06:31:17PM +0100, RenatoUtsch wrote:
 On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander
 wrote:
On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder
wrote:
A quick straw poll.  Do people prefer to have all sources compiled
in a single compiler call, or (more like C++) separate compilation
of each object followed by a link call.

Single compiler call is easier for small projects, but I worry about compile times for larger projects...

Yes, I'm writing a build system for D (that will be pretty damn good, I think, it has some interesting new concepts), and compiling each source separately to an object, and then linking everything will allow easily to make the build parallel, dividing the sources to compile in various threads. Or the compiler already does that if I pass all source files in one call?

[...] I find that the current front-end (common to dmd, gdc, ldc) tends to work better when passed multiple source files at once. It tends to be faster, presumably because it only has to parse commonly-imported files once, and also produces smaller object/executable sizes -- maybe due to fewer duplicated template instantiations? I'm not sure of the exact reasons, but this behaviour appears consistent throughout dmd and gdc, and I presume also ldc (I didn't test that). So based on this, I'd lean toward compiling multiple files at once. However, in very large project, clearly this won't work very well. If it takes half an hour to build the entire system, it makes the code - compile - test cycle very slow, which reduces programmer productivity. So perhaps one possible middle ground would be to link packages separately, but compile all the sources within a single package at once. Presumably, if the project is properly organized, recompiling a single package won't take too long, and has the perk of optimizing for size within packages. This will probably also map to SCons easily, since SCons builds per-directory. T -- They pretend to pay us, and we pretend to work. -- Russian saying
Dec 15 2012
parent reply "RenatoUtsch" <renatoutsch gmail.com> writes:
On Saturday, 15 December 2012 at 18:00:58 UTC, H. S. Teoh wrote:
 On Sat, Dec 15, 2012 at 06:31:17PM +0100, RenatoUtsch wrote:
 On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander
 wrote:
On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder
wrote:
A quick straw poll.  Do people prefer to have all sources 
compiled
in a single compiler call, or (more like C++) separate 
compilation
of each object followed by a link call.

Single compiler call is easier for small projects, but I worry about compile times for larger projects...

Yes, I'm writing a build system for D (that will be pretty damn good, I think, it has some interesting new concepts), and compiling each source separately to an object, and then linking everything will allow easily to make the build parallel, dividing the sources to compile in various threads. Or the compiler already does that if I pass all source files in one call?

[...] I find that the current front-end (common to dmd, gdc, ldc) tends to work better when passed multiple source files at once. It tends to be faster, presumably because it only has to parse commonly-imported files once, and also produces smaller object/executable sizes -- maybe due to fewer duplicated template instantiations? I'm not sure of the exact reasons, but this behaviour appears consistent throughout dmd and gdc, and I presume also ldc (I didn't test that). So based on this, I'd lean toward compiling multiple files at once.

Yeah, I did read about this somewhee.
 However, in very large project, clearly this won't work very 
 well. If it
 takes half an hour to build the entire system, it makes the 
 code -
 compile - test cycle very slow, which reduces programmer 
 productivity.

 So perhaps one possible middle ground would be to link packages
 separately, but compile all the sources within a single package 
 at once.
 Presumably, if the project is properly organized, recompiling a 
 single
 package won't take too long, and has the perk of optimizing for 
 size
 within packages. This will probably also map to SCons easily, 
 since
 SCons builds per-directory.


 T

Well, the idea is good. Small projects usually don't have much packages, so there will be just a few compiler calls. And compiling files concurrently will only have a meaningful efect if the project is large, and a large project will have a lot of packages. Maybe adding an option to choose between compiling all sources at once, per package, or per source. For example, in development and debug builds the compilation is per file or package, but in release builds all sources are compiled at once, or various packages at once. This way release builds will take advantage of this behavior that the frontend has, but developers won't have productivity issues. And, of couse, the behaviour will not be fixed, the devs that are using the build system will choose that.
Dec 15 2012
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Dec 15, 2012 at 07:30:52PM +0100, RenatoUtsch wrote:
 On Saturday, 15 December 2012 at 18:00:58 UTC, H. S. Teoh wrote:

[...]
So perhaps one possible middle ground would be to link packages
separately, but compile all the sources within a single package at
once.  Presumably, if the project is properly organized, recompiling
a single package won't take too long, and has the perk of optimizing
for size within packages. This will probably also map to SCons
easily, since SCons builds per-directory.


[...]
 Well, the idea is good. Small projects usually don't have much
 packages, so there will be just a few compiler calls. And compiling
 files concurrently will only have a meaningful efect if the project
 is large, and a large project will have a lot of packages.

Yes, that's the idea behind it.
 Maybe adding an option to choose between compiling all sources at
 once, per package, or per source. For example, in development and
 debug builds the compilation is per file or package, but in release
 builds all sources are compiled at once, or various packages at once.
 
 This way release builds will take advantage of this behavior that
 the frontend has, but developers won't have productivity issues.
 And, of couse, the behaviour will not be fixed, the devs that are
 using the build system will choose that.

I forgot to mention also, that passing too many source files to the compiler may sometimes cause memory consumption issues, as the compiler has to hold everything in memory. This may not be practical for very large project, where you can't fit everything into RAM. T -- Stop staring at me like that! You'll offend... no, you'll hurt your eyes!
Dec 15 2012
parent "RenatoUtsch" <renatoutsch gmail.com> writes:
On Saturday, 15 December 2012 at 18:44:35 UTC, H. S. Teoh wrote:
 On Sat, Dec 15, 2012 at 07:30:52PM +0100, RenatoUtsch wrote:
 On Saturday, 15 December 2012 at 18:00:58 UTC, H. S. Teoh 
 wrote:

[...]
So perhaps one possible middle ground would be to link 
packages
separately, but compile all the sources within a single 
package at
once.  Presumably, if the project is properly organized, 
recompiling
a single package won't take too long, and has the perk of 
optimizing
for size within packages. This will probably also map to SCons
easily, since SCons builds per-directory.


[...]
 Well, the idea is good. Small projects usually don't have much
 packages, so there will be just a few compiler calls. And 
 compiling
 files concurrently will only have a meaningful efect if the 
 project
 is large, and a large project will have a lot of packages.

Yes, that's the idea behind it.
 Maybe adding an option to choose between compiling all sources 
 at
 once, per package, or per source. For example, in development 
 and
 debug builds the compilation is per file or package, but in 
 release
 builds all sources are compiled at once, or various packages 
 at once.
 
 This way release builds will take advantage of this behavior 
 that
 the frontend has, but developers won't have productivity 
 issues.
 And, of couse, the behaviour will not be fixed, the devs that 
 are
 using the build system will choose that.

I forgot to mention also, that passing too many source files to the compiler may sometimes cause memory consumption issues, as the compiler has to hold everything in memory. This may not be practical for very large project, where you can't fit everything into RAM. T

Well, so compiling by packages seem to be the best approach. When I return home I will do some tests to see what I can do. -- Renato
Dec 15 2012
prev sibling parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/15/2012 10:30 AM, RenatoUtsch wrote:
 Well, the idea is good. Small projects usually don't have much packages,
 so there will be just a few compiler calls. And compiling files
 concurrently will only have a meaningful efect if the project is large,
 and a large project will have a lot of packages.

 Maybe adding an option to choose between compiling all sources at once,
 per package, or per source. For example, in development and debug builds
 the compilation is per file or package, but in release builds all
 sources are compiled at once, or various packages at once.

I always thought it would be cool if my build tool could automatically determine the dependency graph of my d code and use it to intelligently break my project into small enough semi-independent chunks to compile.
Dec 16 2012
prev sibling next sibling parent reply "jerro" <a a.com> writes:
On Saturday, 15 December 2012 at 17:31:19 UTC, RenatoUtsch wrote:
 On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander 
 wrote:
 On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder 
 wrote:
 A quick straw poll.  Do people prefer to have all sources 
 compiled in a
 single compiler call, or (more like C++) separate compilation 
 of each
 object followed by a link call.

Single compiler call is easier for small projects, but I worry about compile times for larger projects...

Yes, I'm writing a build system for D (that will be pretty damn good, I think, it has some interesting new concepts)

I took a look at your github project, there isn't any code yet, but I like the concept. I was actually planing to do something similar, but since you are already doing it, I think my time would be better spent contributing to your project. Will there be some publicly available code in the near future?
Dec 15 2012
parent "RenatoUtsch" <renatoutsch gmail.com> writes:
On Saturday, 15 December 2012 at 18:24:50 UTC, jerro wrote:
 On Saturday, 15 December 2012 at 17:31:19 UTC, RenatoUtsch 
 wrote:
 On Saturday, 15 December 2012 at 17:05:59 UTC, Peter Alexander 
 wrote:
 On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder 
 wrote:
 A quick straw poll.  Do people prefer to have all sources 
 compiled in a
 single compiler call, or (more like C++) separate 
 compilation of each
 object followed by a link call.

Single compiler call is easier for small projects, but I worry about compile times for larger projects...

Yes, I'm writing a build system for D (that will be pretty damn good, I think, it has some interesting new concepts)

I took a look at your github project, there isn't any code yet, but I like the concept. I was actually planing to do something similar, but since you are already doing it, I think my time would be better spent contributing to your project. Will there be some publicly available code in the near future?

I expect to release a first alpha version in about 15~30 days, maybe less, it depends on how much time I will have on the rest of this month.
Dec 15 2012
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/15/2012 9:31 AM, RenatoUtsch wrote:
 Yes, I'm writing a build system for D (that will be pretty damn good, I think,
 it has some interesting new concepts), and compiling each source separately to
 an object, and then linking everything will allow easily to make the build
 parallel, dividing the sources to compile in various threads. Or the compiler
 already does that if I pass all source files in one call?

The compiler does a little multithreading, but not enough to make a difference. I've certainly thought about various schemes to parallelize it, though.
Dec 15 2012
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/15/2012 8:55 AM, Russel Winder wrote:
 A quick straw poll.  Do people prefer to have all sources compiled in a
 single compiler call, or (more like C++) separate compilation of each
 object followed by a link call.

Both are needed, and are suitable for different purposes. It's like asking if you prefer a standard or a philips screwdriver.
Dec 15 2012
prev sibling next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 15.12.2012 17:55, schrieb Russel Winder:
 A quick straw poll.  Do people prefer to have all sources compiled in a
 single compiler call, or (more like C++) separate compilation of each
 object followed by a link call.

 Thanks.

I prefer to compile by package, like in any module supporting language. -- Paulo
Dec 16 2012
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 15 December 2012 at 16:55:39 UTC, Russel Winder 
wrote:
 A quick straw poll.  Do people prefer to have all sources 
 compiled in a
 single compiler call, or (more like C++) separate compilation 
 of each
 object followed by a link call.

 Thanks.

I currently use full compilation. I have no choice because the compiler isn't emitting all symbols I need and get linking errors when doing separate compilation. Compiling my code require more than 2.5Gb of RAM now and is quite slow. To avoid multiple instantiation of the same template by the compiler, I guess the best option to me would be to compile on a by package basis.
Dec 16 2012