www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Reggae v0.0.5 super alpha: A build system in D

reply "Atila Neves" <atila.neves gmail.com> writes:
I wanted to work on this a little more before announcing it, but 
it seems I'm going to be busy working on trying to get 
unit-threaded into std.experimental so here it is:

http://code.dlang.org/packages/reggae

If you're wondering about the name, it's because it's supposed to 
build on dub.

You might wonder at some of the design decisions. Some of them 
are solutions to weird problems caused by writing build 
descriptions in a compiled language, others I'm not too sure of. 
Should compiler flags be an array of strings or a string? I got 
tired of typing square brackets so it's a string for now.

Please let me know if the API is suitable or not, preferably by 
trying to actually use it to build your software.

Existing dub projects might work by just doing this from a build 
directory of your choice: "reggae -b make /path/to/project". That 
should generate a Makefile (or equivalent Ninja ones if `-b 
ninja` is used) to do what `dub build` usually does. It _should_ 
work for all dub projects, but it doesn't right now. For at least 
a few projects it's due to bugs in `dub describe`. For others it 
might be bugs in reggae, I don't know yet. Any dub.json files 
that use dub configurations extensively is likely to not work.

Features:

. Make and Ninja backends (tup will be the next one)
. Automatically imports dub projects and writes the reggae build 
configuration
. Access to all objects to be built with dub (including 
dependencies) when writing custom builds (reggae does this itself)
. Out-of-tree builds, like CMake
. Arbitrary build rules but pre-built ease-of-use higher level 
targets
. Separate compilation. One file changes, only one file gets 
rebuilt
. Automatic dependency detection for D, C, and C++ source files
. Can build itself (but includes too many object files, another 
`dub describe` bug)

There are several runnable examples in the features directory, in 
the form of Cucumber tests. They include linking D code to C++.

I submitted a proposal to talk about this at DConf but I'll be 
talking about testing instead. Maybe next year? Anyway, destroy!

Atila
Apr 03 2015
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Friday, 3 April 2015 at 17:03:35 UTC, Atila Neves wrote:

 . Separate compilation. One file changes, only one file gets 
 rebuilt


This immediately has caught my eye as huge "no" in the 
description. We must ban C style separate compilation, there is 
simply no way to move forward otherwise. At the very least not 
endorse it in any way.
Apr 03 2015
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
Also I don't see any point in yet another meta build system. The 
very point of initial discussion was about getting D only 
cross-platform solution that won't require installing any 
additional software but working D compiler.
Apr 03 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Friday, 3 April 2015 at 17:13:41 UTC, Dicebot wrote:

 Also I don't see any point in yet another meta build system. 
 The very point of initial discussion was about getting D only 
 cross-platform solution that won't require installing any 
 additional software but working D compiler.


I was also thinking of a binary backend (producing a binary 
executable that does the build, kinda like what ctRegex does but 
for builds), and also something that just builds it on the spot.

The thing is, I want to get feedback on the API first and 
foremost, and delegating the whole 
do-I-or-do-I-not-need-to-build-it logic to programs that already 
do that (and well) first was the obvious (for me) choice.

Also, Ninja is _really_ fast.
Apr 03 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Friday, 3 April 2015 at 17:17:50 UTC, Atila Neves wrote:

 On Friday, 3 April 2015 at 17:13:41 UTC, Dicebot wrote:

 Also I don't see any point in yet another meta build system. 
 The very point of initial discussion was about getting D only 
 cross-platform solution that won't require installing any 
 additional software but working D compiler.


 I was also thinking of a binary backend (producing a binary 
 executable that does the build, kinda like what ctRegex does 
 but for builds), and also something that just builds it on the 
 spot.

 The thing is, I want to get feedback on the API first and 
 foremost, and delegating the whole 
 do-I-or-do-I-not-need-to-build-it logic to programs that 
 already do that (and well) first was the obvious (for me) 
 choice.

 Also, Ninja is _really_ fast.


The thing is, it may actually affect API. The way I have 
originally expected it, any legal D code would be allowed for 
build commands instead of pure DSL approach. So instead of 
providing high level abstraction like this:

const mainObj  = Target("main.o",  "dmd -I$project/src -c $in 
-of$out", Target("src/main.d"));
const mathsObj = Target("maths.o", "dmd -c $in -of$out", 
Target("src/maths.d"));
const app = Target("myapp", "dmd -of$out $in", [mainObj, 
mathsObj]);

.. you instead define dependency building blocks in D domain:

struct App
{
     enum  path = "./myapp";
     alias deps = Depends!(mainObj, mathsObj);

     static void generate()
     {
         import std.process;
         enforce(execute([ "dmd",  "-ofmyapp", deps[0].path, 
deps[1].path]).status);
     }
}

And provide higher level helper abstractions on top of that, 
tuned for D projects. This is just random syntax I have just 
invented for example of course. It is already possible to write 
decent cross-platform scripts in D - only dependency tracking 
library is missing. But of course that would make using other 
build systems as backends impossible.
Apr 03 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Friday, 3 April 2015 at 17:40:42 UTC, Dicebot wrote:

 On Friday, 3 April 2015 at 17:17:50 UTC, Atila Neves wrote:

 On Friday, 3 April 2015 at 17:13:41 UTC, Dicebot wrote:

 Also I don't see any point in yet another meta build system. 
 The very point of initial discussion was about getting D only 
 cross-platform solution that won't require installing any 
 additional software but working D compiler.


 I was also thinking of a binary backend (producing a binary 
 executable that does the build, kinda like what ctRegex does 
 but for builds), and also something that just builds it on the 
 spot.

 The thing is, I want to get feedback on the API first and 
 foremost, and delegating the whole 
 do-I-or-do-I-not-need-to-build-it logic to programs that 
 already do that (and well) first was the obvious (for me) 
 choice.

 Also, Ninja is _really_ fast.


 The thing is, it may actually affect API. The way I have 
 originally expected it, any legal D code would be allowed for 
 build commands instead of pure DSL approach. So instead of 
 providing high level abstraction like this:

 const mainObj  = Target("main.o",  "dmd -I$project/src -c $in 
 -of$out", Target("src/main.d"));
 const mathsObj = Target("maths.o", "dmd -c $in -of$out", 
 Target("src/maths.d"));
 const app = Target("myapp", "dmd -of$out $in", [mainObj, 
 mathsObj]);

 .. you instead define dependency building blocks in D domain:

 struct App
 {
     enum  path = "./myapp";
     alias deps = Depends!(mainObj, mathsObj);

     static void generate()
     {
         import std.process;
         enforce(execute([ "dmd",  "-ofmyapp", deps[0].path, 
 deps[1].path]).status);
     }
 }

 And provide higher level helper abstractions on top of that, 
 tuned for D projects. This is just random syntax I have just 
 invented for example of course. It is already possible to write 
 decent cross-platform scripts in D - only dependency tracking 
 library is missing. But of course that would make using other 
 build systems as backends impossible.


Well, I took your advice (and one of my acceptance tests is based 
off of your simplified real-work example) and started with the 
low-level any-command-will-do API first. I built the high-level 
ones on top of that. It doesn't seem crazy to me that certain 
builds can only be done by certain backends. The fact that the 
make backend can track C/C++/D dependencies wasn't a given and 
the implementation is quite ugly.

In any case, the Target structs aren't high-level abstractions, 
they're just data. Data that can be generated by any code. Your 
example is basically how the `dExe` rule works: run dmd at 
run-time, collect dependencies and build all the `Target` 
instances. You could have a D backend that outputs (then compiles 
and runs) your example. The "only" problem I can see is execution 
speed.

Maybe I didn't include enough examples.

I also need to think of your example a bit more.
Apr 03 2015
parent "Dicebot" <public dicebot.lv> writes:
On Friday, 3 April 2015 at 17:59:22 UTC, Atila Neves wrote:

 Well, I took your advice (and one of my acceptance tests is 
 based off of your simplified real-work example) and started 
 with the low-level any-command-will-do API first. I built the 
 high-level ones on top of that. It doesn't seem crazy to me 
 that certain builds can only be done by certain backends. The 
 fact that the make backend can track C/C++/D dependencies 
 wasn't a given and the implementation is quite ugly.

 In any case, the Target structs aren't high-level abstractions, 
 they're just data. Data that can be generated by any code. Your 
 example is basically how the `dExe` rule works: run dmd at 
 run-time, collect dependencies and build all the `Target` 
 instances. You could have a D backend that outputs (then 
 compiles and runs) your example. The "only" problem I can see 
 is execution speed.

 Maybe I didn't include enough examples.

 I also need to think of your example a bit more.


I may have misunderstood how it works judging only by provided 
examples. Give a me bit more time to investigate actual sources 
and I may reconsider :)
Apr 03 2015
prev sibling next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Friday, 3 April 2015 at 17:10:33 UTC, Dicebot wrote:

 On Friday, 3 April 2015 at 17:03:35 UTC, Atila Neves wrote:

 . Separate compilation. One file changes, only one file gets 
 rebuilt


 This immediately has caught my eye as huge "no" in the 
 description. We must ban C style separate compilation, there is 
 simply no way to move forward otherwise. At the very least not 
 endorse it in any way.


I understand that. But:

1. One of D's advantages is fast compilation. I don't think that 
means we should should compile everything all the time because we 
can (it's fast anyway!)
2. There are measureable differences in compile-time. While 
working on reggae I got much faster edit-compile-unittest cycles 
because of separate compilation
3. This is valuable feedback. I was wondering what everybody else 
would think. It could be configureable, your "not endorse it in 
any way" notwithstanding. I for one would rather have it compile 
separately
4. CTFE and memory consumption can go through the roof 
(anecdotally anyway, it's never been a problem for me) when 
compiling everything at once.
Apr 03 2015
parent "Dicebot" <public dicebot.lv> writes:
On Friday, 3 April 2015 at 17:22:42 UTC, Atila Neves wrote:

 On Friday, 3 April 2015 at 17:10:33 UTC, Dicebot wrote:

 On Friday, 3 April 2015 at 17:03:35 UTC, Atila Neves wrote:

 . Separate compilation. One file changes, only one file gets 
 rebuilt


 This immediately has caught my eye as huge "no" in the 
 description. We must ban C style separate compilation, there 
 is simply no way to move forward otherwise. At the very least 
 not endorse it in any way.


 I understand that. But:

 1. One of D's advantages is fast compilation. I don't think 
 that means we should should compile everything all the time 
 because we can (it's fast anyway!)
 2. There are measureable differences in compile-time. While 
 working on reggae I got much faster edit-compile-unittest 
 cycles because of separate compilation
 3. This is valuable feedback. I was wondering what everybody 
 else would think. It could be configureable, your "not endorse 
 it in any way" notwithstanding. I for one would rather have it 
 compile separately
 4. CTFE and memory consumption can go through the roof 
 (anecdotally anyway, it's never been a problem for me) when 
 compiling everything at once.


See 
http://forum.dlang.org/post/nhaoahnqucqkjgdwtxsa forum.dlang.org

tl; dr: separate compilation support is necessary, but not at 
single module level.
Apr 03 2015
prev sibling next sibling parent reply Ben Boeckel via Digitalmars-d-announce writes:
On Fri, Apr 03, 2015 at 17:10:31 +0000, Dicebot via Digitalmars-d-announce
wrote:

 On Friday, 3 April 2015 at 17:03:35 UTC, Atila Neves wrote:

 . Separate compilation. One file changes, only one file gets 
 rebuilt


 
 This immediately has caught my eye as huge "no" in the 
 description. We must ban C style separate compilation, there is 
 simply no way to move forward otherwise. At the very least not 
 endorse it in any way.


Why? Other than the -fversion=... stuff, what is really blocking this? I
personally find unity builds to not be worth it, but I don't see
anything blocking separate compilation for D if dependencies are set up
properly.

--Ben
Apr 03 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Friday, 3 April 2015 at 17:25:51 UTC, Ben Boeckel wrote:

 On Fri, Apr 03, 2015 at 17:10:31 +0000, Dicebot via 
 Digitalmars-d-announce wrote:

 On Friday, 3 April 2015 at 17:03:35 UTC, Atila Neves wrote:

 . Separate compilation. One file changes, only one file gets 
 rebuilt


 
 This immediately has caught my eye as huge "no" in the 
 description. We must ban C style separate compilation, there 
 is simply no way to move forward otherwise. At the very least 
 not endorse it in any way.


 Why? Other than the -fversion=... stuff, what is really 
 blocking this? I
 personally find unity builds to not be worth it, but I don't see
 anything blocking separate compilation for D if dependencies 
 are set up
 properly.

 --Ben


There are 2 big problems with C-style separate compilation:

1)

Complicates whole-program optimization possibilities. Old school 
object files are simply not good enough to preserve information 
necessary to produce optimized builds and we are not in position 
to create own metadata + linker combo to circumvent that. This 
also applies to attribute inference which has become a really 
important development direction to handle growing attribute hell.

During last D Berlin Meetup we had an interesting conversation on 
attribute inference topic with Martin Nowak and dropping legacy 
C-style separate compilation seemed to be recognized as 
unavoidable to implement anything decent in that domain.

2)

Ironically, it is just very slow. Those who come from C world got 
used to using separate compilation to speed up rebuilds but it 
doesn't work that way in D. It may look better if you change only 
1 or 2 module but as amount of modified modules grows, 
incremental rebuild quickly becomes _slower_ than full program 
build with all files processed in one go. It can sometimes result 
in order of magnitude slowdown (personal experience).

Difference from C is that repeated imports are very cheap in D 
(you don't copy-paste module content again and again like with 
headers) but at the same time semantic analysis of imported 
module is more expensive (because D semantics are more 
complicated). When you do separate compilation you discard 
already processed imports and repeat it again and again from the 
very beginning for each new compiled file, accumulating huge 
slowdown for application in total.

To get best compilation speed in D you want to process as many 
modules with shared imports at one time as possible. At the same 
time for really big projects it becomes not feasible at some 
point, especially if CTFE is heavily used and memory consumption 
explodes. In that case best approach is partial separate 
compilation - decoupling parts of a program as static libraries 
and doing parallel compilation of each separate library - but 
still compiling each library in one go. That allows to get 
parallelization without doing the same costly work again and 
again.
Apr 03 2015
next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Friday, 3 April 2015 at 17:55:00 UTC, Dicebot wrote:

 On Friday, 3 April 2015 at 17:25:51 UTC, Ben Boeckel wrote:

 On Fri, Apr 03, 2015 at 17:10:31 +0000, Dicebot via 
 Digitalmars-d-announce wrote:

 On Friday, 3 April 2015 at 17:03:35 UTC, Atila Neves wrote:

 . Separate compilation. One file changes, only one file 
 gets rebuilt


 
 This immediately has caught my eye as huge "no" in the 
 description. We must ban C style separate compilation, there 
 is simply no way to move forward otherwise. At the very least 
 not endorse it in any way.


 Why? Other than the -fversion=... stuff, what is really 
 blocking this? I
 personally find unity builds to not be worth it, but I don't 
 see
 anything blocking separate compilation for D if dependencies 
 are set up
 properly.

 --Ben


 There are 2 big problems with C-style separate compilation:

 1)

 Complicates whole-program optimization possibilities. Old 
 school object files are simply not good enough to preserve 
 information necessary to produce optimized builds and we are 
 not in position to create own metadata + linker combo to 
 circumvent that. This also applies to attribute inference which 
 has become a really important development direction to handle 
 growing attribute hell.

 During last D Berlin Meetup we had an interesting conversation 
 on attribute inference topic with Martin Nowak and dropping 
 legacy C-style separate compilation seemed to be recognized as 
 unavoidable to implement anything decent in that domain.

 2)

 Ironically, it is just very slow. Those who come from C world 
 got used to using separate compilation to speed up rebuilds but 
 it doesn't work that way in D. It may look better if you change 
 only 1 or 2 module but as amount of modified modules grows, 
 incremental rebuild quickly becomes _slower_ than full program 
 build with all files processed in one go. It can sometimes 
 result in order of magnitude slowdown (personal experience).

 Difference from C is that repeated imports are very cheap in D 
 (you don't copy-paste module content again and again like with 
 headers) but at the same time semantic analysis of imported 
 module is more expensive (because D semantics are more 
 complicated). When you do separate compilation you discard 
 already processed imports and repeat it again and again from 
 the very beginning for each new compiled file, accumulating 
 huge slowdown for application in total.

 To get best compilation speed in D you want to process as many 
 modules with shared imports at one time as possible. At the 
 same time for really big projects it becomes not feasible at 
 some point, especially if CTFE is heavily used and memory 
 consumption explodes. In that case best approach is partial 
 separate compilation - decoupling parts of a program as static 
 libraries and doing parallel compilation of each separate 
 library - but still compiling each library in one go. That 
 allows to get parallelization without doing the same costly 
 work again and again.


Interesting.

It's true that it's not always faster to compile each module 
separately, I already knew that. It seems to me, however, that 
when that's actually the case, the practical difference is 
negligible. Even if 10x slower, the linker will take longer 
anyway. Because it'll all still be under a second. That's been my 
experience anyway. i.e. It's either faster or it doesn't make 
much of a difference.

All I know is I've seen a definite improvement in my 
edit-compile-unittest cycle by compiling modules separately.

How would the decoupling happen? Is the user supposed to 
partition the binary into suitable static libraries? Or is the 
system supposed to be smart enough to figure that out?

Atila
Apr 03 2015
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-04-03 20:06, Atila Neves wrote:


 Interesting.

 It's true that it's not always faster to compile each module separately,
 I already knew that. It seems to me, however, that when that's actually
 the case, the practical difference is negligible. Even if 10x slower,
 the linker will take longer anyway. Because it'll all still be under a
 second. That's been my experience anyway. i.e. It's either faster or it
 doesn't make much of a difference.


I just tried compiling one of my project. It has a makefile that does 
separate compilation and a shell script I use for unit testing which 
compiles everything in one go. The makefile takes 5.3 seconds, does not 
including linking since it builds a library. The shell script takes 1.3 
seconds which include compiling unit tests and linking as well.

-- 
/Jacob Carlborg
Apr 03 2015
next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Friday, 3 April 2015 at 19:07:09 UTC, Jacob Carlborg wrote:

 On 2015-04-03 20:06, Atila Neves wrote:


 Interesting.

 It's true that it's not always faster to compile each module 
 separately,
 I already knew that. It seems to me, however, that when that's 
 actually
 the case, the practical difference is negligible. Even if 10x 
 slower,
 the linker will take longer anyway. Because it'll all still be 
 under a
 second. That's been my experience anyway. i.e. It's either 
 faster or it
 doesn't make much of a difference.


 I just tried compiling one of my project. It has a makefile 
 that does separate compilation and a shell script I use for 
 unit testing which compiles everything in one go. The makefile 
 takes 5.3 seconds, does not including linking since it builds a 
 library. The shell script takes 1.3 seconds which include 
 compiling unit tests and linking as well.


change one file and see which one is faster with an incremental 
build.
Apr 03 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Friday, 3 April 2015 at 19:08:58 UTC, weaselcat wrote:

 I just tried compiling one of my project. It has a makefile 
 that does separate compilation and a shell script I use for 
 unit testing which compiles everything in one go. The makefile 
 takes 5.3 seconds, does not including linking since it builds 
 a library. The shell script takes 1.3 seconds which include 
 compiling unit tests and linking as well.


 change one file and see which one is faster with an incremental 
 build.


I don't care if incremental build is 10x faster if full build 
still stays at ~1 second. However I do care (and consider 
unacceptable) if support for incremental builds makes full build 
10 seconds long.
Apr 03 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Friday, 3 April 2015 at 19:54:09 UTC, Dicebot wrote:

 On Friday, 3 April 2015 at 19:08:58 UTC, weaselcat wrote:

 I just tried compiling one of my project. It has a makefile 
 that does separate compilation and a shell script I use for 
 unit testing which compiles everything in one go. The 
 makefile takes 5.3 seconds, does not including linking since 
 it builds a library. The shell script takes 1.3 seconds which 
 include compiling unit tests and linking as well.


 change one file and see which one is faster with an 
 incremental build.


 I don't care if incremental build is 10x faster if full build 
 still stays at ~1 second. However I do care (and consider 
 unacceptable) if support for incremental builds makes full 
 build 10 seconds long.


I'm of the opposite opinion. I don't care if full builds take 1h 
as long as incremental builds are as fast as possible. Why would 
I keep doing full builds? That's like git cloning multiple times. 
What for?

What's clear is that I need to try Andrei's per-package idea, at 
least as an option, if not the default. Having a large D codebase 
to test it on would be nice as well, but I don't know of anything 
bigger than Phobos.

Atila
Apr 04 2015
next sibling parent "Kagamin" <spam here.lot> writes:
On Saturday, 4 April 2015 at 07:44:12 UTC, Atila Neves wrote:

 I'm of the opposite opinion. I don't care if full builds take 
 1h as long as incremental builds are as fast as possible. Why 
 would I keep doing full builds? That's like git cloning 
 multiple times. What for?


Full build is important when you do it only once, e.g. if you 
want to try new version of a program and it's not precompiled, 
you'll need to compile it from source and never recompile.
Apr 04 2015
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 4 April 2015 at 07:44:12 UTC, Atila Neves wrote:

 On Friday, 3 April 2015 at 19:54:09 UTC, Dicebot wrote:

 On Friday, 3 April 2015 at 19:08:58 UTC, weaselcat wrote:

 I just tried compiling one of my project. It has a makefile 
 that does separate compilation and a shell script I use for 
 unit testing which compiles everything in one go. The 
 makefile takes 5.3 seconds, does not including linking since 
 it builds a library. The shell script takes 1.3 seconds 
 which include compiling unit tests and linking as well.


 change one file and see which one is faster with an 
 incremental build.


 I don't care if incremental build is 10x faster if full build 
 still stays at ~1 second. However I do care (and consider 
 unacceptable) if support for incremental builds makes full 
 build 10 seconds long.


 I'm of the opposite opinion. I don't care if full builds take 
 1h as long as incremental builds are as fast as possible. Why 
 would I keep doing full builds? That's like git cloning 
 multiple times. What for?

 What's clear is that I need to try Andrei's per-package idea, 
 at least as an option, if not the default. Having a large D 
 codebase to test it on would be nice as well, but I don't know 
 of anything bigger than Phobos.


At work I often switch between dozen of different projects a day 
with small chunk of changes for each. That means that incremental 
builds are never of any value.

Even if you consistently work with the same project it is 
incredibly rare to have a changeset contained in a single module. 
And if there are at least 5 changed modules (including 
inter-dependencies) it becomes long enough already.

As for test codebase - I know that Martin has been testing his GC 
improvements on Higgs (https://github.com/higgsjs/Higgs), could 
be a suitable test subject for you too.
Apr 04 2015
next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Saturday, 4 April 2015 at 19:56:28 UTC, Dicebot wrote:

 On Saturday, 4 April 2015 at 07:44:12 UTC, Atila Neves wrote:

 On Friday, 3 April 2015 at 19:54:09 UTC, Dicebot wrote:

 On Friday, 3 April 2015 at 19:08:58 UTC, weaselcat wrote:

 I just tried compiling one of my project. It has a makefile 
 that does separate compilation and a shell script I use for 
 unit testing which compiles everything in one go. The 
 makefile takes 5.3 seconds, does not including linking 
 since it builds a library. The shell script takes 1.3 
 seconds which include compiling unit tests and linking as 
 well.


 change one file and see which one is faster with an 
 incremental build.


 I don't care if incremental build is 10x faster if full build 
 still stays at ~1 second. However I do care (and consider 
 unacceptable) if support for incremental builds makes full 
 build 10 seconds long.


 I'm of the opposite opinion. I don't care if full builds take 
 1h as long as incremental builds are as fast as possible. Why 
 would I keep doing full builds? That's like git cloning 
 multiple times. What for?

 What's clear is that I need to try Andrei's per-package idea, 
 at least as an option, if not the default. Having a large D 
 codebase to test it on would be nice as well, but I don't know 
 of anything bigger than Phobos.


 At work I often switch between dozen of different projects a 
 day with small chunk of changes for each. That means that 
 incremental builds are never of any value.

 Even if you consistently work with the same project it is 
 incredibly rare to have a changeset contained in a single 
 module. And if there are at least 5 changed modules (including 
 inter-dependencies) it becomes long enough already.

 As for test codebase - I know that Martin has been testing his 
 GC improvements on Higgs (https://github.com/higgsjs/Higgs), 
 could be a suitable test subject for you too.


It seems our workflows are very different. Half of the time I 
make changes to a file that only contains unit tests. That's 
always self contained, and doing anything else except for 
recompiling that one file and relinking is going to be slower.

It seems to me that different projects might benefit from 
different compilation strategies. It might just be a case of unit 
tests alongside production code vs in separate files. As 
mentioned before, my experience with per-module compilation was 
usually faster, but I'm going to change the default to be per 
package.

Another cool thing about using reggae to build itself was 
building the unit test and production binaries at the same time. 
I couldn't really do that with dub alone.
Apr 04 2015
parent reply "Sergei Nosov" <sergei.nosov gmail.com> writes:
On Sunday, 5 April 2015 at 00:22:35 UTC, Atila Neves wrote:

 It seems to me that different projects might benefit from 
 different compilation strategies. It might just be a case of 
 unit tests alongside production code vs in separate files. As 
 mentioned before, my experience with per-module compilation was 
 usually faster, but I'm going to change the default to be per 
 package.


I want to also share my experience in that regard.

When I was writing a vibe.d based application, I used dub as a 
build system, which sends everything in one go. My application 
was just a couple of files, so I was, practically, just building 
vibe every time.

I was developing the application on a desktop with 4 Gb RAM and 
everything was fine (albeit I was missing the "progress bar" of 
files in progress provided by ninja/make).

But then it was time to deploy the app, and I bought a 1 GB RAM 
virtual node from Linode. After executing dub it told me "Out of 
memory" and exited. And there was nothing I could do.

So I took the only option I saw - I switched to CMake (modified 
for working with D) to provide me a separate compilation build 
(ninja-based) and swore to never again.

I understand the reasoning behind both separate and "throw in 
everything" compilation strategies. And I also understand the 
pros of a middle-ground solution (like, per-package one), which 
is probably the way D will go. But this area seems kind of gray 
to me (like, in my case the "per-package" solution wouldn't work 
either, if I understand it correctly).

So, personally, I will probably stick to separate compilation, 
until I see that:

- The pros of "batch" compilation are clear and, desirably, 
obvious. At the moment it seems to me (seems to me), that faster 
compilation and attribute inference just don't have a significant 
impact.
- There's a way to fine tune between "separate" and "throw in 
everything" compilation if necessary.

Thanks!
Apr 06 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 6 April 2015 at 11:29:20 UTC, Sergei Nosov wrote:

 On Sunday, 5 April 2015 at 00:22:35 UTC, Atila Neves wrote:

 It seems to me that different projects might benefit from 
 different compilation strategies. It might just be a case of 
 unit tests alongside production code vs in separate files. As 
 mentioned before, my experience with per-module compilation 
 was usually faster, but I'm going to change the default to be 
 per package.


 I want to also share my experience in that regard.

 ...


See, the problem with this approach is that you can trivially get 
out of 1GB of memory with DMD even when compiling single module, 
all you need is to do enough compile-time magic. Separate 
compilation here delays the issue but does not actually solve it.

If any effort is to be put into supporting this scenario 
(on-server compilation), it is better to be put in reducing 
actual memory hog of compiler, not supporting another workaround.

Also you can still achieve the similar profile by splitting your 
project in small enough static libraries, so it is not completely 
out of question.
Apr 07 2015
parent "Sergei Nosov" <sergei.nosov gmail.com> writes:
On Tuesday, 7 April 2015 at 08:25:02 UTC, Dicebot wrote:

 See, the problem with this approach is that you can trivially 
 get out of 1GB of memory with DMD even when compiling single 
 module, all you need is to do enough compile-time magic. 
 Separate compilation here delays the issue but does not 
 actually solve it.


Yeah, absolutely agree. But at the moment separate compilation is 
the most "forgiving" one. Like, if it doesn't work - anything 
else won't work either. And given that personally I don't 
recognize the (possibly) increased compilation time as an issue, 
it's the solution that works for me.


 If any effort is to be put into supporting this scenario 
 (on-server compilation), it is better to be put in reducing 
 actual memory hog of compiler, not supporting another 
 workaround.


Agreed, too. The whole "forget about frees" approach sounds a 
little too controversial to me. Especially, after I have faced 
the dark side of it. So, I'm all for improving in that regard. 
But it seems like it's not recognized as a (high-priority) issue 
at the moment. So, we (the users) have to live with that.


 Also you can still achieve the similar profile by splitting 
 your project in small enough static libraries, so it is not 
 completely out of question.


As I described, my project was just a couple of files. Building 
vibe.d was the actual problem. I don't think it is feasible to 
expect that a user of a library will start splitting it into 
"small enough libraries", when faced with this problem. A more 
structured approach is needed.
Apr 07 2015
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/4/15 12:56 PM, Dicebot wrote:

 Even if you consistently work with the same project it is incredibly
 rare to have a changeset contained in a single module. And if there are
 at least 5 changed modules (including inter-dependencies) it becomes
 long enough already.


That's my experience as well. -- Andrei
Apr 05 2015
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/3/15 12:07 PM, Jacob Carlborg wrote:

 On 2015-04-03 20:06, Atila Neves wrote:


 Interesting.

 It's true that it's not always faster to compile each module separately,
 I already knew that. It seems to me, however, that when that's actually
 the case, the practical difference is negligible. Even if 10x slower,
 the linker will take longer anyway. Because it'll all still be under a
 second. That's been my experience anyway. i.e. It's either faster or it
 doesn't make much of a difference.


 I just tried compiling one of my project. It has a makefile that does
 separate compilation and a shell script I use for unit testing which
 compiles everything in one go. The makefile takes 5.3 seconds, does not
 including linking since it builds a library. The shell script takes 1.3
 seconds which include compiling unit tests and linking as well.


Truth be told that's 5.3 seconds for an entire build so the comparison 
is only partially relevant. -- Andrei
Apr 03 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/3/15 11:06 AM, Atila Neves wrote:

 It's true that it's not always faster to compile each module separately,
 I already knew that. It seems to me, however, that when that's actually
 the case, the practical difference is negligible. Even if 10x slower,
 the linker will take longer anyway. Because it'll all still be under a
 second. That's been my experience anyway. i.e. It's either faster or it
 doesn't make much of a difference.


Whoa. The difference is much larger (= day and night) on at least a 
couple of projects at work.


 All I know is I've seen a definite improvement in my
 edit-compile-unittest cycle by compiling modules separately.

 How would the decoupling happen? Is the user supposed to partition the
 binary into suitable static libraries? Or is the system supposed to be
 smart enough to figure that out?


Smarts would be nice, but in first approximation one package = one 
compilation unit is a great policy.


Andrei
Apr 03 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Friday, 3 April 2015 at 19:49:04 UTC, Andrei Alexandrescu 
wrote:

 On 4/3/15 11:06 AM, Atila Neves wrote:

 It's true that it's not always faster to compile each module 
 separately,
 I already knew that. It seems to me, however, that when that's 
 actually
 the case, the practical difference is negligible. Even if 10x 
 slower,
 the linker will take longer anyway. Because it'll all still be 
 under a
 second. That's been my experience anyway. i.e. It's either 
 faster or it
 doesn't make much of a difference.


 Whoa. The difference is much larger (= day and night) on at 
 least a couple of projects at work.


Even when only one file has changed?

Atila
Apr 04 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/4/15 1:30 AM, Atila Neves wrote:

 On Friday, 3 April 2015 at 19:49:04 UTC, Andrei Alexandrescu wrote:

 On 4/3/15 11:06 AM, Atila Neves wrote:

 It's true that it's not always faster to compile each module separately,
 I already knew that. It seems to me, however, that when that's actually
 the case, the practical difference is negligible. Even if 10x slower,
 the linker will take longer anyway. Because it'll all still be under a
 second. That's been my experience anyway. i.e. It's either faster or it
 doesn't make much of a difference.


 Whoa. The difference is much larger (= day and night) on at least a
 couple of projects at work.


 Even when only one file has changed?


Yes; due to interdependencies, it's rare that only one file gets 
compiled. -- Andrei
Apr 04 2015
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 3 April 2015 at 18:06:42 UTC, Atila Neves wrote:

 All I know is I've seen a definite improvement in my 
 edit-compile-unittest cycle by compiling modules separately.

 How would the decoupling happen? Is the user supposed to 
 partition the binary into suitable static libraries? Or is the 
 system supposed to be smart enough to figure that out?


Ideally both. Build system should be smart enough to group into 
static libraries automatically if user doesn't care (Andrei 
suggestion of one package per library makes sense) but option of 
explicit definition of compilation units is still necessary of 
course.
Apr 03 2015
prev sibling next sibling parent "weaselcat" <weaselcat gmail.com> writes:
On Friday, 3 April 2015 at 17:55:00 UTC, Dicebot wrote:

 On Friday, 3 April 2015 at 17:25:51 UTC, Ben Boeckel wrote:

 On Fri, Apr 03, 2015 at 17:10:31 +0000, Dicebot via 
 Digitalmars-d-announce wrote:

 On Friday, 3 April 2015 at 17:03:35 UTC, Atila Neves wrote:

 . Separate compilation. One file changes, only one file 
 gets rebuilt


 
 This immediately has caught my eye as huge "no" in the 
 description. We must ban C style separate compilation, there 
 is simply no way to move forward otherwise. At the very least 
 not endorse it in any way.


 Why? Other than the -fversion=... stuff, what is really 
 blocking this? I
 personally find unity builds to not be worth it, but I don't 
 see
 anything blocking separate compilation for D if dependencies 
 are set up
 properly.

 --Ben


 There are 2 big problems with C-style separate compilation:

 1)

 Complicates whole-program optimization possibilities. Old 
 school object files are simply not good enough to preserve 
 information necessary to produce optimized builds and we are 
 not in position to create own metadata + linker combo to 
 circumvent that. This also applies to attribute inference which 
 has become a really important development direction to handle 
 growing attribute hell.


Not sure about other people, but I do not care about whole 
program optimization during an edit-compile-run cycle. I just 
want it to compile as fast as possible, and if I change one or 
two files I don't want to have to recompile an entire codebase.
Apr 03 2015
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-04-03 19:54, Dicebot wrote:


 2)

 Ironically, it is just very slow. Those who come from C world got used
 to using separate compilation to speed up rebuilds but it doesn't work
 that way in D. It may look better if you change only 1 or 2 module but
 as amount of modified modules grows, incremental rebuild quickly becomes
 _slower_ than full program build with all files processed in one go. It
 can sometimes result in order of magnitude slowdown (personal experience).


BTW, are all the issues with incremental rebuilds solved? I.e. templates 
not outputted to all object files and other problems I can't remember 
right now.

-- 
/Jacob Carlborg
Apr 04 2015
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Friday, 3 April 2015 at 17:55:00 UTC, Dicebot wrote:

 Complicates whole-program optimization possibilities. Old 
 school object files are simply not good enough to preserve 
 information necessary to produce optimized builds and we are 
 not in position to create own metadata + linker combo to 
 circumvent that.


Development builds are usually not whole-program optimized. And 
proper optimizers work with IR and see no problem in separate 
compilation, it's all transparent. Separate compilation is nice 
for RAM too - good in virtualized environment like a CI service.


 This also applies to attribute inference which has become a 
 really important development direction to handle growing 
 attribute hell.


Depends on code style.
Apr 04 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 4 April 2015 at 16:58:23 UTC, Kagamin wrote:

 On Friday, 3 April 2015 at 17:55:00 UTC, Dicebot wrote:

 Complicates whole-program optimization possibilities. Old 
 school object files are simply not good enough to preserve 
 information necessary to produce optimized builds and we are 
 not in position to create own metadata + linker combo to 
 circumvent that.


 Development builds are usually not whole-program optimized. And 
 proper optimizers work with IR and see no problem in separate 
 compilation, it's all transparent. Separate compilation is nice 
 for RAM too - good in virtualized environment like a CI service.


We need solutions that can be reasonably implemented with 
existing resources, not perfect solutions. Storing IR in object 
files and using custom linker is "correct" approach for WPO but 
it is currently unaffordable. Add compilation time problems and 
there seems to be no compelling reasons to go that route for now.


 This also applies to attribute inference which has become a 
 really important development direction to handle growing 
 attribute hell.


 Depends on code style.


I am not aware of any solutions based on coding style. Can you 
elaborate?
Apr 04 2015
parent reply "Kagamin" <spam here.lot> writes:
On Saturday, 4 April 2015 at 19:59:46 UTC, Dicebot wrote:

 We need solutions that can be reasonably implemented with 
 existing resources, not perfect solutions. Storing IR in object 
 files and using custom linker is "correct" approach for WPO but 
 it is currently unaffordable.


Works for me with llvm toolchain.


 Add compilation time problems and there seems to be no 
 compelling reasons to go that route for now.


A compelling reason is memory consumption and exhaustion.


 I am not aware of any solutions based on coding style.


Not sure what you mean, reliance on attribute hell is a coding 
style. You can look at any language, which has no such problem.
Apr 05 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Sunday, 5 April 2015 at 12:17:09 UTC, Kagamin wrote:

 On Saturday, 4 April 2015 at 19:59:46 UTC, Dicebot wrote:

 We need solutions that can be reasonably implemented with 
 existing resources, not perfect solutions. Storing IR in 
 object files and using custom linker is "correct" approach for 
 WPO but it is currently unaffordable.


 Works for me with llvm toolchain.


Unless LDC does some D specific WPO magic I am not aware of this 
is not what your original statement was about.


 I am not aware of any solutions based on coding style.


 Not sure what you mean, reliance on attribute hell is a coding 
 style. You can look at any language, which has no such problem.


Erm. Either it is coding style issue or a language issue. Pick 
one. Only coding style for D I am aware of that deals with 
attribute hell is "ignore most attributes" which is hardly 
solution. Please give any specific example to back your point.
Apr 05 2015
parent reply "Kagamin" <spam here.lot> writes:
On Sunday, 5 April 2015 at 12:22:15 UTC, Dicebot wrote:

 Unless LDC does some D specific WPO magic I am not aware of 
 this is not what your original statement was about.


llvm does normal WPO in a sense that compiled code is not opaque.


 Erm. Either it is coding style issue or a language issue. Pick 
 one. Only coding style for D I am aware of that deals with 
 attribute hell is "ignore most attributes" which is hardly 
 solution.


The problem can't be solved for coding styles, which rely on 
attribute hell, I only said the problem depends on coding style.
Apr 05 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Sunday, 5 April 2015 at 12:50:52 UTC, Kagamin wrote:

 On Sunday, 5 April 2015 at 12:22:15 UTC, Dicebot wrote:

 Unless LDC does some D specific WPO magic I am not aware of 
 this is not what your original statement was about.


 llvm does normal WPO in a sense that compiled code is not 
 opaque.


And I have never been speaking about "normal WPO", only about one 
specific to D semantics.


 Erm. Either it is coding style issue or a language issue. Pick 
 one. Only coding style for D I am aware of that deals with 
 attribute hell is "ignore most attributes" which is hardly 
 solution.


 The problem can't be solved for coding styles, which rely on 
 attribute hell, I only said the problem depends on coding style.


This sentence probably means something but I were not able to 
figure it out even after re-reading it several times. "coding 
style which relies on attribute hell", what kind of weird beast 
that is?
Apr 07 2015
parent "Kagamin" <spam here.lot> writes:
On Tuesday, 7 April 2015 at 08:28:08 UTC, Dicebot wrote:

 And I have never been speaking about "normal WPO", only about 
 one specific to D semantics.


AFAIK, hypothetical D-specific optimizations were never 
implemented (like elision of pure calls and optimization of 
immutable data). But they work on signature level, so they 
shouldn't be affected by separate compilation in any way.


 This sentence probably means something but I were not able to 
 figure it out even after re-reading it several times. "coding 
 style which relies on attribute hell", what kind of weird beast 
 that is?


I suppose your coding style can be an example, you wouldn't be 
interested in attribute hell otherwise.
Apr 07 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/3/15 10:10 AM, Dicebot wrote:

 On Friday, 3 April 2015 at 17:03:35 UTC, Atila Neves wrote:

 . Separate compilation. One file changes, only one file gets rebuilt


 This immediately has caught my eye as huge "no" in the description. We
 must ban C style separate compilation, there is simply no way to move
 forward otherwise. At the very least not endorse it in any way.


Agreed. D build style should be one invocation per package. -- Andrei
Apr 03 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Friday, 3 April 2015 at 19:45:38 UTC, Andrei Alexandrescu 
wrote:

 On 4/3/15 10:10 AM, Dicebot wrote:

 On Friday, 3 April 2015 at 17:03:35 UTC, Atila Neves wrote:

 . Separate compilation. One file changes, only one file gets 
 rebuilt


 This immediately has caught my eye as huge "no" in the 
 description. We
 must ban C style separate compilation, there is simply no way 
 to move
 forward otherwise. At the very least not endorse it in any way.


 Agreed. D build style should be one invocation per package. -- 
 Andrei


Just to clarify, reggae has:

1. Low-level building blocks that can be used for pretty much 
anything
2. High-level convenience rules


doesn't force anything, it's just data definition.


integration spit out build systems that compiler per module but 
that can be easily changed or even configured.

Even now it's perfectly possible to define a build system for a D 
project with per package compilation, it'll just take more typing.

Atila
Apr 04 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-04-03 19:03, Atila Neves wrote:

 I wanted to work on this a little more before announcing it, but it
 seems I'm going to be busy working on trying to get unit-threaded into
 std.experimental so here it is:

 http://code.dlang.org/packages/reggae


One thing I noticed immediately (unless I'm mistaken), compiling a D 
project without dependencies is too complicated. It should just be:

$ cd my_d_project
$ reggae

-- 
/Jacob Carlborg
Apr 03 2015