www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D needs to publicize its speed of compilation

reply Joakim <dlang joakim.fea.st> writes:
This one of the main strengths of D, it is what Walter focuses 
on, yet I have seen almost nothing on the D blog talking about 
this.  What brought me to emphasize this today is this recent 
post about how long it takes to compile the mostly-C++ Chromium 
web browser and the reddit discussion about it:

https://lobste.rs/s/iri1te/chromium_has_compilation_time_problem
https://www.reddit.com/r/programming/comments/7ktzog/chromium_has_a_compilation_time_problem/

I'm tempted to call BS on that 6-7 hour build time, as I used to 
contribute to the Chromium project, and I think it used to take 
me 20 minutes for a release build in a FreeBSD jail on a fairly 
weak, 2008-vintage mini-desktop, a dual-core Celeron with 2 GBs 
of RAM (don't hold me to that build time, just a vague 
recollection, but probably in the ballpark).  Of course, the last 
time I built Chromium was more than 5 years ago, and a lot has 
likely changed since then, such as now using LTO to speed up the 
browser apparently, and maybe the cross-compilation toolchain for 
ARM is slower, though others note similar times for native x64 
compilation also.

That still implies a slowdown of 2-3 orders of magnitude over the 
last 5 years, given the much more powerful hardware he's using, 
which is nuts.

D really needs the community to write blog posts talking about 
how fast it is, publicizing that there is an alternative to these 
glacial build times: who wants to do this?  It doesn't need to be 
on the D blog, could just be on your personal blog, but it is a 
message that really needs to be spread.
Dec 22 2017
next sibling parent reply user1234 <user1234 12.nl> writes:
On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:
 This one of the main strengths of D, it is what Walter focuses 
 on, yet I have seen almost nothing on the D blog talking about 
 this.  What brought me to emphasize this today is this recent 
 post about how long it takes to compile the mostly-C++ Chromium 
 web browser and the reddit discussion about it:

 https://lobste.rs/s/iri1te/chromium_has_compilation_time_problem
 https://www.reddit.com/r/programming/comments/7ktzog/chromium_has_a_compilation_time_problem/

 I'm tempted to call BS on that 6-7 hour build time, as I used 
 to contribute to the Chromium project, and I think it used to 
 take me 20 minutes for a release build in a FreeBSD jail on a 
 fairly weak, 2008-vintage mini-desktop, a dual-core Celeron 
 with 2 GBs of RAM (don't hold me to that build time, just a 
 vague recollection, but probably in the ballpark).  Of course, 
 the last time I built Chromium was more than 5 years ago, and a 
 lot has likely changed since then, such as now using LTO to 
 speed up the browser apparently, and maybe the 
 cross-compilation toolchain for ARM is slower, though others 
 note similar times for native x64 compilation also.

 That still implies a slowdown of 2-3 orders of magnitude over 
 the last 5 years, given the much more powerful hardware he's 
 using, which is nuts.

 D really needs the community to write blog posts talking about 
 how fast it is, publicizing that there is an alternative to 
 these glacial build times: who wants to do this?  It doesn't 
 need to be on the D blog, could just be on your personal blog, 
 but it is a message that really needs to be spread.
If this ever happens, what will be published will have to be based on LDC2. If benchmarks based on DMD are published, the article will be subject to the criticism that is that the shorter build time is due to the optimization pass, since it's known not to be super deep in DMD backend.
Dec 22 2017
next sibling parent thedeemon <dlang thedeemon.com> writes:
On Friday, 22 December 2017 at 11:39:48 UTC, user1234 wrote:

 If benchmarks based on DMD are published, the article will be 
 subject to the criticism that is that the shorter build time is 
 due to the optimization pass, since it's known not to be super 
 deep in DMD backend.
Well, Go folks don't shy away from praising their compilation speed, but the situation is very similar there: Go compiles fast because it doesn't optimize the code. Last time I checked their compiler had zero options mentioning optimization, it's like "always debug build".
Dec 22 2017
prev sibling parent Joakim <dlang joakim.fea.st> writes:
On Friday, 22 December 2017 at 11:39:48 UTC, user1234 wrote:
 If this ever happens, what will be published will have to be 
 based on LDC2.
 If benchmarks based on DMD are published, the article will be 
 subject to the criticism that is that the shorter build time is 
 due to the optimization pass, since it's known not to be super 
 deep in DMD backend.
Sure, ldc is slower but not significantly so, and will still beat the pants off clang, Rust, or Swift. On Friday, 22 December 2017 at 11:46:49 UTC, Chris wrote:
 D does not do itself any favors when it keeps accepting 
 mediocre results in benchmarks.
These are all runtime codegen benchmarks, whereas I was talking about build speed.
 https://www.techempower.com/benchmarks/previews/round15/

 Some years it does not complete the tests, in others it shows 
 results that are below mediocre.

 Vibe.d ( and other D web frameworks ) benching twice slower 
 then scripting languages like PHP/Ruby. That does not advertise 
 D.
Sonke and others have noted that there are issues that need to be worked out with that benchmark submission.
 How about:

 http://benchmarksgame.alioth.debian.org/

 No D there? Performance must be bad because its not listed at 
 all ( for a language that exist 20 years )?
It was on there for many years but was tossed out by a subsequent maintainer, who doesn't want it for some reason, as thedeemon pointed out: http://forum.dlang.org/post/lspybogtpqorauausmmv forum.dlang.org
 Another one:

 https://github.com/costajob/app-servers

 Single threaded ( look at CPU total ), losing to other 
 languages again.
I think vibe.d is meant to be run in multiple process instances instead, and it does second-best to Rust on a single core (maybe beat out by Crystal too?), which is pretty good.
 Impressions are everything when there is a wealth of languages 
 to pick from. Anybody stumbling over these results think: Well, 
 i am better going with Go, Rust, Crystal, ... for a web hosting 
 as they show more consistent high speed results.
I wouldn't say they're "everything," but they intrigue people enough to dig deeper into the language or just to move on without bothering. Of course, you may get great benchmarks with Rust but once you try coding with it, you still may not like using it. On Friday, 22 December 2017 at 14:48:22 UTC, thedeemon wrote:
 On Friday, 22 December 2017 at 11:46:49 UTC, Chris wrote:
 http://benchmarksgame.alioth.debian.org/

 No D there? Performance must be bad because its not listed at 
 all ( for a language that exist 20 years )?
D is not there for the only reason of that benchmark maintainer unwilling to include D. Technically you can take any C solution there, translate it to D (mostly by renaming from .c to .d) and with LDC get the same speed as C shows.
Better than C actually, if you use D libraries instead: http://forum.dlang.org/post/olelrjiaiwdxhjhilagv forum.dlang.org Getting back to the original topic of build speed, highlighting that the dmd compiler itself now builds in seconds after the C++->D translation would seem to be a layup: http://forum.dlang.org/thread/oknuas$2lfp$1 digitalmars.com
Dec 23 2017
prev sibling next sibling parent reply Chris <Chris c.com> writes:
On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:
 This one of the main strengths of D, it is what Walter focuses 
 on, yet I have seen almost nothing on the D blog talking about 
 this.  What brought me to emphasize this today is this recent 
 post about how long it takes to compile the mostly-C++ Chromium 
 web browser and the reddit discussion about it:

 https://lobste.rs/s/iri1te/chromium_has_compilation_time_problem
 https://www.reddit.com/r/programming/comments/7ktzog/chromium_has_a_compilation_time_problem/

 I'm tempted to call BS on that 6-7 hour build time, as I used 
 to contribute to the Chromium project, and I think it used to 
 take me 20 minutes for a release build in a FreeBSD jail on a 
 fairly weak, 2008-vintage mini-desktop, a dual-core Celeron 
 with 2 GBs of RAM (don't hold me to that build time, just a 
 vague recollection, but probably in the ballpark).  Of course, 
 the last time I built Chromium was more than 5 years ago, and a 
 lot has likely changed since then, such as now using LTO to 
 speed up the browser apparently, and maybe the 
 cross-compilation toolchain for ARM is slower, though others 
 note similar times for native x64 compilation also.

 That still implies a slowdown of 2-3 orders of magnitude over 
 the last 5 years, given the much more powerful hardware he's 
 using, which is nuts.

 D really needs the community to write blog posts talking about 
 how fast it is, publicizing that there is an alternative to 
 these glacial build times: who wants to do this?  It doesn't 
 need to be on the D blog, could just be on your personal blog, 
 but it is a message that really needs to be spread.
D does not do itself any favors when it keeps accepting mediocre results in benchmarks. https://www.techempower.com/benchmarks/previews/round15/ Some years it does not complete the tests, in others it shows results that are below mediocre. Vibe.d ( and other D web frameworks ) benching twice slower then scripting languages like PHP/Ruby. That does not advertise D. How about: http://benchmarksgame.alioth.debian.org/ No D there? Performance must be bad because its not listed at all ( for a language that exist 20 years )? Another one: https://github.com/costajob/app-servers Single threaded ( look at CPU total ), losing to other languages again. Impressions are everything when there is a wealth of languages to pick from. Anybody stumbling over these results think: Well, i am better going with Go, Rust, Crystal, ... for a web hosting as they show more consistent high speed results.
Dec 22 2017
parent reply thedeemon <dlang thedeemon.com> writes:
On Friday, 22 December 2017 at 11:46:49 UTC, Chris wrote:
 http://benchmarksgame.alioth.debian.org/

 No D there? Performance must be bad because its not listed at 
 all ( for a language that exist 20 years )?
D is not there for the only reason of that benchmark maintainer unwilling to include D. Technically you can take any C solution there, translate it to D (mostly by renaming from .c to .d) and with LDC get the same speed as C shows.
Dec 22 2017
parent Russel Winder <russel winder.org.uk> writes:
On Fri, 2017-12-22 at 14:48 +0000, thedeemon via Digitalmars-d wrote:
 [=E2=80=A6]
=20
 D is not there for the only reason of that benchmark maintainer=20
 unwilling to include D. Technically you can take any C solution=20
 there, translate it to D (mostly by renaming from .c to .d) and=20
 with LDC get the same speed as C shows.
For performance related stuff Go people use gccgo, just as D people use gdc (or more usually ldc). --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Road m: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk
Dec 22 2017
prev sibling next sibling parent reply aberba <karabutaworld gmail.com> writes:
On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:
 This one of the main strengths of D, it is what Walter focuses 
 on, yet I have seen almost nothing on the D blog talking about 
 this.  What brought me to emphasize this today is this recent 
 post about how long it takes to compile the mostly-C++ Chromium 
 web browser and the reddit discussion about it:

 [...]
Memory consumption is a deal breaker.
Dec 24 2017
parent reply Joakim <dlang joakim.fea.st> writes:
On Sunday, 24 December 2017 at 22:55:12 UTC, aberba wrote:
 On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:
 This one of the main strengths of D, it is what Walter focuses 
 on, yet I have seen almost nothing on the D blog talking about 
 this.  What brought me to emphasize this today is this recent 
 post about how long it takes to compile the mostly-C++ 
 Chromium web browser and the reddit discussion about it:

 [...]
Memory consumption is a deal breaker.
The dmd frontend prioritized speed over memory, which can cause problems with large or CTFE-heavy projects on low-memory systems: http://www.drdobbs.com/cpp/increasing-compiler-speed-by-over-75/240158941 On Wednesday, 27 December 2017 at 21:23:15 UTC, Atila Neves wrote:
 I thought D did market itself that way.
It has historically been a selling point, but perhaps it has been taken for granted. I'm saying it needs a marketing push now. I just went and checked the front page of the website, and I see a new marketing slogan that was added a couple weeks ago, "write fast, read fast, and run fast. Fast code, fast:" https://github.com/dlang/dlang.org/pull/1965 However, "build fast" is not included in that list. You could argue it's implied by "write fast" or the last "code, fast," but it's not mentioned in the list under "write fast" below. So even the front page doesn't highlight build speed, one of the core advantages of D.
 However, my experience has been that D has fast builds from 
 scratch, but is really really slow for incremental builds. 
 Here's the thing: personally I don't care about anything except 
 incremental builds. It's great that Phobos compiles in seconds 
 when I do a git fetch after months. Win! Oh wait, I'm working 
 on a bug, changed one character and it still takes the same 
 amount of time...

 Given that I write D code every day now, this incremental 
 compilation situation is driving me nuts, and I'm definitely 
 going to do something about it.

 It's getting so bad that the day at work I had to write C++ I 
 was happy. Because the edit-compile-link-test cycle was 
 _faster_. Think about that for a bit - I enjoyed writing C++ 
 (!!! either it or Scala are the slowest languages to compile 
 ever) because I got faster feedback. Epic fail.

 It's also possible I use templates and CTFE more that the 
 regular D programmer. But... they're some of the main reasons I 
 use D in the first place! I don't want to write Java in D 
 syntax.
What do you plan to do about it?
Dec 27 2017
parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 28 December 2017 at 07:05:25 UTC, Joakim wrote:
 On Sunday, 24 December 2017 at 22:55:12 UTC, aberba wrote:
 On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:
What do you plan to do about it?
I have an idea or two. I just need time/mental energy to do this. Atila
Dec 28 2017
parent reply Seb <seb wilzba.ch> writes:
On Thursday, 28 December 2017 at 13:20:34 UTC, Atila Neves wrote:
 On Thursday, 28 December 2017 at 07:05:25 UTC, Joakim wrote:
 On Sunday, 24 December 2017 at 22:55:12 UTC, aberba wrote:
 On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:
What do you plan to do about it?
I have an idea or two. I just need time/mental energy to do this. Atila
Are you aware of this great work: https://github.com/dlang/dmd/pull/7239?
Dec 28 2017
parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 28 December 2017 at 13:26:55 UTC, Seb wrote:
 On Thursday, 28 December 2017 at 13:20:34 UTC, Atila Neves 
 wrote:
 On Thursday, 28 December 2017 at 07:05:25 UTC, Joakim wrote:
 On Sunday, 24 December 2017 at 22:55:12 UTC, aberba wrote:
 On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:
What do you plan to do about it?
I have an idea or two. I just need time/mental energy to do this. Atila
Are you aware of this great work: https://github.com/dlang/dmd/pull/7239?
I wasn't, thanks for pointing it out. Looks interesting, but all the compile times were still over a second, which is my pain threshold. Atila
Dec 29 2017
prev sibling next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:
 This one of the main strengths of D, it is what Walter focuses 
 on, yet I have seen almost nothing on the D blog talking about 
 this.  What brought me to emphasize this today is this recent 
 post about how long it takes to compile the mostly-C++ Chromium 
 web browser and the reddit discussion about it:

 https://lobste.rs/s/iri1te/chromium_has_compilation_time_problem
 https://www.reddit.com/r/programming/comments/7ktzog/chromium_has_a_compilation_time_problem/

 I'm tempted to call BS on that 6-7 hour build time, as I used 
 to contribute to the Chromium project, and I think it used to 
 take me 20 minutes for a release build in a FreeBSD jail on a 
 fairly weak, 2008-vintage mini-desktop, a dual-core Celeron 
 with 2 GBs of RAM (don't hold me to that build time, just a 
 vague recollection, but probably in the ballpark).  Of course, 
 the last time I built Chromium was more than 5 years ago, and a 
 lot has likely changed since then, such as now using LTO to 
 speed up the browser apparently, and maybe the 
 cross-compilation toolchain for ARM is slower, though others 
 note similar times for native x64 compilation also.

 That still implies a slowdown of 2-3 orders of magnitude over 
 the last 5 years, given the much more powerful hardware he's 
 using, which is nuts.

 D really needs the community to write blog posts talking about 
 how fast it is, publicizing that there is an alternative to 
 these glacial build times: who wants to do this?  It doesn't 
 need to be on the D blog, could just be on your personal blog, 
 but it is a message that really needs to be spread.
I thought D did market itself that way. However, my experience has been that D has fast builds from scratch, but is really really slow for incremental builds. Here's the thing: personally I don't care about anything except incremental builds. It's great that Phobos compiles in seconds when I do a git fetch after months. Win! Oh wait, I'm working on a bug, changed one character and it still takes the same amount of time... Given that I write D code every day now, this incremental compilation situation is driving me nuts, and I'm definitely going to do something about it. It's getting so bad that the day at work I had to write C++ I was happy. Because the edit-compile-link-test cycle was _faster_. Think about that for a bit - I enjoyed writing C++ (!!! either it or Scala are the slowest languages to compile ever) because I got faster feedback. Epic fail. It's also possible I use templates and CTFE more that the regular D programmer. But... they're some of the main reasons I use D in the first place! I don't want to write Java in D syntax. Atila Atila
Dec 27 2017
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/27/2017 1:23 PM, Atila Neves wrote:
 However, my experience has been that D has fast builds from scratch, but is 
 really really slow for incremental builds.
You can do fast incremental builds with D by not putting all the files on the command line. Just put one.
Dec 27 2017
parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 28 December 2017 at 07:21:28 UTC, Walter Bright 
wrote:
 On 12/27/2017 1:23 PM, Atila Neves wrote:
 However, my experience has been that D has fast builds from 
 scratch, but is really really slow for incremental builds.
You can do fast incremental builds with D by not putting all the files on the command line. Just put one.
I don't build the compiler command-line myself, nor do I want to. Even then, recompiling one file isn't useful to me, what is is to recompile what's needed and run the tests. Emacs/flycheck already highlights compilation errors in just one file. Atila
Dec 28 2017
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Dec 28, 2017 at 01:22:13PM +0000, Atila Neves via Digitalmars-d wrote:
 On Thursday, 28 December 2017 at 07:21:28 UTC, Walter Bright wrote:
 On 12/27/2017 1:23 PM, Atila Neves wrote:
 However, my experience has been that D has fast builds from
 scratch, but is really really slow for incremental builds.
You can do fast incremental builds with D by not putting all the files on the command line. Just put one.
I don't build the compiler command-line myself, nor do I want to. Even then, recompiling one file isn't useful to me, what is is to recompile what's needed and run the tests.
[...] Isn't that what a build system is for? T -- GEEK = Gatherer of Extremely Enlightening Knowledge
Dec 28 2017
parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 28 December 2017 at 15:54:03 UTC, H. S. Teoh wrote:
 On Thu, Dec 28, 2017 at 01:22:13PM +0000, Atila Neves via 
 Digitalmars-d wrote:
 On Thursday, 28 December 2017 at 07:21:28 UTC, Walter Bright 
 wrote:
 On 12/27/2017 1:23 PM, Atila Neves wrote:
 However, my experience has been that D has fast builds 
 from scratch, but is really really slow for incremental 
 builds.
You can do fast incremental builds with D by not putting all the files on the command line. Just put one.
I don't build the compiler command-line myself, nor do I want to. Even then, recompiling one file isn't useful to me, what is is to recompile what's needed and run the tests.
[...] Isn't that what a build system is for? T
Yes, and I wrote one. It turns out that I misunderstood what Walter said though. Atila
Dec 29 2017
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/28/2017 5:22 AM, Atila Neves wrote:
 On Thursday, 28 December 2017 at 07:21:28 UTC, Walter Bright wrote:
 On 12/27/2017 1:23 PM, Atila Neves wrote:
 However, my experience has been that D has fast builds from scratch, but is 
 really really slow for incremental builds.
You can do fast incremental builds with D by not putting all the files on the command line. Just put one.
I don't build the compiler command-line myself, nor do I want to. Even then, recompiling one file isn't useful to me, what is is to recompile what's needed and run the tests. Emacs/flycheck already highlights compilation errors in just one file.
I don't understand. C++ compiles files one at a time. You can do the same in D. How can this model be useful in C++ but not with D?
Dec 28 2017
next sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 28 December 2017 at 22:58, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 12/28/2017 5:22 AM, Atila Neves wrote:
 On Thursday, 28 December 2017 at 07:21:28 UTC, Walter Bright wrote:
 On 12/27/2017 1:23 PM, Atila Neves wrote:
 However, my experience has been that D has fast builds from scratch, but
 is really really slow for incremental builds.
You can do fast incremental builds with D by not putting all the files on the command line. Just put one.
I don't build the compiler command-line myself, nor do I want to. Even then, recompiling one file isn't useful to me, what is is to recompile what's needed and run the tests. Emacs/flycheck already highlights compilation errors in just one file.
I don't understand. C++ compiles files one at a time. You can do the same in D. How can this model be useful in C++ but not with D?
D modules are like C++ headers. Change one, and you need to recompile everything that imports it.
Dec 28 2017
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/28/2017 2:16 PM, Iain Buclaw wrote:
 D modules are like C++ headers.  Change one, and you need to recompile
 everything that imports it.
That's right. That doesn't mean that D should incrementally compile slower than C++.
Dec 29 2017
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 28 December 2017 at 21:58:07 UTC, Walter Bright 
wrote:
 On 12/28/2017 5:22 AM, Atila Neves wrote:
 On Thursday, 28 December 2017 at 07:21:28 UTC, Walter Bright 
 wrote:
 On 12/27/2017 1:23 PM, Atila Neves wrote:
 However, my experience has been that D has fast builds from 
 scratch, but is really really slow for incremental builds.
You can do fast incremental builds with D by not putting all the files on the command line. Just put one.
I don't build the compiler command-line myself, nor do I want to. Even then, recompiling one file isn't useful to me, what is is to recompile what's needed and run the tests. Emacs/flycheck already highlights compilation errors in just one file.
I don't understand. C++ compiles files one at a time. You can do the same in D. How can this model be useful in C++ but not with D?
Because: . As Iain pointed out independently, D modules are like C/C++ headers. All dependencies must also be recompiled. If I edit a C++ source file, only that source file has to be recompiled. I can use .di files, and they can even be generated for me (nice!) but there's no build system to do this properly. This is one of the ideas I alluded to earlier - I want to put this sort of thing in reggae. And now that dmd is a dub package I can even just use the actual compiler code. i.e. Generate the .di, if it's the same as before don't overwrite the old one to not mess with the timestamp. . C++ doesn't have built-in unit tests. Which means a framework has to be used and since nobody wants to put their tests in #ifdef "blocks" they live in separate files. Changing the implementation doesn't require the tests to be rebuilt. Yes, I can separate them in D too, but then I lose DDoc benefits and it's actually nicer to see tests next to the code. The fact that all D files are effectively headers means editing a unittest means recompiling all modules that depend on the current one. . Related to the previous point: compiling separately in D is actually slower overall than by package, because of all of the dependencies (I measured). Yes, I can compile by package, but I don't want to do this by hand. To the extent of my knowledge the only way to get per-package compilation by default or otherwise in the D ecosystem is reggae. But: . Until recently, it wasn't possible to use __traits(getUnitTests) with separate compilation. I've run into other issues with when doing so (I can't remember what they were). Even with the bug fix editing a test entails compiling several modules and it's slower than just one C++ file. In C++ editing a test source file always means recompiling exactly one file. . Speaking of dependencies, there's no easy way currently to know what they are. _If_ I want to build per package/module and only build what's needed, then I need to either hand-roll a Makefile (points gun to head) or use reggae. If there's any other alternative, I don't know about it. Even Phobos recompiles the world. That's not a good example to set. . C++ test framework's hacky usage of global variable assignment to register tests seems to be faster to compile than D's compile-time reflection to achieve the same result. I'd have to benchmark to be sure though. . If one has dub dependencies (and in all likelihood one has), then either one is a reggae early adopter (I keep mentioning it to highlight why I wrote it in the first place), or in the 99.9% of other users typing either `dub build` or `dub test`. There is no per-package build option. There's a per-module option, but it's single-threaded, therefore slow. Also, no dependency tracking, so let's recompile the whole dub package! In one thread. Zzzzzzzzz. What I really want is to hit F5 in emacs, rebuild the current file _and_ all of its dependencies, cache the dependency build per file so that if they haven't change don't rebuild, link and run the unit tests. But build _only_ what's needed to avoid linker errors in order to be able to run the unit tests for _one_ file. I spent an afternoon writing the necessary elisp to get this done and it worked! Then I tried a different project and it failed miserably. So did the 3rd one. I think I just remembered one of the separate compilation bugs I mentioned above. I know, I know, if I don't file it it won't get fixed (even if I file it I have a sneaky suspicion it'll be me fixing it anyway), but figuring out what went wrong takes time and energy and I haven't had enough of either to take this off my TODO list. To summarise: . Dependencies. . There's no standard build system that isn't called dub. . There's only one alternative build system that can do any of what's needed. . That alternative build system hasn't implemented my .di idea yet. . Separate compilation bugs. That nobody ever hits because dub builds everything at once. Atila
Dec 29 2017
prev sibling next sibling parent reply I Love Stuffing <ilovestuffing gmail.com> writes:
On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:
 This one of the main strengths of D, it is what Walter focuses 
 on, yet I have seen almost nothing on the D blog talking about 
 this.  What brought me to emphasize this today is this recent 
 post about how long it takes to compile the mostly-C++ Chromium 
 web browser and the reddit discussion about it:

 https://lobste.rs/s/iri1te/chromium_has_compilation_time_problem
 https://www.reddit.com/r/programming/comments/7ktzog/chromium_has_a_compilation_time_problem/

 I'm tempted to call BS on that 6-7 hour build time, as I used 
 to contribute to the Chromium project, and I think it used to 
 take me 20 minutes for a release build in a FreeBSD jail on a 
 fairly weak, 2008-vintage mini-desktop, a dual-core Celeron 
 with 2 GBs of RAM (don't hold me to that build time, just a 
 vague recollection, but probably in the ballpark).  Of course, 
 the last time I built Chromium was more than 5 years ago, and a 
 lot has likely changed since then, such as now using LTO to 
 speed up the browser apparently, and maybe the 
 cross-compilation toolchain for ARM is slower, though others 
 note similar times for native x64 compilation also.

 That still implies a slowdown of 2-3 orders of magnitude over 
 the last 5 years, given the much more powerful hardware he's 
 using, which is nuts.

 D really needs the community to write blog posts talking about 
 how fast it is, publicizing that there is an alternative to 
 these glacial build times: who wants to do this?  It doesn't 
 need to be on the D blog, could just be on your personal blog, 
 but it is a message that really needs to be spread.
To be fair, you are basing this off C++. C++ has an obnoxiously slow build system at times and everyone knows this, have known this, and have even addressed some parts of it. Yet they still use it. I don't think build times are that big of a deal. It's a nice to have, but as long as a fairly sizable project compiles in a less than a minute, you won't stand out in a bad way. If anything, D should leverage the fast compile times to take advantage of the leeway it gives so the compiler can do a bit of extra work to give better error messages.
Dec 29 2017
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, December 29, 2017 22:37:56 I Love Stuffing via Digitalmars-d 
wrote:
 To be fair, you are basing this off C++. C++ has an obnoxiously
 slow build system at times and everyone knows this, have known
 this, and have even addressed some parts of it. Yet they still
 use it. I don't think build times are that big of a deal. It's a
 nice to have, but as long as a fairly sizable project compiles in
 a less than a minute, you won't stand out in a bad way.
So much about build times and how much they matter seems to be about perception. For instance, I used to work at a company whose main project took over 3 - 4 hours to do a clean build. After the project was redone in Java, the build times were about 10 minutes, and folks complained about how long they were even though they were worlds better than the C++ build times had been. It also depends on your personality. Personally, while I like builds being near instantaneous when dealing with unit tests, if they take a few seconds, it's not a big deal, but I know folks who get seriously annoyed if they can notice much of a difference time-wise between hitting enter to run the build command and hitting enter with no command. Either way though, having build times be more than a few seconds when dealing with unit tests tends to cause serious problems even though having a build for an entire project take a few minutes usually isn't a big deal unless you're really impatient. Having the delay between making changes and testing them be very long can _really_ slow down development, and so that's where build times really matter. - Jonathan M Davis
Dec 29 2017
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 29, 2017 at 03:49:07PM -0700, Jonathan M Davis via Digitalmars-d
wrote:
[...]
 It also depends on your personality. Personally, while I like builds
 being near instantaneous when dealing with unit tests, if they take a
 few seconds, it's not a big deal, but I know folks who get seriously
 annoyed if they can notice much of a difference time-wise between
 hitting enter to run the build command and hitting enter with no
 command. Either way though, having build times be more than a few
 seconds when dealing with unit tests tends to cause serious problems
 even though having a build for an entire project take a few minutes
 usually isn't a big deal unless you're really impatient. Having the
 delay between making changes and testing them be very long can
 _really_ slow down development, and so that's where build times really
 matter.
[...] If you think a few minutes is bad... I've been on the receiving end of builds that take 30-45 *minutes* for 1-line code changes (*ahem*cough*C++ header files*cough*ahem*). It put a serious damper on my ability to make any progress with the code. It's *possible*, yes, but it's not something I'd wish on anyone. T -- MSDOS = MicroSoft's Denial Of Service
Dec 29 2017
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, December 29, 2017 15:51:53 H. S. Teoh via Digitalmars-d wrote:
 On Fri, Dec 29, 2017 at 03:49:07PM -0700, Jonathan M Davis via
 Digitalmars-d wrote: [...]

 It also depends on your personality. Personally, while I like builds
 being near instantaneous when dealing with unit tests, if they take a
 few seconds, it's not a big deal, but I know folks who get seriously
 annoyed if they can notice much of a difference time-wise between
 hitting enter to run the build command and hitting enter with no
 command. Either way though, having build times be more than a few
 seconds when dealing with unit tests tends to cause serious problems
 even though having a build for an entire project take a few minutes
 usually isn't a big deal unless you're really impatient. Having the
 delay between making changes and testing them be very long can
 _really_ slow down development, and so that's where build times really
 matter.
[...] If you think a few minutes is bad... I've been on the receiving end of builds that take 30-45 *minutes* for 1-line code changes (*ahem*cough*C++ header files*cough*ahem*). It put a serious damper on my ability to make any progress with the code. It's *possible*, yes, but it's not something I'd wish on anyone.
Oh, certainly. But my point is that once it starts taking minutes to be able to build and run the unit tests for what you're currently doing, it becomes infeasible to edit-test-edit-etc. in any kind of reasonable cycle. It doesn't have to get anywhere near the 30 minute mark for it to be a serious problem. So, of course, 30+ minutes is an utter disaster if you're looking to test stuff as you go. - Jonathan M Davis
Dec 29 2017
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 29, 2017 at 05:52:36PM -0700, Jonathan M Davis via Digitalmars-d
wrote:
[...]
 Oh, certainly. But my point is that once it starts taking minutes to
 be able to build and run the unit tests for what you're currently
 doing, it becomes infeasible to edit-test-edit-etc. in any kind of
 reasonable cycle. It doesn't have to get anywhere near the 30 minute
 mark for it to be a serious problem. So, of course, 30+ minutes is an
 utter disaster if you're looking to test stuff as you go.
[...] Yeah... recently I've been resorting to `dmd -unittest -main module.d` on single modules for faster turnaround time, because in spite of D's famed compilation speeds, some things still make it go far too slow, e.g. if you use template-heavy / CTFE-heavy code, or there are just too many files to compile. This has also been encouraging me to write code in a far more modular fashion than even traditional modularists would do, in the sense that where possible, even dependent modules won't be imported, but instead left to a template parameter, to be instantiated by the main program with the dependent module. Contrived example: ----- void printSomething(alias stdio = selfImport!"std.stdio", Args...)(string fmt, Args args) { stdio.writefln(fmt, args); } unittest { struct TestHarness { static void writefln() { /* fake implementation here */ } } printSomething!TestHarness("%s", "blah"); } ----- The idea is that importing std.stdio won't even happen when compiling with -unittest -main. This approach lets you completely detach a module from its dependencies, so you never have to worry about what else it depends on when compiling with -unittest -main, and also conveniently allows you to substitute complicated types with cheaper substitutes that suffice for testing, e.g., instead of writing to stdout or actually causing side-effects visible outside the program, have a substitute module (or just struct!) that redirects output or filesystem changes to a log that's verified by the unittest to be the correct sequence of operations. The substitute type doesn't even have to implement the complete API of the module/object in question; just enough of it for the unittest to do what it needs to do. Being able to substitute a struct in place of a module inside a unittest is one of the awesome things about D that make me love it so. :-D And a side perk is that you'll get automatic attribute inference for all such functions, if the original function would otherwise be just a non-template function. (On that note, nowadays I'm tempted to just stick empty compile-time parameters on every function I can; it enables attribute inference and also reduces executable bloat if the function is never actually called from anywhere. The only bad thing is the compiler won't run semantic on potentially broken code... but then again, if you never use it, it doesn't matter, and if it ought to matter, there should be a unittest for it anyway.) T -- "You are a very disagreeable person." "NO."
Dec 29 2017
parent reply Atila Neves <atila.neves gmail.com> writes:
On Saturday, 30 December 2017 at 01:09:59 UTC, H. S. Teoh wrote:
 On Fri, Dec 29, 2017 at 05:52:36PM -0700, Jonathan M Davis via 
 Digitalmars-d wrote: [...]
 [...]
[...] Yeah... recently I've been resorting to `dmd -unittest -main module.d` on single modules for faster turnaround time, because in spite of D's famed compilation speeds, some things still make it go far too slow, e.g. if you use template-heavy / CTFE-heavy code, or there are just too many files to compile. [...]
This is pretty interesting. Unfortunately I can see it getting messy with multiple dependent modules and/or if the function has "real" template parameters on top. I have to think about how to use your strategy in my own code. Atila
Jan 02
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Jan 02, 2018 at 04:00:11PM +0000, Atila Neves via Digitalmars-d wrote:
 On Saturday, 30 December 2017 at 01:09:59 UTC, H. S. Teoh wrote:
[...]
 Yeah... recently I've been resorting to `dmd -unittest -main
 module.d` on single modules for faster turnaround time, because in
 spite of D's famed compilation speeds, some things still make it go
 far too slow, e.g. if you use template-heavy / CTFE-heavy code, or
 there are just too many files to compile.
 
 [...]
This is pretty interesting. Unfortunately I can see it getting messy with multiple dependent modules and/or if the function has "real" template parameters on top. I have to think about how to use your strategy in my own code.
[...] Yeah, when you have dependent modules it can get pretty messy. Which is why I've been moving towards modularising my code beyond what people would usually do, i.e., turn even symbols in dependent modules into template parameters so that you can make the module completely independent. Here's one actual code example I'm particularly proud of: ------ /** * Expands ' '-directives in a range of strings. * * Returns: A range of strings with lines that begin with ' ' substituted with * the contents of the file named by the rest of the line. */ auto expandFileDirectives(File = std.stdio.File, R)(R args) if (isInputRange!R && is(ElementType!R : const(char)[])) { import std.algorithm.iteration : joiner, map; import std.algorithm.searching : startsWith; import std.range : only; import std.range.interfaces : InputRange, inputRangeObject; import std.typecons : No; return args.map!(arg => arg.startsWith(' ') ? cast(InputRange!string) inputRangeObject( File(arg[1 .. $]).byLineCopy(No.keepTerminator)) : cast(InputRange!string) inputRangeObject(only(arg))) .joiner; } unittest { static struct FakeFile { static string[][string] files; string[] contents; this(string filename) { contents = files[filename]; } auto byLineCopy(KeepTerminator dummy) { return contents; } } FakeFile.files["file1"] = [ "x=1", "y=2" ]; FakeFile.files["file2"] = [ "z=3" ]; import std.algorithm.comparison : equal; assert([ "p=abc", " file1", "q=def", " file2", "e=ghi" ].expandFileDirectives!FakeFile.equal([ "p=abc", "x=1", "y=2", "q=def", "z=3", "e=ghi" ])); } ------ As you can see, I made File into a template parameter so that the unittest can simulate a virtual filesystem with a struct, without ever needing to deal with the messiness of actually creating and managing temporary files on a real filesystem. This is (mostly) invisible to user code (i.e., other code that uses this function) because if you don't specify what File should be, it defaults to the usual std.stdio.File. Note also that not all of File's API needed to be implemented; since the function only uses .byLineCopy, that's all I needed to implement in my surrogate FakeFile struct. Not all code can be cut up in this way, though. I've only achieved this in a few low-level modules; for other modules I'm still facing the problem of having to specify dependent modules on the command-line. But in trying to attain to this goal, it really forces you to think about structuring your code in a better way than just throwing in haphazard dependencies that may not strictly be necessary. T -- Just because you can, doesn't mean you should.
Jan 02
parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 01/02/2018 10:13 AM, H. S. Teoh wrote:

 As you can see, I made File into a template parameter so that the
 unittest can simulate a virtual filesystem with a struct
All I'm hearing is "blah blah blog." ;) Ali
Jan 02