www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D compiles fast, right? Right??

reply Atila Neves <atila.neves gmail.com> writes:
Fast code fast, they said. It'll be fun, they said. Here's a D 
file:

     import std.path;


Yep, that's all there is to it. Let's compile it on my laptop:

     /tmp % time dmd -c  foo.d
     dmd -c foo.d  0.12s user 0.02s system 98% cpu 0.139 total


That... doesn't seem too fast to me. But wait, there's more:

     /tmp % time dmd -c -unittest foo.d
     dmd -c -unittest foo.d  0.46s user 0.06s system 99% cpu 0.525 
total


Half. A. Second. AKA "an eternity" in dog years, err, CPU time. I 
know this has been brought up before, and recently even, but, 
just... just... sigh.

So I wondered how fast it'd be in Go, since it's got a reputation 
for speedy compilation:

     package foo
     import "path"
     func Foo() string {
	    return path.Base("foo")
     }

     /tmp % time go tool compile foo.go
     go tool compile foo.go  0.01s user 0.01s system 117% cpu 
0.012 total


See, now that's what I'd consider fast. It has actual code in the 
file because otherwise it complains the file isn't using the 
imported package, because, Go things. It compiled so fast I had 
to check I'd generated an object file, and then I learned you 
can't use objdump on Go .o files, because... more Go things (go 
tool objdump for the curious).

Ok, so how about C++, surely that will make D look good?

     #include <experimental/filesystem>  // yes, also a one-liner

     /tmp % time /usr/bin/clang++ -std=c++17 -c foo.cpp
     /usr/bin/clang++ -std=c++17 -c foo.cpp  0.45s user 0.03s 
system 96% cpu 0.494 total

     /tmp % time /usr/bin/g++ -std=c++17 -c foo.cpp
     /usr/bin/g++ -std=c++17 -c foo.cpp  0.39s user 0.04s system 
99% cpu 0.429 total


So.... yeeaaaaaaaaah. If one is compiling unit tests, which I 
happen to pretty much only exclusively do, then trying to do 
anything with paths in D is

1. Comparable to C++ in build times
2. Actually _slower_ than C++ (who'd've thunk it?) *
3. Gets lapped around Captain America vs The Falcon style about 
50 times by Go.

And that's assuming there's a crazy D programmer out there (hint: 
me) that actually tries to compile minimal units at a time (with 
actual dependency tracking!) instead of the whole project at 
once, otherwise it'll take even longer. And this to just import 
`std.path`, then there's the actual work you were trying to get 
to.

Today actually made me want to write Go. I'm going to take a 
shower now.

Atila

* Building a whole project in C++ still takes a lot longer since 
D scales much better, but that's not my typical worflow, nor 
should it be anyone else's.
Mar 30 2018
next sibling parent Meta <jared771 gmail.com> writes:
On Friday, 30 March 2018 at 16:12:44 UTC, Atila Neves wrote:

 Fast code fast, they said. It'll be fun, they said. Here's a D 
 file:

     import std.path;


 Yep, that's all there is to it. Let's compile it on my laptop:

     /tmp % time dmd -c  foo.d
     dmd -c foo.d  0.12s user 0.02s system 98% cpu 0.139 total


 That... doesn't seem too fast to me. But wait, there's more:

     /tmp % time dmd -c -unittest foo.d
     dmd -c -unittest foo.d  0.46s user 0.06s system 99% cpu 
 0.525 total


 Half. A. Second. AKA "an eternity" in dog years, err, CPU time. 
 I know this has been brought up before, and recently even, but, 
 just... just... sigh.

 So I wondered how fast it'd be in Go, since it's got a 
 reputation for speedy compilation:

     package foo
     import "path"
     func Foo() string {
 	    return path.Base("foo")
     }

     /tmp % time go tool compile foo.go
     go tool compile foo.go  0.01s user 0.01s system 117% cpu 
 0.012 total


 See, now that's what I'd consider fast. It has actual code in 
 the file because otherwise it complains the file isn't using 
 the imported package, because, Go things. It compiled so fast I 
 had to check I'd generated an object file, and then I learned 
 you can't use objdump on Go .o files, because... more Go things 
 (go tool objdump for the curious).

 Ok, so how about C++, surely that will make D look good?

     #include <experimental/filesystem>  // yes, also a one-liner

     /tmp % time /usr/bin/clang++ -std=c++17 -c foo.cpp
     /usr/bin/clang++ -std=c++17 -c foo.cpp  0.45s user 0.03s 
 system 96% cpu 0.494 total

     /tmp % time /usr/bin/g++ -std=c++17 -c foo.cpp
     /usr/bin/g++ -std=c++17 -c foo.cpp  0.39s user 0.04s system 
 99% cpu 0.429 total


 So.... yeeaaaaaaaaah. If one is compiling unit tests, which I 
 happen to pretty much only exclusively do, then trying to do 
 anything with paths in D is

 1. Comparable to C++ in build times
 2. Actually _slower_ than C++ (who'd've thunk it?) *
 3. Gets lapped around Captain America vs The Falcon style about 
 50 times by Go.

 And that's assuming there's a crazy D programmer out there 
 (hint: me) that actually tries to compile minimal units at a 
 time (with actual dependency tracking!) instead of the whole 
 project at once, otherwise it'll take even longer. And this to 
 just import `std.path`, then there's the actual work you were 
 trying to get to.

 Today actually made me want to write Go. I'm going to take a 
 shower now.

 Atila

 * Building a whole project in C++ still takes a lot longer 
 since D scales much better, but that's not my typical worflow, 
 nor should it be anyone else's.


Yeah, that's pretty bad, relatively, for a no-op build. Probably 
some CTFE or template stuff that gets pulled in by one of the 
imports.
Mar 30 2018
prev sibling next sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Friday, 30 March 2018 at 16:12:44 UTC, Atila Neves wrote:

 Fast code fast, they said. It'll be fun, they said. Here's a D 
 file:

     import std.path;


 Yep, that's all there is to it. Let's compile it on my laptop:

     /tmp % time dmd -c  foo.d
     dmd -c foo.d  0.12s user 0.02s system 98% cpu 0.139 total


 That... doesn't seem too fast to me. But wait, there's more:

     /tmp % time dmd -c -unittest foo.d
     dmd -c -unittest foo.d  0.46s user 0.06s system 99% cpu 
 0.525 total


 Half. A. Second. AKA "an eternity" in dog years, err, CPU time. 
 I know this has been brought up before, and recently even, but, 
 just... just... sigh.

 So I wondered how fast it'd be in Go, since it's got a 
 reputation for speedy compilation:

     package foo
     import "path"
     func Foo() string {
 	    return path.Base("foo")
     }

     /tmp % time go tool compile foo.go
     go tool compile foo.go  0.01s user 0.01s system 117% cpu 
 0.012 total


 See, now that's what I'd consider fast. It has actual code in 
 the file because otherwise it complains the file isn't using 
 the imported package, because, Go things. It compiled so fast I 
 had to check I'd generated an object file, and then I learned 
 you can't use objdump on Go .o files, because... more Go things 
 (go tool objdump for the curious).

 Ok, so how about C++, surely that will make D look good?

     #include <experimental/filesystem>  // yes, also a one-liner

     /tmp % time /usr/bin/clang++ -std=c++17 -c foo.cpp
     /usr/bin/clang++ -std=c++17 -c foo.cpp  0.45s user 0.03s 
 system 96% cpu 0.494 total

     /tmp % time /usr/bin/g++ -std=c++17 -c foo.cpp
     /usr/bin/g++ -std=c++17 -c foo.cpp  0.39s user 0.04s system 
 99% cpu 0.429 total


 So.... yeeaaaaaaaaah. If one is compiling unit tests, which I 
 happen to pretty much only exclusively do, then trying to do 
 anything with paths in D is

 1. Comparable to C++ in build times
 2. Actually _slower_ than C++ (who'd've thunk it?) *
 3. Gets lapped around Captain America vs The Falcon style about 
 50 times by Go.

 And that's assuming there's a crazy D programmer out there 
 (hint: me) that actually tries to compile minimal units at a 
 time (with actual dependency tracking!) instead of the whole 
 project at once, otherwise it'll take even longer. And this to 
 just import `std.path`, then there's the actual work you were 
 trying to get to.

 Today actually made me want to write Go. I'm going to take a 
 shower now.

 Atila

 * Building a whole project in C++ still takes a lot longer 
 since D scales much better, but that's not my typical worflow, 
 nor should it be anyone else's.


Seems like you're comparing apples to oranges.

Go's path.go is very small, a 215 line file:
https://github.com/golang/go/blob/master/src/path/path.go
Documentation: https://golang.org/pkg/path/


Dlang's std.path is much more comprehensive with 4181 lines: 
https://github.com/dlang/phobos/blob/master/std/path.d
Documentation: https://dlang.org/phobos/std_path.html

It's over an order of magnitude more code and only takes twice as 
long to compile without unittests, and it's only fair to compare 
the "non-unittest" version of std.path with Go, since Go does not 
include unittests.

I'm not sure why you would compile the standard library unittests 
every time you compile anything. Probably a consequence of not 
having `-unittest=<pattern>`.  timotheecour suggested we add 
support for this and I agree for cases like this, where druntime 
and phobos would be exclude by default (just like we do with -i), 
meaning that your compilation example would not have compiled 
phobos unittests.
Mar 30 2018
parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 30 March 2018 at 16:41:42 UTC, Jonathan Marler wrote:

 Seems like you're comparing apples to oranges.


No, I'm comparing one type of apple to another with regards to 
weight in my shopping bag before I've even taken a bite.


 Go's path.go is very small, a 215 line file:
 https://github.com/golang/go/blob/master/src/path/path.go
 Documentation: https://golang.org/pkg/path/


gocloc has it at 123 SLOC.


 Dlang's std.path is much more comprehensive with 4181 lines: 
 https://github.com/dlang/phobos/blob/master/std/path.d
 Documentation: https://dlang.org/phobos/std_path.html


dscanner says 1857 SLOC. Also, that includes unit tests, of which 
there are 72 so probably some 700SLOC there.

I don't think how big the files are is revelant for me, a user of 
the standard library. If I want to do something with paths and 
don't want to roll my own code, I pay a price for it in D, 
whereas it's relatively free with Go. It makes me want to 
substitute every usage of std.path.buildPath in my code with just 
`foo ~ "/" ~ bar ~ ...`.


 It's over an order of magnitude more code


More lines of code is a liability, not an advantage.


 and only takes twice as long to compile without unittests,


No... that's ~11.583x with no unittests and ~43.75x with. The 
former number not being interesting to me in the slightest.


 and it's only fair to compare the "non-unittest" version of 
 std.path with Go, since Go does not include unittests.


Absolutely not.

There is *0* compile-time penalty on Go programmers when they 
test their programs, whereas my compile times go up by a factor 
of 3 on a one-line program. And that's >3 multiplied by "already 
slow to begin with".

Atila
Apr 02 2018
next sibling parent Seb <seb wilzba.ch> writes:
On Monday, 2 April 2018 at 12:33:37 UTC, Atila Neves wrote:

 I don't think how big the files are is revelant for me, a user 
 of the standard library. If I want to do something with paths 
 and don't want to roll my own code, I pay a price for it in D, 
 whereas it's relatively free with Go. It makes me want to 
 substitute every usage of std.path.buildPath in my code with 
 just `foo ~ "/" ~ bar ~ ...`.


That will also be _faster_ at runtime ;-)
std.path is a performance nightmare.
My favorite example is buildNormalizedPath:

https://github.com/dlang/phobos/blob/2e105c72a9e5fa028f31f1898ec8d479a9bae4a1/std/path.d#L1738

We ought to start to separate: the good, the bad and the ugly 
modules.
Apr 02 2018
prev sibling next sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Monday, 2 April 2018 at 12:33:37 UTC, Atila Neves wrote:

 On Friday, 30 March 2018 at 16:41:42 UTC, Jonathan Marler wrote:

 Seems like you're comparing apples to oranges.


 No, I'm comparing one type of apple to another with regards to 
 weight in my shopping bag before I've even taken a bite.


My point was that GO's path library is very different from 
dlang's std.path library.  It has an order of magnitude less code 
so the point was that you're comparing a very small library with 
much less functionality to a very large one.


 It's over an order of magnitude more code


 More lines of code is a liability, not an advantage.


I didn't say anything about whether it was advantageous, the 
point was that it's more code so you should take that into 
account when you evaluate performance.  You're post was 
misleading because it was assuming that both libraries were 
comparable when in reality they appear to be very different.



 and it's only fair to compare the "non-unittest" version of 
 std.path with Go, since Go does not include unittests.


 Absolutely not.

 There is *0* compile-time penalty on Go programmers when they 
 test their programs, whereas my compile times go up by a factor 
 of 3 on a one-line program. And that's >3 multiplied by 
 "already slow to begin with".


My point was that if you want to compare "compile-time" 
performance, you should not include the unittests in D's time 
since Go does not include unittests.  In practicality, D should 
not be compiling in the standard library unittest by default.  
This is a problem that should be fixed but still doesn't change 
the fact that not taking this into consideration would be an 
unfair comparison.
Apr 02 2018
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, April 02, 2018 18:52:14 Jonathan Marler via Digitalmars-d wrote:

 On Monday, 2 April 2018 at 12:33:37 UTC, Atila Neves wrote:

 On Friday, 30 March 2018 at 16:41:42 UTC, Jonathan Marler wrote:

 Seems like you're comparing apples to oranges.


 No, I'm comparing one type of apple to another with regards to
 weight in my shopping bag before I've even taken a bite.


 My point was that GO's path library is very different from
 dlang's std.path library.  It has an order of magnitude less code
 so the point was that you're comparing a very small library with
 much less functionality to a very large one.


 It's over an order of magnitude more code


 More lines of code is a liability, not an advantage.


 I didn't say anything about whether it was advantageous, the
 point was that it's more code so you should take that into
 account when you evaluate performance.  You're post was
 misleading because it was assuming that both libraries were
 comparable when in reality they appear to be very different.


 and it's only fair to compare the "non-unittest" version of
 std.path with Go, since Go does not include unittests.


 Absolutely not.

 There is *0* compile-time penalty on Go programmers when they
 test their programs, whereas my compile times go up by a factor
 of 3 on a one-line program. And that's >3 multiplied by
 "already slow to begin with".


 My point was that if you want to compare "compile-time"
 performance, you should not include the unittests in D's time
 since Go does not include unittests.  In practicality, D should
 not be compiling in the standard library unittest by default.
 This is a problem that should be fixed but still doesn't change
 the fact that not taking this into consideration would be an
 unfair comparison.


You both have good points. On the one hand, yes, std.path is doing more, so
it's not surprising that it takes longer to compile, and in that sense, it's
comparing apples and oranges. However, from the standpoint of the user,
they're just calling these functions to get something done, and the
implementation details don't really matter. So, from the user's standpoint,
as far as compilation time goes, std.path is just worse. The reasons why are
kind of irrelevant from that perspective. So, in a way, you're both right.

Now, ultimately, given how D works and the functionality in std.path, I
don't know that it really makes sense for it to compile as fast as Go's
solution if Go's solution is doing so much less, but regardless, we should
be trying to at least eliminate the unnecessary slowdowns for things like
compiling unit tests, and we should be looking into how to speed up the
stuff that's slow to compile (e.g. improving the compilation speed of
templates in general would be a huge boon). std.path doesn't need to be as
slow to compile as it is to do what it does.

- Jonathan M Davis
Apr 02 2018
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 2 April 2018 at 18:52:14 UTC, Jonathan Marler wrote:


 My point was that GO's path library is very different from 
 dlang's std.path library.  It has an order of magnitude less 
 code so the point was that you're comparing a very small 
 library with much less functionality to a very large one.


I understood your point. I'm not sure you understood mine, which 
is: I don't care. I want to get work done, and I don't want to 
wait for the computer.


 I didn't say anything about whether it was advantageous, the 
 point was that it's more code so you should take that into 
 account when you evaluate performance.  You're post was 
 misleading because it was assuming that both libraries were 
 comparable when in reality they appear to be very different.


I disagree. They're very similar in the sense that, if I want to 
build a path and want to rely on the standard library, it takes 
vastly different amounts of time to compile my code in one 
situation vs the other.



 My point was that if you want to compare "compile-time" 
 performance, you should not include the unittests in D's time 
 since Go does not include unittests.


"Go does not include unittests"? Under some interpretations I 
guess that could be viewed as correct, but in practical terms I 
can write Go tests without an external library 
(https://golang.org/pkg/testing/)/ Whether it's a language 
keyword or not is irrelevant.

What _is_ relevant (to me) is that I can write Go code that 
manipulates paths and test it with everything building in less 
time that it takes to render a frame in a videogame, whereas in 
D...


 In practicality, D should not be compiling in the standard 
 library unittest by default.


I think everyone is in agreement here.


 This is a problem that should be fixed but still doesn't change 
 the fact that not taking this into consideration would be an 
 unfair comparison.


No, no, no, a thousand times more no.

We can't make a marketing point of D compiling so fast it might 
as well be a scripting language when it's not even true. I get a 
better edit-compile-test cycle in *C++*, which is embarassing.

Atila
Apr 03 2018
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Apr 03, 2018 at 10:24:15AM +0000, Atila Neves via Digitalmars-d wrote:
[...]

 We can't make a marketing point of D compiling so fast it might as
 well be a scripting language when it's not even true. I get a better
 edit-compile-test cycle in *C++*, which is embarassing.


[...]

+1.  And this is why our "fast code fast" slogan makes me cringe.  We
need to do some serious work on delivering that "fast" promise.


T

-- 
Caffeine underflow. Brain dumped.
Apr 03 2018
prev sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Tuesday, 3 April 2018 at 10:24:15 UTC, Atila Neves wrote:

 On Monday, 2 April 2018 at 18:52:14 UTC, Jonathan Marler wrote:


 My point was that GO's path library is very different from 
 dlang's std.path library.  It has an order of magnitude less 
 code so the point was that you're comparing a very small 
 library with much less functionality to a very large one.


 I understood your point. I'm not sure you understood mine, 
 which is: I don't care. I want to get work done, and I don't 
 want to wait for the computer.


You still missed my point.  You're post was saying that "D does 
not compile as fast as GO". But the libraries you're comparing 
are vastly different.  If you're post was saying, "dlang's 
std.path compiles much slower than GO's" then you would be fine.  
However, you're post was misleading saying the Go compile's 
faster than D in general, and I was pointing out that the use 
case you provided doesn't apply in the general case, it only 
applies to a library with the same name/type of functionality.



 I didn't say anything about whether it was advantageous, the 
 point was that it's more code so you should take that into 
 account when you evaluate performance.  You're post was 
 misleading because it was assuming that both libraries were 
 comparable when in reality they appear to be very different.


 I disagree. They're very similar in the sense that, if I want 
 to build a path and want to rely on the standard library, it 
 takes vastly different amounts of time to compile my code in 
 one situation vs the other.


I refer to my previous answer.  Your example shows that dlang's 
std.path compiles slower than GO's, but that doesn't say anything 
about the compile performance for both languages in the general 
case.  To make such a claim you should compare the exact same 
"functionality" implemented in both languages.



 My point was that if you want to compare "compile-time" 
 performance, you should not include the unittests in D's time 
 since Go does not include unittests.


 "Go does not include unittests"? Under some interpretations I 
 guess that could be viewed as correct, but in practical terms I 
 can write Go tests without an external library 
 (https://golang.org/pkg/testing/)/ Whether it's a language 
 keyword or not is irrelevant.

 What _is_ relevant (to me) is that I can write Go code that 
 manipulates paths and test it with everything building in less 
 time that it takes to render a frame in a videogame, whereas in 
 D...


You're totally misunderstanding me.  I was just saying that if 
you want to compare the compile speed of D vs GO (IN THE GENERAL 
CASE), you should not include the unittests in D's performance 
because you weren't including them in your GO example.


 This is a problem that should be fixed but still doesn't 
 change the fact that not taking this into consideration would 
 be an unfair comparison.


 No, no, no, a thousand times more no.

 We can't make a marketing point of D compiling so fast it might 
 as well be a scripting language when it's not even true. I get 
 a better edit-compile-test cycle in *C++*, which is embarassing.

 Atila


You totally misunderstood what I was saying once again.  I agree 
with what you said here, but it has nothing to do with what I was 
saying.

If your point is that it takes too long to access std.path's 
functionality then I completely agree. What I am arguing against 
is that your example is not evidence that GO compiles faster than 
D in general. You're example is comparing 2 different libraries 
in 2 different languages, not about the languages themselves.
Apr 03 2018
next sibling parent reply Rubn <where is.this> writes:
On Tuesday, 3 April 2018 at 19:07:54 UTC, Jonathan Marler wrote:

 On Tuesday, 3 April 2018 at 10:24:15 UTC, Atila Neves wrote:

 On Monday, 2 April 2018 at 18:52:14 UTC, Jonathan Marler wrote:

 My point was that if you want to compare "compile-time" 
 performance, you should not include the unittests in D's time 
 since Go does not include unittests.


 "Go does not include unittests"? Under some interpretations I 
 guess that could be viewed as correct, but in practical terms 
 I can write Go tests without an external library 
 (https://golang.org/pkg/testing/)/ Whether it's a language 
 keyword or not is irrelevant.

 What _is_ relevant (to me) is that I can write Go code that 
 manipulates paths and test it with everything building in less 
 time that it takes to render a frame in a videogame, whereas 
 in D...


 You're totally misunderstanding me.  I was just saying that if 
 you want to compare the compile speed of D vs GO (IN THE 
 GENERAL CASE), you should not include the unittests in D's 
 performance because you weren't including them in your GO 
 example.


I feel that's probably the case for any comparisons across two 
languages, you are going to have a person that is more 
knowledgeable in one language than another. Mistakes are going to 
be made, but I think it should be blatantly obvious that one 
language is going to compiler slower if it is compiling all the 
unittests for a library compared to one that isn't. That's just 
blatant bias against D, not a mistake from misunderstanding Go.
Apr 03 2018
parent reply bachmeier <no spam.net> writes:
On Tuesday, 3 April 2018 at 21:17:35 UTC, Rubn wrote:

 I feel that's probably the case for any comparisons across two 
 languages, you are going to have a person that is more 
 knowledgeable in one language than another. Mistakes are going 
 to be made, but I think it should be blatantly obvious that one 
 language is going to compiler slower if it is compiling all the 
 unittests for a library compared to one that isn't. That's just 
 blatant bias against D, not a mistake from misunderstanding Go.


Yeah, I don't understand that either. Unit tests can be 
arbitrarily large, so no matter how fast the compiler, it would 
always be possible to make it take longer than any other language.
Apr 03 2018
next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 3 April 2018 at 21:53:35 UTC, bachmeier wrote:

 On Tuesday, 3 April 2018 at 21:17:35 UTC, Rubn wrote:

 I feel that's probably the case for any comparisons across two 
 languages, you are going to have a person that is more 
 knowledgeable in one language than another. Mistakes are going 
 to be made, but I think it should be blatantly obvious that 
 one language is going to compiler slower if it is compiling 
 all the unittests for a library compared to one that isn't. 
 That's just blatant bias against D, not a mistake from 
 misunderstanding Go.


 Yeah, I don't understand that either. Unit tests can be 
 arbitrarily large, so no matter how fast the compiler, it would 
 always be possible to make it take longer than any other 
 language.


There were 0 tests in my example.
Apr 03 2018
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/03/2018 05:53 PM, bachmeier wrote:

 On Tuesday, 3 April 2018 at 21:17:35 UTC, Rubn wrote:

 I feel that's probably the case for any comparisons across two 
 languages, you are going to have a person that is more knowledgeable 
 in one language than another. Mistakes are going to be made, but I 
 think it should be blatantly obvious that one language is going to 
 compiler slower if it is compiling all the unittests for a library 
 compared to one that isn't. That's just blatant bias against D, not a 
 mistake from misunderstanding Go.


 
 Yeah, I don't understand that either. Unit tests can be arbitrarily 
 large, so no matter how fast the compiler, it would always be possible 
 to make it take longer than any other language.


Exactly, which is why I'm insisting this - and not compiler 
benchmarking, let alone idle chattaroo in the forums - is where we need 
to hit. What we have here, ladies and gentlemen, is a high-impact 
preapproved item of great general interest. Shall we start the auction?
Apr 03 2018
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, April 03, 2018 21:08:48 Andrei Alexandrescu via Digitalmars-d 
wrote:

 On 04/03/2018 05:53 PM, bachmeier wrote:

 On Tuesday, 3 April 2018 at 21:17:35 UTC, Rubn wrote:

 I feel that's probably the case for any comparisons across two
 languages, you are going to have a person that is more knowledgeable
 in one language than another. Mistakes are going to be made, but I
 think it should be blatantly obvious that one language is going to
 compiler slower if it is compiling all the unittests for a library
 compared to one that isn't. That's just blatant bias against D, not a
 mistake from misunderstanding Go.


 Yeah, I don't understand that either. Unit tests can be arbitrarily
 large, so no matter how fast the compiler, it would always be possible
 to make it take longer than any other language.


 Exactly, which is why I'm insisting this - and not compiler
 benchmarking, let alone idle chattaroo in the forums - is where we need
 to hit. What we have here, ladies and gentlemen, is a high-impact
 preapproved item of great general interest. Shall we start the auction?


Steven already said somewhere in here that he's planning to work on this
during the dconf hackathon. So, at least one person has stepped forward and
said that they'll look into it.

- Jonathan M Davis
Apr 03 2018
prev sibling next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 4 April 2018 at 01:08:48 UTC, Andrei Alexandrescu 
wrote:

 [ ... ]

 Exactly, which is why I'm insisting this - and not compiler 
 benchmarking, let alone idle chattaroo in the forums - is where 
 we need to hit. What we have here, ladies and gentlemen, is a 
 high-impact preapproved item of great general interest. Shall 
 we start the auction?


I have analyzed the problem;
And in this case it points to _std.uni_ as the main time-taker.

I have also created and uploaded a video where I show how to do 
this kind of profiling.
It got a bit carried away at the beginning so you can skip to 
10:45 :)
https://www.youtube.com/watch?v=28bOtu64CCM

For those are not up to watching me talk slowly:
Phobos is in dire need of having dependencies broken.

And having dedicated compile-time profiling tools is pretty 
useful.
Even if they are very rudimentary it's better then semi-randomly 
commenting out code.

--
Stefan
Apr 04 2018
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Wednesday, 4 April 2018 at 19:25:43 UTC, Stefan Koch wrote:

 On Wednesday, 4 April 2018 at 01:08:48 UTC, Andrei Alexandrescu 
 wrote:

 [ ... ]

 Exactly, which is why I'm insisting this - and not compiler 
 benchmarking, let alone idle chattaroo in the forums - is 
 where we need to hit. What we have here, ladies and gentlemen, 
 is a high-impact preapproved item of great general interest. 
 Shall we start the auction?


 I have analyzed the problem;
 And in this case it points to _std.uni_ as the main time-taker.


Could be because of huge tables. Does it point specifically to 
std.uni or its dependencies might count as well?

Anyway to import a binary file as an array? Turbo Pascal had a 
feature like that, it was awesome ;)

On other thought - maybe just .di + separate compilation is it.
Apr 04 2018
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 4 April 2018 at 20:02:56 UTC, Dmitry Olshansky 
wrote:

 On Wednesday, 4 April 2018 at 19:25:43 UTC, Stefan Koch wrote:

 On Wednesday, 4 April 2018 at 01:08:48 UTC, Andrei 
 Alexandrescu wrote:

 [ ... ]

 Exactly, which is why I'm insisting this - and not compiler 
 benchmarking, let alone idle chattaroo in the forums - is 
 where we need to hit. What we have here, ladies and 
 gentlemen, is a high-impact preapproved item of great general 
 interest. Shall we start the auction?


 I have analyzed the problem;
 And in this case it points to _std.uni_ as the main time-taker.


 Could be because of huge tables. Does it point specifically to 
 std.uni or its dependencies might count as well?

 Anyway to import a binary file as an array? Turbo Pascal had a 
 feature like that, it was awesome ;)

 On other thought - maybe just .di + separate compilation is it.


There is the string import feature, which can give you a char 
array which you can then cast to ubyte slice and from there to a 
ubyte pointer you cast that to a pointer of the target type and 
slice it. This requires the target type to have no pointers 
inside of it.
Apr 04 2018
prev sibling parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Wednesday, 4 April 2018 at 01:08:48 UTC, Andrei Alexandrescu 
wrote:

 Exactly, which is why I'm insisting this - and not compiler 
 benchmarking, let alone idle chattaroo in the forums - is where 
 we need to hit. What we have here, ladies and gentlemen, is a 
 high-impact preapproved item of great general interest. Shall 
 we start the auction?


Are you aware of this PR? https://github.com/dlang/dmd/pull/8124
Apr 04 2018
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 4 April 2018 at 20:04:04 UTC, Jack Stouffer wrote:

 On Wednesday, 4 April 2018 at 01:08:48 UTC, Andrei Alexandrescu 
 wrote:

 Exactly, which is why I'm insisting this - and not compiler 
 benchmarking, let alone idle chattaroo in the forums - is 
 where we need to hit. What we have here, ladies and gentlemen, 
 is a high-impact preapproved item of great general interest. 
 Shall we start the auction?


 Are you aware of this PR? https://github.com/dlang/dmd/pull/8124


This is but a layer of paint over the real problem.
Unneeded Dependencies. Programming should not be a game of jenga.
Piling things on top of other things rarely works out.
Apr 04 2018
next sibling parent Jack Stouffer <jack jackstouffer.com> writes:
On Wednesday, 4 April 2018 at 20:29:19 UTC, Stefan Koch wrote:

 On Wednesday, 4 April 2018 at 20:04:04 UTC, Jack Stouffer wrote:

 On Wednesday, 4 April 2018 at 01:08:48 UTC, Andrei 
 Alexandrescu wrote:

 Exactly, which is why I'm insisting this - and not compiler 
 benchmarking, let alone idle chattaroo in the forums - is 
 where we need to hit. What we have here, ladies and 
 gentlemen, is a high-impact preapproved item of great general 
 interest. Shall we start the auction?


 Are you aware of this PR? 
 https://github.com/dlang/dmd/pull/8124


 This is but a layer of paint over the real problem.
 Unneeded Dependencies. Programming should not be a game of 
 jenga.
 Piling things on top of other things rarely works out.


I encourage you to leave a review on the PR then with corrective 
measures.
Apr 04 2018
prev sibling next sibling parent Jonathan Marler <johnnymarler gmail.com> writes:
On Wednesday, 4 April 2018 at 20:29:19 UTC, Stefan Koch wrote:

 On Wednesday, 4 April 2018 at 20:04:04 UTC, Jack Stouffer wrote:

 On Wednesday, 4 April 2018 at 01:08:48 UTC, Andrei 
 Alexandrescu wrote:

 Exactly, which is why I'm insisting this - and not compiler 
 benchmarking, let alone idle chattaroo in the forums - is 
 where we need to hit. What we have here, ladies and 
 gentlemen, is a high-impact preapproved item of great general 
 interest. Shall we start the auction?


 Are you aware of this PR? 
 https://github.com/dlang/dmd/pull/8124


 This is but a layer of paint over the real problem.
 Unneeded Dependencies. Programming should not be a game of 
 jenga.
 Piling things on top of other things rarely works out.


Having unittests included from precompiled libraries is a problem 
in and of itself.  This is causing many templates to be 
instantiated that will never be used by the application, killing 
compilation time.

There are also other problems...here's a link to my description 
of "Lazy Imports" that I think would help other issues we 
currently have.

https://github.com/marler8997/dlangfeatures#lazy-imports
Apr 04 2018
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Apr 04, 2018 at 08:29:19PM +0000, Stefan Koch via Digitalmars-d wrote:

 On Wednesday, 4 April 2018 at 20:04:04 UTC, Jack Stouffer wrote:

 On Wednesday, 4 April 2018 at 01:08:48 UTC, Andrei Alexandrescu wrote:

 Exactly, which is why I'm insisting this - and not compiler
 benchmarking, let alone idle chattaroo in the forums - is where we
 need to hit. What we have here, ladies and gentlemen, is a
 high-impact preapproved item of great general interest. Shall we
 start the auction?


 
 Are you aware of this PR? https://github.com/dlang/dmd/pull/8124


 
 This is but a layer of paint over the real problem.  Unneeded
 Dependencies. Programming should not be a game of jenga.  Piling
 things on top of other things rarely works out.


Unneeded dependencies is (partly) addressed by this PR, by isolating
user code from the unittests (and by extension, their dependencies) of
external libraries that said user code has no interest in.

True, this does not solve the entire problem, which is that too much of
Phobos is involved in a hairball of inextricable dependencies.  The
situation actually has already improved since 3-4 years ago, when it was
much, much worse.  It's just that we have still some ways to go.

On the flip side, though, why *shouldn't* unittests in a Phobos module
make use of other Phobos modules?  It's one thing to say that Phobos
code proper (i.e., outside of unittests) should be as independent as
possible, and that I agree with, strongly.  But unittests should be free
to import other stuff in order to make testing easier, and also ddoc'd
unittest examples more relevant to the reader.  By separating the import
of a module from the importing of its unittests (which doesn't even make
sense from an API perspective -- why should downstream user code need to
pull in upstream unittests, which are supposed to be used only for
upstream development?), we provide freedom for upstream library authors
to make use of external components in order to test their code, without
penalizing downstream user code with dependencies they don't care about.

I'd say this PR strikes at the heart of this problem, it's not just
papering over the issue.  It's not a full solution, but it's an
important step towards a full solution.


T

-- 
English is useful because it is a mess. Since English is a mess, it maps well
onto the problem space, which is also a mess, which we call reality. Similarly,
Perl was designed to be a mess, though in the nicest of all possible ways. --
Larry Wall
Apr 04 2018
prev sibling next sibling parent reply burjui <bytefu gmail.com> writes:
Atila laid it out pretty clear: he doesn't care about the 
differences, he wants the work to be done. And I'm with him on 
that. Go and it's standard library may be way simpler, but it 
get's the job done (which is trivial in both cases, by the way) 
almost instantaneously, which is a much bigger deal than it seems 
to be. When your edit-compile cycle is that fast, it changes the 
way you write code, you develop a habit of writing smaller pieces 
of code and testing them more frequently. Remember that Linus 
Torvalds' talk about Git at Google?
https://www.youtube.com/watch?v=4XpnKHJAok8&t=3025

H. S. Teoh is not the only one here cringing at "fast code fast" 
on the main page. I use D from time to time for over 10 years 
now, and even used it at work and it was a relatively positive 
experience, thanks to vibe.d. But compilation times are just 
horrible - minimum 3 seconds for a 1500 lines project (on a 
8-core 4GHz CPU with 16 GB RAM), and that's after I ditched 
std.regex, made all imports qualified (didn't help that much, 
though) and switched to ld.gold. And I would be ok with slow 
compilation if DMD was smart enough, doing some graph magic, like 
extensive control flow analysis, and insane optimizations, but it 
doesn't. For example, Rust compilation times are no picnic 
either, but it's obvious why - you get nice good-looking error 
messages, tons of useful warnings and very fast programs free of 
memory corruption bugs. It's not the case with DMD, though. The 
language may be better than C++, but it's fastest compiler is 
slower and produces worse code? I'd rather not boast about speed 
at the main page in this situation. And god save us from ridicule 
by Goers.
Apr 03 2018
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Apr 03, 2018 at 11:09:10PM +0000, burjui via Digitalmars-d wrote:
[...]

 H. S. Teoh is not the only one here cringing at "fast code fast" on
 the main page. I use D from time to time for over 10 years now, and
 even used it at work and it was a relatively positive experience,
 thanks to vibe.d. But compilation times are just horrible - minimum 3
 seconds for a 1500 lines project (on a 8-core 4GHz CPU with 16 GB
 RAM), and that's after I ditched std.regex, made all imports qualified
 (didn't help that much, though) and switched to ld.gold.


3 seconds for 1500 lines?  Is that because you're using dub with its
unbearably slow compulsive network lookups?  Or because you imported
monsters like std.format?  Or because you have heavily-templated code?
Vibe.d is pretty heavy on templates (Diet templates, while pretty cool
from a geekiness POV, also slow things down like a hog because of heavy
template + CTFE usage).

Also, which compiler version did you use?  If you use nested templates
heavily (like UFCS chains of ranges), in older releases you may have run
into the exponential symbol size problem, where most of the compilation
time is spent generating, looking up, and reading symbols that are tens
of megabytes long.  After Rainers' symbol compression PR was merged,
compilation times on such code was hugely improved. As of a release or
two ago, this is no longer a problem.

I've noticed that if I don't use certain slow things, dmd is actually
lightning fast at compiling up to several thousand LOCs.  But as soon as
std.format enters the picture, or if you have recursive templates or
heavy CTFE, it rapidly deteriorates.  (Note, though, that relatively
simple template code doesn't significantly slow things down; it's when
you start doing things like manipulating type tuples AKA AliasSeq's,
recursive templates, compile-time loops like static foreach over
introspection, heavy CTFE, that things start grinding. Simple
parametrized types are still lightning fast, and last time I checked a
lot faster than, say, equivalent C++ code compiled with g++.)



 And I would be ok with slow compilation if DMD was smart enough, doing
 some graph magic, like extensive control flow analysis, and insane
 optimizations, but it doesn't. For example, Rust compilation times are
 no picnic either, but it's obvious why - you get nice good-looking
 error messages, tons of useful warnings and very fast programs free of
 memory corruption bugs. It's not the case with DMD, though. The
 language may be better than C++, but it's fastest compiler is slower
 and produces worse code? I'd rather not boast about speed at the main
 page in this situation.  And god save us from ridicule by Goers.


These days, I don't even look at benchmarks of dmd-compiled code
anymore.  For anything even remotely performance-related, I look at
gdc/ldc.  They do compile noticeably slower than dmd, but with the huge
benefit of far superior backends that gives me top runtime performance.

If Andrei & Walter are serious about this "fast code fast" thing, then
we'd better get our act together and do some serious optimization work,
both on the compiler itself, and on the quality of its codegen.


T

-- 
Bomb technician: If I'm running, try to keep up.
Apr 03 2018
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 3 April 2018 at 19:07:54 UTC, Jonathan Marler wrote:

 On Tuesday, 3 April 2018 at 10:24:15 UTC, Atila Neves wrote:

 On Monday, 2 April 2018 at 18:52:14 UTC, Jonathan Marler wrote:






 You still missed my point.


I got your point. I'm disagreeing.


 You're post was saying that "D does not compile as fast as GO".


Please show me where in my post where you think I said that.


 But the libraries you're comparing are vastly different.


Their sizes are different. I disagree that they're vastly 
different.


 If you're post was saying, "dlang's std.path compiles much 
 slower than GO's" then you would be fine.


That is exactly what I said.


 However, you're post was misleading saying the Go compile's 
 faster than D in general,


I never said that.


 and I was pointing out that the use case you provided doesn't 
 apply in the general case,


Maybe it applies in the general case, maybe it doesn't. I have no 
idea.


 it only applies to a library with the same name/type of 
 functionality.


I don't know about “only".



 You're totally misunderstanding me.  I was just saying that if 
 you want to compare the compile speed of D vs GO (IN THE 
 GENERAL CASE), you should not include the unittests in D's 
 performance because you weren't including them in your GO 
 example.


Include what? The Go standard library's own tests? libstdc++'s?

All the code I compiled was in that post. The only reason the Go 
file isn't just a one liner is because the silly opinionated 
language won't let me.

I showed how long it takes to compile the minimum amount of code 
necessary to import the part of the standard library responsible 
for paths in 3 languages. Then I showed how much slower it got in 
D with -unittest on the exact same one liner.

There isn't an equivalent in Go or C++. And yet one can write 
tests in them. And when one does, the compile-time penalty is 0.


 What I am arguing against is that your example is not evidence 
 that GO compiles faster than D in general.


I have no idea why you're arguing against something I never 
stated.


 You're example is comparing 2 different libraries in 2 
 different languages, not about the languages themselves.


No, I compared importing path functionality in files that did 
nothing else (except for some dummy code in Go) in *3* different 
languages. Then I showed that compiling the one liner in D with 
-unittest was slower than C++ by just a bit and nearly 50 slower 
than Go. With no actual tests in sight.
Apr 03 2018
parent Jonathan Marler <johnnymarler gmail.com> writes:
On Tuesday, 3 April 2018 at 23:29:34 UTC, Atila Neves wrote:

 On Tuesday, 3 April 2018 at 19:07:54 UTC, Jonathan Marler wrote:

 On Tuesday, 3 April 2018 at 10:24:15 UTC, Atila Neves wrote:

 On Monday, 2 April 2018 at 18:52:14 UTC, Jonathan Marler 
 wrote:






 You still missed my point.


 I got your point. I'm disagreeing.


I don't know why you keep saying you are "disagreeing" with me.  
It looks like we agree.  You're example is showing that DLANG's 
std.path compiles slower than GO's path library.  I agree.  All I 
was saying is that this example doesn't show that GO code 
compiles faster than D.  That was my one and only point.
Apr 03 2018
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/2/18 8:33 AM, Atila Neves wrote:


 No... that's ~11.583x with no unittests and ~43.75x with. The former 
 number not being interesting to me in the slightest.


Here is an interesting tidbit as well -- importing with -unittest takes 
4x as long AND NO CODE IS GENERATED OR WILL EVER BE GENERATED. 
Literally, the compiler is parsing, semantically analyzing, consuming 
300% more cycles just to "compile" code it will never actually use or 
generate files for, and as far as I know, it has no way of doing so.

I did a simple test:

dmd -c -v testpath.d | grep import | wc -l
60
dmd -c -v -unittest testpath.d | grep import | wc -l
87

So there are 27 more modules processed by DMD that make this 300% 
increase happen (Not to mention, ~60 imports for building paths...). 
Doing some bash-script work, I get the following differences


 import    core.atomic
 import    core.bitop
 import    core.checkedint
 import    core.math
 import    core.stdc.fenv
 import    core.stdc.math
 import    core.sync.exception
 import    core.sync.mutex
 import    core.sys.darwin.mach.port
 import    core.sys.darwin.mach.thread_act
 import    core.sys.darwin.pthread
 import    core.sys.posix.pthread
 import    core.sys.posix.sched
 import    core.sys.posix.semaphore
 import    core.sys.posix.stdlib
 import    core.sys.posix.sys.wait
 import    core.thread
 import    std.algorithm
 import    std.algorithm.internal
 import    std.algorithm.setops
 import    std.algorithm.sorting
 import    std.bitmanip
 import    std.math
 import    std.process
 import    std.random
 import    std.system
 import    std.utf


One disconcerting thing here is, not all of these imports come from 
std.path. Some of them come from unit tests in modules imported by 
std.path unittests. Because inside a unit test you want to have free 
access to all of phobos to do whatever the hell you want for testing, 
it's quite easy to import ALL of phobos when you do unit tests.

We really need to change the unittest import strategy.

-Steve
Apr 02 2018
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Apr 02, 2018 at 03:28:02PM -0400, Steven Schveighoffer via
Digitalmars-d wrote:
[...]

 We really need to change the unittest import strategy.


[...]

I think this has been established beyond reasonable doubt for the last
little while.  What about we start hashing out a solution?

AFAIK, the current proposal is to make it so that `-unittest` only takes
effect on modules that are being compiled (i.e., specified on the
command-line).  Imported modules will *not* have unittests compiled,
unless they have also been specified on the command-line.

The only thing blocking this proposal that I'm aware of, is that it will
break __traits(getUnitTests).  But I'm not sure if that's actually a
serious problem at all.  What if we make it so that a unittest block in
an imported module that isn't specified on the command-line is basically
treated as a template? I.e., parse the AST, but don't do anything with
it.  Don't bother running semantic, don't bother resolving identifiers
(in particular, import statements inside the unittest), etc.. Just leave
it in the AST as essentially translated syntax.  Then if
__traits(getUnitTests) is ever invoked, run semantic on the unittests in
the targeted module.

I'm not 100% certain, but I have a suspicion that this will mitigate the
breakage to __traits(getUnitTests) without compromising on the "don't
compile unittests outside the current list of modules to compile" fix.


T

-- 
Questions are the beginning of intelligence, but the fear of God is the
beginning of wisdom.
Apr 02 2018
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/2/18 3:55 PM, H. S. Teoh wrote:

 On Mon, Apr 02, 2018 at 03:28:02PM -0400, Steven Schveighoffer via
Digitalmars-d wrote:
 [...]

 We really need to change the unittest import strategy.


 [...]
 
 I think this has been established beyond reasonable doubt for the last
 little while.  What about we start hashing out a solution?
 
 AFAIK, the current proposal is to make it so that `-unittest` only takes
 effect on modules that are being compiled (i.e., specified on the
 command-line).  Imported modules will *not* have unittests compiled,
 unless they have also been specified on the command-line.
 
 The only thing blocking this proposal that I'm aware of, is that it will
 break __traits(getUnitTests).  But I'm not sure if that's actually a
 serious problem at all.  What if we make it so that a unittest block in
 an imported module that isn't specified on the command-line is basically
 treated as a template? I.e., parse the AST, but don't do anything with
 it.  Don't bother running semantic, don't bother resolving identifiers
 (in particular, import statements inside the unittest), etc.. Just leave
 it in the AST as essentially translated syntax.  Then if
 __traits(getUnitTests) is ever invoked, run semantic on the unittests in
 the targeted module.
 
 I'm not 100% certain, but I have a suspicion that this will mitigate the
 breakage to __traits(getUnitTests) without compromising on the "don't
 compile unittests outside the current list of modules to compile" fix.


Yes, I think this approach (at least, parsing unittests but deferring 
semantic until invoked) is the correct way to avoid breakage. It may be 
easy enough to do this without "treating it as a template", but I'm not 
sure.

-Steve
Apr 02 2018
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, April 02, 2018 12:55:05 H. S. Teoh via Digitalmars-d wrote:

 On Mon, Apr 02, 2018 at 03:28:02PM -0400, Steven Schveighoffer via
 Digitalmars-d wrote: [...]


 We really need to change the unittest import strategy.


 [...]

 I think this has been established beyond reasonable doubt for the last
 little while.  What about we start hashing out a solution?

 AFAIK, the current proposal is to make it so that `-unittest` only takes
 effect on modules that are being compiled (i.e., specified on the
 command-line).  Imported modules will *not* have unittests compiled,
 unless they have also been specified on the command-line.

 The only thing blocking this proposal that I'm aware of, is that it will
 break __traits(getUnitTests).  But I'm not sure if that's actually a
 serious problem at all.  What if we make it so that a unittest block in
 an imported module that isn't specified on the command-line is basically
 treated as a template? I.e., parse the AST, but don't do anything with
 it.  Don't bother running semantic, don't bother resolving identifiers
 (in particular, import statements inside the unittest), etc.. Just leave
 it in the AST as essentially translated syntax.  Then if
 __traits(getUnitTests) is ever invoked, run semantic on the unittests in
 the targeted module.

 I'm not 100% certain, but I have a suspicion that this will mitigate the
 breakage to __traits(getUnitTests) without compromising on the "don't
 compile unittests outside the current list of modules to compile" fix.


Having never used __traits(getUnitTest), I'm not very familiar with it, but
depending on what it does, it might be enough to register the fact that a
unittest block exists in the file, and then the unittest block itself only
needs to be analyzed enough to parse passed it. But I don't know. I'll have
to study up on what it does exactly to say much intelligent about it.

One concern I have is version(unittest) blocks. In order to avoid code
breakage, those would need to still be compiled in. I know that I've
personally used version(unittest) blocks that had package access level and
were then imported in the unit tests within that package in order to avoid
declaring helper types or functions in each module. So, treating
version(unittest) blocks like they're not there when importing a module
would definitely break som existing code. It could be argued that such uses
should be deprecated in some manner, but simply not compiling in anything
related to -unittest in imported modules would be a problem. However, aside
from maybe __traits(getUnitTests), I don't see why it would be a problem to
ignore code inside a unittest block except for when that module is being
compiled (or when that unittest block is inside a template that's being
compiled).

- Jonathan M Davis
Apr 02 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/2/18 4:05 PM, Jonathan M Davis wrote:


 One concern I have is version(unittest) blocks. In order to avoid code
 breakage, those would need to still be compiled in. I know that I've
 personally used version(unittest) blocks that had package access level and
 were then imported in the unit tests within that package in order to avoid
 declaring helper types or functions in each module. So, treating
 version(unittest) blocks like they're not there when importing a module
 would definitely break som existing code. It could be argued that such uses
 should be deprecated in some manner, but simply not compiling in anything
 related to -unittest in imported modules would be a problem. However, aside
 from maybe __traits(getUnitTests), I don't see why it would be a problem to
 ignore code inside a unittest block except for when that module is being
 compiled (or when that unittest block is inside a template that's being
 compiled).


version(unittest) blocks would have to be rethought in your code at some 
point. I think they have to go through the same rules -- only if 
__traits(getUnitTests) is used would it be included.

I have run into horrible problems with version(unittest), where I 
actually defined a virtual function (by accident, of course) for 
unittests only, and then when you compile/link with different modes of 
-unittest you have spectacular failure. There have been similar problems 
in phobos: https://github.com/dlang/phobos/pull/5932

There are other ways to achieve the same things, and while not as 
convenient, they would be safer and less troublesome for imports.

-Steve
Apr 02 2018
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, April 02, 2018 17:15:40 Steven Schveighoffer via Digitalmars-d 
wrote:

 On 4/2/18 4:05 PM, Jonathan M Davis wrote:

 One concern I have is version(unittest) blocks. In order to avoid code
 breakage, those would need to still be compiled in. I know that I've
 personally used version(unittest) blocks that had package access level
 and were then imported in the unit tests within that package in order
 to avoid declaring helper types or functions in each module. So,
 treating version(unittest) blocks like they're not there when importing
 a module would definitely break som existing code. It could be argued
 that such uses should be deprecated in some manner, but simply not
 compiling in anything related to -unittest in imported modules would be
 a problem. However, aside from maybe __traits(getUnitTests), I don't
 see why it would be a problem to ignore code inside a unittest block
 except for when that module is being compiled (or when that unittest
 block is inside a template that's being compiled).


 version(unittest) blocks would have to be rethought in your code at some
 point. I think they have to go through the same rules -- only if
 __traits(getUnitTests) is used would it be included.

 I have run into horrible problems with version(unittest), where I
 actually defined a virtual function (by accident, of course) for
 unittests only, and then when you compile/link with different modes of
 -unittest you have spectacular failure. There have been similar problems
 in phobos: https://github.com/dlang/phobos/pull/5932

 There are other ways to achieve the same things, and while not as
 convenient, they would be safer and less troublesome for imports.


I do think that there's some value in being able to do version(unittest) to
code and import it, but I can certainly live with losing that ability if
it's really needed to fix the situation with unit tests being compiled in
when they shouldn't. Regardless, my main point was that there is existing
code which relies on the current behavior. So, if we want to change that
behavior without just pulling the rug out from under existing code, we're
going to need some sort of deprecation process for it, whereas I would think
that it would be possible to stop compiling unittest blocks when importing
modules without breaking any code.

- Jonathan M Davis
Apr 02 2018
prev sibling next sibling parent bachmeier <no spam.net> writes:
On Friday, 30 March 2018 at 16:12:44 UTC, Atila Neves wrote:


     /tmp % time dmd -c  foo.d
     dmd -c foo.d  0.12s user 0.02s system 98% cpu 0.139 total


 That... doesn't seem too fast to me. But wait, there's more:

     /tmp % time dmd -c -unittest foo.d
     dmd -c -unittest foo.d  0.46s user 0.06s system 99% cpu 
 0.525 total


I did the same timings on my computer, and I got 0.128s and 
0.272s. I'm on DMD 2.075.1, so something must have changed in 
std.path since then.
Mar 30 2018
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/30/18 12:12 PM, Atila Neves wrote:

 Fast code fast, they said. It'll be fun, they said. Here's a D file:
 
      import std.path;
 
 
 Yep, that's all there is to it. Let's compile it on my laptop:
 
      /tmp % time dmd -c  foo.d
      dmd -c foo.d  0.12s user 0.02s system 98% cpu 0.139 total


Could be faster.


 That... doesn't seem too fast to me. But wait, there's more:
 
      /tmp % time dmd -c -unittest foo.d
      dmd -c -unittest foo.d  0.46s user 0.06s system 99% cpu 0.525 total


Not fast. We need to make -unittest only affect the built module. Even 
though it breaks certain uses of __traits(getUnittests). No two ways 
about it. Who can work on that?

Andrei
Mar 30 2018
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 30 March 2018 at 20:17:39 UTC, Andrei Alexandrescu 
wrote:

 On 3/30/18 12:12 PM, Atila Neves wrote:

 Fast code fast, they said. It'll be fun, they said. Here's a D 
 file:
 
      import std.path;
 
 
 Yep, that's all there is to it. Let's compile it on my laptop:
 
      /tmp % time dmd -c  foo.d
      dmd -c foo.d  0.12s user 0.02s system 98% cpu 0.139 total


 Could be faster.


 That... doesn't seem too fast to me. But wait, there's more:
 
      /tmp % time dmd -c -unittest foo.d
      dmd -c -unittest foo.d  0.46s user 0.06s system 99% cpu 
 0.525 total


 Not fast. We need to make -unittest only affect the built 
 module. Even though it breaks certain uses of 
 __traits(getUnittests). No two ways about it. Who can work on 
 that?

 Andrei


unittests by nature usually have a short list of dependencies and 
therefore their compilation can be parallelized.
There are many performance leaks we can fix, before we need to 
think about breaking useful features!
Mar 30 2018
parent Seb <seb wilzba.ch> writes:
On Friday, 30 March 2018 at 20:40:16 UTC, Stefan Koch wrote:

 On Friday, 30 March 2018 at 20:17:39 UTC, Andrei Alexandrescu 
 wrote:

 On 3/30/18 12:12 PM, Atila Neves wrote:

 Fast code fast, they said. It'll be fun, they said. Here's a 
 D file:
 
      import std.path;
 
 
 Yep, that's all there is to it. Let's compile it on my laptop:
 
      /tmp % time dmd -c  foo.d
      dmd -c foo.d  0.12s user 0.02s system 98% cpu 0.139 total


 Could be faster.


 That... doesn't seem too fast to me. But wait, there's more:
 
      /tmp % time dmd -c -unittest foo.d
      dmd -c -unittest foo.d  0.46s user 0.06s system 99% cpu 
 0.525 total


 Not fast. We need to make -unittest only affect the built 
 module. Even though it breaks certain uses of 
 __traits(getUnittests). No two ways about it. Who can work on 
 that?

 Andrei


 unittests by nature usually have a short list of dependencies 
 and therefore their compilation can be parallelized.
 There are many performance leaks we can fix, before we need to 
 think about breaking useful features!


No one wants to run the std.regex's testsuite when they pass 
-unittest or instantiate and run all tests for all Tuple 
combinations present.
If you can make dmd so fast that this happens unnoticed, that 
would be amazing, otherwise I suggest we don't run Phobos's 
templated unittests in user-code.
Mar 30 2018
prev sibling next sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Friday, 30 March 2018 at 20:17:39 UTC, Andrei Alexandrescu 
wrote:

 On 3/30/18 12:12 PM, Atila Neves wrote:

 Fast code fast, they said. It'll be fun, they said. Here's a D 
 file:
 
      import std.path;
 
 
 Yep, that's all there is to it. Let's compile it on my laptop:
 
      /tmp % time dmd -c  foo.d
      dmd -c foo.d  0.12s user 0.02s system 98% cpu 0.139 total


 Could be faster.


 That... doesn't seem too fast to me. But wait, there's more:
 
      /tmp % time dmd -c -unittest foo.d
      dmd -c -unittest foo.d  0.46s user 0.06s system 99% cpu 
 0.525 total


 Not fast. We need to make -unittest only affect the built 
 module. Even though it breaks certain uses of 
 __traits(getUnittests). No two ways about it. Who can work on 
 that?

 Andrei


If you approve of the -unittest=<pattern> approach then 
timotheecour has already offered to implement this.  It's pattern 
matching would work the same as -i and would also use the 
"implied standard exclusions" that -i uses, namely,

-unittest=-std -unittest=-etc -unittest=-core

This would mean that by default, just passing "-unittest" would 
exclude druntime/phobos just like "-i" by itself also does.
Mar 31 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, March 31, 2018 08:28:31 Jonathan Marler via Digitalmars-d 
wrote:

 On Friday, 30 March 2018 at 20:17:39 UTC, Andrei Alexandrescu

 wrote:

 On 3/30/18 12:12 PM, Atila Neves wrote:

 Fast code fast, they said. It'll be fun, they said. Here's a D

 file:
      import std.path;

 Yep, that's all there is to it. Let's compile it on my laptop:
      /tmp % time dmd -c  foo.d
      dmd -c foo.d  0.12s user 0.02s system 98% cpu 0.139 total


 Could be faster.


 That... doesn't seem too fast to me. But wait, there's more:
      /tmp % time dmd -c -unittest foo.d
      dmd -c -unittest foo.d  0.46s user 0.06s system 99% cpu

 0.525 total


 Not fast. We need to make -unittest only affect the built
 module. Even though it breaks certain uses of
 __traits(getUnittests). No two ways about it. Who can work on
 that?

 Andrei


 If you approve of the -unittest=<pattern> approach then
 timotheecour has already offered to implement this.  It's pattern
 matching would work the same as -i and would also use the
 "implied standard exclusions" that -i uses, namely,

 -unittest=-std -unittest=-etc -unittest=-core

 This would mean that by default, just passing "-unittest" would
 exclude druntime/phobos just like "-i" by itself also does.


And every time you used another library, you'd have the same problem and
have to add -unittest=- whatever for each and every one of them, or you
would have to use -unittest= with everything from your application or
library rather than using -unittest. I really don't see how that scales
well, and it's way too manual and too easy to screw up. It might be a decent
idea for a workaround, but it's not a great solution.

IMHO, this is really something that should be handled by the compiler. It
simply shouldn't be compiling in the unittest blocks for modules that you're
not compiling directly. And if that's done right, then this whole problem
goes away without having to make sure that every project you work on is
configured correctly to avoid pulling in the unit tests from everything that
it depends on.

And maybe figuring out what to do about __traits(getUnittests) complicates
things, but it seems like the fact that we're even having this problem is
due to a flaw in the design of D's unit tests and that that should be fixed,
not worked around.

- Jonathan M Davis
Mar 31 2018
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/18 5:37 PM, Jonathan M Davis wrote:

 And every time you used another library, you'd have the same problem and
 have to add -unittest=- whatever for each and every one of them, or you
 would have to use -unittest= with everything from your application or
 library rather than using -unittest. I really don't see how that scales
 well, and it's way too manual and too easy to screw up. It might be a decent
 idea for a workaround, but it's not a great solution.


Nod. We really need to get to the point where the application of 
unittest is modular.
Mar 31 2018
prev sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Saturday, 31 March 2018 at 21:37:13 UTC, Jonathan M Davis 
wrote:

 On Saturday, March 31, 2018 08:28:31 Jonathan Marler via 
 Digitalmars-d wrote:

 On Friday, 30 March 2018 at 20:17:39 UTC, Andrei Alexandrescu

 wrote:

 On 3/30/18 12:12 PM, Atila Neves wrote:

 Fast code fast, they said. It'll be fun, they said. Here's 
 a D

 file:
      import std.path;

 Yep, that's all there is to it. Let's compile it on my 
 laptop:
      /tmp % time dmd -c  foo.d
      dmd -c foo.d  0.12s user 0.02s system 98% cpu 0.139 
 total


 Could be faster.


 That... doesn't seem too fast to me. But wait, there's more:
      /tmp % time dmd -c -unittest foo.d
      dmd -c -unittest foo.d  0.46s user 0.06s system 99% cpu

 0.525 total


 Not fast. We need to make -unittest only affect the built 
 module. Even though it breaks certain uses of 
 __traits(getUnittests). No two ways about it. Who can work 
 on that?

 Andrei


 If you approve of the -unittest=<pattern> approach then 
 timotheecour has already offered to implement this.  It's 
 pattern matching would work the same as -i and would also use 
 the "implied standard exclusions" that -i uses, namely,

 -unittest=-std -unittest=-etc -unittest=-core

 This would mean that by default, just passing "-unittest" 
 would exclude druntime/phobos just like "-i" by itself also 
 does.


 And every time you used another library, you'd have the same 
 problem and have to add -unittest=- whatever for each and every 
 one of them, or you would have to use -unittest= with 
 everything from your application or library rather than using 
 -unittest. I really don't see how that scales well, and it's 
 way too manual and too easy to screw up. It might be a decent 
 idea for a workaround, but it's not a great solution.

 IMHO, this is really something that should be handled by the 
 compiler. It simply shouldn't be compiling in the unittest 
 blocks for modules that you're not compiling directly. And if 
 that's done right, then this whole problem goes away without 
 having to make sure that every project you work on is 
 configured correctly to avoid pulling in the unit tests from 
 everything that it depends on.

 And maybe figuring out what to do about __traits(getUnittests) 
 complicates things, but it seems like the fact that we're even 
 having this problem is due to a flaw in the design of D's unit 
 tests and that that should be fixed, not worked around.

 - Jonathan M Davis


Let's make this conversation a bit more concrete, I'm not sure we 
are discussing the exact same thing.

The proposed solution is to have -unittest mean "compile 
unittests for all 'compiled modules' according to the pattern 
rules".  The default pattern rule is to include all modules 
except druntime/phobos.

Say you have two "packages" foo and bar that contain modules 
inside their respective directories.  With the proposed 
-unittest=<pattern> this is the semantics you would get.


foo/*.d and bar/*.d

foo/*.d

bar/*.d

foo/x.d

foo/*.d

Note that the default behavior makes sense, but this mechanism 
also allows you more fine-graned control to limit unittesting to 
certain packages or modules.  This degree of control would be 
quite helpful to me, any of the previously listed use cases 
represent valid scenarios that I would like to be able to do.  Do 
you have another solution that would provide this functionality?  
I don't see any reason not to support these use cases.
Mar 31 2018
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, April 01, 2018 01:25:41 Jonathan Marler via Digitalmars-d wrote:

 On Saturday, 31 March 2018 at 21:37:13 UTC, Jonathan M Davis

 wrote:

 On Saturday, March 31, 2018 08:28:31 Jonathan Marler via

 Digitalmars-d wrote:

 On Friday, 30 March 2018 at 20:17:39 UTC, Andrei Alexandrescu

 wrote:

 On 3/30/18 12:12 PM, Atila Neves wrote:

 Fast code fast, they said. It'll be fun, they said. Here's
 a D

 file:
      import std.path;

 Yep, that's all there is to it. Let's compile it on my

 laptop:
      /tmp % time dmd -c  foo.d
      dmd -c foo.d  0.12s user 0.02s system 98% cpu 0.139

 total


 Could be faster.


 That... doesn't seem too fast to me. But wait, there's more:
      /tmp % time dmd -c -unittest foo.d
      dmd -c -unittest foo.d  0.46s user 0.06s system 99% cpu

 0.525 total


 Not fast. We need to make -unittest only affect the built
 module. Even though it breaks certain uses of
 __traits(getUnittests). No two ways about it. Who can work
 on that?

 Andrei


 If you approve of the -unittest=<pattern> approach then
 timotheecour has already offered to implement this.  It's
 pattern matching would work the same as -i and would also use
 the "implied standard exclusions" that -i uses, namely,

 -unittest=-std -unittest=-etc -unittest=-core

 This would mean that by default, just passing "-unittest"
 would exclude druntime/phobos just like "-i" by itself also
 does.


 And every time you used another library, you'd have the same
 problem and have to add -unittest=- whatever for each and every
 one of them, or you would have to use -unittest= with
 everything from your application or library rather than using
 -unittest. I really don't see how that scales well, and it's
 way too manual and too easy to screw up. It might be a decent
 idea for a workaround, but it's not a great solution.

 IMHO, this is really something that should be handled by the
 compiler. It simply shouldn't be compiling in the unittest
 blocks for modules that you're not compiling directly. And if
 that's done right, then this whole problem goes away without
 having to make sure that every project you work on is
 configured correctly to avoid pulling in the unit tests from
 everything that it depends on.

 And maybe figuring out what to do about __traits(getUnittests)
 complicates things, but it seems like the fact that we're even
 having this problem is due to a flaw in the design of D's unit
 tests and that that should be fixed, not worked around.

 - Jonathan M Davis


 Let's make this conversation a bit more concrete, I'm not sure we
 are discussing the exact same thing.

 The proposed solution is to have -unittest mean "compile
 unittests for all 'compiled modules' according to the pattern
 rules".  The default pattern rule is to include all modules
 except druntime/phobos.

 Say you have two "packages" foo and bar that contain modules
 inside their respective directories.  With the proposed
 -unittest=<pattern> this is the semantics you would get.


 foo/*.d and bar/*.d

 foo/*.d

 bar/*.d

 foo/x.d

 foo/*.d

 Note that the default behavior makes sense, but this mechanism
 also allows you more fine-graned control to limit unittesting to
 certain packages or modules.  This degree of control would be
 quite helpful to me, any of the previously listed use cases
 represent valid scenarios that I would like to be able to do.  Do
 you have another solution that would provide this functionality?
 I don't see any reason not to support these use cases.


My point is that this is not a good solution for the problem of dependencies
having their unit tests included in your project. That should simply not be
happening.

Discussions of ways to tell the compiler to enable unit tests for some
modules and not others within your own project are IMHO therefore a separate
issue. Now, as for the specific proposal trying to solve the issue of
running only certain unit tests in your project, I think that it's overly
complicated once you start having to deal with stuff like exclusions. I'd
argue for simply compiling the module or modules that you care about with
-unittest and that you not use -unittest when compiling the other modules.
All of this is going to have to be controlled by whatever build system
you're using anyway, since it really doesn't scale to keep calling the
compiler directly like that, and once you're dealing with a build system
like dub or make or whatever, then it can have support for something like

dub test --only=foo

where it then compiles only the module foo with -unittest, and compiles the
rest with out -unittest. There shouldn't be any need to change dmd for that,
just the build tool.

But regardless, I don't think that such a feature is a good solution for the
problem of the unit tests from dependencies being compiled in. The compiler
should be fixed to take care of that.

- Jonathan M Davis
Mar 31 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/30/2018 1:17 PM, Andrei Alexandrescu wrote:

 Could be faster.


It's been a fair amount of time since somebody has done profiling of dmd. It 
needs to be done. There's probably plenty of low hanging fruit. Speculating 
about why it is slow is pointless without data.
Mar 31 2018
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 1 April 2018 at 02:40:26 UTC, Walter Bright wrote:

 On 3/30/2018 1:17 PM, Andrei Alexandrescu wrote:

 Could be faster.


 It's been a fair amount of time since somebody has done 
 profiling of dmd. It needs to be done. There's probably plenty 
 of low hanging fruit. Speculating about why it is slow is 
 pointless without data.


have a look at https://github.com/dlang/dmd/pull/7792 for a 
little profiling utility which tells you about which parts of a 
programm draw compiletime.

Whenever I see long compile times 90% of it is due to templates.
Apr 01 2018
prev sibling next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Sunday, 1 April 2018 at 02:40:26 UTC, Walter Bright wrote:

 On 3/30/2018 1:17 PM, Andrei Alexandrescu wrote:

 Could be faster.


 It's been a fair amount of time since somebody has done 
 profiling of dmd. It needs to be done. There's probably plenty 
 of low hanging fruit. Speculating about why it is slow is 
 pointless without data.


I profiled it as soon as I had the numbers. I didn't bother to 
mention it because I thought I'd just work on making dmd faster 
instead.

I seem to be the one who feels the most pain by this, it'd be 
silly of me to expect anyone else to work on it.
Apr 02 2018
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 2 April 2018 at 12:35:03 UTC, Atila Neves wrote:

 On Sunday, 1 April 2018 at 02:40:26 UTC, Walter Bright wrote:

 On 3/30/2018 1:17 PM, Andrei Alexandrescu wrote:

 Could be faster.


 It's been a fair amount of time since somebody has done 
 profiling of dmd. It needs to be done. There's probably plenty 
 of low hanging fruit. Speculating about why it is slow is 
 pointless without data.


 I profiled it as soon as I had the numbers. I didn't bother to 
 mention it because I thought I'd just work on making dmd faster 
 instead.

 I seem to be the one who feels the most pain by this, it'd be 
 silly of me to expect anyone else to work on it.


Making certain parts of the compiler faster is also on my agenda.
Feel free to talk to me anytime :)
Apr 02 2018
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Apr 02, 2018 at 12:35:03PM +0000, Atila Neves via Digitalmars-d wrote:

 On Sunday, 1 April 2018 at 02:40:26 UTC, Walter Bright wrote:

 On 3/30/2018 1:17 PM, Andrei Alexandrescu wrote:

 Could be faster.


 
 It's been a fair amount of time since somebody has done profiling of
 dmd. It needs to be done. There's probably plenty of low hanging
 fruit.  Speculating about why it is slow is pointless without data.


 
 I profiled it as soon as I had the numbers. I didn't bother to mention
 it because I thought I'd just work on making dmd faster instead.
 
 I seem to be the one who feels the most pain by this, it'd be silly of
 me to expect anyone else to work on it.


Recently I've also started feeling the sting of slow compile times.  I
don't know why I didn't notice it before... either I was jaded by C++
compile times in the past and even D's "slower" compile times are fast
by comparison, or maybe dmd performance has degraded over time? It's a
possibility we should investigate.  Or most likely, the recent
templatization of certain parts of druntime and Phobos has exacerbated
the problem to the point that it's now very noticeable.  The recent
fiasco with __switch and `import std.format` come to mind.

While currently all the fingers seem to be pointing at templates, I have
to say that I'm a big fan of templated code, and would rather see an
improvement in how the compiler deals with templates, than a reduction
in the usage of templates. Some of D's key selling features being
templates and meta-programming, how they are implemented is pretty
important.


T

-- 
Береги платье снову, а здоровье смолоду.
Apr 02 2018
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/02/2018 08:35 AM, Atila Neves wrote:

 On Sunday, 1 April 2018 at 02:40:26 UTC, Walter Bright wrote:

 On 3/30/2018 1:17 PM, Andrei Alexandrescu wrote:

 Could be faster.


 It's been a fair amount of time since somebody has done profiling of 
 dmd. It needs to be done. There's probably plenty of low hanging 
 fruit. Speculating about why it is slow is pointless without data.


 
 I profiled it as soon as I had the numbers. I didn't bother to mention 
 it because I thought I'd just work on making dmd faster instead.
 
 I seem to be the one who feels the most pain by this, it'd be silly of 
 me to expect anyone else to work on it.


A large and obvious time sink is that unittests in library code are 
built when user code is to be unittested. I'd recommend doing this 
before any other optimization.
Apr 02 2018
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Apr 02, 2018 at 12:09:01PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 04/02/2018 08:35 AM, Atila Neves wrote:

 On Sunday, 1 April 2018 at 02:40:26 UTC, Walter Bright wrote:

 On 3/30/2018 1:17 PM, Andrei Alexandrescu wrote:

 Could be faster.


 
 It's been a fair amount of time since somebody has done profiling
 of dmd. It needs to be done. There's probably plenty of low
 hanging fruit. Speculating about why it is slow is pointless
 without data.


 
 I profiled it as soon as I had the numbers. I didn't bother to
 mention it because I thought I'd just work on making dmd faster
 instead.
 
 I seem to be the one who feels the most pain by this, it'd be silly
 of me to expect anyone else to work on it.


 
 A large and obvious time sink is that unittests in library code are
 built when user code is to be unittested. I'd recommend doing this
 before any other optimization.


Lately this has been mentioned more and more frequently.  So what's the
status on this?  Are we going to move forward with making it so that
-unittest only applies to modules supplied on the command-line?

Has there been an investigation into how __traits(getUnitTests) could be
made to work in spite of this change?


T

-- 
Indifference will certainly be the downfall of mankind, but who cares? --
Miquel van Smoorenburg
Apr 02 2018
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/02/2018 12:22 PM, H. S. Teoh wrote:

 Lately this has been mentioned more and more frequently.  So what's the
 status on this?  Are we going to move forward with making it so that
 -unittest only applies to modules supplied on the command-line?
 
 Has there been an investigation into how __traits(getUnitTests) could be
 made to work in spite of this change?


Walter and I are willing to go with the change assuming no showstopper 
presents itself, but we don't have time budgeted for it. So it needs a 
strong champion.
Apr 02 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/2/18 12:28 PM, Andrei Alexandrescu wrote:

 On 04/02/2018 12:22 PM, H. S. Teoh wrote:

 Lately this has been mentioned more and more frequently.  So what's the
 status on this?  Are we going to move forward with making it so that
 -unittest only applies to modules supplied on the command-line?

 Has there been an investigation into how __traits(getUnitTests) could be
 made to work in spite of this change?


 
 Walter and I are willing to go with the change assuming no showstopper 
 presents itself, but we don't have time budgeted for it. So it needs a 
 strong champion.


If nobody else volunteers, I might give it a shot at the hackathon. I'd 
need help figuring out where the important bits are, so I may be bugging 
compiler devs that day :)

-Steve
Apr 02 2018
parent Seb <seb wilzba.ch> writes:
On Monday, 2 April 2018 at 19:32:33 UTC, Steven Schveighoffer 
wrote:

 On 4/2/18 12:28 PM, Andrei Alexandrescu wrote:

 On 04/02/2018 12:22 PM, H. S. Teoh wrote:

 Lately this has been mentioned more and more frequently.  So 
 what's the
 status on this?  Are we going to move forward with making it 
 so that
 -unittest only applies to modules supplied on the 
 command-line?

 Has there been an investigation into how 
 __traits(getUnitTests) could be
 made to work in spite of this change?


 
 Walter and I are willing to go with the change assuming no 
 showstopper presents itself, but we don't have time budgeted 
 for it. So it needs a strong champion.


 If nobody else volunteers, I might give it a shot at the 
 hackathon. I'd need help figuring out where the important bits 
 are, so I may be bugging compiler devs that day :)

 -Steve


Wow. I think we better get an extra hackathon day then :)
(there's a lot more work just "like this" lying around)
Apr 02 2018
prev sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 1 April 2018 at 02:40:26 UTC, Walter Bright wrote:

 On 3/30/2018 1:17 PM, Andrei Alexandrescu wrote:

 Could be faster.


 It's been a fair amount of time since somebody has done 
 profiling of dmd. It needs to be done. There's probably plenty 
 of low hanging fruit. Speculating about why it is slow is 
 pointless without data.


Well Surprisingly enough the biggest function in a profile for 
sane code, (no templates) is called spellerY.
Apr 03 2018
prev sibling next sibling parent Seb <seb wilzba.ch> writes:
On Friday, 30 March 2018 at 16:12:44 UTC, Atila Neves wrote:

 Fast code fast, they said. It'll be fun, they said. Here's a D 
 file:

     import std.path;


 Yep, that's all there is to it. Let's compile it on my laptop:

     /tmp % time dmd -c  foo.d
     dmd -c foo.d  0.12s user 0.02s system 98% cpu 0.139 total


We had a recent discussion about this:

https://github.com/dlang/phobos/pull/5916

tl;dr:
- e.g. all unicode tables are probably imported and CTFE-ed 
(that's a huge chunk)
- Vladimir has written an excellent tool (-> 
https://github.com/CyberShadow/DBuildStat)
Mar 30 2018
prev sibling parent reply Kagamin <spam here.lot> writes:
On Friday, 30 March 2018 at 16:12:44 UTC, Atila Neves wrote:

 * Building a whole project in C++ still takes a lot longer 
 since D scales much better, but that's not my typical worflow, 
 nor should it be anyone else's.


I can write code for days without even saving :)
What's the point to compile code that's not supposed to work?
Apr 03 2018
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 3 April 2018 at 20:47:48 UTC, Kagamin wrote:

 On Friday, 30 March 2018 at 16:12:44 UTC, Atila Neves wrote:

 * Building a whole project in C++ still takes a lot longer 
 since D scales much better, but that's not my typical worflow, 
 nor should it be anyone else's.


 I can write code for days without even saving :)


That sentence might as well be fingernails on a blackboard for 
me! I save compulsively. Whenever I stop typing, C-x C-s it is 
for me.


 What's the point to compile code that's not supposed to work?


TDD.
Apr 03 2018
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Apr 03, 2018 at 10:59:13PM +0000, Atila Neves via Digitalmars-d wrote:

 On Tuesday, 3 April 2018 at 20:47:48 UTC, Kagamin wrote:

 On Friday, 30 March 2018 at 16:12:44 UTC, Atila Neves wrote:

 * Building a whole project in C++ still takes a lot longer since D
 scales much better, but that's not my typical worflow, nor should
 it be anyone else's.


 
 I can write code for days without even saving :)


 
 That sentence might as well be fingernails on a blackboard for me! I
 save compulsively. Whenever I stop typing, C-x C-s it is for me.


[...]

Yeah, after having lost valuable work more often than I'd care to admit
due to unexpected power outages and other hardware problems, I've
developed a compulsive twitch in my fingers that automatically saves my
file every other minute.  It has become so automatic that it's basically
unconscious now.  I can't imagine not saving for long periods of time.


T

-- 
All men are mortal. Socrates is mortal. Therefore all men are Socrates.
Apr 03 2018
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 3 Apr 2018 16:15:35 -0700
schrieb "H. S. Teoh" <hsteoh quickfur.ath.cx>:


 On Tue, Apr 03, 2018 at 10:59:13PM +0000, Atila Neves via Digitalmars-d w=


rote:

 That sentence might as well be fingernails on a blackboard for me! I
 save compulsively. Whenever I stop typing, C-x C-s it is for me. =20


=20
 [=E2=80=A6] I can't imagine not saving for long periods of time.


Me too, but for me it was Delphi 2005/2006 with frequent IDE
crashes that had me start doing this and dmd eating into swap
memory when I write messy CTFE code that keeps me doing
Ctrl+S before before any action that has any likelihood to
stall the system. I also don't trust my own code.

--=20
Marco
Apr 03 2018
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, April 03, 2018 16:15:35 H. S. Teoh via Digitalmars-d wrote:

 On Tue, Apr 03, 2018 at 10:59:13PM +0000, Atila Neves via Digitalmars-d 


wrote:

 On Tuesday, 3 April 2018 at 20:47:48 UTC, Kagamin wrote:

 On Friday, 30 March 2018 at 16:12:44 UTC, Atila Neves wrote:

 * Building a whole project in C++ still takes a lot longer since D
 scales much better, but that's not my typical worflow, nor should
 it be anyone else's.


 I can write code for days without even saving :)


 That sentence might as well be fingernails on a blackboard for me! I
 save compulsively. Whenever I stop typing, C-x C-s it is for me.


 [...]

 Yeah, after having lost valuable work more often than I'd care to admit
 due to unexpected power outages and other hardware problems, I've
 developed a compulsive twitch in my fingers that automatically saves my
 file every other minute.  It has become so automatic that it's basically
 unconscious now.  I can't imagine not saving for long periods of time.


LOL. I save probably closer to every ten seconds, though I'm not sure
exactly how often it is because of how automatic it is. It's highly
dependent on when I finish a set of commands in vim, so with code, I tend to
save _very_ frequently, whereas with text, it's a lot less frequent, because
there, I often stay in insert mode for while. In addition to just generally
being a good idea, it eliminates the need for vim's swap files, and I hate
it when programs create files like that (be it to restore what you didn't
save when the program crashes or as a backup file every time you save). The
file is what is on disk, and I don't want the program trying to keep track
of unsaved data or old data and make it easy for me to restore it when my
computer crashes. That's what saving is for, and if I want older versions,
I'll use source control. But I guess that there are folks who do a lot of
typing without saving. I sure couldn't though.

Now, it is true that I frequently write a lot of code before bothering to
compile anything - not days worth at a time though.

- Jonathan M Davis
Apr 03 2018