www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Compiler benchmarker for D, C, C++, Go, Rust with more to come

reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
After reading

https://forum.dlang.org/thread/nlhybmioaoxkydicmtjj forum.dlang.org

it's about time I announce my little Python snippet that 
exercises the compilation speed of D and other languages with a 
superficial test program generator at

https://github.com/nordlow/compiler-benchmark

Run as

    ./benchmark.py

This generates a single source file for each supported language, 
currently D, C, C++, Go and Rust, containing 50k number of 
trivial functions that from the main function are called and 
reduced to a single integer which is returned at the end of main. 
The benchmark then measures the time it takes for each compiler 
to run semantic analysis on the source file (excluding output of 
any object files).

My only question is:
How is it possible for Rust be to 50x slower than D in this 
test!!!??? Am I calling rustc in the wrong way? Especially taking 
into account the amount of time (and money) that has been put 
into making Rust fast(er). Through fancy caching of IR-files and 
more.

I very much welcome PRs that add more languages.

Output on Ubuntu 18.04 system is

Code-generation:
Generating generated/c/foo.c took 0.073 seconds (C)
Generating generated/c++/foo.c++ took 0.073 seconds (C++)
Generating generated/d/foo.d took 0.076 seconds (D)
Generating generated/rust/foo.rs took 0.076 seconds (Rust)
Generating generated/zig/foo.zig took 0.077 seconds (Zig)
Generating generated/go/foo.go took 0.074 seconds (Go)

Clang:
- Checking of generated/c/foo.c took 1.373 seconds (using 
"/usr/bin/clang-7")
- Checking of generated/c++/foo.c++ took 1.345 seconds (using 
"/usr/bin/clang++-7")

GCC:
- Checking of generated/c/foo.c took 0.385 seconds (using 
"/usr/bin/gcc-5")
- Checking of generated/c++/foo.c++ took 0.675 seconds (using 
"/usr/bin/g++-5")
- Checking of generated/c/foo.c took 0.551 seconds (using 
"/usr/bin/gcc-6")
- Checking of generated/c/foo.c took 0.518 seconds (using 
"/usr/bin/gcc-7")
- Checking of generated/c++/foo.c++ took 1.294 seconds (using 
"/usr/bin/g++-7")
- Checking of generated/c/foo.c took 0.527 seconds (using 
"/usr/bin/gcc-8")
- Checking of generated/c++/foo.c++ took 1.363 seconds (using 
"/usr/bin/g++-8")

D:
- Checking of generated/d/foo.d took 0.403 seconds (using 
"/usr/bin/dmd")
- Checking of generated/d/foo.d took 0.352 seconds (using 
"/home/per/.local/ldc2-1.15.0-beta1-linux-x86_64/bin/ldmd2")

Go:
- Checking of generated/go/foo.go took 1.716 seconds (using 
"/usr/bin/gccgo")

Rust:
- Checking of generated/rust/foo.rs took 17.828 seconds (using 
"/home/per/.cargo/bin/rustc")

Zig:
Speedups:
- D to gcc-8: 1.50
- D to g++-8: 3.88
- D to Clang: 3.90
- D to Clang++: 3.83
- D to Go: 4.88
- D to Rust: 50.71
Mar 16
parent reply Jacob Carlborg <doob me.com> writes:
On 2019-03-16 22:23, Per Nordlöw wrote:
 After reading
 
 https://forum.dlang.org/thread/nlhybmioaoxkydicmtjj forum.dlang.org
 
 it's about time I announce my little Python snippet that exercises the 
 compilation speed of D and other languages with a superficial test 
 program generator at
 
 https://github.com/nordlow/compiler-benchmark
 
 Run as
 
     ./benchmark.py
 
 This generates a single source file for each supported language, 
 currently D, C, C++, Go and Rust, containing 50k number of trivial 
 functions that from the main function are called and reduced to a single 
 integer which is returned at the end of main. The benchmark then 
 measures the time it takes for each compiler to run semantic analysis on 
 the source file (excluding output of any object files).
 
 My only question is:
 How is it possible for Rust be to 50x slower than D in this test!!!??? 
 Am I calling rustc in the wrong way? Especially taking into account the 
 amount of time (and money) that has been put into making Rust fast(er). 
 Through fancy caching of IR-files and more.
 
 I very much welcome PRs that add more languages.
 
 Output on Ubuntu 18.04 system is
 
 Code-generation:
 Generating generated/c/foo.c took 0.073 seconds (C)
 Generating generated/c++/foo.c++ took 0.073 seconds (C++)
 Generating generated/d/foo.d took 0.076 seconds (D)
 Generating generated/rust/foo.rs took 0.076 seconds (Rust)
 Generating generated/zig/foo.zig took 0.077 seconds (Zig)
 Generating generated/go/foo.go took 0.074 seconds (Go)
 
 Clang:
 - Checking of generated/c/foo.c took 1.373 seconds (using 
 "/usr/bin/clang-7")
 - Checking of generated/c++/foo.c++ took 1.345 seconds (using 
 "/usr/bin/clang++-7")
 
 GCC:
 - Checking of generated/c/foo.c took 0.385 seconds (using "/usr/bin/gcc-5")
 - Checking of generated/c++/foo.c++ took 0.675 seconds (using 
 "/usr/bin/g++-5")
 - Checking of generated/c/foo.c took 0.551 seconds (using "/usr/bin/gcc-6")
 - Checking of generated/c/foo.c took 0.518 seconds (using "/usr/bin/gcc-7")
 - Checking of generated/c++/foo.c++ took 1.294 seconds (using 
 "/usr/bin/g++-7")
 - Checking of generated/c/foo.c took 0.527 seconds (using "/usr/bin/gcc-8")
 - Checking of generated/c++/foo.c++ took 1.363 seconds (using 
 "/usr/bin/g++-8")
 
 D:
 - Checking of generated/d/foo.d took 0.403 seconds (using "/usr/bin/dmd")
 - Checking of generated/d/foo.d took 0.352 seconds (using 
 "/home/per/.local/ldc2-1.15.0-beta1-linux-x86_64/bin/ldmd2")
 
 Go:
 - Checking of generated/go/foo.go took 1.716 seconds (using 
 "/usr/bin/gccgo")
 
 Rust:
 - Checking of generated/rust/foo.rs took 17.828 seconds (using 
 "/home/per/.cargo/bin/rustc")
 
 Zig:
 Speedups:
 - D to gcc-8: 1.50
 - D to g++-8: 3.88
 - D to Clang: 3.90
 - D to Clang++: 3.83
 - D to Go: 4.88
 - D to Rust: 50.71
It would be interesting to measure D's speed with DMD compiled with LDC as the host D compiler. -- /Jacob Carlborg
Mar 17
parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Sunday, 17 March 2019 at 09:17:06 UTC, Jacob Carlborg wrote:
 It would be interesting to measure D's speed with DMD compiled 
 with LDC as the host D compiler.
I thought that already is the case...
Mar 17
parent reply Jacob Carlborg <doob me.com> writes:
On 2019-03-17 21:09, Per Nordlöw wrote:

 I thought that already is the case...
No, the official binaries are built with DMD as the host compiler. -- /Jacob Carlborg
Mar 18
next sibling parent reply aliak <something something.com> writes:
On Monday, 18 March 2019 at 10:05:40 UTC, Jacob Carlborg wrote:
 On 2019-03-17 21:09, Per Nordlöw wrote:

 I thought that already is the case...
No, the official binaries are built with DMD as the host compiler.
How come they're not built with LDC (for example) and then distributed? I'd think it'd be shit tonnes more performant?
Mar 18
parent reply Seb <seb wilzba.ch> writes:
On Monday, 18 March 2019 at 11:02:55 UTC, aliak wrote:
 On Monday, 18 March 2019 at 10:05:40 UTC, Jacob Carlborg wrote:
 On 2019-03-17 21:09, Per Nordlöw wrote:

 I thought that already is the case...
No, the official binaries are built with DMD as the host compiler.
How come they're not built with LDC (for example) and then distributed? I'd think it'd be shit tonnes more performant?
Yes it would be _a lot_ faster, but the release process is pretty complicated [1] and at the moment Martin is the only one who fully understands it and he's pretty busy with his day job and other tasks. [1] https://github.com/dlang/installer
Mar 18
next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Monday, 18 March 2019 at 12:33:12 UTC, Seb wrote:
 On Monday, 18 March 2019 at 11:02:55 UTC, aliak wrote:
 On Monday, 18 March 2019 at 10:05:40 UTC, Jacob Carlborg wrote:
 On 2019-03-17 21:09, Per Nordlöw wrote:

 I thought that already is the case...
No, the official binaries are built with DMD as the host compiler.
How come they're not built with LDC (for example) and then distributed? I'd think it'd be shit tonnes more performant?
Yes it would be _a lot_ faster, but the release process is pretty complicated [1] and at the moment Martin is the only one who fully understands it and he's pretty busy with his day job and other tasks. [1] https://github.com/dlang/installer
It would be great if a campaign (donations) could be created to overwork the release process. There are some quite important topics like 64 bit DMD for windows or the LDC builds. Also the process should be easier and maybe even documented. Kind regards Andre
Mar 18
prev sibling parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Monday, 18 March 2019 at 12:33:12 UTC, Seb wrote:
 [1] https://github.com/dlang/installer
Does this include a script for building dmd with ldc or this not yet possible?
Mar 18
parent reply Seb <seb wilzba.ch> writes:
On Monday, 18 March 2019 at 21:34:40 UTC, Per Nordlöw wrote:
 On Monday, 18 March 2019 at 12:33:12 UTC, Seb wrote:
 [1] https://github.com/dlang/installer
Does this include a script for building dmd with ldc or this not yet possible?
Well, you don't really need a script: git clone https://github.com/dlang/dmd HOST_DMD=ldmd2 make -f posix.mak -j10 Additionally, you can add `ENABLE_RELEASE=1` to create a fully optimized release binary. This has been possible for a very long time and actually is even tested on every PR with SemaphoreCI (Linux) and AppVeyor (Windows). BTW, the Arch DMD package [1] is the only package that I know of which is actually using LDC to compile DMD for the released binaries. [1] https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/dmd#n54
Mar 19
next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 19 March 2019 at 10:10:28 UTC, Seb wrote:
 On Monday, 18 March 2019 at 21:34:40 UTC, Per Nordlöw wrote:
 On Monday, 18 March 2019 at 12:33:12 UTC, Seb wrote:
 [1] https://github.com/dlang/installer
Does this include a script for building dmd with ldc or this not yet possible?
Well, you don't really need a script: git clone https://github.com/dlang/dmd HOST_DMD=ldmd2 make -f posix.mak -j10
HOST_DMD=ldmd2 make -f posix.mak -j`nproc` FTFY. It's bad enough make is a dinosaur that defaults to one thread, let's not hard-code the number of cores.
 Additionally, you can add `ENABLE_RELEASE=1` to create a fully 
 optimized release binary.
Awesome.
 BTW, the Arch DMD package [1] is the only package that I know 
 of which is actually using LDC to compile DMD for the released 
 binaries.
Huh, I didn't know that. Yay Arch! And yet, compilation times are still my mortal enemy...
Mar 19
prev sibling next sibling parent Laurent =?UTF-8?B?VHLDqWd1aWVy?= <laurent.treguier.sink gmail.com> writes:
On Tuesday, 19 March 2019 at 10:10:28 UTC, Seb wrote:
 BTW, the Arch DMD package [1] is the only package that I know 
 of which is actually using LDC to compile DMD for the released 
 binaries.

 [1] 
 https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/dmd#n54
I'm building DMD packages for RPM-based distros [1], and I'm using LDC too to compile it on Fedora since it's available in the official Fedora repositories. [1] https://copr.fedorainfracloud.org/coprs/tcg/devel/package/dmd/
Mar 19
prev sibling parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Tuesday, 19 March 2019 at 10:10:28 UTC, Seb wrote:
 Well, you don't really need a script:

 git clone https://github.com/dlang/dmd
 HOST_DMD=ldmd2 make -f posix.mak -j10

 Additionally, you can add `ENABLE_RELEASE=1` to create a fully 
 optimized release binary.
Thanks!
Mar 20
prev sibling parent reply Andrea Fontana <nospam example.org> writes:
On Monday, 18 March 2019 at 10:05:40 UTC, Jacob Carlborg wrote:
 On 2019-03-17 21:09, Per Nordlöw wrote:

 I thought that already is the case...
No, the official binaries are built with DMD as the host compiler.
Is v2.0xx compiled with v2.0xx-1 or with v2.0xx itself? Andrea
Mar 18
parent Jacob Carlborg <doob me.com> writes:
On 2019-03-18 15:49, Andrea Fontana wrote:

 Is v2.0xx compiled with v2.0xx-1 or with v2.0xx itself?
With v2.0xx-1, as far as I understand. -- /Jacob Carlborg
Mar 18