www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D compiler benchmarks

reply Robert Clipsham <robert octarineparrot.com> writes:
Hi all,

I have set up some benchmarks for dmd, ldc and gdc at 
http://dbench.octarineparrot.com/.

There are currently only 6 tests all from 
http://shootout.alioth.debian.org/gp4/d.php. My knowledge of phobos is 
not great enough to port the others to tango (I've chosen tango as ldc 
does not support phobos currently, so it make sense to choose tango as 
all compilers support it). If you would like to contribute new tests or 
improve on the current ones let me know and I'll include them next time 
I run them.

All source code can be found at 
http://hg.octarineparrot.com/dbench/file/tip.

Let me know if you have any ideas for how I can improve the benchmarks, 
I currently plan to add compile times, size of the final executable and 
memory usage (if anyone knows an easy way to get the memory usage of a 
process in D, let me know :D).
Mar 07 2009
next sibling parent reply The Anh Tran <trtheanh gmail.com> writes:
1. Could you add "some" bench that Alioth currently run on q6600 ubuntu?
2. A gnu c++ for a reference would be great.
I'm very eager to port C++ entries to D :)

Robert Clipsham wrote:
 Hi all,
 
 I have set up some benchmarks for dmd, ldc and gdc at 
 http://dbench.octarineparrot.com/.
 
 There are currently only 6 tests all from 
 http://shootout.alioth.debian.org/gp4/d.php. My knowledge of phobos is 
 not great enough to port the others to tango (I've chosen tango as ldc 
 does not support phobos currently, so it make sense to choose tango as 
 all compilers support it). If you would like to contribute new tests or 
 improve on the current ones let me know and I'll include them next time 
 I run them.
 
 All source code can be found at 
 http://hg.octarineparrot.com/dbench/file/tip.
 
 Let me know if you have any ideas for how I can improve the benchmarks, 
 I currently plan to add compile times, size of the final executable and 
 memory usage (if anyone knows an easy way to get the memory usage of a 
 process in D, let me know :D).

Mar 07 2009
parent reply Robert Clipsham <robert octarineparrot.com> writes:
The Anh Tran wrote:
 1. Could you add "some" bench that Alioth currently run on q6600 ubuntu?
 2. A gnu c++ for a reference would be great.
 I'm very eager to port C++ entries to D :)

1. All the benchmarks currently up are just tango ports of tests from alioth, if that's what you mean? 2. I wasn't planning on adding in C,C++ etc benchmarks as then it will just become a clone of the shootout. I don't mind adding a reference in for each test from C/C++ if there is enough demand, I would rather avoid it and make it purely for D benchmarks if possible. Please feel free to port benchmarks to D/tango, I'll be more than happy to incorporate them into the suite (which is currently fairly minimal).
Mar 07 2009
parent reply The Anh Tran <trtheanh gmail.com> writes:
Could you provide D compiler version that you're using?
Download-able compiler packages if you don't mind :D. I'm spoiled by EasyD.
Mar 07 2009
parent Robert Clipsham <robert octarineparrot.com> writes:
The Anh Tran wrote:
 Could you provide D compiler version that you're using?
 Download-able compiler packages if you don't mind :D. I'm spoiled by EasyD.

is dmd 1.041 and a tango package, ldc is from hg/tango from svn.
Mar 07 2009
prev sibling next sibling parent reply The Anh Tran <trtheanh gmail.com> writes:
3. Do you use multithread or single thread?
Mar 07 2009
parent reply Robert Clipsham <robert octarineparrot.com> writes:
The Anh Tran wrote:
 3. Do you use multithread or single thread?

I'm not sure what you mean here. All the current benchmarks are single threaded as the multi threaded benchmarks use std.thread, and my knowledge of phobos is not good enough to port them. If you mean the machine itself, it does support multithreading, so tests could benefit from that.
Mar 07 2009
parent reply The Anh Tran <trtheanh gmail.com> writes:
Robert Clipsham wrote:
 The Anh Tran wrote:
 3. Do you use multithread or single thread?

I'm not sure what you mean here. All the current benchmarks are single threaded as the multi threaded benchmarks use std.thread, and my knowledge of phobos is not good enough to port them. If you mean the machine itself, it does support multithreading, so tests could benefit from that.

Sorry, my english is bad. Does your bench split into single/multi thread categories like Alioth's: http://shootout.alioth.debian.org/u64q/ http://shootout.alioth.debian.org/u64/ They use affinity to emulate single core bench. But i think we can add a number to command line for that. Ie: fankuch 10 4 // run fankuch bench with 4 threads, array size is 10
Mar 07 2009
parent Robert Clipsham <robert octarineparrot.com> writes:
The Anh Tran wrote:
 Robert Clipsham wrote:
 The Anh Tran wrote:
 3. Do you use multithread or single thread?

I'm not sure what you mean here. All the current benchmarks are single threaded as the multi threaded benchmarks use std.thread, and my knowledge of phobos is not good enough to port them. If you mean the machine itself, it does support multithreading, so tests could benefit from that.

Sorry, my english is bad. Does your bench split into single/multi thread categories like Alioth's: http://shootout.alioth.debian.org/u64q/ http://shootout.alioth.debian.org/u64/ They use affinity to emulate single core bench. But i think we can add a number to command line for that. Ie: fankuch 10 4 // run fankuch bench with 4 threads, array size is 10

No, I don't do that, tests just run as they come.
Mar 07 2009
prev sibling next sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Robert Clipsham wrote:
 ...
 
 (if anyone knows an easy way to get the memory usage of a
 process in D, let me know :D).

There's a way to do it in Phobos;
 import gc=std.gc;
 import gcstats;

 void main()
 {
     GCStats gcst;
     gc.getStats(gcst);
 }

I went hunting through Tango, and it looks like it has the same method; it just isn't exposed. Probably because of this comment: // NOTE: This routine is experimental. The stats or function name may // change before it is made officially available. None the less, if you want it now, you could try adding this to your code somewhere:
 struct GCStats
 {
     size_t poolsize;        // total size of pool
     size_t usedsize;        // bytes allocated
     size_t freeblocks;      // number of blocks marked FREE
     size_t freelistsize;    // total of memory on free lists
     size_t pageblocks;      // number of blocks marked PAGE
 }

 extern(C) GCStats gc_stats();

You should probably whack this in a module so you can replace it easily if and when it changes. -- Daniel
Mar 07 2009
parent reply Robert Clipsham <robert octarineparrot.com> writes:
Daniel Keep wrote:
 
 Robert Clipsham wrote:
 ...

 (if anyone knows an easy way to get the memory usage of a
 process in D, let me know :D).

There's a way to do it in Phobos;
 import gc=std.gc;
 import gcstats;

 void main()
 {
     GCStats gcst;
     gc.getStats(gcst);
 }

I went hunting through Tango, and it looks like it has the same method; it just isn't exposed. Probably because of this comment: // NOTE: This routine is experimental. The stats or function name may // change before it is made officially available. None the less, if you want it now, you could try adding this to your code somewhere:
 struct GCStats
 {
     size_t poolsize;        // total size of pool
     size_t usedsize;        // bytes allocated
     size_t freeblocks;      // number of blocks marked FREE
     size_t freelistsize;    // total of memory on free lists
     size_t pageblocks;      // number of blocks marked PAGE
 }

 extern(C) GCStats gc_stats();

You should probably whack this in a module so you can replace it easily if and when it changes. -- Daniel

app I'm using for benchmarking, it's in the repository if you're interested). It's rather difficult to get the memory usage in run.d so I can put it straight into the stats page if it's being got from within benchmark, and doing this would probably affect the benchmark some amount which I would ideally like to avoid.
Mar 07 2009
next sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Robert Clipsham wrote:
 I was thinking more a way of getting the memory usage using run.d (the
 app I'm using for benchmarking, it's in the repository if you're
 interested). It's rather difficult to get the memory usage in run.d so I
 can put it straight into the stats page if it's being got from within
 benchmark, and doing this would probably affect the benchmark some
 amount which I would ideally like to avoid.

Ah. From the alioth FAQ:
 How did you measure memory use?

 By sampling GTop proc_mem for the program and it's child processes
 every 0.2 seconds. Obviously those measurements are unlikely to be
 reliable for programs that run for less than 0.2 seconds.

Probably best to ensure this sampling thread is running on a different hardware thread to the tested program... -- Daniel
Mar 07 2009
parent Robert Clipsham <robert octarineparrot.com> writes:
Daniel Keep wrote:
 
 From the alioth FAQ:
 
 How did you measure memory use?

 By sampling GTop proc_mem for the program and it's child processes
 every 0.2 seconds. Obviously those measurements are unlikely to be
 reliable for programs that run for less than 0.2 seconds.


I had read this, but that's as far as I got with it!
 
 Probably best to ensure this sampling thread is running on a different
 hardware thread to the tested program...
 
   -- Daniel

It should be running in an entirely different process, but that depends on how tango.sys.Process deals with processes.
Mar 07 2009
prev sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Incidentally, this might be of assistance:

http://shootout.alioth.debian.org/u32q/faq.php#measurementscripts

  -- Daniel
Mar 07 2009
parent Robert Clipsham <robert octarineparrot.com> writes:
Daniel Keep wrote:
 Incidentally, this might be of assistance:
 
 http://shootout.alioth.debian.org/u32q/faq.php#measurementscripts
 
   -- Daniel

Thanks! I've actually already downloaded these, but being me completely overlooked them. If I remember correctly they were python scripts, and my current testing app is in D - http://hg.octarineparrot.com/dbench/file/tip/run.d
Mar 07 2009
prev sibling parent reply Georg Wrede <georg.wrede iki.fi> writes:
Robert Clipsham wrote:
 Hi all,
 
 I have set up some benchmarks for dmd, ldc and gdc at 
 http://dbench.octarineparrot.com/.
 
 There are currently only 6 tests all from 
 http://shootout.alioth.debian.org/gp4/d.php. My knowledge of phobos is 
 not great enough to port the others to tango (I've chosen tango as ldc 
 does not support phobos currently, so it make sense to choose tango as 
 all compilers support it). If you would like to contribute new tests or 
 improve on the current ones let me know and I'll include them next time 
 I run them.
 
 All source code can be found at 
 http://hg.octarineparrot.com/dbench/file/tip.
 
 Let me know if you have any ideas for how I can improve the benchmarks, 
 I currently plan to add compile times, size of the final executable and 
 memory usage (if anyone knows an easy way to get the memory usage of a 
 process in D, let me know :D).

The first run should not be included in the average.
Mar 08 2009
next sibling parent reply Robert Clipsham <robert octarineparrot.com> writes:
Georg Wrede wrote:
 Robert Clipsham wrote:
 Hi all,

 I have set up some benchmarks for dmd, ldc and gdc at 
 http://dbench.octarineparrot.com/.

 There are currently only 6 tests all from 
 http://shootout.alioth.debian.org/gp4/d.php. My knowledge of phobos is 
 not great enough to port the others to tango (I've chosen tango as ldc 
 does not support phobos currently, so it make sense to choose tango as 
 all compilers support it). If you would like to contribute new tests 
 or improve on the current ones let me know and I'll include them next 
 time I run them.

 All source code can be found at 
 http://hg.octarineparrot.com/dbench/file/tip.

 Let me know if you have any ideas for how I can improve the 
 benchmarks, I currently plan to add compile times, size of the final 
 executable and memory usage (if anyone knows an easy way to get the 
 memory usage of a process in D, let me know :D).

The first run should not be included in the average.

Could you explain your reasoning for this? I can't see why it shouldn't be included personally.
Mar 08 2009
next sibling parent Frank Benoit <keinfarbton googlemail.com> writes:
Robert Clipsham schrieb:
 Georg Wrede wrote:
 Robert Clipsham wrote:
 Hi all,

 I have set up some benchmarks for dmd, ldc and gdc at
 http://dbench.octarineparrot.com/.

 There are currently only 6 tests all from
 http://shootout.alioth.debian.org/gp4/d.php. My knowledge of phobos
 is not great enough to port the others to tango (I've chosen tango as
 ldc does not support phobos currently, so it make sense to choose
 tango as all compilers support it). If you would like to contribute
 new tests or improve on the current ones let me know and I'll include
 them next time I run them.

 All source code can be found at
 http://hg.octarineparrot.com/dbench/file/tip.

 Let me know if you have any ideas for how I can improve the
 benchmarks, I currently plan to add compile times, size of the final
 executable and memory usage (if anyone knows an easy way to get the
 memory usage of a process in D, let me know :D).

The first run should not be included in the average.

Could you explain your reasoning for this? I can't see why it shouldn't be included personally.

fill up of disk and memory caches. That is why the first run has a different timing to the other runs.
Mar 08 2009
prev sibling parent reply Georg Wrede <georg.wrede iki.fi> writes:
Robert Clipsham wrote:
 Georg Wrede wrote:
 Robert Clipsham wrote:
 Hi all,

 I have set up some benchmarks for dmd, ldc and gdc at 
 http://dbench.octarineparrot.com/.

 There are currently only 6 tests all from 
 http://shootout.alioth.debian.org/gp4/d.php. My knowledge of phobos 
 is not great enough to port the others to tango (I've chosen tango as 
 ldc does not support phobos currently, so it make sense to choose 
 tango as all compilers support it). If you would like to contribute 
 new tests or improve on the current ones let me know and I'll include 
 them next time I run them.

 All source code can be found at 
 http://hg.octarineparrot.com/dbench/file/tip.

 Let me know if you have any ideas for how I can improve the 
 benchmarks, I currently plan to add compile times, size of the final 
 executable and memory usage (if anyone knows an easy way to get the 
 memory usage of a process in D, let me know :D).

The first run should not be included in the average.

Could you explain your reasoning for this? I can't see why it shouldn't be included personally.

Suppose you have run the same program very recently before the test. Then the executable will be in memory already, any other files it may want to access are in memory too. This makes execution much faster than if it were the first time ever this program is run. If things were deterministic, then you wouldn't run several times and average the results, right?
Mar 08 2009
next sibling parent Robert Clipsham <robert octarineparrot.com> writes:
Bill Baxter wrote:
 On Mon, Mar 9, 2009 at 3:15 AM, Georg Wrede <georg.wrede iki.fi> wrote:
 Robert Clipsham wrote:
 Georg Wrede wrote:
 Robert Clipsham wrote:
 Hi all,

 I have set up some benchmarks for dmd, ldc and gdc at
 http://dbench.octarineparrot.com/.

 There are currently only 6 tests all from
 http://shootout.alioth.debian.org/gp4/d.php. My knowledge of phobos is not
 great enough to port the others to tango (I've chosen tango as ldc does not
 support phobos currently, so it make sense to choose tango as all compilers
 support it). If you would like to contribute new tests or improve on the
 current ones let me know and I'll include them next time I run them.

 All source code can be found at
 http://hg.octarineparrot.com/dbench/file/tip.

 Let me know if you have any ideas for how I can improve the benchmarks,
 I currently plan to add compile times, size of the final executable and
 memory usage (if anyone knows an easy way to get the memory usage of a
 process in D, let me know :D).


included personally.

the executable will be in memory already, any other files it may want to access are in memory too. This makes execution much faster than if it were the first time ever this program is run. If things were deterministic, then you wouldn't run several times and average the results, right?

Also I think standard practice for benchmarks is not to average but to take the minimum time. To the extent that things are not deterministic it is generally because of factors outside of your program's control -- virtual memory page fault kicking in, some other process stealing cycles, etc. Or put another way, there is no way for the measured run time of your program to come out artificially too low, but there are lots of ways it could come out too high. The reason you average measurements in other scenarios is because of an expectation that the measurements form a normal distribution around the true value. That is not the case for measurements of computer program running times. Measurements will basically always be higher than the true intrinsic run-time for your program. --bb

By minimum time, do you mean the fastest time or the slowest time?
Mar 08 2009
prev sibling parent reply Robert Clipsham <robert octarineparrot.com> writes:
Georg Wrede wrote:
 Suppose you have run the same program very recently before the test. 
 Then the executable will be in memory already, any other files it may 
 want to access are in memory too.
 
 This makes execution much faster than if it were the first time ever 
 this program is run.
 
 If things were deterministic, then you wouldn't run several times and 
 average the results, right?

Ok, I will rerun the tests later today and disregard the first test. I may also take the minimum value rather than taking an average (thanks to Bill Baxter for this idea).
Mar 08 2009
parent Isaac Gouy <igouy2 yahoo.com> writes:
Robert Clipsham Wrote:

 Georg Wrede wrote:
 Suppose you have run the same program very recently before the test. 
 Then the executable will be in memory already, any other files it may 
 want to access are in memory too.
 
 This makes execution much faster than if it were the first time ever 
 this program is run.
 
 If things were deterministic, then you wouldn't run several times and 
 average the results, right?

Ok, I will rerun the tests later today and disregard the first test. I may also take the minimum value rather than taking an average (thanks to Bill Baxter for this idea).

As you're re-inventing functionality that's in the benchmarks game measurement scripts, let me suggest that there are 2 phases involved: 1) record measurements 2) analyze measurements As long as you keep the measurements in the order they were made in and keep the measurements for each different configuration in their own file, you can decide to do different selections from those measurements at some later date. You can throw away the first measurement or not, you can take the fastest or the median, you can ... without doing new measurements. As you are only trying to measure a couple of language implementations, measure them across a dozen different input values rather than one or two - leaving the computer churning overnight will help keep your home warm :-)
Mar 08 2009
prev sibling parent Bill Baxter <wbaxter gmail.com> writes:
On Mon, Mar 9, 2009 at 3:15 AM, Georg Wrede <georg.wrede iki.fi> wrote:
 Robert Clipsham wrote:
 Georg Wrede wrote:
 Robert Clipsham wrote:
 Hi all,

 I have set up some benchmarks for dmd, ldc and gdc at
 http://dbench.octarineparrot.com/.

 There are currently only 6 tests all from
 http://shootout.alioth.debian.org/gp4/d.php. My knowledge of phobos is not
 great enough to port the others to tango (I've chosen tango as ldc does not
 support phobos currently, so it make sense to choose tango as all compilers
 support it). If you would like to contribute new tests or improve on the
 current ones let me know and I'll include them next time I run them.

 All source code can be found at
 http://hg.octarineparrot.com/dbench/file/tip.

 Let me know if you have any ideas for how I can improve the benchmarks,
 I currently plan to add compile times, size of the final executable and
 memory usage (if anyone knows an easy way to get the memory usage of a
 process in D, let me know :D).

The first run should not be included in the average.

Could you explain your reasoning for this? I can't see why it shouldn't be included personally.

Suppose you have run the same program very recently before the test. Then the executable will be in memory already, any other files it may want to access are in memory too. This makes execution much faster than if it were the first time ever this program is run. If things were deterministic, then you wouldn't run several times and average the results, right?

Also I think standard practice for benchmarks is not to average but to take the minimum time. To the extent that things are not deterministic it is generally because of factors outside of your program's control -- virtual memory page fault kicking in, some other process stealing cycles, etc. Or put another way, there is no way for the measured run time of your program to come out artificially too low, but there are lots of ways it could come out too high. The reason you average measurements in other scenarios is because of an expectation that the measurements form a normal distribution around the true value. That is not the case for measurements of computer program running times. Measurements will basically always be higher than the true intrinsic run-time for your program. --bb
Mar 08 2009