www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - The Computer Languages Shootout Game

reply Isaac Gouy <igouy2 yahoo.com> writes:
On 10/31/2010 14:41 PM, Eric Poggel wrote:

 What can we do to get it back on there?
1) What you can do is - make your own measurements and publish them. To make that easy I packaged the Python script used to make the benchmarks game measurements. A) Notes and download (Alioth issue their own security certificate, so your browser will complain) - https://alioth.debian.org/frs/shownotes.php?release_id=1484 B) Download (from the benchmarks game website Help page) http://shootout.alioth.debian.org/help.php#languagex 2) Per Bothner used the script to make and publish measurements of his Kawa-Scheme implementation http://per.bothner.com/blog/2010/Kawa-in-shootout/
Nov 01 2010
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/1/10 12:23 PM, Isaac Gouy wrote:
 On 10/31/2010 14:41 PM, Eric Poggel wrote:

 What can we do to get it back on there?
1) What you can do is - make your own measurements and publish them. To make that easy I packaged the Python script used to make the benchmarks game measurements. A) Notes and download (Alioth issue their own security certificate, so your browser will complain) - https://alioth.debian.org/frs/shownotes.php?release_id=1484 B) Download (from the benchmarks game website Help page) http://shootout.alioth.debian.org/help.php#languagex 2) Per Bothner used the script to make and publish measurements of his Kawa-Scheme implementation http://per.bothner.com/blog/2010/Kawa-in-shootout/
Thanks Isaac for weighing in. Well we now have a good answer for "Re: What can the community do to help D?" thread. Andrei
Nov 01 2010
prev sibling next sibling parent reply Isaac Gouy <igouy2 yahoo.com> writes:
--- On Mon, 11/1/10, Walter Bright <newshound2 digitalmars.com> wrote:

 From: Walter Bright <newshound2 digitalmars.com>
 Subject: Re: The Computer Languages Shootout Game
 To: digitalmars-d puremagic.com
 Date: Monday, November 1, 2010, 11:02 AM
 Isaac Gouy wrote:
 On 10/31/2010 5:13 PM, Walter Bright wrote:
 The thing about that shootout is D used to be on
there, until the
 maintainer of the site removed it. 
D still is on there. The maintainer of the site did
not remove it.
 
 http://shootout.alioth.debian.org/gp4/measurements.php?lang=dlang
Thanks, but I don't find any links to that from the home page.
The home page shows the current benchmarks game. The link to the out-of-date measurements is on the help page http://shootout.alioth.debian.org/help.php#history D is one of the 30 or so language implementations not measured on Q6600 that were measured on Pentium 4. (Incidentally if you would prefer D not to be shown in the out-of-date measurements - because you feel it's unrepresentative of the current state of D language implementations - just ask for it to be removed.)
 He refuses to include D on the benchmarks.
Being included in the current benchmarks game isn't an
entitlement.
 What I refuse to do is let you guys dictate how I
spend my free time. I'm not happy with your choice, but I don't dictate to you or anyone else. It's your site and you can do as you please with it.
Maybe someone in the D community will make the effort and produce comparison performance measurements, and then you can choose to publish them (or not) on the D website.
Nov 01 2010
parent Walter Bright <newshound2 digitalmars.com> writes:
Isaac Gouy wrote:
 I'm not happy with your choice, but I don't dictate to you or anyone else.
 It's your site and you can do as you please with it.
Maybe someone in the D community will make the effort and produce comparison performance measurements, and then you can choose to publish them (or not) on the D website.
Nobody would believe benchmarks on the D web site. Heck, I don't believe any benchmarks published by the developers of any language. I was even once accused of literally sabotaging DMC++ in order to make DMD look better.
Nov 01 2010
prev sibling next sibling parent reply Isaac Gouy <igouy2 yahoo.com> writes:
--- On Mon, 11/1/10, Walter Bright <newshound2 digitalmars.com> wrote:

 From: Walter Bright <newshound2 digitalmars.com>
 Subject: Re: The Computer Languages Shootout Game
 To: digitalmars-d puremagic.com
 Date: Monday, November 1, 2010, 11:54 AM
 Isaac Gouy wrote:
 I'm not happy with your choice, but I don't
dictate to you or anyone else.
 It's your site and you can do as you please with
it.
 Maybe someone in the D community will make the effort
and produce comparison
 performance measurements, and then you can choose to
publish them (or not) on
 the D website.
Nobody would believe benchmarks on the D web site. Heck, I don't believe any benchmarks published by the developers of any language.
When you publish the source code of the programs, and the compile and build logs, and the compiler and linker versions, and the OS the measurements were made on, ... others don't have to just "believe" because they can try to confirm the measurements for themselves. Which is why the benchmarks game measurement scripts markup source code and log the build - http://shootout.alioth.debian.org/u32q/program.php?test=meteor&lang=gpp&id=4#log
Nov 01 2010
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Isaac Gouy wrote:
 Nobody would believe benchmarks on the D web site. Heck, I don't believe
 any benchmarks published by the developers of any language.
When you publish the source code of the programs, and the compile and build logs, and the compiler and linker versions, and the OS the measurements were made on, ... others don't have to just "believe" because they can try to confirm the measurements for themselves.
It's true, they can, but they don't. I have nearly 30 years of experience with this.
Nov 01 2010
parent retard <re tard.com.invalid> writes:
Mon, 01 Nov 2010 13:39:07 -0700, Walter Bright wrote:

 Isaac Gouy wrote:
 Nobody would believe benchmarks on the D web site. Heck, I don't
 believe any benchmarks published by the developers of any language.
When you publish the source code of the programs, and the compile and build logs, and the compiler and linker versions, and the OS the measurements were made on, ... others don't have to just "believe" because they can try to confirm the measurements for themselves.
It's true, they can, but they don't. I have nearly 30 years of experience with this.
Instead of this useless bikeshedding someone could provide the required help to Isaac here. I'm guessing he also accepts money and other kinds of gifts as a dedication of gratitude. I personally thank him now, the test has revealed many useful languages to me and I've found it very informative. What some language distributions have done is they include the language shootout tests in the standard distribution. The language community can do all sorts of things to make the up to date test results a reality: up to date open source compiler for all required platforms, debian packages, help with the scripts to make the tests run and so on. The site is a great advertisement - I'm pretty sure the Go developers have realized that already.
Nov 01 2010
prev sibling parent Rainer Deyke <rainerd eldwood.com> writes:
On 11/1/2010 13:04, Isaac Gouy wrote:
 --- On Mon, 11/1/10, Walter Bright <newshound2 digitalmars.com>
 wrote:
 Nobody would believe benchmarks on the D web site. Heck, I don't
 believe any benchmarks published by the developers of any
 language.
When you publish the source code of the programs, and the compile and build logs, and the compiler and linker versions, and the OS the measurements were made on, ... others don't have to just "believe" because they can try to confirm the measurements for themselves.
I tend to assume that benchmarks posted by the developers of the language are correct, but deliberately chosen to make the language look good and the competition look bad. Reproducing the tests locally would only catch the most blatant forms of cheating. -- Rainer Deyke - rainerd eldwood.com
Nov 01 2010
prev sibling next sibling parent reply Isaac Gouy <igouy2 yahoo.com> writes:
--- On Mon, 11/1/10, Walter Bright <newshound2 digitalmars.com> wrote:

 From: Walter Bright <newshound2 digitalmars.com>
 Subject: Re: The Computer Languages Shootout Game
 To: digitalmars-d puremagic.com
 Date: Monday, November 1, 2010, 1:39 PM
 Isaac Gouy wrote:
 Nobody would believe benchmarks on the D web site.
Heck, I don't believe
 any benchmarks published by the developers of any
language.
 When you publish the source code of the programs, and
the compile and build
 logs, and the compiler and linker versions, and the OS
the measurements were
 made on, ... others don't have to just "believe"
because they can try to
 confirm the measurements for themselves.
It's true, they can, but they don't. I have nearly 30 years of experience with this.
It doesn't matter whether they try to confirm the measurements for themselves or not - what matters is that they are provided with the all the information required to do so. I only have 5 years experience publishing the measurements for the benchmarks game - and I've come across a handful of people who did try to confirm the measurements for themselves. (The most interesting example compared a couple of language implementations on one particular task but measured at 2 dozen different input values. That nicely demonstrated that the same language implementation wasn't always faster across all the input values. The 3 different input values shown on the benchmarks game isn't usually enough to demonstrate that kind of thing.)
Nov 01 2010
parent sybrandy <sybrandy gmail.com> writes:
 It doesn't matter whether they try to confirm the measurements for themselves
or not - what matters is that they are provided with the all the information
required to do so.


 I only have 5 years experience publishing the measurements for the benchmarks
game - and I've come across a handful of people who did try to confirm the
measurements for themselves.

 (The most interesting example compared a couple of language implementations on
one particular task but measured at 2 dozen different input values. That nicely
demonstrated that the same language implementation wasn't always faster across
all the input values. The 3 different input values shown on the benchmarks game
isn't usually enough to demonstrate that kind of thing.)
That's an interesting observation. I didn't even think of that before, but it does make sense. I was debating on posting this, but I figured it couldn't hurt: the biggest problem I have with the benchmarks they use is that, at least from my perspective, they're not all very common algorithms. Some things I'd love to see are B-Trees, which are common in databases, encryption, compression, etc. as they are very common and therefore provide more useful comparisons. Even MapReduce would be good since that's becoming very popular. Taking it a step further, there needs to be well-defined standard implementations and alternative implementations. The standard implementations would be designed to be straight-forward designs that don't use any trickery so that we can actually compare language implementations. The alternative ones would then show how you can make the implementations faster. I mention this because a buddy of mine submitted a C version of one benchmark, but implemented his own thread pooling code. It was rejected even though the C++ version used Boost, which also, from what I'm told, uses thread pooling. A standard implementation could be used to define if things like thread pooling can/should be used. I'd argue not in this case as not every language supports and/or requires it. E.g. Erlang. Of course, this is all just some ideas that I'm not going to try to implement as it's just going to be too much work to do and I don't have the resources to do it right. Even then, how do we make it truly fair and accurate? Based on what I've seen in this thread, it's a pretty hard problem if even the data can affect a languages performance. Casey
Nov 02 2010
prev sibling next sibling parent Isaac Gouy <igouy2 yahoo.com> writes:
=0A=0A--- On Mon, 11/1/10, Rainer Deyke <rainerd eldwood.com> wrote:=0A=0A>=
 From: Rainer Deyke <rainerd eldwood.com>=0A> Subject: Re: The Computer Lan=
guages Shootout Game=0A> To: digitalmars-d puremagic.com=0A> Date: Monday, =
November 1, 2010, 2:01 PM=0A> On 11/1/2010 13:04, Isaac Gouy=0A> wrote:=0A>=
 --- On Mon, 11/1/10, Walter Bright <newshound2 digitalmars.com>=0A> > wr=
ote:=0A> >> Nobody would believe benchmarks on the D web site.=0A> Heck, I = don't=0A> >> believe any benchmarks published by the developers=0A> of any= =0A> >> language.=0A> > =0A> > When you publish the source code of the prog= rams, and=0A> the compile and=0A> > build logs, and the compiler and linker= versions, and=0A> the OS the=0A> > measurements were made on, ... others d= on't have to=0A> just "believe"=0A> > because they can try to confirm the m= easurements for=0A> themselves.=0A> =0A> I tend to assume that benchmarks p= osted by the developers=0A> of the=0A> language are correct, but deliberate= ly chosen to make the=0A> language look=0A> good and the competition look b= ad.=A0 Reproducing the=0A> tests locally would=0A> only catch the most blat= ant forms of cheating.=0A=0A=0AGiven this discussion was about The Computer= Language Benchmarks Game I had assumed those were the benchmarks under dis= cussion - in which case I don't really see how your concern about benchmark= s "deliberately chosen to make the language look good" would be relevant.= =0A=0A=0A
Nov 01 2010
prev sibling parent Isaac Gouy <igouy2 yahoo.com> writes:
--- On Mon, 11/1/10, retard <re tard.com.invalid> wrote:

 From: retard <re tard.com.invalid>
 Subject: Re: The Computer Languages Shootout Game
 To: digitalmars-d puremagic.com
 Date: Monday, November 1, 2010, 5:23 PM
 Mon, 01 Nov 2010 13:39:07 -0700,
 I'm guessing he also accepts money and
 other kinds of gifts as a dedication of gratitude. 
No.
Nov 01 2010