www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Benchmark: OOPack

reply Thomas Kuehne <thomas-dloop kuehne.cn> writes:
Another benchmark. OOPack tries to evaluate the cost of using OOP against
the cost of the "plain C" style.

The D port is a quick 'n' dirty hack ...

gdc-0.8(gcc-3.4.2): -O2 -frename-registers -fomit-frame-pointer -fweb
-frelease -finline-functions -march=i686
                         Seconds       Mflops
Test       Iterations     C    OOP     C    OOP  Ratio
----       ----------  -----------  -----------  -----
Iterator       100000  0.530 28.990  377.358 6.899  54.698
Complex          1000  0.020 801.01  400.000 0.010  40050.500
Matrix           1000  0.890 7.420  280.899 33.693  8.337
Max            100000  0.860 3.510  116.279 28.490  4.081

dmd-0.105: -O -release -inline
                         Seconds       Mflops
Test       Iterations     C    OOP     C    OOP  Ratio
----       ----------  -----------  -----------  -----
Iterator       100000  1.060 9.910  188.679 20.182  9.349
Complex          1000  0.040 3.990  200.000 2.005  99.750
Matrix           1000  1.460 6.720  171.233 37.202  4.603
Max            100000  0.790 3.530  126.582 28.329  4.468

g++-3.4.2: -O3 -march=i686
                         Seconds       Mflops
Test       Iterations     C    OOP     C    OOP  Ratio
----       ----------  -----------  -----------  -----
Iterator       100000    0.5   0.5  377.4 377.4    1.0
Complex          1000    0.0   0.1  400.0  57.1    7.0
Matrix           1000    0.9   0.7  274.7 342.5    0.8
Max            100000    0.8   0.8  126.6 126.6    1.0

icc-8.0: -O3 -march=i686
                         Seconds       Mflops
Test       Iterations     C    OOP     C    OOP  Ratio
----       ----------  -----------  -----------  -----
Iterator       100000    0.5   0.6  444.4 357.1    1.2
Complex          1000    0.0   0.0  400.0 400.0    1.0
Matrix           1000    0.7   0.7  384.6 384.6    1.0
Max            100000    0.7   0.7  137.0 137.0    1.0

gdc's Iterator and Complex test seem really bad. Both the frontend and the
backend generate better results. This might be an integration problem or an
optimization where the init code was in-lined into the OOP benchmark
function.

Interestingly gcc's Matrix-C function is slower than it's Matrix-OPP
function.

Intel's results are that good, because the bought the company(kai.com) that
wrote this benchmark along with their compiler(KAI C++) and tuned icc for
this benchmark.

OOPack (C++ & D) the original C++ page has vanished
svn://svn.kuehne.cn/dstress/benchmark/oopack

Thomas
Nov 04 2004
next sibling parent reply no where.com writes:
This result looks not good for D.  I have never seen a benchmark where D
performs so bad like this compared with C++.  Is it just because "The D port is
a quick 'n' dirty hack"; or some other reasons.

D is designed to be a simpler language than C++, the object model should be more
efficient, right?


In article <cmcp5c$41g$4 digitaldaemon.com>, Thomas Kuehne says...
Another benchmark. OOPack tries to evaluate the cost of using OOP against
the cost of the "plain C" style.

The D port is a quick 'n' dirty hack ...

gdc-0.8(gcc-3.4.2): -O2 -frename-registers -fomit-frame-pointer -fweb
-frelease -finline-functions -march=i686
                         Seconds       Mflops
Test       Iterations     C    OOP     C    OOP  Ratio
----       ----------  -----------  -----------  -----
Iterator       100000  0.530 28.990  377.358 6.899  54.698
Complex          1000  0.020 801.01  400.000 0.010  40050.500
Matrix           1000  0.890 7.420  280.899 33.693  8.337
Max            100000  0.860 3.510  116.279 28.490  4.081

dmd-0.105: -O -release -inline
                         Seconds       Mflops
Test       Iterations     C    OOP     C    OOP  Ratio
----       ----------  -----------  -----------  -----
Iterator       100000  1.060 9.910  188.679 20.182  9.349
Complex          1000  0.040 3.990  200.000 2.005  99.750
Matrix           1000  1.460 6.720  171.233 37.202  4.603
Max            100000  0.790 3.530  126.582 28.329  4.468

g++-3.4.2: -O3 -march=i686
                         Seconds       Mflops
Test       Iterations     C    OOP     C    OOP  Ratio
----       ----------  -----------  -----------  -----
Iterator       100000    0.5   0.5  377.4 377.4    1.0
Complex          1000    0.0   0.1  400.0  57.1    7.0
Matrix           1000    0.9   0.7  274.7 342.5    0.8
Max            100000    0.8   0.8  126.6 126.6    1.0

icc-8.0: -O3 -march=i686
                         Seconds       Mflops
Test       Iterations     C    OOP     C    OOP  Ratio
----       ----------  -----------  -----------  -----
Iterator       100000    0.5   0.6  444.4 357.1    1.2
Complex          1000    0.0   0.0  400.0 400.0    1.0
Matrix           1000    0.7   0.7  384.6 384.6    1.0
Max            100000    0.7   0.7  137.0 137.0    1.0

gdc's Iterator and Complex test seem really bad. Both the frontend and the
backend generate better results. This might be an integration problem or an
optimization where the init code was in-lined into the OOP benchmark
function.

Interestingly gcc's Matrix-C function is slower than it's Matrix-OPP
function.

Intel's results are that good, because the bought the company(kai.com) that
wrote this benchmark along with their compiler(KAI C++) and tuned icc for
this benchmark.

OOPack (C++ & D) the original C++ page has vanished
svn://svn.kuehne.cn/dstress/benchmark/oopack

Thomas

Nov 04 2004
parent reply Thomas Kuehne <thomas-dloop kuehne.cn> writes:
no where.com schrieb am Donnerstag, 4. November 2004 23:19:
 This result looks not good for D.  I have never seen a benchmark where D
 performs so bad like this compared with C++.  Is it just because "The D
 port is a quick 'n' dirty hack"; or some other reasons.

The D port is "quick 'n' dirty hack" because it neither uses static/const attribute nor D's on constructs like "foreach". Currently it's only in the state where it might uncover serious problems like gdc's "Complex" test case. This benchmark requires review before it can provide comparable data. Comeon every one and tune this monster. ;) svn://svn.kuehne.cn/dstress/benchmark/oopack
 D is designed to be a simpler language than C++, the object model should
 be more efficient, right?

current D compilers are relativly young. To figure out what is going on someone with some time to spare and a dual boot Linux(dmd,gdc,gcc)/Windows(dmc,dmd,gdc,gcc) box should run the benchmark. This way it should be possible to figure if the problem lies within the frontent, backend, benchmark port or the interaction of the tool chain. Thomas
Nov 04 2004
next sibling parent reply "Walter" <newshound digitalmars.com> writes:
"Thomas Kuehne" <thomas-dloop kuehne.cn> wrote in message
news:cmei3a$34n$1 digitaldaemon.com...
 svn://svn.kuehne.cn/dstress/benchmark/oopack

The link doesn't work. Can you check it, please?
Nov 05 2004
parent reply =?ISO-8859-1?Q?Anders_F_Bj=F6rklund?= <afb algonet.se> writes:
Walter wrote:

svn://svn.kuehne.cn/dstress/benchmark/oopack

The link doesn't work. Can you check it, please?

It seems to work fine from here: (it's a Subversion link)
 # svn checkout svn://svn.kuehne.cn/dstress/benchmark/oopack                   
                          [6]
 A  oopack/oopack_v1p8_d.d
 A  oopack/oopack_v1p8_cpp.cpp
 Checked out revision 125.

Maybe Thomas could provide zipped snapshots too ? (http) --anders PS. Subversion software at http://subversion.tigris.org/
Nov 05 2004
parent reply "Walter" <newshound digitalmars.com> writes:
"Anders F Björklund" <afb algonet.se> wrote in message
news:cmfj31$21j4$1 digitaldaemon.com...
 Walter wrote:

svn://svn.kuehne.cn/dstress/benchmark/oopack

The link doesn't work. Can you check it, please?

It seems to work fine from here: (it's a Subversion link)
 # svn checkout svn://svn.kuehne.cn/dstress/benchmark/oopack


 A  oopack/oopack_v1p8_d.d
 A  oopack/oopack_v1p8_cpp.cpp
 Checked out revision 125.

Maybe Thomas could provide zipped snapshots too ? (http)

Or just make .txt files out of the code and link to them? That way we can examine the code without needing to download, unzip, etc.
Nov 05 2004
parent reply =?ISO-8859-1?Q?Anders_F_Bj=F6rklund?= <afb algonet.se> writes:
Walter wrote:

Maybe Thomas could provide zipped snapshots too ? (http)

Or just make .txt files out of the code and link to them? That way we can examine the code without needing to download, unzip, etc.

AFAIK, there should be web interfaces to Subversion that let's you do both things automatically from the repository... I know such things exists for CVS, at least. --anders
Nov 05 2004
parent reply "Thomas Kuehne" <thomas-dloop kuehne.cn> writes:
Anders F Björklund schrieb:
Maybe Thomas could provide zipped snapshots too ? (http)

Or just make .txt files out of the code and link to them? That way we can examine the code without needing to download, unzip, etc.

AFAIK, there should be web interfaces to Subversion that let's you do both things automatically from the repository... I know such things exists for CVS, at least.

I am going to enable http access soon - before that I have to make sure that the different access rights are correct. Thomas
Nov 05 2004
parent reply "Thomas Kuehne" <thomas-dloop kuehne.cn> writes:
Maybe Thomas could provide zipped snapshots too ? (http)

Or just make .txt files out of the code and link to them? That way we can examine the code without needing to download, unzip, etc.

AFAIK, there should be web interfaces to Subversion that let's you do both things automatically from the repository... I know such things exists for CVS, at least.

I am going to enable http access soon - before that I have to make sure that the different access rights are correct.

Http access to DStress and the benchmark sources is up and running. http://svn.kuehne.cn/dstress svn://svn.kuehne.cn/dstress Thomas
Nov 06 2004
parent reply "Walter" <newshound digitalmars.com> writes:
"Thomas Kuehne" <thomas-dloop kuehne.cn> wrote in message
news:cmjgps$2t9i$1 digitaldaemon.com...
 Http access to DStress and the benchmark sources is up and running.

 http://svn.kuehne.cn/dstress
 svn://svn.kuehne.cn/dstress

Thanks. The reason the HAVE_COMPLEX D benchmark is so slow is that class Complex, in D, is created on the heap, whereas it is used as strictly a value object. The first solution is to make it a struct instead of a class, which will make it a value object. Even better, just replace class Complex with cdouble, which is D's complex double native type.
Nov 06 2004
parent reply no where.com writes:
In article <cmjtmn$c0f$1 digitaldaemon.com>, Walter says...
"Thomas Kuehne" <thomas-dloop kuehne.cn> wrote in message
news:cmjgps$2t9i$1 digitaldaemon.com...
 Http access to DStress and the benchmark sources is up and running.

 http://svn.kuehne.cn/dstress
 svn://svn.kuehne.cn/dstress

Thanks. The reason the HAVE_COMPLEX D benchmark is so slow is that class Complex, in D, is created on the heap, whereas it is used as strictly a

Sometime ago, there's a post about "Dropping the distinction between objects and references may hinder performance", do you think this issue will be addressed in the future? digitalmars.D/11960
value object. The first solution is to make it a struct instead of a class,
which will make it a value object. Even better, just replace class Complex
with cdouble, which is D's complex double native type.

Nov 08 2004
next sibling parent "Walter" <newshound digitalmars.com> writes:
<no where.com> wrote in message news:cmolmi$107v$1 digitaldaemon.com...
 In article <cmjtmn$c0f$1 digitaldaemon.com>, Walter says...
"Thomas Kuehne" <thomas-dloop kuehne.cn> wrote in message
news:cmjgps$2t9i$1 digitaldaemon.com...
 Http access to DStress and the benchmark sources is up and running.

 http://svn.kuehne.cn/dstress
 svn://svn.kuehne.cn/dstress

Thanks. The reason the HAVE_COMPLEX D benchmark is so slow is that class Complex, in D, is created on the heap, whereas it is used as strictly a

Sometime ago, there's a post about "Dropping the distinction between

 references may hinder performance", do you think this issue will be

 the future?

 digitalmars.D/11960

I understand the issue with it, but I want to run with what we've got for a while.
Nov 09 2004
prev sibling parent Ilya Minkov <minkov cs.tum.edu> writes:
no where.com schrieb:

 Sometime ago, there's a post about "Dropping the distinction between objects
and
 references may hinder performance", do you think this issue will be addressed
in
 the future?
 
 digitalmars.D/11960

The suggestion from this message cannot work without breaking polymorphy. Reason: every object can be substituted by an instance of a derived object, which may be larger than the original one. Thus it is unclear how much space should be left within the container to accomodate the object. Thus, it *has* to be allocated dynamically. Suggested solution: allow structs to inherit from classes - the resulting struct is reliable, that is it cannot be substituted by anything else. We attempt to write a complete, detailed proposal, with reasoning and details for suggested implementations, as soon as Walter calls out to design D 2.0. Until then, it is not realistic to expect such features. -eye
Nov 09 2004
prev sibling next sibling parent reply Dave <Dave_member pathlink.com> writes:
In article <cmei3a$34n$1 digitaldaemon.com>, Thomas Kuehne says...
no where.com schrieb am Donnerstag, 4. November 2004 23:19:
 This result looks not good for D.  I have never seen a benchmark where D
 performs so bad like this compared with C++.  Is it just because "The D
 port is a quick 'n' dirty hack"; or some other reasons.

The D port is "quick 'n' dirty hack" because it neither uses static/const attribute nor D's on constructs like "foreach". Currently it's only in the state where it might uncover serious problems like gdc's "Complex" test case. This benchmark requires review before it can provide comparable data. Comeon every one and tune this monster. ;) svn://svn.kuehne.cn/dstress/benchmark/oopack

"finalizing" all of the classes gives much better results for everything except Complex - the main problem there is that several new Complex objects are created for each iteration of the oop_test.. Anyone have a solution for that?
 D is designed to be a simpler language than C++, the object model should
 be more efficient, right?

current D compilers are relativly young. To figure out what is going on someone with some time to spare and a dual boot Linux(dmd,gdc,gcc)/Windows(dmc,dmd,gdc,gcc) box should run the benchmark. This way it should be possible to figure if the problem lies within the frontent, backend, benchmark port or the interaction of the tool chain. Thomas

Nov 05 2004
parent reply "Thomas Kuehne" <thomas-dloop kuehne.cn> writes:
Dave schrieb:
The D port is "quick 'n' dirty hack" because it neither uses static/const
attributes nor D's own constructs like "foreach". Currently it's only in the
state where it might uncover serious problems like gdc's "Complex" test
case. This benchmark requires review before it can provide comparable data.
Come on every one and tune this monster. ;)
svn://svn.kuehne.cn/dstress/benchmark/oopack

"finalizing" all of the classes gives much better results for everything except Complex - the main problem there is that several new Complex objects are created for each iteration of the oop_test.. Anyone have a solution for that?

Could you please send me a diff of your finalized version? Thomas
Nov 05 2004
parent Dave <Dave_member pathlink.com> writes:
In article <cmh819$2ij6$1 digitaldaemon.com>, Thomas Kuehne says...
Dave schrieb:
The D port is "quick 'n' dirty hack" because it neither uses static/const
attributes nor D's own constructs like "foreach". Currently it's only in the
state where it might uncover serious problems like gdc's "Complex" test
case. This benchmark requires review before it can provide comparable data.
Come on every one and tune this monster. ;)
svn://svn.kuehne.cn/dstress/benchmark/oopack

"finalizing" all of the classes gives much better results for everything except Complex - the main problem there is that several new Complex objects are created for each iteration of the oop_test.. Anyone have a solution for that?

Could you please send me a diff of your finalized version? Thomas

Ok - here that is: 322c322 < final class MaxBenchmark : Benchmark { ---
 class MaxBenchmark : Benchmark {

< final class MatrixBenchmark: public Benchmark { ---
 class MatrixBenchmark: public Benchmark {

< final class Matrix { ---
       class Matrix {

< final class IteratorBenchmark: public Benchmark { ---
 class IteratorBenchmark: public Benchmark {

< final class Iterator { ---
       class Iterator {

< final class ComplexBenchmark: public Benchmark { ---
 class ComplexBenchmark: public Benchmark {

< final class Complex { ---
       class Complex {

- Dave
Nov 06 2004
prev sibling parent Dave <Dave_member pathlink.com> writes:
In article <cmei3a$34n$1 digitaldaemon.com>, Thomas Kuehne says...
no where.com schrieb am Donnerstag, 4. November 2004 23:19:
 This result looks not good for D.  I have never seen a benchmark where D
 performs so bad like this compared with C++.  Is it just because "The D
 port is a quick 'n' dirty hack"; or some other reasons.

The D port is "quick 'n' dirty hack" because it neither uses static/const attribute nor D's on constructs like "foreach". Currently it's only in the state where it might uncover serious problems like gdc's "Complex" test case. This benchmark requires review before it can provide comparable data. Comeon every one and tune this monster. ;) svn://svn.kuehne.cn/dstress/benchmark/oopack
 D is designed to be a simpler language than C++, the object model should
 be more efficient, right?

current D compilers are relativly young. To figure out what is going on someone with some time to spare and a dual boot Linux(dmd,gdc,gcc)/Windows(dmc,dmd,gdc,gcc) box should run the benchmark. This way it should be possible to figure if the problem lies within the frontent, backend, benchmark port or the interaction of the tool chain. Thomas

In the NG archives there has been talk about the compiler being able to 'auto-finalize' classes and methods because of how the D language has been designed. Once that is done a lot of the abstraction penalty should disappear w/o the programmer being concerned with using language constructs to accomplish that. - Dave
Nov 06 2004
prev sibling next sibling parent reply =?ISO-8859-1?Q?Anders_F_Bj=F6rklund?= <afb algonet.se> writes:
Thomas Kuehne wrote:

 Another benchmark. OOPack tries to evaluate the cost of using
 OOP against the cost of the "plain C" style.
 
 The D port is a quick 'n' dirty hack ...

Here are the test results from GDC, running on Mac OS X 10.3: g++ 3.4.2: (g++ -O3 -mcpu=G4) Seconds Mflops Test Iterations C OOP C OOP Ratio ---- ---------- ----------- ----------- ----- Iterator 100000 0.7 0.8 281.7 266.7 1.1 Complex 1000 0.0 0.2 400.0 42.1 9.5 Matrix 1000 0.9 0.9 280.9 274.7 1.0 Max 100000 1.5 1.9 65.8 52.1 1.3 gdc 0.8: (gdc -O2 -fweb -frelease -finline-functions -mcpu=G4) Seconds Mflops Test Iterations C OOP C OOP Ratio ---- ---------- ----------- ----------- ----- Iterator 0 0.000 0.000 nan nan nan Complex 0 0.000 0.000 nan nan nan Matrix 0 0.000 0.000 nan nan nan Max 0 0.000 0.000 nan nan nan Oops! Hmmm, seems like the clock functions are a bit off: std/c/time.d:
version (darwin) {
const uint CLOCKS_PER_SEC = 100;
const uint CLK_TCK        = 100;
} else {
const uint CLOCKS_PER_SEC = 1000;
const uint CLK_TCK        = 1000;
}

And the strtol function has the wrong return type: (64, not 32) oopack_v1p8_d.d:
version (darwin) {
extern (C) int strtol(char *nptr, char **endptr, int base);
} else {
extern (C) long strtol(char *nptr, char **endptr, int base);
}

Seconds Mflops Test Iterations C OOP C OOP Ratio ---- ---------- ----------- ----------- ----- Iterator 100000 0.070 1.279 2857.143 156.372 18.271 Complex 1000 0.002 0.483 4000.000 16.563 241.500 Matrix 1000 0.095 0.753 2631.579 332.005 7.926 Max 100000 0.233 0.375 429.185 266.667 1.609 Still seems off by a factor ten, but it did take way longer... Both were using the same local gcc/gdc build, by the way. (i.e. I used the g++ compiler from the gdc installation) --anders PS. "For a run to be accepted, adjust the number of iterations for each test so that each time reported is greater than 10 seconds." (reported by program) I just used the same #.
Nov 05 2004
parent "Thomas Kuehne" <thomas-dloop kuehne.cn> writes:
Anders F Björklund schrieb:
 And the strtol function has the wrong return type: (64, not 32)

 oopack_v1p8_d.d:
  >version (darwin) {
  >extern (C) int strtol(char *nptr, char **endptr, int base);
  >} else {
  >extern (C) long strtol(char *nptr, char **endptr, int base);
  >}

Thanks, fixed it. Thomas
Nov 05 2004
prev sibling parent Thomas Kuehne <thomas-dloop kuehne.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

After roughly 2 years(
http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digita
mars.D&artnum=12277 )
I've generated new test results for OOPack:
http://dstress.kuehne.cn/benchmark/oopack/oopack-results.pdf

extract:

dmd-0.169 -O -release -inline
                         Seconds       Mflops
Test       Iterations     C    OOP     C    OOP  Ratio
- ----       ----------  -----------  -----------  -----
Iterator       400000  3.280 4.520  243.902 176.991  1.378
Complex        200000  2.020 158.09 792.079 10.121  78.262
Matrix           3000  3.270 4.560  229.358 164.474  1.394
Max           1000000  3.760 3.990  265.957 250.627  1.061

gdc-0.19(gcc-3.4.6) -c -O3 -frelease -finline-functions -m32
                         Seconds       Mflops
Test       Iterations     C    OOP     C    OOP  Ratio
- ----       ----------  -----------  -----------  -----
Iterator       400000  1.090 4.260  733.945 187.793  3.908
Complex        200000  1.360 241.1 1176.471  6.636 177.279
Matrix           3000  1.140 3.530  657.895 212.465  3.096
Max           1000000  4.100 4.100  243.902 243.902  1.000

g++-4.1.1 -O3 -m32
                         Seconds       Mflops
Test       Iterations     C    OOP     C    OOP  Ratio
- ----       ----------  -----------  -----------  -----
Iterator       400000    1.1   1.1  733.9 740.7    1.0
Complex        200000    1.1   1.1  1428.6 1403.5  1.0
Matrix           3000    1.1   1.1  663.7 663.7    1.0
Max           1000000    3.0   3.1  327.9 326.8    1.0


While the gaps between GCC, GDC and DMD have narrowed, "Complex" still
poses a serious problem for GDC and DMD.

The C version of "Iterator" seems to be surprisingly hard for DMD:
DMD / GDC / GCC = 243.902 / 733.945 / 733.9 Mflops

It would be interesting to see how DM, DMD and GDC compete on a
MSWindows system.

Thomas


-----BEGIN PGP SIGNATURE-----

iD8DBQFFL4X7LK5blCcjpWoRAqNYAJ44mG/ooXg0Wpv4OMpp/wP5jTjDeACeOy2I
ThDvTrHyC2/zdQzBbfTEd0U=
=LXOf
-----END PGP SIGNATURE-----
Oct 13 2006