www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Csmith - C compiler stress test

reply Robert Clipsham <robert octarineparrot.com> writes:
I've just come across Csmith:

http://embed.cs.utah.edu/csmith/

The tool randomly generates valid C99 code to stress test C compilers 
and has found 350+ bugs in GCC/LLVM alone over the past couple of years. 
It might be interesting to run dmc through this, and maybe adapt the 
tool to generate D test cases?

-- 
Robert
http://octarineparrot.com/
Apr 13 2011
next sibling parent reply Trass3r <un known.com> writes:
Am 13.04.2011, 16:13 Uhr, schrieb Robert Clipsham  =

<robert octarineparrot.com>:
 I've just come across Csmith:

 http://embed.cs.utah.edu/csmith/

 The tool randomly generates valid C99 code to stress test C compilers =

 and has found 350+ bugs in GCC/LLVM alone over the past couple of year=

 It might be interesting to run dmc through this, and maybe adapt the  =

 tool to generate D test cases?

I think I also came across such a tool but I'm not sure if it was this o= ne. Since it uses a grammar (subset) to generate the programs it should be = perfectly possible to adapt it for D. Though it "is mainly intended to =EF=AC=81nd bugs in the parts of a comp= iler that = perform transformations on an intermediate representation".
Apr 13 2011
next sibling parent David Nadlinger <see klickverbot.at> writes:
On 4/13/11 5:16 PM, Trass3r wrote:
 I think I also came across such a tool but I'm not sure if it was this one.
 Since it uses a grammar (subset) to generate the programs it should be
 perfectly possible to adapt it for D.
 Though it "is mainly intended to find bugs in the parts of a compiler
 that perform transformations on an intermediate representation".

From what I recall from a quick look at the Csmith sources and Xuejun Yang's talk at the LLVM developer meeting, Csmith is tailored rather specifically to C, and isn't based on a grammar representation that could be easily swapped out… David
Apr 13 2011
prev sibling parent Emil Madsen <sovende gmail.com> writes:
--001636e0aa5ef34fd804a0cf3d10
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 13 April 2011 17:27, David Nadlinger <see klickverbot.at> wrote:

 On 4/13/11 5:16 PM, Trass3r wrote:

 I think I also came across such a tool but I'm not sure if it was this
 one.
 Since it uses a grammar (subset) to generate the programs it should be
 perfectly possible to adapt it for D.
 Though it "is mainly intended to =EF=AC=81nd bugs in the parts of a comp=


 that perform transformations on an intermediate representation".

From what I recall from a quick look at the Csmith sources and Xuejun Yang's talk at the LLVM developer meeting, Csmith is tailored rather specifically to C, and isn't based on a grammar representation that could=

 easily swapped out=E2=80=A6

 David

*Can Csmith be altered to emit programs in a language other than C or C++?*= Not easily. http://embed.cs.utah.edu/csmith/using.html --=20 // Yours sincerely // Emil 'Skeen' Madsen --001636e0aa5ef34fd804a0cf3d10 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On 13 April 2011 17:27, David Nadlinger <span di= r=3D"ltr">&lt;<a href=3D"mailto:see klickverbot.at">see klickverbot.at</a>&= gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 = 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"> <div class=3D"im">On 4/13/11 5:16 PM, Trass3r wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> I think I also came across such a tool but I&#39;m not sure if it was this = one.<br> Since it uses a grammar (subset) to generate the programs it should be<br> perfectly possible to adapt it for D.<br> Though it &quot;is mainly intended to =EF=AC=81nd bugs in the parts of a co= mpiler<br> that perform transformations on an intermediate representation&quot;.<br> </blockquote> <br></div>
From what I recall from a quick look at the Csmith sources and Xuejun Yang&=

cally to C, and isn&#39;t based on a grammar representation that could be e= asily swapped out=E2=80=A6<br> <font color=3D"#888888"> <br> David<br> </font></blockquote></div><br><span class=3D"Apple-style-span" style=3D"fon= t-family: &#39;Times New Roman&#39;; font-size: medium; "><b>Can Csmith be = altered to emit programs in a language other than C or C++?</b>=C2=A0Not ea= sily.</span><div> <font class=3D"Apple-style-span" face=3D"&#39;Times New Roman&#39;" size=3D= "3"><a href=3D"http://embed.cs.utah.edu/csmith/using.html">http://embed.cs.= utah.edu/csmith/using.html</a><br clear=3D"all"></font><br>-- <br>// Yours = sincerely<br> // Emil &#39;Skeen&#39; Madsen<br> </div> --001636e0aa5ef34fd804a0cf3d10--
Apr 13 2011
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/13/2011 7:13 AM, Robert Clipsham wrote:
 I've just come across Csmith:

 http://embed.cs.utah.edu/csmith/

 The tool randomly generates valid C99 code to stress test C compilers and has
 found 350+ bugs in GCC/LLVM alone over the past couple of years. It might be
 interesting to run dmc through this, and maybe adapt the tool to generate D
test
 cases?

I talked to one of the others, he didn't think it would be that hard to adapt it to D, but we'd have to do it ourselves. I think this could be fairly valuable for D compilers. Anyone looking for a project?
Apr 13 2011
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 13 Apr 2011 14:45:17 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 4/13/2011 7:13 AM, Robert Clipsham wrote:
 I've just come across Csmith:

 http://embed.cs.utah.edu/csmith/

 The tool randomly generates valid C99 code to stress test C compilers  
 and has
 found 350+ bugs in GCC/LLVM alone over the past couple of years. It  
 might be
 interesting to run dmc through this, and maybe adapt the tool to  
 generate D test
 cases?

I talked to one of the others, he didn't think it would be that hard to adapt it to D, but we'd have to do it ourselves. I think this could be fairly valuable for D compilers. Anyone looking for a project?

As much as this might be a fun project, and might find bugs for D, I have to point out that there are *plenty* of bugs in dmd that need fixing. Any would be takers for this project should also consider lending a hand at fixing existing bugs. Then again, fixing dmd bugs might be less fun, so I could understand if someone preferred not to. I think this tool would be great once dmd is fully functional. -Steve
Apr 13 2011
next sibling parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 13.04.2011 20:57, schrieb Steven Schveighoffer:
 On Wed, 13 Apr 2011 14:45:17 -0400, Walter Bright
 <newshound2 digitalmars.com> wrote:
 
 On 4/13/2011 7:13 AM, Robert Clipsham wrote:
 I've just come across Csmith:

 http://embed.cs.utah.edu/csmith/

 The tool randomly generates valid C99 code to stress test C compilers
 and has
 found 350+ bugs in GCC/LLVM alone over the past couple of years. It
 might be
 interesting to run dmc through this, and maybe adapt the tool to
 generate D test
 cases?

I talked to one of the others, he didn't think it would be that hard to adapt it to D, but we'd have to do it ourselves. I think this could be fairly valuable for D compilers. Anyone looking for a project?

As much as this might be a fun project, and might find bugs for D, I have to point out that there are *plenty* of bugs in dmd that need fixing. Any would be takers for this project should also consider lending a hand at fixing existing bugs. Then again, fixing dmd bugs might be less fun, so I could understand if someone preferred not to. I think this tool would be great once dmd is fully functional. -Steve

As developing it probably needs some time anyway it may make sense to start developing it before dmd is mostly bugfree. Cheers, - Daniel
Apr 13 2011
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/13/2011 11:57 AM, Steven Schveighoffer wrote:
 As much as this might be a fun project, and might find bugs for D, I have to
 point out that there are *plenty* of bugs in dmd that need fixing. Any would be
 takers for this project should also consider lending a hand at fixing existing
 bugs. Then again, fixing dmd bugs might be less fun, so I could understand if
 someone preferred not to.

 I think this tool would be great once dmd is fully functional.

I think it's a mistake to avoid looking for bugs since we already have bugs to fix. We have bugzilla to keep track of them, and sure it might take time to fix them, but at least they will be there. Also, many people submit bug fix patches. Different people are interested in different kinds of bugs, and having some integer math bugs (which is what csmith finds) may appeal to a particular fixer.
Apr 13 2011
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/13/2011 1:22 PM, Steven Schveighoffer wrote:
 I'm not saying, don't write a program to find bugs. I'm saying, if you want to
 help D, a better option than trying to find bugs is to fix the bugs we have.

Often, people come to me with "what can I do for D?" I give a list of suggestions, and never once has the person done any of them. I can't tell people what to work on. People work on what they want to. They're self-starters and are self-motivated. If someone finds adapting Csmith to D a fun project, I'm going to encourage that :-)
Apr 13 2011
prev sibling next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
== Quote from Walter Bright (newshound2 digitalmars.com)'s article
 On 4/13/2011 7:13 AM, Robert Clipsham wrote:
 I've just come across Csmith:

 http://embed.cs.utah.edu/csmith/

 The tool randomly generates valid C99 code to stress test C compilers and has
 found 350+ bugs in GCC/LLVM alone over the past couple of years. It might be
 interesting to run dmc through this, and maybe adapt the tool to generate D
test
 cases?

to D, but we'd have to do it ourselves. I think this could be fairly valuable for D compilers. Anyone looking for a project?

As coincidences goes, I was considering about writing a fuzz testing tool for D just the other day (whilst in the middle of brooding over several bugs found in GDC that were not covered in the D2 testsuite). Will bookmark this for later I think. ;)
Apr 13 2011
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/13/2011 11:45 AM, Walter Bright wrote:
 I talked to one of the others, he didn't think it would be that hard to adapt
it
 to D, but we'd have to do it ourselves.

s/others/authors/
Apr 13 2011
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 13 Apr 2011 16:00:46 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 4/13/2011 11:57 AM, Steven Schveighoffer wrote:
 As much as this might be a fun project, and might find bugs for D, I  
 have to
 point out that there are *plenty* of bugs in dmd that need fixing. Any  
 would be
 takers for this project should also consider lending a hand at fixing  
 existing
 bugs. Then again, fixing dmd bugs might be less fun, so I could  
 understand if
 someone preferred not to.

 I think this tool would be great once dmd is fully functional.

I think it's a mistake to avoid looking for bugs since we already have bugs to fix. We have bugzilla to keep track of them, and sure it might take time to fix them, but at least they will be there.

I'm not saying, don't write a program to find bugs. I'm saying, if you want to help D, a better option than trying to find bugs is to fix the bugs we have. If all you want to do is find bugs, then I don't have any issue with that, and welcome the additional tools. But we have a wealth of bug finders, and few fixers. Someone who is good enough to write a D-code-generating program might have the same skillset as it takes to work on the compiler. I'd love for that person to spend time fixing bugs rather than building a tool to find them :) In other words, D doesn't *need* more bugs (though we will happily accept more bugs in bugzilla), it needs more fixes. I didn't mean to sound like I don't think we should promote such a tool or that I don't like someone working on it. I just want to also bring up that we need people to fix bugs too, and if you are talented enough and like doing either, I think it would be more appreciated by the community if bugs were fixed rather than generated.
 Also, many people submit bug fix patches. Different people are  
 interested in different kinds of bugs, and having some integer math bugs  
 (which is what csmith finds) may appeal to a particular fixer.

I hope that a csmith-like tool will be written and that it helps more people enjoy fixing bugs in dmd. I hope *more* that the existing const, inout, alias this, template overload, property, etc. bugs are fixed. -Steve
Apr 13 2011