www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - RTest, a random testing framework

reply Fawzi Mohamed <fmohamed mac.com> writes:
= RTest
== RTest a random testing framework

I wrote a framework to quickly write tests that check 
property/functions using randomly generated data or all combinations of 
some values (full coverage).
This was inspired by Haskell's Quickcheck, but the result is quite different.

the code is at
	http://github.com/fawzi/rtest/tree/master

The idea is to be able to write tests as quickly and as painlessly as possible.
Typical use is as follow:
{{{
    import frm.rtest.RTest;

    private mixin testInit!() autoInitTst;

    void myTests(){
        // define a collection for my tests
        TestCollection myTests=new TestCollection("myTests",__LINE__,__FILE__);

        // define a test
        autoInitTst.testTrue("testName",functionToTest,__LINE__,__FILE__);
        // for example
        autoInitTst.testTrue("(2*x)%2==0",(int x){ return 
((2*x)%2==0);},__LINE__,__FILE__);

        // run the tests
        myTests.runTests();
    }
}}}
If everything goes well not much should happen, because by default the 
printer does not write successes.
You can change the default controller as follows:
{{{
    SingleRTest.defaultTestController=new TextController(
        TextController.OnFailure.StopTest,
        TextController.PrintLevel.AllShort,Stdout);
}}}
and it should write out something like
{{{
    test`testName`          failures-passes/totalTests(totalCombinatorialRuns)
}}}
i.e.:
{{{
    test`assert(x*x<100)`                 0-100/100(100)
    test`assert(x*x<100)`                 0- 56/100(100)
}}}
If one wants to run three times as many tests:
{{{
    myTests.runTests(3);
}}}
If a test fails then it will print out something like this
{{{
    test`(2*x)%4==0 || (2*x)%4==2` failed (returned false instead of true)
    arg0: -802454419

    To reproduce:
     intial rng state: 
CMWC000000003ade6df6_00000020_595a6207_2a7a7b53_e59a5471_492be655_75b9b464_f45bb6b8_c5af6b1d_1eb47eb9_ff49627d_fe4cecb1_fa196181_ab208cf5_cc398818_d75acbbc_92212c68_ceaff756_c47bf07b_c11af291_c1b66dc4_ac48aabe_462ec397_21bf4b7a_803338ab_c214db41_dc162ebe_41a762a8_7b914689_ba74dba0_d0e7fa35_7fb2df5a_3beb71fb_6dcee941_0000001f_2a9f30df_00000000_00000000
 

    counter: [0]
    ERROR test `(2*x)%4==0 || (2*x)%4==2` from `test.d:35` FAILED!!
    -----------------------------------------------------------
    test`(2*x)%4==0 || (2*x)%4==2`   1-  0/  1(  1)
}}}
from it you should see the arguments that made the test fail.
If you want to re-run it you can add .runTests(1,seed,counter) to it, i.e.:
{{{
autoInitTst.testTrue("(2*x)%4==0 || (2*x)%4==2 (should fail)",(int x){ 
return ((2*x)%4==0 || (2*x)%4==2);},
    
__LINE__,__FILE__).runTests(1,"CMWC000000003ade6df6_00000020_595a6207_2a7a7b53_e59a5471_492be655_75b9b464_f45bb6b8_c5af6b1d_1eb47eb9_ff49627d_fe4cecb1_fa196181_ab208cf5_cc398818_d75acbbc_92212c68_ceaff756_c47bf07b_c11af291_c1b66dc4_ac48aabe_462ec397_21bf4b7a_803338ab_c214db41_dc162ebe_41a762a8_7b914689_ba74dba0_d0e7fa35_7fb2df5a_3beb71fb_6dcee941_0000001f_2a9f30df_00000000_00000000",[0])
}}}

If 

the default generator is not good enough you can create tests that use 
a custom generator like this:
{{{
    private mixin testInit!(manualInit,checkInit) customTst;
}}}
in manualInit you have the following variables:
  arg0,arg1,... : variable of the first,second,... argument that you 
can initialize
  arg0_i,arg0_i,... : index variable for combinatorial (extensive) coverage.
    if you use it you probably want to initialize the next variable
  arg0_max, arg1_max,...: variable that can be initialized to an 
integer that gives
    the maximum value of arg0_i+1, arg1_i+1,... giving it a value makes 
the combinatorial
    machine work, and does not set test.hasRandom to true for this variable
If an argument is not defined the default generation procedure
{{{
    Rand r=...;
    argI=generateRandom!(typeof(argI))(r);
}}}
is used.
checkInit can be used if the generation of the random configurations is 
mostly good,
  but might contain some configurations that should be skipped. In 
checkInit one
  should set the boolean variable "acceptable" to false if the configuration
  should be skipped.

For example:
{{{
    private mixin testInit!("arg0=r.uniformR(10);") smallIntTst;
}}}
then gets used as follow:
{{{
    smallIntTst.testTrue("x*x<100",(int x){ return 
(x*x<100);},__LINE__,__FILE__).runTests();
}}}
by the way this is also a faster way to perform a test, as you can see 
you don't need to define a collection (but probably it is a good idea 
to define one)

enjoy

Fawzi Mohamed
Jul 21 2008
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Fawzi Mohamed:
 = RTest
 == RTest a random testing framework

This code: template isFloat(T){ static if(is(T==float)||is(T==double)||is(T==real)){ const bool isFloat=true; } else { const bool isFloat=false; } } Can be written as: template isFloat(T){ const bool isFloat = is(T == float) || is(T == double) || is(T == real); } Or using a nicer template: template isFloat(T){ const bool isFloat = IsType!(T, float, double, real); } Bye, bearophile
Jul 21 2008
prev sibling next sibling parent reply BCS <ao pathlink.com> writes:
Reply to Fawzi,

I'm not quite following that, it seems to randomly select test values to 
supply to a function.
While I like that Idea, the implementation doesn't appeal to me (I have never 
liked string mixins if anything else is usable). I had an idea a while ago 
to might make for a better interface and might even allow better application 
of constraints:

double x,y;
Assert(1/x=pow(x,-1)).
    Where(x).NotZero.
    TestRandom();

Assert(x<y).
    Where(x).LessThan(y).
    Where(x).InRange(-10,10).
    TestEdges(1000); //< 1000 test points

Assert's arg would be lazy bool and the delegate would be storeed, the Where 
would use ref so it can pick up a pointer to x. Some sort of internal magic 
would keep track of what constraints apply to what and let the Test* functions 
inelegantly search the envelope.
Jul 21 2008
parent reply Fawzi Mohamed <fmohamed mac.com> writes:
On 2008-07-22 00:12:51 +0200, BCS <ao pathlink.com> said:

 Reply to Fawzi,
 
 I'm not quite following that, it seems to randomly select test values 
 to supply to a function.

Hi BCS, that is exactly the idea: test a function with random values as input, so that many test cases can be generated with little effort from your side (I dislike very much writing test cases :). These random values do not have to be basic types, they can be structures, classes,typedefs, actually if one is using too much custom generators probably he should define his type, and write T randomGenerate(T:MyType) for it.
 While I like that Idea, the implementation doesn't appeal to me (I have 
 never liked string mixins if anything else is usable).

I agree the string mixins are a feature that should be avoided if possible, but I didn't see how to do it with a similar effort without them. I see them as the equivalent of dynamically typed languages vs statically typed, like in that case you loose many checks, but you can gain simplicity and expressiveness. There are many discussions about it but it seems clear to me that each choice has it own pros and cons, and depending on the problem it might be more or less suited.
  I had an idea a while ago to might make for a better interface and 
 might even allow better application of constraints:
 
 double x,y;
 Assert(1/x=pow(x,-1)).
     Where(x).NotZero.
     TestRandom();
 
 Assert(x<y).
     Where(x).LessThan(y).
     Where(x).InRange(-10,10).
     TestEdges(1000); //< 1000 test points
 
 Assert's arg would be lazy bool and the delegate would be storeed, the 
 Where would use ref so it can pick up a pointer to x. Some sort of 
 internal magic would keep track of what constraints apply to what and 
 let the Test* functions inelegantly search the envelope.

I like your proposal but I fail to see how it can scale to more complex cases. Indeed my examples where very simple, but a "real" case looks like this: You have a function that solves linear systems of equations, and you want to test it. The matrix should be square, and the b vector should have the same size as the matrix dimension. So either you define an ad-hoc structure, or you write a custom generator for it (it is quite unlikely that the constraint are satisfied just by chance and you would spend all your time waiting for a valid test case). Then (if detA>0) you can check that the solution really solves the system of equations with a small residual error. Your test can fail in many ways also due to the internal checks of the equation solver, and you want to always have a nice report that lets you reproduce the problem. Another typical use case is when you have a slow reference implementation for something and a fast one, and you want to be sure they are the same. I think that my approach works well in those cases, and I don't see how your "magic" could work, but I would like to be shown wrong :) Fawzi
Jul 22 2008
parent reply BCS <ao pathlink.com> writes:
Reply to Fawzi,

 On 2008-07-22 00:12:51 +0200, BCS <ao pathlink.com> said:
 
 Reply to Fawzi,
 
 I'm not quite following that, it seems to randomly select test values
 to supply to a function.
 

that is exactly the idea: test a function with random values as input, so that many test cases can be generated with little effort from your side (I dislike very much writing test cases :). These random values do not have to be basic types, they can be structures, classes,typedefs, actually if one is using too much custom generators probably he should define his type, and write T randomGenerate(T:MyType) for it.
 While I like that Idea, the implementation doesn't appeal to me (I
 have never liked string mixins if anything else is usable).
 

possible, but I didn't see how to do it with a similar effort without them. I see them as the equivalent of dynamically typed languages vs statically typed, like in that case you loose many checks, but you can gain simplicity and expressiveness. There are many discussions about it but it seems clear to me that each choice has it own pros and cons, and depending on the problem it might be more or less suited.
 I had an idea a while ago to might make for a better interface and
 might even allow better application of constraints:
 
 double x,y;
 Assert(1/x=pow(x,-1)).
 Where(x).NotZero.
 TestRandom();
 Assert(x<y).
 Where(x).LessThan(y).
 Where(x).InRange(-10,10).
 TestEdges(1000); //< 1000 test points
 Assert's arg would be lazy bool and the delegate would be storeed,
 the Where would use ref so it can pick up a pointer to x. Some sort
 of internal magic would keep track of what constraints apply to what
 and let the Test* functions inelegantly search the envelope.
 

complex cases. Indeed my examples where very simple, but a "real" case looks like this: You have a function that solves linear systems of equations, and you want to test it. The matrix should be square, and the b vector should have the same size as the matrix dimension. So either you define an ad-hoc structure, or you write a custom generator for it (it is quite unlikely that the constraint are satisfied just by chance and you would spend all your time waiting for a valid test case). Then (if detA>0) you can check that the solution really solves the system of equations with a small residual error. Your test can fail in many ways also due to the internal checks of the equation solver, and you want to always have a nice report that lets you reproduce the problem. Another typical use case is when you have a slow reference implementation for something and a fast one, and you want to be sure they are the same. I think that my approach works well in those cases, and I don't see how your "magic" could work, but I would like to be shown wrong :)

The way the magic above would work is that the Where clauses are not just tested, they are used to define the test envelope. e.g. the clause Where(x).InRange(-10,10) would define a radome number generator that sets x to a radome number in the given range, then the Where(x).LessThan(y) clause would be evaluated and a radome number generated for y that is always in the specified range (in this cases [-inf, x]). Thus in the given cases, ALL test cases would be valid. Other 'Where' predicates could be defended that don't drive the generator but are only checked to see if the test cases is valid (like the det()!=0 example you gave) or even only used to determine the expected result so that correct failure cases could be checked as well. Other variants of the where clause could be used like: With(arg).InRange(0,8).Eval(matrix.SquareArrray(arg)). // set arg in [0,10] then eval the expression The overloading needed to mix ref and lazy parameters with constant arguments would take some work but I think it is doable. There would also need to be quite a bit of magic in the final test function to find test cases within the provided constraints but for many cases this could be made a lot better than guess and check. I would love to take a crack at the problem but I have about 4 different project in the queue already.
 Fawzi
 

p.s. typo: Assert(1/x=pow(x,-1)). -> Assert(1/x==pow(x,-1)).
Jul 22 2008
parent reply Fawzi Mohamed <fmohamed mac.com> writes:
On 2008-07-22 19:22:19 +0200, BCS <ao pathlink.com> said:

 Reply to Fawzi,
 
 On 2008-07-22 00:12:51 +0200, BCS <ao pathlink.com> said:
 
 Reply to Fawzi,
 
 [...]
 I had an idea a while ago to might make for a better interface and
 might even allow better application of constraints:
 
 double x,y;
 Assert(1/x=pow(x,-1)).
 Where(x).NotZero.
 TestRandom();
 Assert(x<y).
 Where(x).LessThan(y).
 Where(x).InRange(-10,10).
 TestEdges(1000); //< 1000 test points
 Assert's arg would be lazy bool and the delegate would be storeed,
 the Where would use ref so it can pick up a pointer to x. Some sort
 of internal magic would keep track of what constraints apply to what
 and let the Test* functions inelegantly search the envelope.
 

complex cases. [...]

The way the magic above would work is that the Where clauses are not just tested, they are used to define the test envelope. e.g. the clause Where(x).InRange(-10,10) would define a radome number generator that sets x to a radome number in the given range, then the Where(x).LessThan(y) clause would be evaluated and a radome number generated for y that is always in the specified range (in this cases [-inf, x]). Thus in the given cases, ALL test cases would be valid. Other 'Where' predicates could be defended that don't drive the generator but are only checked to see if the test cases is valid (like the det()!=0 example you gave) or even only used to determine the expected result so that correct failure cases could be checked as well. Other variants of the where clause could be used like: With(arg).InRange(0,8).Eval(matrix.SquareArrray(arg)). // set arg in [0,10] then eval the expression The overloading needed to mix ref and lazy parameters with constant arguments would take some work but I think it is doable. There would also need to be quite a bit of magic in the final test function to find test cases within the provided constraints but for many cases this could be made a lot better than guess and check. I would love to take a crack at the problem but I have about 4 different project in the queue already.

ok I see how this could work, you have one where clause for each generator, and that is used to pick up the type and the address of the variable and store them is some structure (like a Variant). Then subsequent messages would set the generator, and maybe constraints. It could be done, and would be an interesting and challenging project... Would it be easier to use or simpler to implement than string mixins? I don't know, probably not, still an interesting approach. thanks for the feedback BCS! Fawzi
 p.s. typo: Assert(1/x=pow(x,-1)).  -> Assert(1/x==pow(x,-1)).

and the second where should be Where(y) :)
Jul 22 2008
parent reply BCS <ao pathlink.com> writes:
Reply to Fawzi,


 ok I see how this could work, you have one where clause for each
 generator, and that is used to pick up the type and the address of the
 variable and store them is some structure (like a Variant).
 Then subsequent messages would set the generator, and maybe
 constraints.
 It could be done, and would be an interesting and challenging
 project...
 Would it be easier to use or simpler to implement than string mixins?
 I don't know, probably not, still an interesting approach.
 

Easier to use, yes. For one, it would syntax highlight correctly! Also the parse errors get better messages. Easier to write? probably not, but it might not be harder either.
 thanks for the feedback BCS!
 
 Fawzi
 
 p.s. typo: Assert(1/x=pow(x,-1)).  -> Assert(1/x==pow(x,-1)).
 


There was a reason I did it that way... Darn I forget what. But you are correct that line is wrong (but switching to y might not the the solution).
Jul 22 2008
parent reply Fawzi Mohamed <fmohamed mac.com> writes:
On 2008-07-23 00:54:49 +0200, BCS <ao pathlink.com> said:

 Reply to Fawzi,
 
 
 ok I see how this could work, you have one where clause for each
 generator, and that is used to pick up the type and the address of the
 variable and store them is some structure (like a Variant).
 Then subsequent messages would set the generator, and maybe
 constraints.
 It could be done, and would be an interesting and challenging
 project...
 Would it be easier to use or simpler to implement than string mixins?
 I don't know, probably not, still an interesting approach.
 

Easier to use, yes. For one, it would syntax highlight correctly!

fair enough
  Also the parse errors get better messages.

well I spent some effort in making that better, if there is a syntax error it caches it and writes your a message saying there is and error, your arguments and the core part of the generated mixin. Not perfect, but much better than the default behavior.
  Easier to write? probably not, but it might not be harder either.

true, in my case expression mixin (a D 2.0 feature) would have made the interface and usage a little bit better, but well... Fawzi
Jul 23 2008
next sibling parent Fawzi Mohamed <fmohamed mac.com> writes:
On 2008-07-23 09:20:47 +0200, Fawzi Mohamed <fmohamed mac.com> said:

 On 2008-07-23 00:54:49 +0200, BCS <ao pathlink.com> said:
 
 Reply to Fawzi,
 
 
 ok I see how this could work, you have one where clause for each
 generator, and that is used to pick up the type and the address of the
 variable and store them is some structure (like a Variant).
 Then subsequent messages would set the generator, and maybe
 constraints.
 It could be done, and would be an interesting and challenging
 project...
 Would it be easier to use or simpler to implement than string mixins?
 I don't know, probably not, still an interesting approach.
 

Easier to use, yes. For one, it would syntax highlight correctly!

fair enough
  Also the parse errors get better messages.

well I spent some effort in making that better, if there is a syntax error it caches it and writes your a message saying there is and error, your arguments and the core part of the generated mixin. Not perfect, but much better than the default behavior.

and I forgot to say that my hope is that the most common case will be of using the default generator for each type (so instantiate the template with no arguments). Quickcheck enforces it, if you want another generator you need to define a typedef, I leave more freedom, one can (and should) do it if he thinks that this generator is going to be used often.
 
  Easier to write? probably not, but it might not be harder either.

true, in my case expression mixin (a D 2.0 feature) would have made the interface and usage a little bit better, but well... Fawzi

Jul 23 2008
prev sibling parent reply BCS <ao pathlink.com> writes:
Reply to Fawzi,

 On 2008-07-23 00:54:49 +0200, BCS <ao pathlink.com> said:
 
 Also the parse errors get better messages.
 

error it caches it and writes your a message saying there is and error, your arguments and the core part of the generated mixin. Not perfect, but much better than the default behavior.

the better thing about non string mixin code is that the error happens at the point of the error, there is no way to tell where the string is defined and generate an error there.
Jul 23 2008
next sibling parent Fawzi Mohamed <fmohamed mac.com> writes:
On 2008-07-23 18:48:50 +0200, BCS <ao pathlink.com> said:

 Reply to Fawzi,
 
 On 2008-07-23 00:54:49 +0200, BCS <ao pathlink.com> said:
 
 Also the parse errors get better messages.
 

error it caches it and writes your a message saying there is and error, your arguments and the core part of the generated mixin. Not perfect, but much better than the default behavior.

the better thing about non string mixin code is that the error happens at the point of the error, there is no way to tell where the string is defined and generate an error there.

well I could get rid of the mixin for the manual init and force people to use typedefs, mmhh, first I will try to use it a little, than I will re-evaluate the decision. For the exclusion of bad cases I think that the mixin is still the best solution, well actually an expression mixin would be better, but in D1.0...
Jul 23 2008
prev sibling parent reply Don <nospam nospam.com.au> writes:
BCS wrote:
 Reply to Fawzi,
 
 On 2008-07-23 00:54:49 +0200, BCS <ao pathlink.com> said:

 Also the parse errors get better messages.

error it caches it and writes your a message saying there is and error, your arguments and the core part of the generated mixin. Not perfect, but much better than the default behavior.

the better thing about non string mixin code is that the error happens at the point of the error, there is no way to tell where the string is defined and generate an error there.

Yes there is! If you detect an error, instead of returning your mixed-in string, you return `static assert(0, "Found an error in your code");` And then the error points the line of the user's code. That's actually much better than you can do with templates.
Jul 24 2008
parent BCS <ao pathlink.com> writes:
Reply to don,

 BCS wrote:
 
 Reply to Fawzi,
 
 On 2008-07-23 00:54:49 +0200, BCS <ao pathlink.com> said:
 
 Also the parse errors get better messages.
 

error it caches it and writes your a message saying there is and error, your arguments and the core part of the generated mixin. Not perfect, but much better than the default behavior.

happens at the point of the error, there is no way to tell where the string is defined and generate an error there.

mixed-in string, you return `static assert(0, "Found an error in your code");` And then the error points the line of the user's code. That's actually much better than you can do with templates.

option 1; int Foo(char[] foo)() { return mixin(foo); } int Bob()() { Foo!("blab")(); return 0; } void main() { Bob!()(); } errors: t2.d(1): Error: undefined identifier blab // the relevant error message t2.d(2): template instance t2.Foo!("blab") error instantiating // the relevant line# t2.d(3): template instance t2.Bob!() error instantiating note that there could be an arbitrary number of errors between line 1 & 2 and between 2 & 3 depending on how much processing is done. option 2: int Foo2(T)(lazy T t) { return t(); } int Bob2()() { Foo2!(int)(blab); return 0; } void main() { Bob2!()(); } errors: t2.d(2): Error: undefined identifier blab // the relevant one t2.d(3): template instance t2.Bob2!() error instantiating I think it's better to have the same error message give me the line number of the actual error and what's wrong is a lot better than splitting them across who knows how much space. And to boot it's at the top of the list.
Jul 24 2008
prev sibling next sibling parent JAnderson <ask me.com> writes:
Fawzi Mohamed wrote:
 = RTest
 == RTest a random testing framework
 
 I wrote a framework to quickly write tests that check property/functions 
 using randomly generated data or all combinations of some values (full 
 coverage).
 This was inspired by Haskell's Quickcheck, but the result is quite 
 different.
 
 the code is at
     http://github.com/fawzi/rtest/tree/master
 
 The idea is to be able to write tests as quickly and as painlessly as 
 possible.
 Typical use is as follow:
 {{{
    import frm.rtest.RTest;
 
    private mixin testInit!() autoInitTst;
 
    void myTests(){
        // define a collection for my tests
        TestCollection myTests=new 
 TestCollection("myTests",__LINE__,__FILE__);
 
        // define a test
        autoInitTst.testTrue("testName",functionToTest,__LINE__,__FILE__);
        // for example
        autoInitTst.testTrue("(2*x)%2==0",(int x){ return 
 ((2*x)%2==0);},__LINE__,__FILE__);
 
        // run the tests
        myTests.runTests();
    }
 }}}
 If everything goes well not much should happen, because by default the 
 printer does not write successes.
 You can change the default controller as follows:
 {{{
    SingleRTest.defaultTestController=new TextController(
        TextController.OnFailure.StopTest,
        TextController.PrintLevel.AllShort,Stdout);
 }}}
 and it should write out something like
 {{{
    test`testName`          
 failures-passes/totalTests(totalCombinatorialRuns)
 }}}
 i.e.:
 {{{
    test`assert(x*x<100)`                 0-100/100(100)
    test`assert(x*x<100)`                 0- 56/100(100)
 }}}
 If one wants to run three times as many tests:
 {{{
    myTests.runTests(3);
 }}}
 If a test fails then it will print out something like this
 {{{
    test`(2*x)%4==0 || (2*x)%4==2` failed (returned false instead of true)
    arg0: -802454419
 
    To reproduce:
     intial rng state: 
 CMWC000000003ade6df6_00000020_595a6207_2a7a7b53_e59a5471_492be655_75b9b464_f45bb6b8_c5af6b1d_1eb47eb9_ff49627d_fe4cecb1_fa196181_ab208cf5_cc398818_d75acbbc_92212c68_ceaff756_c47bf07b_c11af291_c1b66dc4_ac48aabe_462ec397_21bf4b7a_803338ab_c214db41_dc162ebe_41a762a8_7b914689_ba74dba0_d0e7fa35_7fb2df5a_3beb71fb_6dcee941_0000001f_2a9f30
f_00000000_00000000 
 
 
 
    counter: [0]
    ERROR test `(2*x)%4==0 || (2*x)%4==2` from `test.d:35` FAILED!!
    -----------------------------------------------------------
    test`(2*x)%4==0 || (2*x)%4==2`   1-  0/  1(  1)
 }}}
 from it you should see the arguments that made the test fail.
 If you want to re-run it you can add .runTests(1,seed,counter) to it, i.e.:
 {{{
 autoInitTst.testTrue("(2*x)%4==0 || (2*x)%4==2 (should fail)",(int x){ 
 return ((2*x)%4==0 || (2*x)%4==2);},
    
 __LINE__,__FILE__).runTests(1,"CMWC000000003ade6df6_00000020_595a6207_2a7a7b53_e59a5471_492be655_75b9b464_f45bb6b8_c5af6b1d_1eb47eb9_ff49627d_fe4cecb1_fa196181_ab208cf5_cc398818_d75acbbc_92212c68_ceaff756_c47bf07b_c11af291_c1b66dc4_ac48aabe_462ec397_21bf4b7a_803338ab_c214db41_dc162ebe_41a762a8_7b914689_ba74dba0_d0e7fa35_7fb2df5a_3beb71fb_6dcee941_0000001f_2a9f30df_000
0000_00000000",[0]) 
 
 }}}
 
 If
 the default generator is not good enough you can create tests that use a 
 custom generator like this:
 {{{
    private mixin testInit!(manualInit,checkInit) customTst;
 }}}
 in manualInit you have the following variables:
  arg0,arg1,... : variable of the first,second,... argument that you can 
 initialize
  arg0_i,arg0_i,... : index variable for combinatorial (extensive) coverage.
    if you use it you probably want to initialize the next variable
  arg0_max, arg1_max,...: variable that can be initialized to an integer 
 that gives
    the maximum value of arg0_i+1, arg1_i+1,... giving it a value makes 
 the combinatorial
    machine work, and does not set test.hasRandom to true for this variable
 If an argument is not defined the default generation procedure
 {{{
    Rand r=...;
    argI=generateRandom!(typeof(argI))(r);
 }}}
 is used.
 checkInit can be used if the generation of the random configurations is 
 mostly good,
  but might contain some configurations that should be skipped. In 
 checkInit one
  should set the boolean variable "acceptable" to false if the configuration
  should be skipped.
 
 For example:
 {{{
    private mixin testInit!("arg0=r.uniformR(10);") smallIntTst;
 }}}
 then gets used as follow:
 {{{
    smallIntTst.testTrue("x*x<100",(int x){ return 
 (x*x<100);},__LINE__,__FILE__).runTests();
 }}}
 by the way this is also a faster way to perform a test, as you can see 
 you don't need to define a collection (but probably it is a good idea to 
 define one)
 
 enjoy
 
 Fawzi Mohamed
 

Nice! although I'm not exactly sure what the process is above with your code. I've often though about writing a tool that automatically creates unit-tests. Something that you could give an object and it would test every function in many random ways and sequences. It would validate based on the outputs of the function. So more or less it would be a test to make sure that the function behavior doesn't change, rather then a check to make sure it works in the first place. However you could look at the results that where generated and touch it up a little to give it better coverage. -Joel
Jul 22 2008
prev sibling parent reply "Bruce Adams" <tortoise_74 yeah.who.co.uk> writes:
On Mon, 21 Jul 2008 22:30:58 +0100, Fawzi Mohamed <fmohamed mac.com> wrote:

 = RTest
 == RTest a random testing framework

 I wrote a framework to quickly write tests that check property/functions  
 using randomly generated data or all combinations of some values (full  
 coverage).
 This was inspired by Haskell's Quickcheck, but the result is quite  
 different.

cases need to be deterministic and repeatable otherwise you don't have much chance of tracking down problems when your tests fail. That said, automatically generating test cases from a deterministic pseudo-random number generator might still have its uses. It might be idea to add something that checks you have sufficient coverage of the range/domain's involved statistically speaking. Regards, Bruce.
Jul 22 2008
next sibling parent reply BCS <ao pathlink.com> writes:
Reply to Bruce,

 On Mon, 21 Jul 2008 22:30:58 +0100, Fawzi Mohamed <fmohamed mac.com>
 wrote:
 
 = RTest
 == RTest a random testing framework
 I wrote a framework to quickly write tests that check
 property/functions
 using randomly generated data or all combinations of some values
 (full
 coverage).
 This was inspired by Haskell's Quickcheck, but the result is quite
 different.

Test cases need to be deterministic and repeatable otherwise you don't have much chance of tracking down problems when your tests fail. That said, automatically generating test cases from a deterministic pseudo-random number generator might still have its uses. It might be idea to add something that checks you have sufficient coverage of the range/domain's involved statistically speaking. Regards, Bruce.

I think there was something in the OP about dumping the entropy needed to reproduce failed test cases. As to checking the domain, My proposal might be of use there as it could explicitly define the edges (where most of the interesting stuff happens) and then concentrate checks there.
Jul 22 2008
parent reply dsimcha <dsimcha yahoo.com> writes:
I disagree.  Random testing can be a great way to find subtle bugs in relatively
complex algorithms that have a simpler but less efficient equivalent.  For
example, let's say you're trying to write a super-efficient implementation of a
hash table with lots of little speed hacks that could hide subtle bugs in
something that's only called a relatively small percentage of the time to begin
with, like collision resolution.  Then, let's say that this bug only shows up
under some relatively specific combination of inputs.  An easy way to be
reasonably sure that you don't have these kinds of subtle bugs would be to also
implement an associative array as a linear search just for testing.  This is
trivial to implement, so unlike your uber-optimized hash table, if it looks
right
it probably is.  In any event, it's even less likely to be wrong in the same way
as your hash table.  Then generate a ton of random data and put it in both your
hash table and your linear search and make sure it all reads back properly.  If
the bug is subtle enough, or if you don't think of it, it may just be near
impossible to manually generate enough test cases to find it.
Jul 22 2008
next sibling parent reply BCS <ao pathlink.com> writes:
Reply to dsimcha,

 I disagree. 

I'm not sure you do as I'm not sure what you are disagreeing with. All I was saying is that most (not all) errors are edge cases so spend more time (but not all of it) plugging away there. If 90% of the errors can be found in 3% of the domain, I'd rather spend 90% of my time in that 3% Aside from that, I have no issues with your assertions.
 Random testing can be a great way to find subtle bugs in
 relatively complex algorithms that have a simpler but less efficient
 equivalent.  For example, let's say you're trying to write a
 super-efficient implementation of a hash table with lots of little
 speed hacks that could hide subtle bugs in something that's only
 called a relatively small percentage of the time to begin with, like
 collision resolution.  Then, let's say that this bug only shows up
 under some relatively specific combination of inputs.  An easy way to
 be reasonably sure that you don't have these kinds of subtle bugs
 would be to also implement an associative array as a linear search
 just for testing.  This is trivial to implement, so unlike your
 uber-optimized hash table, if it looks right it probably is.  In any
 event, it's even less likely to be wrong in the same way as your hash
 table.  Then generate a ton of random data and put it in both your
 hash table and your linear search and make sure it all reads back
 properly.  If the bug is subtle enough, or if you don't think of it,
 it may just be near impossible to manually generate enough test cases
 to find it.
 

Jul 22 2008
parent reply Fawzi Mohamed <fmohamed mac.com> writes:
On 2008-07-23 06:43:53 +0200, Jesse Phillips <jessekphillips gmail.com> said:

 [...]
 I agree with Bruce that test cases need to be deterministic. The reason
 for this is that in order to debug on must be able to reproduce the
 problem at hand, if a random is applied that causes an assert to fail,
 you will not be able to track down where the problem lies. Such a system
 is just as bad as running your application and a crash occurring. You
 have successfully produced the random data set needed to create a crash,
 but no way of tracking it down.
 
 The only way a random test case could be of use is if the random value is
 captured and reported at crash time. This would allow it to be analyzed
 and be added as a static test case to prevent future regressions. I have
 not read the suggested code to see if this is the case, but the adding of
 the test case as an unchanging value is vital to the assurance of bug
 free code.

and this is exactly what my framework does. It prints the arguments it had generated for the function (often that is enough to understand what is wrong) *and* it prints the Rng initial state and counter number you need to reproduce exactly that run and (as BCS noted, I had written in the initial post) you just need to append a .runTests(1,seed,counter) to the test to do it. The counter number is used to have a full coverage by performing all combinations of discrete sets. For example you know that 0 and 1 will be corner cases for the first argument and 2,4,8 for the second argument you can easily define a generator that does all possible combinations of them. You can also mix combinatorial arguments and random ones. I did this (that for example Quickcheck cannot do easily) because while random coverage is good if you have few cases that you want to check, the probability that at least one will be missed is greater that what one expects, so having both is (I think) a good idea. Fawzi
Jul 23 2008
parent JAnderson <ask me.com> writes:
Fawzi Mohamed wrote:
 On 2008-07-23 06:43:53 +0200, Jesse Phillips <jessekphillips gmail.com> 
 said:
 
 [...]
 I agree with Bruce that test cases need to be deterministic. The reason
 for this is that in order to debug on must be able to reproduce the
 problem at hand, if a random is applied that causes an assert to fail,
 you will not be able to track down where the problem lies. Such a system
 is just as bad as running your application and a crash occurring. You
 have successfully produced the random data set needed to create a crash,
 but no way of tracking it down.

 The only way a random test case could be of use is if the random value is
 captured and reported at crash time. This would allow it to be analyzed
 and be added as a static test case to prevent future regressions. I have
 not read the suggested code to see if this is the case, but the adding of
 the test case as an unchanging value is vital to the assurance of bug
 free code.

and this is exactly what my framework does. It prints the arguments it had generated for the function (often that is enough to understand what is wrong) *and* it prints the Rng initial state and counter number you need to reproduce exactly that run and (as BCS noted, I had written in the initial post) you just need to append a .runTests(1,seed,counter) to the test to do it. The counter number is used to have a full coverage by performing all combinations of discrete sets. For example you know that 0 and 1 will be corner cases for the first argument and 2,4,8 for the second argument you can easily define a generator that does all possible combinations of them. You can also mix combinatorial arguments and random ones. I did this (that for example Quickcheck cannot do easily) because while random coverage is good if you have few cases that you want to check, the probability that at least one will be missed is greater that what one expects, so having both is (I think) a good idea. Fawzi

At work I use UnitTest++. That allows me to run the program though a debugger when I need to. Often its simply enough to know that you changed something that broke the unittest however perhaps you could print out the entire unit test when it fails. Then people could simply run that piece of code though a debugger. Another though that I've thought would be cool for these sort of unit tests is something that monitors/records your code and automatically generates units tests on the macro scale. ie it would write unit test files out which would essentially be recording of what your app did. I imagine you could come up with some sort of template to do this: ie: I have: foo(5); foo2(20); Now I want to record foo and foo2 as well so I change the definition to: Record(foo)(5); //Run foo and record it Record(foo2)(10);//Run foo2 and record it //Output from application run foo(5); foo2(10); Of course this would also be useful for playing back code in smoke tests. It could of course be made more advanced, like being able to change the script to wait for a certain function to return a certain result etc... -Joel
Jul 23 2008
prev sibling parent Jesse Phillips <jessekphillips gmail.com> writes:
On Tue, 22 Jul 2008 21:13:31 +0000, BCS wrote:

 Reply to dsimcha,
 
 I disagree.

I'm not sure you do as I'm not sure what you are disagreeing with. All I was saying is that most (not all) errors are edge cases so spend more time (but not all of it) plugging away there. If 90% of the errors can be found in 3% of the domain, I'd rather spend 90% of my time in that 3% Aside from that, I have no issues with your assertions.
 Random testing can be a great way to find subtle bugs in relatively
 complex algorithms that have a simpler but less efficient equivalent. 
 For example, let's say you're trying to write a super-efficient
 implementation of a hash table with lots of little speed hacks that
 could hide subtle bugs in something that's only called a relatively
 small percentage of the time to begin with, like collision resolution. 
 Then, let's say that this bug only shows up under some relatively
 specific combination of inputs.  An easy way to be reasonably sure that
 you don't have these kinds of subtle bugs would be to also implement an
 associative array as a linear search just for testing.  This is trivial
 to implement, so unlike your uber-optimized hash table, if it looks
 right it probably is.  In any event, it's even less likely to be wrong
 in the same way as your hash table.  Then generate a ton of random data
 and put it in both your hash table and your linear search and make sure
 it all reads back properly.  If the bug is subtle enough, or if you
 don't think of it, it may just be near impossible to manually generate
 enough test cases to find it.


His reply got misplaced, it was to go to Bruce Adams's post. And the rest of my reply is to go to dsimcha. I agree with Bruce that test cases need to be deterministic. The reason for this is that in order to debug on must be able to reproduce the problem at hand, if a random is applied that causes an assert to fail, you will not be able to track down where the problem lies. Such a system is just as bad as running your application and a crash occurring. You have successfully produced the random data set needed to create a crash, but no way of tracking it down. The only way a random test case could be of use is if the random value is captured and reported at crash time. This would allow it to be analyzed and be added as a static test case to prevent future regressions. I have not read the suggested code to see if this is the case, but the adding of the test case as an unchanging value is vital to the assurance of bug free code.
Jul 22 2008
prev sibling parent "Bruce Adams" <tortoise_74 yeah.who.co.uk> writes:
On Tue, 22 Jul 2008 22:01:37 +0100, dsimcha <dsimcha yahoo.com> wrote:

 I disagree.  Random testing can be a great way to find subtle bugs in  
 relatively
 complex algorithms that have a simpler but less efficient equivalent.   
 For
 example, let's say you're trying to write a super-efficient  
 implementation of a
 hash table with lots of little speed hacks that could hide subtle bugs in
 something that's only called a relatively small percentage of the time  
 to begin
 with, like collision resolution.  Then, let's say that this bug only  
 shows up
 under some relatively specific combination of inputs.  An easy way to be
 reasonably sure that you don't have these kinds of subtle bugs would be  
 to also
 implement an associative array as a linear search just for testing.   
 This is
 trivial to implement, so unlike your uber-optimized hash table, if it  
 looks right
 it probably is.  In any event, it's even less likely to be wrong in the  
 same way
 as your hash table.  Then generate a ton of random data and put it in  
 both your
 hash table and your linear search and make sure it all reads back  
 properly.  If
 the bug is subtle enough, or if you don't think of it, it may just be  
 near
 impossible to manually generate enough test cases to find it.

I agree with the strategy of using a slow version to test a fast version of an algorithm. I often use it myself. I would still be less keen on throwing random numbers at it. Rather I would try to write interfaces that exposes the bit where you're being clever. In this case maybe the collision resolution dohickey. I try to test things at the lowest level possible first. Really test the units and then the other unit tests become more like integration tests. They are mainly there to check that the logic of calling the simpler cases is correct.
Jul 24 2008