www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Unit tests and verifying pre/post-conditions and invariants

reply Jonathan M Davis <jmdavisprog gmail.com> writes:
Is there a standard and/or acceptable way to make sure that pre-conditions, 
post-conditions, or invariants _fail_ when running unit tests? That is, lets
say 
I had a function like this

void func(int x)
in
{
    assert(x < 8);
}
body
{
  //...
}


and I wanted to test to make sure that func() couldn't be called with any int 
greater or equal to 8, what would I do? The best that I can think of is to
catch 
an AssertError and ignore it. e.g.

unittest
{
    try
    {
        func(8);
        assert(0);
    }
    catch(AssertionError ae)
    {}
}


But catching AssertionErrors is generally a bad idea. As I understand it, when 
anything derived from Error is thrown, code is not left in a proper state, with 
stuff like destructors being skipped. Would it be okay to catch it in a case
like 
this, or is it a really bad idea? If it's a bad idea, I haven't a clue how to 
verify that pre-condition, post-conditions, and invariants are correct.

- Jonathan M Davis
Aug 14 2010
next sibling parent Lutger <lutger.blijdestijn gmail.com> writes:
Jonathan M Davis wrote:


 Is there a standard and/or acceptable way to make sure that pre-conditions,
 post-conditions, or invariants _fail_ when running unit tests? That is, lets
 say I had a function like this
 
 void func(int x)
 in
 {
     assert(x < 8);
 }
 body
 {
   //...
 }
 
 
 and I wanted to test to make sure that func() couldn't be called with any int
 greater or equal to 8, what would I do? The best that I can think of is to
 catch an AssertError and ignore it. e.g.
 
 unittest
 {
     try
     {
         func(8);
         assert(0);
     }
     catch(AssertionError ae)
     {}
 }
 
 
 But catching AssertionErrors is generally a bad idea. As I understand it, when
 anything derived from Error is thrown, code is not left in a proper state,
 with stuff like destructors being skipped. Would it be okay to catch it in a
 case like this, or is it a really bad idea? If it's a bad idea, I haven't a
 clue how to verify that pre-condition, post-conditions, and invariants are
 correct.
 
 - Jonathan M Davis


Thats a good one. It used to be so that dmd would assume a halt and you could 
get segfaults for continuing on. I *believe* this has changed and it is ok to 
catch AssertError in this scenario. The only thing I can think of is that 
nothrow functions can be optimized by the compiler, but are not typechecked for 
throwing Error. 

Even if this is correct, you still have the problem foo could call other 
functions that assert, which invalidates at least two contract checks. This may
 
require some care. Perhaps an idea is to define a PreconditionError and 
PostconditionError and wrap them in a function similar to enforce, so you can 
differentiate more easily where the error is coming from. This way you also get 
better error messages and have one point where to change the behavior if this 
may be required.
Aug 15 2010
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sat, 14 Aug 2010 23:49:18 -0400, Jonathan M Davis  
<jmdavisprog gmail.com> wrote:


 Is there a standard and/or acceptable way to make sure that  
 pre-conditions,
 post-conditions, or invariants _fail_ when running unit tests? That is,  
 lets say
 I had a function like this

 void func(int x)
 in
 {
     assert(x < 8);
 }
 body
 {
   //...
 }


 and I wanted to test to make sure that func() couldn't be called with  
 any int
 greater or equal to 8, what would I do?


Hm... unit testing your unit tests :)

input contracts and unit tests are supposed to be simple, provable code so  
you don't have to test them.  The above function is obviously a simple  
example, you don't really need to unit test it (right?), so what would a  
complicated in contract look like?

It's also a good idea to avoid complex expressions in unit tests.  If you  
have a complex expression, split it out into several lines to avoid having  
to work through the logic in your head.  Performance/minimal LOC is not a  
goal you need in unit tests/contracts.

-Steve
Aug 16 2010
parent reply Jonathan M Davis <jmdavisprog gmail.com> writes:
On Monday, August 16, 2010 05:43:08 Steven Schveighoffer wrote:

 On Sat, 14 Aug 2010 23:49:18 -0400, Jonathan M Davis
 
 <jmdavisprog gmail.com> wrote:

 Is there a standard and/or acceptable way to make sure that
 pre-conditions,
 post-conditions, or invariants _fail_ when running unit tests? That is,
 lets say
 I had a function like this
 
 void func(int x)
 in
 {
 
     assert(x < 8);
 
 }
 body
 {
 
   //...
 
 }
 
 
 and I wanted to test to make sure that func() couldn't be called with
 any int
 greater or equal to 8, what would I do?


 
 Hm... unit testing your unit tests :)
 
 input contracts and unit tests are supposed to be simple, provable code so
 you don't have to test them.  The above function is obviously a simple
 example, you don't really need to unit test it (right?), so what would a
 complicated in contract look like?
 
 It's also a good idea to avoid complex expressions in unit tests.  If you
 have a complex expression, split it out into several lines to avoid having
 to work through the logic in your head.  Performance/minimal LOC is not a
 goal you need in unit tests/contracts.
 
 -Steve


Ideally, unit tests and contracts would be simple. However, I do believe that 
there is some value in verifying that you wrote you contracts correctly. 
Ideally, you could use a wrapper function/template to call the function to 
verify that an AssertError was thrown, and the unit test would then be simple.

Of greater value is testing that normal exceptions are thrown when they're 
supposed to - especially for bad input. Testing that is essentially the same as 
testing for AssertErrors except that they're regular exceptions, so you don't 
have the whole issue with Errors not getting cleaned up properly. In the case
of 
normal exceptions though, it is arguably part of the API (albeit not part of
the 
signature), while for AssertError it's a contract which isn't really part of
the 
API so much as verifying that your code is correct.

In any case, I definitely see some value in testing that sort of thing. You
don't 
necessarily want to write unit tests for all of your contracts, but sometimes
it 
can be valuable. I would argue though that you generally _do_ want to write
code 
for normal exceptions that are thrown due to bad input (like with enforce) 
because you want to guarantee the behavior of the function, and that is part of 
the behavior and stays regardless of the -release flag.

- Jonathan M Davis
Aug 16 2010
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 16 Aug 2010 12:57:22 -0400, Jonathan M Davis  
<jmdavisprog gmail.com> wrote:


 Ideally, unit tests and contracts would be simple. However, I do believe  
 that
 there is some value in verifying that you wrote you contracts correctly.
 Ideally, you could use a wrapper function/template to call the function  
 to
 verify that an AssertError was thrown, and the unit test would then be  
 simple.


I don't see why that wouldn't work.  If you really feel the need to test  
your tests.

One thing I've found invaluable, especially when writing unit tests for  
templated items, is to write the unit tests *inside* the item.  Then  
instantiate the item with all the different parameter types you want.

For example, in dcollections, all my unit tests are generic, and work with  
all value types that are integral.  Then at the end of the file I just  
have:

unittest
{
    ArrayList!ubyte al1;
    ArrayList!byte al2;
    ArrayList!ushort al3;
    ...
}

In all, I have about 8-10 different flavors of unit tests that run on each  
collection type.  I found some very obscure compiler/runtime bugs that way  
:)


 Of greater value is testing that normal exceptions are thrown when  
 they're
 supposed to - especially for bad input. Testing that is essentially the  
 same as
 testing for AssertErrors except that they're regular exceptions, so you  
 don't
 have the whole issue with Errors not getting cleaned up properly. In the  
 case of
 normal exceptions though, it is arguably part of the API (albeit not  
 part of the
 signature), while for AssertError it's a contract which isn't really  
 part of the
 API so much as verifying that your code is correct.


I whole-heartedly agree, and the code I use for this is:

bool caughtException = false;
try
{
    functionThatShouldThrow(badInput);
}
catch(SpecificException e)
{
    caughtException = true;
}
assert(caughtException);

An example in dcollections:  
http://www.dsource.org/projects/dcollections/browser/branches/d2/dcollections/HashMap.d#L757

Note another useful idiom I found: In some cases, your unit tests only  
work for certain template instantiations.  In dcollections, all my input  
to initialize the collections is done with integral literals.  So if you  
happened to instantiate an ArrayList!string, for instance, the code would  
fail to compile, because you can't add (1, 2, 3, 4, 5) to that array list.

So I define an enum bool doUnittest in each object, which looks something  
like this:

enum bool doUnittest = isIntegral!V;

Then I use the doUnittest flag to statically disable all unit tests for  
things like ArrayList of string.  You could probably define multiple  
booleans if necessary if you had unittests that did support strings.

The only annoyance is this boolean is still in the release code, but it is  
only used at compile time and it does not add to the object size, it's  
just an extra data point in the static code.


 In any case, I definitely see some value in testing that sort of thing.  
 You don't
 necessarily want to write unit tests for all of your contracts, but  
 sometimes it
 can be valuable. I would argue though that you generally _do_ want to  
 write code
 for normal exceptions that are thrown due to bad input (like with  
 enforce)
 because you want to guarantee the behavior of the function, and that is  
 part of
 the behavior and stays regardless of the -release flag.


I agree, you should test your normal exceptions, especially if they are  
thrown on invalid user input (i.e. deterministic).  contract asserts, I'm  
not so sure.  But I can see a reason to do that.

-Steve
Aug 16 2010
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
Steven Schveighoffer wrote:

 On Mon, 16 Aug 2010 12:57:22 -0400, Jonathan M Davis
 <jmdavisprog gmail.com> wrote:



 Of greater value is testing that normal exceptions are thrown when
 they're
 supposed to - especially for bad input. Testing that is essentially
 the same as
 testing for AssertErrors except that they're regular exceptions, so
 you don't
 have the whole issue with Errors not getting cleaned up properly. In
 the case of
 normal exceptions though, it is arguably part of the API (albeit not
 part of the
 signature), while for AssertError it's a contract which isn't really
 part of the
 API so much as verifying that your code is correct.


 I whole-heartedly agree, and the code I use for this is:

 bool caughtException = false;
 try
 {
    functionThatShouldThrow(badInput);
 }
 catch(SpecificException e)
 {
    caughtException = true;
 }
 assert(caughtException);


I've used the same. :) We should have something like ensure_throws:

   ensure_throws!SpecificException(functionThatShouldThrow(badInput));

Perhaps some of the existing unit test libraries already have that.

Ali
Aug 16 2010
parent Jonathan M Davis <jmdavisprog gmail.com> writes:
On Monday, August 16, 2010 14:46:47 Ali =C3=87ehreli wrote:

 I've used the same. :) We should have something like ensure_throws:
=20
    ensure_throws!SpecificException(functionThatShouldThrow(badInput));
=20
 Perhaps some of the existing unit test libraries already have that.


http://d.puremagic.com/issues/show_bug.cgi?id=3D4644
Aug 16 2010
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sat, 14 Aug 2010 23:49:18 -0400, Jonathan M Davis  
<jmdavisprog gmail.com> wrote:



 and I wanted to test to make sure that func() couldn't be called with  
 any int
 greater or equal to 8, what would I do? The best that I can think of is  
 to catch
 an AssertError and ignore it. e.g.

 unittest
 {
     try
     {
         func(8);
         assert(0);
     }
     catch(AssertionError ae)
     {}
 }


BTW, I just realized, this code doesn't test anything.  You want something  
like this:

unittest
{
    bool caughtAssert = false;
    try
    {
       func(8);
    }
    catch(AssertionError ae)
    {
       caughtAssert = true;
    }
    assert(caughtAssert);
}

-Steve
Aug 16 2010
parent Jonathan M Davis <jmdavisprog gmail.com> writes:
On Monday, August 16, 2010 14:36:13 Steven Schveighoffer wrote:

 On Sat, 14 Aug 2010 23:49:18 -0400, Jonathan M Davis
 
 <jmdavisprog gmail.com> wrote:

 and I wanted to test to make sure that func() couldn't be called with
 any int
 greater or equal to 8, what would I do? The best that I can think of is
 to catch
 an AssertError and ignore it. e.g.
 
 unittest
 {
 
     try
     {
     
         func(8);
         assert(0);
     
     }
     catch(AssertionError ae)
     {}
 
 }


 
 BTW, I just realized, this code doesn't test anything.  You want something
 like this:
 
 unittest
 {
     bool caughtAssert = false;
     try
     {
        func(8);
     }
     catch(AssertionError ae)
     {
        caughtAssert = true;
     }
     assert(caughtAssert);
 }
 
 -Steve


Ah, that would be true, since assert throws an AssertException. Whoops. It does 
work for normal exceptions though (assuming that you changed the type in the 
catch to the appropriate exception type).

- Jonathan M Davis
Aug 16 2010