www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Unit tests and verifying pre/post-conditions and invariants

reply Jonathan M Davis <jmdavisprog gmail.com> writes:
Is there a standard and/or acceptable way to make sure that pre-conditions, 
post-conditions, or invariants _fail_ when running unit tests? That is, lets
say 
I had a function like this

void func(int x)
in
{
    assert(x < 8);
}
body
{
  //...
}


and I wanted to test to make sure that func() couldn't be called with any int 
greater or equal to 8, what would I do? The best that I can think of is to
catch 
an AssertError and ignore it. e.g.

unittest
{
    try
    {
        func(8);
        assert(0);
    }
    catch(AssertionError ae)
    {}
}


But catching AssertionErrors is generally a bad idea. As I understand it, when 
anything derived from Error is thrown, code is not left in a proper state, with 
stuff like destructors being skipped. Would it be okay to catch it in a case
like 
this, or is it a really bad idea? If it's a bad idea, I haven't a clue how to 
verify that pre-condition, post-conditions, and invariants are correct.

- Jonathan M Davis
Aug 14 2010
next sibling parent Lutger <lutger.blijdestijn gmail.com> writes:
Jonathan M Davis wrote:

 Is there a standard and/or acceptable way to make sure that pre-conditions,
 post-conditions, or invariants _fail_ when running unit tests? That is, lets
 say I had a function like this
 
 void func(int x)
 in
 {
     assert(x < 8);
 }
 body
 {
   //...
 }
 
 
 and I wanted to test to make sure that func() couldn't be called with any int
 greater or equal to 8, what would I do? The best that I can think of is to
 catch an AssertError and ignore it. e.g.
 
 unittest
 {
     try
     {
         func(8);
         assert(0);
     }
     catch(AssertionError ae)
     {}
 }
 
 
 But catching AssertionErrors is generally a bad idea. As I understand it, when
 anything derived from Error is thrown, code is not left in a proper state,
 with stuff like destructors being skipped. Would it be okay to catch it in a
 case like this, or is it a really bad idea? If it's a bad idea, I haven't a
 clue how to verify that pre-condition, post-conditions, and invariants are
 correct.
 
 - Jonathan M Davis

Thats a good one. It used to be so that dmd would assume a halt and you could get segfaults for continuing on. I *believe* this has changed and it is ok to catch AssertError in this scenario. The only thing I can think of is that nothrow functions can be optimized by the compiler, but are not typechecked for throwing Error. Even if this is correct, you still have the problem foo could call other functions that assert, which invalidates at least two contract checks. This may require some care. Perhaps an idea is to define a PreconditionError and PostconditionError and wrap them in a function similar to enforce, so you can differentiate more easily where the error is coming from. This way you also get better error messages and have one point where to change the behavior if this may be required.
Aug 15 2010
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sat, 14 Aug 2010 23:49:18 -0400, Jonathan M Davis  
<jmdavisprog gmail.com> wrote:

 Is there a standard and/or acceptable way to make sure that  
 pre-conditions,
 post-conditions, or invariants _fail_ when running unit tests? That is,  
 lets say
 I had a function like this

 void func(int x)
 in
 {
     assert(x < 8);
 }
 body
 {
   //...
 }


 and I wanted to test to make sure that func() couldn't be called with  
 any int
 greater or equal to 8, what would I do?

Hm... unit testing your unit tests :) input contracts and unit tests are supposed to be simple, provable code so you don't have to test them. The above function is obviously a simple example, you don't really need to unit test it (right?), so what would a complicated in contract look like? It's also a good idea to avoid complex expressions in unit tests. If you have a complex expression, split it out into several lines to avoid having to work through the logic in your head. Performance/minimal LOC is not a goal you need in unit tests/contracts. -Steve
Aug 16 2010
parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
Steven Schveighoffer wrote:
 On Mon, 16 Aug 2010 12:57:22 -0400, Jonathan M Davis
 <jmdavisprog gmail.com> wrote:

 Of greater value is testing that normal exceptions are thrown when
 they're
 supposed to - especially for bad input. Testing that is essentially
 the same as
 testing for AssertErrors except that they're regular exceptions, so
 you don't
 have the whole issue with Errors not getting cleaned up properly. In
 the case of
 normal exceptions though, it is arguably part of the API (albeit not
 part of the
 signature), while for AssertError it's a contract which isn't really
 part of the
 API so much as verifying that your code is correct.

I whole-heartedly agree, and the code I use for this is: bool caughtException = false; try { functionThatShouldThrow(badInput); } catch(SpecificException e) { caughtException = true; } assert(caughtException);

I've used the same. :) We should have something like ensure_throws: ensure_throws!SpecificException(functionThatShouldThrow(badInput)); Perhaps some of the existing unit test libraries already have that. Ali
Aug 16 2010
prev sibling next sibling parent Jonathan M Davis <jmdavisprog gmail.com> writes:
On Monday, August 16, 2010 05:43:08 Steven Schveighoffer wrote:
 On Sat, 14 Aug 2010 23:49:18 -0400, Jonathan M Davis
 
 <jmdavisprog gmail.com> wrote:
 Is there a standard and/or acceptable way to make sure that
 pre-conditions,
 post-conditions, or invariants _fail_ when running unit tests? That is,
 lets say
 I had a function like this
 
 void func(int x)
 in
 {
 
     assert(x < 8);
 
 }
 body
 {
 
   //...
 
 }
 
 
 and I wanted to test to make sure that func() couldn't be called with
 any int
 greater or equal to 8, what would I do?

Hm... unit testing your unit tests :) input contracts and unit tests are supposed to be simple, provable code so you don't have to test them. The above function is obviously a simple example, you don't really need to unit test it (right?), so what would a complicated in contract look like? It's also a good idea to avoid complex expressions in unit tests. If you have a complex expression, split it out into several lines to avoid having to work through the logic in your head. Performance/minimal LOC is not a goal you need in unit tests/contracts. -Steve

Ideally, unit tests and contracts would be simple. However, I do believe that there is some value in verifying that you wrote you contracts correctly. Ideally, you could use a wrapper function/template to call the function to verify that an AssertError was thrown, and the unit test would then be simple. Of greater value is testing that normal exceptions are thrown when they're supposed to - especially for bad input. Testing that is essentially the same as testing for AssertErrors except that they're regular exceptions, so you don't have the whole issue with Errors not getting cleaned up properly. In the case of normal exceptions though, it is arguably part of the API (albeit not part of the signature), while for AssertError it's a contract which isn't really part of the API so much as verifying that your code is correct. In any case, I definitely see some value in testing that sort of thing. You don't necessarily want to write unit tests for all of your contracts, but sometimes it can be valuable. I would argue though that you generally _do_ want to write code for normal exceptions that are thrown due to bad input (like with enforce) because you want to guarantee the behavior of the function, and that is part of the behavior and stays regardless of the -release flag. - Jonathan M Davis
Aug 16 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 16 Aug 2010 12:57:22 -0400, Jonathan M Davis  
<jmdavisprog gmail.com> wrote:

 Ideally, unit tests and contracts would be simple. However, I do believe  
 that
 there is some value in verifying that you wrote you contracts correctly.
 Ideally, you could use a wrapper function/template to call the function  
 to
 verify that an AssertError was thrown, and the unit test would then be  
 simple.

I don't see why that wouldn't work. If you really feel the need to test your tests. One thing I've found invaluable, especially when writing unit tests for templated items, is to write the unit tests *inside* the item. Then instantiate the item with all the different parameter types you want. For example, in dcollections, all my unit tests are generic, and work with all value types that are integral. Then at the end of the file I just have: unittest { ArrayList!ubyte al1; ArrayList!byte al2; ArrayList!ushort al3; ... } In all, I have about 8-10 different flavors of unit tests that run on each collection type. I found some very obscure compiler/runtime bugs that way :)
 Of greater value is testing that normal exceptions are thrown when  
 they're
 supposed to - especially for bad input. Testing that is essentially the  
 same as
 testing for AssertErrors except that they're regular exceptions, so you  
 don't
 have the whole issue with Errors not getting cleaned up properly. In the  
 case of
 normal exceptions though, it is arguably part of the API (albeit not  
 part of the
 signature), while for AssertError it's a contract which isn't really  
 part of the
 API so much as verifying that your code is correct.

I whole-heartedly agree, and the code I use for this is: bool caughtException = false; try { functionThatShouldThrow(badInput); } catch(SpecificException e) { caughtException = true; } assert(caughtException); An example in dcollections: http://www.dsource.org/projects/dcollections/browser/branches/d2/dcollections/HashMap.d#L757 Note another useful idiom I found: In some cases, your unit tests only work for certain template instantiations. In dcollections, all my input to initialize the collections is done with integral literals. So if you happened to instantiate an ArrayList!string, for instance, the code would fail to compile, because you can't add (1, 2, 3, 4, 5) to that array list. So I define an enum bool doUnittest in each object, which looks something like this: enum bool doUnittest = isIntegral!V; Then I use the doUnittest flag to statically disable all unit tests for things like ArrayList of string. You could probably define multiple booleans if necessary if you had unittests that did support strings. The only annoyance is this boolean is still in the release code, but it is only used at compile time and it does not add to the object size, it's just an extra data point in the static code.
 In any case, I definitely see some value in testing that sort of thing.  
 You don't
 necessarily want to write unit tests for all of your contracts, but  
 sometimes it
 can be valuable. I would argue though that you generally _do_ want to  
 write code
 for normal exceptions that are thrown due to bad input (like with  
 enforce)
 because you want to guarantee the behavior of the function, and that is  
 part of
 the behavior and stays regardless of the -release flag.

I agree, you should test your normal exceptions, especially if they are thrown on invalid user input (i.e. deterministic). contract asserts, I'm not so sure. But I can see a reason to do that. -Steve
Aug 16 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sat, 14 Aug 2010 23:49:18 -0400, Jonathan M Davis  
<jmdavisprog gmail.com> wrote:


 and I wanted to test to make sure that func() couldn't be called with  
 any int
 greater or equal to 8, what would I do? The best that I can think of is  
 to catch
 an AssertError and ignore it. e.g.

 unittest
 {
     try
     {
         func(8);
         assert(0);
     }
     catch(AssertionError ae)
     {}
 }

BTW, I just realized, this code doesn't test anything. You want something like this: unittest { bool caughtAssert = false; try { func(8); } catch(AssertionError ae) { caughtAssert = true; } assert(caughtAssert); } -Steve
Aug 16 2010
prev sibling next sibling parent Jonathan M Davis <jmdavisprog gmail.com> writes:
On Monday, August 16, 2010 14:36:13 Steven Schveighoffer wrote:
 On Sat, 14 Aug 2010 23:49:18 -0400, Jonathan M Davis
 
 <jmdavisprog gmail.com> wrote:
 and I wanted to test to make sure that func() couldn't be called with
 any int
 greater or equal to 8, what would I do? The best that I can think of is
 to catch
 an AssertError and ignore it. e.g.
 
 unittest
 {
 
     try
     {
     
         func(8);
         assert(0);
     
     }
     catch(AssertionError ae)
     {}
 
 }

BTW, I just realized, this code doesn't test anything. You want something like this: unittest { bool caughtAssert = false; try { func(8); } catch(AssertionError ae) { caughtAssert = true; } assert(caughtAssert); } -Steve

Ah, that would be true, since assert throws an AssertException. Whoops. It does work for normal exceptions though (assuming that you changed the type in the catch to the appropriate exception type). - Jonathan M Davis
Aug 16 2010
prev sibling parent Jonathan M Davis <jmdavisprog gmail.com> writes:
On Monday, August 16, 2010 14:46:47 Ali =C3=87ehreli wrote:
 I've used the same. :) We should have something like ensure_throws:
=20
    ensure_throws!SpecificException(functionThatShouldThrow(badInput));
=20
 Perhaps some of the existing unit test libraries already have that.

http://d.puremagic.com/issues/show_bug.cgi?id=3D4644
Aug 16 2010