www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Simple bolt-on unittest improvement

reply Justin Johansson <procode adam-dott-com.au> writes:
Hope it doesn't sound like I just discovered America but being a D newbie one
is keen to play with all the language features.  So soon getting round to
playing with D's unit test facility with JUnit (Java test unit) experience in
mind,  found that I wanted all my tests to run even if one or the tests would
otherwise fail on some assert.  Googled for DUnit as a hunch such beast might
exist and sure enough found DUnit for D somewhere after the Delphi hits.

http://www.dsource.org/projects/dmocks/wiki/DUnit

Sure enough DUnit writeup spoke about need for continuous testing in D as had
occurred to me.  Then it dawned upon me that there really wasn't any need to go
away from D's built-in unit test facility to achieve this and hence no need
really for the DUnit approach.

The idea is to keep working within D's built in unit test facility; just don't
use assert statements in current fashion as if one fails then whole test stops
running.  Simply replace assert statements with a function call that tests and
records the assert condition instead .. much like in JUnit where you have
functions like assertTrue, assertEquals etc.  Then just inside your main
program you call up your unit test pretty print report and decide then (say
based upon number of failed tests relative to total number of tests) whether to
continue with your mainline code or bail out.

The regime now looks something like this:

Class1:
	unittest {
		jjunit.expectEquals( __FILE__, __LINE__, some_x, some_y);
		jjunit.expectTrue( __FILE__, __LINE__, somecond);
	}


Class2:
	unittest {
		jjunit.expectGreaterThan( __FILE__, __LINE__, some_a, some_b);
		jjunit.expectFalse( __FILE__, __LINE__, some_c);
	}

App:
void main()
{
	// print unittest success/fail report and exit if more than 2 failed
	debug if ( dunit.report() > 2) exit( 1)

	// continue with application
	// ...
}


I can't imagine that my approach is anything new.  What do other people do to
achieve similar continuous unit testing goal?

Cheers
Justin Johansson
Sep 11 2009
next sibling parent Christopher Wright <dhasenan gmail.com> writes:
Justin Johansson wrote:

 Hope it doesn't sound like I just discovered America but being a D newbie one
is keen to play with all the language features.  So soon getting round to
playing with D's unit test facility with JUnit (Java test unit) experience in
mind,  found that I wanted all my tests to run even if one or the tests would
otherwise fail on some assert.  Googled for DUnit as a hunch such beast might
exist and sure enough found DUnit for D somewhere after the Delphi hits.
 
 http://www.dsource.org/projects/dmocks/wiki/DUnit
 
 Sure enough DUnit writeup spoke about need for continuous testing in D as had
occurred to me.  Then it dawned upon me that there really wasn't any need to go
away from D's built-in unit test facility to achieve this and hence no need
really for the DUnit approach.
 
 The idea is to keep working within D's built in unit test facility; just don't
use assert statements in current fashion as if one fails then whole test stops
running.  Simply replace assert statements with a function call that tests and
records the assert condition instead .. much like in JUnit where you have
functions like assertTrue, assertEquals etc.  Then just inside your main
program you call up your unit test pretty print report and decide then (say
based upon number of failed tests relative to total number of tests) whether to
continue with your mainline code or bail out.
 
 The regime now looks something like this:
 
 Class1:
 	unittest {
 		jjunit.expectEquals( __FILE__, __LINE__, some_x, some_y);
 		jjunit.expectTrue( __FILE__, __LINE__, somecond);
 	}


And you don't want to use dunit's assertions because they throw 
exceptions, which means you don't see more than one error when running 
the application.

The problem with this is that you are going to continue running a test 
after an assertion fails. This is fine for a lot of unittests -- in fact 
beneficial -- but in others it's going to be guaranteed failure:

auto foo = buildFoo();
expect(foo !is null);
expect(foo.bar == 17);

If you really want to work with d's builtin unittests, your strategy is 
required. Personally, I dislike it. That's why dunit is as it is.
Sep 11 2009
prev sibling parent reply Lutger <lutger.blijdestijn gmail.com> writes:
I'm rewriting my testing stuff at the moment, I hoped to use the runtime 
features to be able to user regular asserts too but that didn't work out.

Do you know you can get rid of the need to pass __FILE__ and __LINE__?

Define the functions this way:

void expectEquals(T)(T some_x, T some_y,
                     string file = __FILE__,
                     int line = __LINE__);

Default parameter initialization does the right thing here (in D2, dunno 
about D1).

I like the testing overhead to be as minimal as possible, for that purpose I 
have something hackish right now. Basically something like this:

void test(string testname)(void delegate () testClosure,
                           int LineNumber = __LINE__,
                           string FileName = __FILE__)
{
    writeln("running test ", name, " in ", FileName,"(", LineNumber,")");
    try
       testClosure();
    catch(Exception ex)
       // print test failure
    /* print test success. (actually first checks some flag set by expect*** 
functions to see if test passed)*/
}

And this:

void expectTrue(lazy bool exp, int LineNumber = __LINE__, 
                string FileName = __FILE__)
{
    if  (!exp())
        writeln("not true! ", FileName,"(", LineNumber,")");
}

usage:

unittest
{
    int a = 42;

    test!"foo" = {
        auto b = 42;
        expectEquals( a, b );
        expectTrue( a == a  );
        throw new Exception("fail!");
    };
}
Sep 12 2009
parent reply Justin Johansson <procode adam-dott-com.au> writes:
Lutger Wrote:


 I'm rewriting my testing stuff at the moment, I hoped to use the runtime 
 features to be able to user regular asserts too but that didn't work out.
 
 Do you know you can get rid of the need to pass __FILE__ and __LINE__?
 
 Define the functions this way:
 
 void expectEquals(T)(T some_x, T some_y,
                      string file = __FILE__,
                      int line = __LINE__);
 
 Default parameter initialization does the right thing here (in D2, dunno 
 about D1).
 


Thanks to everyone for responses; to keep forum churn low if I haven't thanked
you please consider yourself thanked ;-)

Lutger, I'm using D1.0 and, being an old C++ hacker, immediately tried the
default params trick for __FILE__ and __LINE__ but it gave me the info for the
called function and not the point of call so seems I'm stuck with the noise
unfortunately.

Moving on.  Extending the JUnit functions set which includes assertTrue,
assertGreaterThan et. al. and taking the lead from DUnit to prefix the
funcnames with "expect" rather than "assert", I've implemented things like
"expectAssertFail" and "expectException" as added tests just to verify that my
assert and exception code works.  These latter functions take functions and/or
delegates as parameters and are wrapped in a try/catch block themselves so that
the tests can be continuous.

All in all, and taking into account other replies on this thread, me still
thinks that the stock-standard D idiom for unit testing is all you need for a
decent unit testing regime.  Its just a matter of fine tuning your set of
"assert-but-don't-bail-out-on-just-one-error" functions.

If I haven't misread you, sounds we are on the same track.

Cheers
Justin
Sep 12 2009
next sibling parent reply Lutger <lutger.blijdestijn gmail.com> writes:
Justin Johansson wrote:
...

 If I haven't misread you, sounds we are on the same track.


For sure, one thing I have come to appreciate is how Walter made things so 
simple: one file containing related code, documentation and the testsuite. 
As soon as you have more than one of anything there is bound to be some 
hassle involved :)
Sep 12 2009
parent Justin Johansson <procode adam-dott-com.au> writes:
Lutger Wrote:


 Justin Johansson wrote:
 ...

 If I haven't misread you, sounds we are on the same track.


 
 For sure, one thing I have come to appreciate is how Walter made things so 
 simple: one file containing related code, documentation and the testsuite. 
 As soon as you have more than one of anything there is bound to be some 
 hassle involved :)


Hear, hear (as they say in parliament .. dunno what they say in congress or the
kremlin though).

Thanks Walter for the unittest facility!
Sep 12 2009
prev sibling parent Bill Baxter <wbaxter gmail.com> writes:
On Sat, Sep 12, 2009 at 7:09 AM, Justin Johansson
<procode adam-dott-com.au> wrote:

 Lutger Wrote:


 I'm rewriting my testing stuff at the moment, I hoped to use the runtime
 features to be able to user regular asserts too but that didn't work out.

 Do you know you can get rid of the need to pass __FILE__ and __LINE__?




You can use a string mixin.  That's the only way I know of in D1.  But
then the syntax becomes something like

     mixin(expectEquals("message"));

The expectEquals then has to return a string of code which contains
the __FILE__ and __LINE__.

--bb
Sep 12 2009