www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Seperating unit tests from programs

reply Alex Stevenson <ans104 cs.york.ac.uk> writes:
Hi,

I've been a lurker on the D newsgroup for a while now and I'm a definite 
  D devotee - after having done several months of C, I appreciate the 
features of D greatly. I _finally_ have something to say... It's a 
feature suggestion to throw to the wolves:

Would it be possible/desirable/moronic to add a command line option to 
get DMD to generate unit tests as a seperate executable to the main code?

There are situations where it is desirable to run unit tests seperately 
and not with the main program. I'm not very good at explaining things, 
so I'll give an example from my experience:

I've done a little work in a large-ish C project (30ish developers, 
similar number of testers) where code is built on a machine that is not 
capable of running the program produced (due to software setup/hardware 
present) even though both the build and target machine use Linux and x86 
hardware.

In this case, unit tests were built and run on the build machine after 
the main program had successfully built since none of the unit tests 
relied on the specific hardware of the target machines.

The build was not 'released' to the testers unless the unit tests had 
completed successfully. Having the unit tests seperate from the program 
let the unit tests be run on the build machine without tying up test 
machines.

Also in this case once the unit tests (possibly time-consuming and 
complex) have been successful once for each build, there is no 
additional benefit from running them on program start - in fact it could 
be harmful as the program ran as part of a cluster and made guarantees 
about time taken to start up/restart the program.

As far as I can see generating a unit test only executable would be as 
simple as generating a main function which only ran 	
_moduleCtor();
_moduleUnitTests();

and _not_ run the user-supplied main

Hopefully someone can get my meaning from these ramblings - I think 
seperating unit test code from program could be a useful aid to large 
scale projects or programs which rely on specific hardware.

Alex Stevenson
Feb 07 2005
next sibling parent Andy Friesen <andy ikagames.com> writes:
Alex Stevenson wrote:

 Hi,
 
 I've been a lurker on the D newsgroup for a while now and I'm a definite 
  D devotee - after having done several months of C, I appreciate the 
 features of D greatly. I _finally_ have something to say... It's a 
 feature suggestion to throw to the wolves:
 
 Would it be possible/desirable/moronic to add a command line option to 
 get DMD to generate unit tests as a seperate executable to the main code?
 
 There are situations where it is desirable to run unit tests seperately 
 and not with the main program.
 
 As far as I can see generating a unit test only executable would be as 
 simple as generating a main function which only ran    
 _moduleCtor();
 _moduleUnitTests();
 
 and _not_ run the user-supplied main


This would be a huge timesaver for just about any project using 
test-driven development.

Two other things that would be exceptionally useful, especially in 
combination with this, would be the ability to name unit tests, and 
having the resulting executable run every single test instead of 
stopping at the first failure.

  -- andy
Feb 07 2005
prev sibling next sibling parent pragma <pragma_member pathlink.com> writes:
In article <cu88hv$2uq6$1 digitaldaemon.com>, Alex Stevenson says...

Would it be possible/desirable/moronic to add a command line option to 
get DMD to generate unit tests as a seperate executable to the main code?


It's not a bad suggestion.  In fact, similar things have been brought up in the
past on this group.  However, it has also been noted that DMD is really designed
to be at the bottom of a tool-chain, hence why a behavior like this may not be
justifiable for the compiler itself.

my $0.02:

In your particular build environment, it looks like all you need are some custom
bits added to your makefile and/or use a shell script.  Just don't use the
'unittest{}' block and compile your application against a unit-test oriented
'int main()' as a separate build step.  With a little clever coding, you can use
this to catch exceptions so you can run a report of which tests failed, and
which passed; its much more flexible than a basic 'unittest{}' this way.

The shell-script option would also allow you to branch based on the result of
'unittest.exe' (or whatever you want to call it), so you can skip the call to
compile the target application.  If the test passes, then the script can then
build your main executable and run it (if need be).

If shell scripting's not your thing, you could just write your builder in D.


 // builder program (untested, w/o useful error messages)
 import std.process;
 static const int SUCCESS = 0; //errors are usually non-zero
 void main(){
     int result;
     if(std.process.system("make unittest") != SUCCESS) return;
     if(std.process.system("unittest.exe") != SUCCESS) return; 
     if(std.process.system("make application") != SUCCESS) return;
     std.process.system("application.exe");
 }


- EricAnderton at yahoo
Feb 07 2005
prev sibling next sibling parent "Matthew" <admin stlsoft.dot.dot.dot.dot.org> writes:
 Hi,

 I've been a lurker on the D newsgroup for a while now and I'm a 
 definite D devotee - after having done several months of C, I 
 appreciate the features of D greatly. I _finally_ have something to 
 say... It's a feature suggestion to throw to the wolves:

 Would it be possible/desirable/moronic to add a command line option to 
 get DMD to generate unit tests as a seperate executable to the main 
 code?

 There are situations where it is desirable to run unit tests 
 seperately and not with the main program. I'm not very good at 
 explaining things, so I'll give an example from my experience:

 I've done a little work in a large-ish C project (30ish developers, 
 similar number of testers) where code is built on a machine that is 
 not capable of running the program produced (due to software 
 setup/hardware present) even though both the build and target machine 
 use Linux and x86 hardware.

 In this case, unit tests were built and run on the build machine after 
 the main program had successfully built since none of the unit tests 
 relied on the specific hardware of the target machines.

 The build was not 'released' to the testers unless the unit tests had 
 completed successfully. Having the unit tests seperate from the 
 program let the unit tests be run on the build machine without tying 
 up test machines.

 Also in this case once the unit tests (possibly time-consuming and 
 complex) have been successful once for each build, there is no 
 additional benefit from running them on program start - in fact it 
 could be harmful as the program ran as part of a cluster and made 
 guarantees about time taken to start up/restart the program.

 As far as I can see generating a unit test only executable would be as 
 simple as generating a main function which only ran _moduleCtor();
 _moduleUnitTests();

 and _not_ run the user-supplied main

 Hopefully someone can get my meaning from these ramblings - I think 
 seperating unit test code from program could be a useful aid to large 
 scale projects or programs which rely on specific hardware.


Seems quite sensible to me. STLSoft has unittests in all (well, that's 
the theory) components, which are enabled by definition of the 
STLSOFT_UNITTEST preprocessor symbol. One needs to include any/all 
STLSoft headers in a main program and compile/link to a (ridiculously) 
simple component that iterates through them and executes them.

There's no automatic provision for unit-tests to be run as part of 
programs using STLSoft code, and while people _could_ do that if they 
wished, it seems a very odd thing to want to do, since, as you say, once 
they've run successfully, they will always do so. (Notwithstanding 
cosmic rays, overclocking and quantum fluctuations, of course. <g>)

IMO, unit-tests are quite different (at least in terms of their purpose) 
from contracts. Your post is very useful, since I don't think anyone's 
pointed this out before, yet it's one of those things that seem both 
obvious and important once raised.
Feb 07 2005
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
It would be pretty simple to modify internal\dmain2.d for your application
to just run the unit tests and not call main().
Feb 08 2005
parent reply "Alex Stevenson" <ans104 cs.york.ac.uk> writes:
On Tue, 8 Feb 2005 12:55:42 -0800, Walter <newshound digitalmars.com>  
wrote:


 It would be pretty simple to modify internal\dmain2.d for your  
 application
 to just run the unit tests and not call main().


Yes, looking at the code it seems like a fairly trivial matter, but is  
there any advantage to running unit tests on every program start? Once  
they've passed once the binary will always pass them unless you're using  
them for purposes of a non-unit test nature (like testing for the original  
Pentium FP problems). Unless there's something I'm overlooking, I can't  
think of a case where you'd want to run integrated unit tests and program.


-- 
Using Opera's revolutionary e-mail client: http://www.opera.com/m2/
Feb 08 2005
parent reply "Walter" <newshound digitalmars.com> writes:
"Alex Stevenson" <ans104 cs.york.ac.uk> wrote in message
news:opslwfn3ir08qma6 mjolnir.spamnet.local...

 On Tue, 8 Feb 2005 12:55:42 -0800, Walter <newshound digitalmars.com>
 wrote:


 It would be pretty simple to modify internal\dmain2.d for your
 application
 to just run the unit tests and not call main().


 Yes, looking at the code it seems like a fairly trivial matter, but is
 there any advantage to running unit tests on every program start? Once
 they've passed once the binary will always pass them unless you're using
 them for purposes of a non-unit test nature (like testing for the original
 Pentium FP problems). Unless there's something I'm overlooking, I can't
 think of a case where you'd want to run integrated unit tests and program.


They're not really meant for including in a released binary. Just an easy
way to get them run.
Feb 10 2005
next sibling parent reply Benji Smith <dlanguage xxagg.com> writes:
On Thu, 10 Feb 2005 00:13:24 -0800, "Walter"
<newshound digitalmars.com> wrote:


They're not really meant for including in a released binary. Just an easy
way to get them run.


I'm not too familiar with the D unit tests, so please ignore my
suggestion if it's already possible.

But when I do my java development, I always put my unit tests into a
different source tree than my application code. The test_src directory
has a package structure that mirrors the package hierarchy of the
application code's src directory. Here's an example...

/src
   /net
      /benjismith
         /myproject
            /MyClass.java
/test_src
   /net
      /benjismith
         /myproject
            /MyTestClass.java

When I build my project, it's easy to either include or exclude the
classes in my unit-testing hierarchy. I don't know whether such a
separation is possible in D, but this is how I prefer to structure my
projects.

--Benji Smith
Feb 10 2005
parent =?ISO-8859-1?Q?Anders_F_Bj=F6rklund?= <afb algonet.se> writes:
Benji Smith wrote:


 I'm not too familiar with the D unit tests, so please ignore my
 suggestion if it's already possible.
 
 But when I do my java development, I always put my unit tests into a
 different source tree than my application code. The test_src directory
 has a package structure that mirrors the package hierarchy of the
 application code's src directory. Here's an example...



 When I build my project, it's easy to either include or exclude the
 classes in my unit-testing hierarchy. I don't know whether such a
 separation is possible in D, but this is how I prefer to structure my
 projects.


D makes quite a deal of including unittest { } into the core language.
Thus the tests go in the usual code, and not in a separate hierarchy.

They are enabled with the use of -unittest, and disabled with -release
(actually just by not adding the -unittest option in the first place)

See the various Phobos std modules for examples on how this looks...


Building the "main" program for the unit test is much more of a manual
process with D, since you have to make sure to reference all modules
that you want to test, and since there is no reflection like in Java.

Again, see "phobos/unittest.d" that comes with DMD for an example.

--anders

PS. In case you ask about "JavaDoc", that's usually done with Doxygen...
     Ant is working on some replacement: http://leds.sourceforge.net/
Feb 10 2005
prev sibling parent reply =?iso-8859-1?q?Knud_S=F8rensen?= <knud NetRunner.all-technology.com> writes:
On Thu, 10 Feb 2005 00:13:24 -0800, Walter wrote:



 They're not really meant for including in a released binary. Just an easy
 way to get them run.


Hi

So, far I seen that people would like to:

1) Split the test program from the main program.(Alex)
2) Make sure that the tests get executed. (Walter)
3) Be able to run single named tests (Andy)
4) Be able to run unit test on -release programs. (Anders)

A way to make all this happen would be 
for the compiler to generate a separate unit test executable (1)
and make the compiler run the test executable as the last step (2).
If the test executable could be called with parameters we got (3)
and if dynamic linked to the main code would give us (4).

Knud

ps.)
Some time ago I posted some unit test suggestions on the dlanguage wiki at
http://dlanguage.netunify.com/59

But the wiki seams to have gone away.
Feb 11 2005
parent reply Derek <derek psych.ward> writes:
On Fri, 11 Feb 2005 22:40:56 +0100, Knud Sørensen wrote:


 On Thu, 10 Feb 2005 00:13:24 -0800, Walter wrote:
 
 

 They're not really meant for including in a released binary. Just an easy
 way to get them run.


 
 Hi
 
 So, far I seen that people would like to:
 
 1) Split the test program from the main program.(Alex)
 2) Make sure that the tests get executed. (Walter)
 3) Be able to run single named tests (Andy)
 4) Be able to run unit test on -release programs. (Anders)


And don't forget the desire to run *all* the unittests rather than crash
out on the first assert failure.

-- 
Derek
Melbourne, Australia
Feb 11 2005
next sibling parent reply "Alex Stevenson" <ans104 cs.york.ac.uk> writes:
On Sat, 12 Feb 2005 08:53:45 +1100, Derek <derek psych.ward> wrote:


 On Fri, 11 Feb 2005 22:40:56 +0100, Knud Sørensen wrote:


 On Thu, 10 Feb 2005 00:13:24 -0800, Walter wrote:



 They're not really meant for including in a released binary. Just an  
 easy
 way to get them run.


 Hi

 So, far I seen that people would like to:

 1) Split the test program from the main program.(Alex)
 2) Make sure that the tests get executed. (Walter)
 3) Be able to run single named tests (Andy)
 4) Be able to run unit test on -release programs. (Anders)


 And don't forget the desire to run *all* the unittests rather than crash
 out on the first assert failure.


Yes, this is very useful for large projects which do nightly builds.  
Checking unit test output can be a part of the build team's duties - they  
can arrive at work, find all the failures from the build and throw them at  
the appropriate person. (Who would catch them, rethrow them etc until it's  
time to go home).

-- 
Using Opera's revolutionary e-mail client: http://www.opera.com/m2/
Feb 11 2005
parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
"Alex Stevenson" <ans104 cs.york.ac.uk> wrote in message 
news:opsl183yzi08qma6 mjolnir.spamnet.local...

 On Sat, 12 Feb 2005 08:53:45 +1100, Derek <derek psych.ward> wrote:


 On Fri, 11 Feb 2005 22:40:56 +0100, Knud Sørensen wrote:


 On Thu, 10 Feb 2005 00:13:24 -0800, Walter wrote:



 They're not really meant for including in a released binary. Just an 
 easy
 way to get them run.


 Hi

 So, far I seen that people would like to:

 1) Split the test program from the main program.(Alex)
 2) Make sure that the tests get executed. (Walter)
 3) Be able to run single named tests (Andy)
 4) Be able to run unit test on -release programs. (Anders)


 And don't forget the desire to run *all* the unittests rather than crash
 out on the first assert failure.


 Yes, this is very useful for large projects which do nightly builds. 
 Checking unit test output can be a part of the build team's duties - they 
 can arrive at work, find all the failures from the build and throw them at 
 the appropriate person. (Who would catch them, rethrow them etc until it's 
 time to go home).


I see the D unittests as something that a code change must pass in order to 
get accepted. I can't see checking in code that breaks the unittests - such 
code should be rejected. If by some accident a change breaks a unittest it 
should be backed out.

A complete testing infrastructure would catch more subtle system bugs that 
could creep in by mistake when a change in one area accidentally causes 
another to fail. For that system I would want to catch failures and generate 
nice logs etc etc.

Then if one really wants to control the unittest harness it isn't hard to do 
that by hand or by modifying phobos.
Feb 11 2005
parent reply "Alex Stevenson" <ans104 cs.york.ac.uk> writes:
On Fri, 11 Feb 2005 20:46:08 -0500, Ben Hinkle <ben.hinkle gmail.com>  
wrote:


 "Alex Stevenson" <ans104 cs.york.ac.uk> wrote in message
 news:opsl183yzi08qma6 mjolnir.spamnet.local...

 On Sat, 12 Feb 2005 08:53:45 +1100, Derek <derek psych.ward> wrote:


 On Fri, 11 Feb 2005 22:40:56 +0100, Knud Sørensen wrote:


 On Thu, 10 Feb 2005 00:13:24 -0800, Walter wrote:



 They're not really meant for including in a released binary. Just an
 easy
 way to get them run.


 Hi

 So, far I seen that people would like to:

 1) Split the test program from the main program.(Alex)
 2) Make sure that the tests get executed. (Walter)
 3) Be able to run single named tests (Andy)
 4) Be able to run unit test on -release programs. (Anders)


 And don't forget the desire to run *all* the unittests rather than  
 crash
 out on the first assert failure.


 Yes, this is very useful for large projects which do nightly builds.
 Checking unit test output can be a part of the build team's duties -  
 they
 can arrive at work, find all the failures from the build and throw them  
 at
 the appropriate person. (Who would catch them, rethrow them etc until  
 it's
 time to go home).


 I see the D unittests as something that a code change must pass in order  
 to
 get accepted. I can't see checking in code that breaks the unittests -  
 such
 code should be rejected. If by some accident a change breaks a unittest  
 it
 should be backed out.


I agree, that's how I see unit tests - but since multiple code changes may  
be simultaneously integrated in a multiple developer environment, the unit  
test is a necessary first line of defence for catching code which causes  
problems when a formal build is produced. It's not that developers  
shouldn't run the unit tests before checking in code (they should!), but  
that it's also a good first step towards verifying a particular build of  
code, since you can't always trust programmers to folow procedure and lots  
of code/code changes interacting can produce unforseen complications.

 A complete testing infrastructure would catch more subtle system bugs  
 that
 could creep in by mistake when a change in one area accidentally causes
 another to fail. For that system I would want to catch failures and  
 generate
 nice logs etc etc.


Of course unit testing is just the first stage of testing - A set of  
automatic sanity checks for code consistency to help determine whether  
code does what you think it does (later testing should pick up more subtle  
things, like whether what it does is really what you wanted). Unit testing  
is great for catching inconsistencies before the testing proceeds to more  
involved tests which tie up test resource (Hardware or personnel)

 Then if one really wants to control the unittest harness it isn't hard  
 to do
 that by hand or by modifying phobos.


True. It is easy enough to do manually, but is it sufficiently useful to  
warrent automation (as a compiler option or version flag)?




-- 
Using Opera's revolutionary e-mail client: http://www.opera.com/m2/



 Feb 11 2005



  parent  "Walter" <newshound digitalmars.com>  writes:


"Alex Stevenson" <ans104 cs.york.ac.uk> wrote in message
news:opsl2aeynd08qma6 mjolnir.spamnet.local...

 True. It is easy enough to do manually, but is it sufficiently useful to
 warrent automation (as a compiler option or version flag)?


Sometimes having more compiler switches is more confusing than simply
editting dmain2.d to do what you need.



 Feb 11 2005



prev sibling  parent reply "Walter" <newshound digitalmars.com>  writes:


"Derek" <derek psych.ward> wrote in message
news:hmd6k9c3pmlc$.1q2ml1nysyh0$.dlg 40tude.net...

 And don't forget the desire to run *all* the unittests rather than crash
 out on the first assert failure.


No problem. There are many options:

1) Instead of using:
    assert(e);
in the unit tests, write:
    myassert(e, "message");
and write the myassert() to log any errors to the suitable log file.

2) Provide a custom implementation of std.asserterror to do the logging.

3) Catch any AssertError exceptions, log them, and proceed with the unit
tests:

    unittest
    {
        try
        {
                assert(...);
                ...
        }
        catch (AssertError ae)
        {
            ae.print();
        }
    }

The compiler & language doesn't care what code is between the { } of the
unittest blocks. It can be any valid D code that does whatever you need it
to do.



 Feb 11 2005



  parent  Derek <derek psych.ward>  writes:


On Fri, 11 Feb 2005 22:19:58 -0800, Walter wrote:


 "Derek" <derek psych.ward> wrote in message
 news:hmd6k9c3pmlc$.1q2ml1nysyh0$.dlg 40tude.net...

 And don't forget the desire to run *all* the unittests rather than crash
 out on the first assert failure.


 
 No problem. There are many options:
 
 1) Instead of using:
     assert(e);
 in the unit tests, write:
     myassert(e, "message");
 and write the myassert() to log any errors to the suitable log file.
 
 2) Provide a custom implementation of std.asserterror to do the logging.
 
 3) Catch any AssertError exceptions, log them, and proceed with the unit
 tests:
 
     unittest
     {
         try
         {
                 assert(...);
                 ...
         }
         catch (AssertError ae)
         {
             ae.print();
         }
     }
 
 The compiler & language doesn't care what code is between the { } of the
 unittest blocks. It can be any valid D code that does whatever you need it
 to do.


D'oh! Of course! Thanks for these hints, Walter.

-- 
Derek
Melbourne, Australia



 Feb 12 2005