www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Unit tests called multiple times?

reply "William" <squidkidsignup gmail.com> writes:
I'm compiling a medium-sized project written in D with the 
-unittests flag.  I have some writeln statements in my unit 
tests, for debug purposes, and I've noticed that certain unit 
tests are being run more than once.  Sample output is along these 
lines:

DEBUG: unit test started...
DEBUG: loop found whatever
DEBUG: iteration successful
DEBUG: bar is equal to bar
DEBUG: unit test passed...
DEBUG: unit test started...
DEBUG: loop found whatever
DEBUG: iteration successful
DEBUG: bar is equal to bar
DEBUG: unit test passed...
DEBUG: unit test started...
DEBUG: loop found whatever
DEBUG: iteration successful
DEBUG: bar is equal to bar
DEBUG: unit test passed...

is this standard behavior for DMD?  Is it documented anywhere?  
It's not much of a problem, but it strikes me as rather odd, and 
makes the debug output difficult to read.
Apr 12 2013
next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 18:52:50 -0400, William <squidkidsignup gmail.com>  
wrote:

 I'm compiling a medium-sized project written in D with the -unittests  
 flag.  I have some writeln statements in my unit tests, for debug  
 purposes, and I've noticed that certain unit tests are being run more  
 than once.  Sample output is along these lines:

 DEBUG: unit test started...
 DEBUG: loop found whatever
 DEBUG: iteration successful
 DEBUG: bar is equal to bar
 DEBUG: unit test passed...
 DEBUG: unit test started...
 DEBUG: loop found whatever
 DEBUG: iteration successful
 DEBUG: bar is equal to bar
 DEBUG: unit test passed...
 DEBUG: unit test started...
 DEBUG: loop found whatever
 DEBUG: iteration successful
 DEBUG: bar is equal to bar
 DEBUG: unit test passed...

 is this standard behavior for DMD?  Is it documented anywhere?  It's not  
 much of a problem, but it strikes me as rather odd, and makes the debug  
 output difficult to read.

Are your unit tests inside templates? If so, they are instantiated once per template. To fix, move outside the template. This actually can be a nice feature, and at the same time horrible. It's nice because you can parameterize your unit tests! It's horrible because you have to be careful your unit tests are valid for all possible instantiations! -Steve
Apr 12 2013
prev sibling next sibling parent "William" <squidkidsignup gmail.com> writes:
On Friday, 12 April 2013 at 23:01:55 UTC, Steven Schveighoffer 
wrote:
 On Fri, 12 Apr 2013 18:52:50 -0400, William 
 <squidkidsignup gmail.com> wrote:

 I'm compiling a medium-sized project written in D with the 
 -unittests flag.  I have some writeln statements in my unit 
 tests, for debug purposes, and I've noticed that certain unit 
 tests are being run more than once.  Sample output is along 
 these lines:

 DEBUG: unit test started...
 DEBUG: loop found whatever
 DEBUG: iteration successful
 DEBUG: bar is equal to bar
 DEBUG: unit test passed...
 DEBUG: unit test started...
 DEBUG: loop found whatever
 DEBUG: iteration successful
 DEBUG: bar is equal to bar
 DEBUG: unit test passed...
 DEBUG: unit test started...
 DEBUG: loop found whatever
 DEBUG: iteration successful
 DEBUG: bar is equal to bar
 DEBUG: unit test passed...

 is this standard behavior for DMD?  Is it documented anywhere?
  It's not much of a problem, but it strikes me as rather odd, 
 and makes the debug output difficult to read.

Are your unit tests inside templates? If so, they are instantiated once per template. To fix, move outside the template. This actually can be a nice feature, and at the same time horrible. It's nice because you can parameterize your unit tests! It's horrible because you have to be careful your unit tests are valid for all possible instantiations! -Steve

Thank you! That was exactly it. I tend to put my unittests directly after the functions/classes they test, so my opApply for an implementation of n-dimensional associative arrays, (with parameterized keys/values) was having its associated unittest called every time the template was instantiated. Moving it to the module level fixed the problem, although now it reads as though the test is intended to test the whole class, when in fact it only verifies the correctness of opApply. Still, feels good to have weird behavior demystified.
Apr 12 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Apr 13, 2013 at 01:08:38AM +0200, William wrote:
 On Friday, 12 April 2013 at 23:01:55 UTC, Steven Schveighoffer
 wrote:
On Fri, 12 Apr 2013 18:52:50 -0400, William
<squidkidsignup gmail.com> wrote:

I'm compiling a medium-sized project written in D with the
-unittests flag.  I have some writeln statements in my unit
tests, for debug purposes, and I've noticed that certain unit
tests are being run more than once.  Sample output is along
these lines:

DEBUG: unit test started...
DEBUG: loop found whatever
DEBUG: iteration successful
DEBUG: bar is equal to bar
DEBUG: unit test passed...
DEBUG: unit test started...
DEBUG: loop found whatever
DEBUG: iteration successful
DEBUG: bar is equal to bar
DEBUG: unit test passed...
DEBUG: unit test started...
DEBUG: loop found whatever
DEBUG: iteration successful
DEBUG: bar is equal to bar
DEBUG: unit test passed...

is this standard behavior for DMD?  Is it documented anywhere?
 It's not much of a problem, but it strikes me as rather odd,
and makes the debug output difficult to read.

Are your unit tests inside templates? If so, they are instantiated once per template. To fix, move outside the template. This actually can be a nice feature, and at the same time horrible. It's nice because you can parameterize your unit tests! It's horrible because you have to be careful your unit tests are valid for all possible instantiations! -Steve

Thank you! That was exactly it. I tend to put my unittests directly after the functions/classes they test, so my opApply for an implementation of n-dimensional associative arrays, (with parameterized keys/values) was having its associated unittest called every time the template was instantiated. Moving it to the module level fixed the problem, although now it reads as though the test is intended to test the whole class, when in fact it only verifies the correctness of opApply. Still, feels good to have weird behavior demystified.

You could use static if to make the unittest only instantiate for a particular template instance, e.g.: struct S(T) { ... void method(T t) { ... } static if (T==int) unittest { auto s = S!int(123); s.method(123); assert( ... ); } } This way, you can still keep unittests together with the methods they test. Beware, though, that you actually instantiate S!int somewhere outside, otherwise the unittest may never actually be instantiated and run. :) (This does give rise to an interesting idiom of unittests catered for specific types that only run when your code actually uses those particular instantiations. Hmm...) T -- Just because you can, doesn't mean you should.
Apr 12 2013
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Friday, 12 April 2013 at 23:01:55 UTC, Steven Schveighoffer 
wrote:
 On Fri, 12 Apr 2013 18:52:50 -0400, William 
 <squidkidsignup gmail.com> wrote:

 I'm compiling a medium-sized project written in D with the 
 -unittests flag.  I have some writeln statements in my unit 
 tests, for debug purposes, and I've noticed that certain unit 
 tests are being run more than once.  Sample output is along 
 these lines:

 DEBUG: unit test started...
 DEBUG: loop found whatever
 DEBUG: iteration successful
 DEBUG: bar is equal to bar
 DEBUG: unit test passed...
 DEBUG: unit test started...
 DEBUG: loop found whatever
 DEBUG: iteration successful
 DEBUG: bar is equal to bar
 DEBUG: unit test passed...
 DEBUG: unit test started...
 DEBUG: loop found whatever
 DEBUG: iteration successful
 DEBUG: bar is equal to bar
 DEBUG: unit test passed...

 is this standard behavior for DMD?  Is it documented anywhere?
  It's not much of a problem, but it strikes me as rather odd, 
 and makes the debug output difficult to read.

Are your unit tests inside templates? If so, they are instantiated once per template. To fix, move outside the template. This actually can be a nice feature, and at the same time horrible. It's nice because you can parameterize your unit tests! It's horrible because you have to be careful your unit tests are valid for all possible instantiations! -Steve

That's a really nice feature. By running the unittests a developer could verify that there isn't a latent bug in the library code revealed by their particular types (or a bug in their types of course). If we could leverage this more then bugs in phobos could be identified quicker.
Apr 13 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 19:16:05 -0400, H. S. Teoh <hsteoh quickfur.ath.cx>  
wrote:

 On Sat, Apr 13, 2013 at 01:08:38AM +0200, William wrote:
 On Friday, 12 April 2013 at 23:01:55 UTC, Steven Schveighoffer
 wrote:
On Fri, 12 Apr 2013 18:52:50 -0400, William
<squidkidsignup gmail.com> wrote:

I'm compiling a medium-sized project written in D with the
-unittests flag.  I have some writeln statements in my unit
tests, for debug purposes, and I've noticed that certain unit
tests are being run more than once.  Sample output is along
these lines:

DEBUG: unit test started...
DEBUG: loop found whatever
DEBUG: iteration successful
DEBUG: bar is equal to bar
DEBUG: unit test passed...
DEBUG: unit test started...
DEBUG: loop found whatever
DEBUG: iteration successful
DEBUG: bar is equal to bar
DEBUG: unit test passed...
DEBUG: unit test started...
DEBUG: loop found whatever
DEBUG: iteration successful
DEBUG: bar is equal to bar
DEBUG: unit test passed...

is this standard behavior for DMD?  Is it documented anywhere?
 It's not much of a problem, but it strikes me as rather odd,
and makes the debug output difficult to read.

Are your unit tests inside templates? If so, they are instantiated once per template. To fix, move outside the template. This actually can be a nice feature, and at the same time horrible. It's nice because you can parameterize your unit tests! It's horrible because you have to be careful your unit tests are valid for all possible instantiations! -Steve

Thank you! That was exactly it. I tend to put my unittests directly after the functions/classes they test, so my opApply for an implementation of n-dimensional associative arrays, (with parameterized keys/values) was having its associated unittest called every time the template was instantiated. Moving it to the module level fixed the problem, although now it reads as though the test is intended to test the whole class, when in fact it only verifies the correctness of opApply. Still, feels good to have weird behavior demystified.

You could use static if to make the unittest only instantiate for a particular template instance, e.g.: struct S(T) { ... void method(T t) { ... } static if (T==int) unittest { auto s = S!int(123); s.method(123); assert( ... ); } } This way, you can still keep unittests together with the methods they test. Beware, though, that you actually instantiate S!int somewhere outside, otherwise the unittest may never actually be instantiated and run. :)

This is similar to what dcollections does: https://github.com/schveiguy/dcollections/blob/master/dcollections/ArrayList.d#L53 https://github.com/schveiguy/dcollections/blob/master/dcollections/ArrayList.d#L146
 (This does give rise to an interesting idiom of unittests catered for
 specific types that only run when your code actually uses those
 particular instantiations. Hmm...)

The issue mainly is that unit tests generally provide the parameters, so if the unit test is parameterized, you have to parameterize your literals! It's not very easy. For example, see how I have to use casts in the ArrayList unit test above. I also have helper functions in some cases that allow me to construct the correct type. -Steve
Apr 15 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sat, 13 Apr 2013 05:01:05 -0400, John Colvin  
<john.loughran.colvin gmail.com> wrote:

 That's a really nice feature. By running the unittests a developer could  
 verify that there isn't a latent bug in the library code revealed by  
 their particular types (or a bug in their types of course). If we could  
 leverage this more then bugs in phobos could be identified quicker.

I actually find it very useful. For example, in dcollections, I test with every possible integer instantiation (ubyte, byte, ushort, short, uint, int, ulong long), and it has caught quite a few bugs, even compiler bugs! Only drawback is it takes a long time to compile with all those instantiations! -Steve
Apr 15 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 4/15/13, Steven Schveighoffer <schveiguy yahoo.com> wrote:
 This is similar to what dcollections does:

 https://github.com/schveiguy/dcollections/blob/master/dcollections/ArrayList.d#L53

OT: Is dcollection now moved to github or is it just a mirror? The old site[1] doesn't mention it. [1] : http://www.dsource.org/projects/dcollections
Apr 15 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 15 Apr 2013 12:00:43 -0400, Andrej Mitrovic  
<andrej.mitrovich gmail.com> wrote:

 On 4/15/13, Steven Schveighoffer <schveiguy yahoo.com> wrote:
 This is similar to what dcollections does:

 https://github.com/schveiguy/dcollections/blob/master/dcollections/ArrayList.d#L53

OT: Is dcollection now moved to github or is it just a mirror? The old site[1] doesn't mention it. [1] : http://www.dsource.org/projects/dcollections

I did it one afternoon fully intending to port everything to the github repository. But all I did was port the source history. Given that I haven't touched dcollections in a long time, it's effectively a mirror. I will only do development in github from now on. But I need to spend some time to copy all the documentation/wiki stuff over to github. And find the time to do it :) -Steve
Apr 15 2013
prev sibling parent "Jesper" <jolsen1974 gmail.com> writes:
On Friday, 12 April 2013 at 23:18:00 UTC, H. S. Teoh wrote:
 On Sat, Apr 13, 2013 at 01:08:38AM +0200, William wrote:
 On Friday, 12 April 2013 at 23:01:55 UTC, Steven Schveighoffer
 wrote:
On Fri, 12 Apr 2013 18:52:50 -0400, William
<squidkidsignup gmail.com> wrote:

I'm compiling a medium-sized project written in D with the
-unittests flag.  I have some writeln statements in my unit
tests, for debug purposes, and I've noticed that certain unit
tests are being run more than once.  Sample output is along
these lines:

DEBUG: unit test started...
DEBUG: loop found whatever
DEBUG: iteration successful
DEBUG: bar is equal to bar
DEBUG: unit test passed...
DEBUG: unit test started...
DEBUG: loop found whatever
DEBUG: iteration successful
DEBUG: bar is equal to bar
DEBUG: unit test passed...
DEBUG: unit test started...
DEBUG: loop found whatever
DEBUG: iteration successful
DEBUG: bar is equal to bar
DEBUG: unit test passed...

is this standard behavior for DMD?  Is it documented 
anywhere?
 It's not much of a problem, but it strikes me as rather odd,
and makes the debug output difficult to read.

Are your unit tests inside templates? If so, they are instantiated once per template. To fix, move outside the template. This actually can be a nice feature, and at the same time horrible. It's nice because you can parameterize your unit tests! It's horrible because you have to be careful your unit tests are valid for all possible instantiations! -Steve

Thank you! That was exactly it. I tend to put my unittests directly after the functions/classes they test, so my opApply for an implementation of n-dimensional associative arrays, (with parameterized keys/values) was having its associated unittest called every time the template was instantiated. Moving it to the module level fixed the problem, although now it reads as though the test is intended to test the whole class, when in fact it only verifies the correctness of opApply. Still, feels good to have weird behavior demystified.

You could use static if to make the unittest only instantiate for a particular template instance, e.g.: struct S(T) { ... void method(T t) { ... } static if (T==int) unittest { auto s = S!int(123); s.method(123); assert( ... ); } } This way, you can still keep unittests together with the methods they test. Beware, though, that you actually instantiate S!int somewhere outside, otherwise the unittest may never actually be instantiated and run. :) (This does give rise to an interesting idiom of unittests catered for specific types that only run when your code actually uses those particular instantiations. Hmm...) T

I have also noticed this, that unless you instantiate a class from a path from your main-function the unittest will not be run at all. Actually come to think of it I believe this also is valid for the very compilation of the code..no compilation of code if it's not instantiated, but does this only happen for template-classes? I also have another problem with unittests, because my unittests would fail because exceptions was thrown and I would just get a completely useless message like: rdmd -unittest c:\kode\datastructures\datastructures.d DataStructures.ExceptionIndexOutOfBounds c:\kode\datastructures\datastructures.d (68): Index was out of bounds: 1 [2-3] ---------------- 0x004155A8 0x00415433 0x0040294C 0x00408BA6 0x00411A48 0x0040BBE9 0x00408C48 0x765D336A in BaseThreadInitThunk 0x77119F72 in RtlInitializeExceptionChain 0x77119F45 in RtlInitializeExceptionChain where 'ExceptionIndexOutOfBounds' is an exception I made myself and it is thrown when the unittests are run. The line-number given as a result points to the constructor of the 'ExceptionIndexOutOfBounds'-exception and not to the unittest that failed. The mem-addresses below I figure are supposed to be a stacktrace, but as you can see it lacks some basic information to make it useful. Has anyone seen this before and what am I doing wrong? (or is this the designed behaviour?..i hope not).
Oct 24 2013