www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Counting passed/failed unit tests

reply =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
Hi,

Is there any way to count the amount of passed/failed unit tests in a 
module or a program as a whole? From what I can see, druntime simply 
invokes a function on each module which triggers all actual tests. If 
this is all the information that is available, I guess it's impossible, 
but I'm _very_ open to suggestions/ideas here...

Thanks!

- Alex
Oct 24 2011
next sibling parent reply Justin Whear <justin economicmodeling.com> writes:
core.runtime allows a user-supplied unittest runner. Never used it, but 
check out 
http://www.digitalmars.com/d/2.0/phobos/core_runtime.html#moduleUnitTester

Justin


Alex Rønne Petersen wrote:

 Hi,
 
 Is there any way to count the amount of passed/failed unit tests in a
 module or a program as a whole? From what I can see, druntime simply
 invokes a function on each module which triggers all actual tests. If
 this is all the information that is available, I guess it's impossible,
 but I'm _very_ open to suggestions/ideas here...
 
 Thanks!
 
 - Alex

Oct 24 2011
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I'm not sure why it just stops after the first failing unittest
though. What is the point of that 'failed' counter?
Oct 24 2011
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Ok I can see why, ModuleInfo.unitTest just returns a pointer to a
function that calls all unittests on its own. That's not very
flexible.
Oct 24 2011
parent =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 24-10-2011 20:27, Andrej Mitrovic wrote:
 Ok I can see why, ModuleInfo.unitTest just returns a pointer to a
 function that calls all unittests on its own. That's not very
 flexible.

Exactly. I was hoping there would be some obscure workaround, but it doesn't seem like luck is on my side here. :( - Alex
Oct 24 2011
prev sibling next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?

It's a long standing issue that when one unit test fails within a module, no more within that module are run (though fortunately, a while back it was fixed so that other modules' unit tests will still run). As I recall, there had to be a change to the compiler to fix it, but I don't known/remember the details. Certainly, the issue still stands. - Jonathan M Davis
Oct 24 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-24 22:08, Jonathan M Davis wrote:
 On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?

It's a long standing issue that when one unit test fails within a module, no more within that module are run (though fortunately, a while back it was fixed so that other modules' unit tests will still run). As I recall, there had to be a change to the compiler to fix it, but I don't known/remember the details. Certainly, the issue still stands. - Jonathan M Davis

A workaround is to catch AssertErrors, hook it up with some library code and you get a minimal unit test framework: https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTester.d Example of usage: https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d -- /Jacob Carlborg
Oct 25 2011
next sibling parent reply David Gileadi <gileadis NSPMgmail.com> writes:
On 10/25/11 4:04 AM, Jacob Carlborg wrote:
 On 2011-10-24 22:08, Jonathan M Davis wrote:
 On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?

It's a long standing issue that when one unit test fails within a module, no more within that module are run (though fortunately, a while back it was fixed so that other modules' unit tests will still run). As I recall, there had to be a change to the compiler to fix it, but I don't known/remember the details. Certainly, the issue still stands. - Jonathan M Davis

A workaround is to catch AssertErrors, hook it up with some library code and you get a minimal unit test framework: https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTester.d Example of usage: https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d

As an argument for continuing to run tests after one fails, I'm taking a TDD class and the instructor asserted that for unit tests you should generally only have one or two assertions per test method. His reasoning is that when something breaks you immediately know the extent of your breakage by counting the number of failed methods. This argument is pretty convincing to me.
Oct 26 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-26 17:40, David Gileadi wrote:
 On 10/25/11 4:04 AM, Jacob Carlborg wrote:
 On 2011-10-24 22:08, Jonathan M Davis wrote:
 On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?

It's a long standing issue that when one unit test fails within a module, no more within that module are run (though fortunately, a while back it was fixed so that other modules' unit tests will still run). As I recall, there had to be a change to the compiler to fix it, but I don't known/remember the details. Certainly, the issue still stands. - Jonathan M Davis

A workaround is to catch AssertErrors, hook it up with some library code and you get a minimal unit test framework: https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTester.d Example of usage: https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d

As an argument for continuing to run tests after one fails, I'm taking a TDD class and the instructor asserted that for unit tests you should generally only have one or two assertions per test method. His reasoning is that when something breaks you immediately know the extent of your breakage by counting the number of failed methods. This argument is pretty convincing to me.

Well, in my library, if an assert error is thrown in a block (passed to the "it" method), the whole block is canceled and it will continue with the next block. So it's up to the user how the asserts should be laid out. -- /Jacob Carlborg
Oct 26 2011
next sibling parent David Gileadi <gileadis NSPMgmail.com> writes:
On 10/26/11 3:45 PM, Jonathan M Davis wrote:
 On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:
 On 2011-10-26 17:40, David Gileadi wrote:
 On 10/25/11 4:04 AM, Jacob Carlborg wrote:
 On 2011-10-24 22:08, Jonathan M Davis wrote:
 On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?

It's a long standing issue that when one unit test fails within a module, no more within that module are run (though fortunately, a while back it was fixed so that other modules' unit tests will still run). As I recall, there had to be a change to the compiler to fix it, but I don't known/remember the details. Certainly, the issue still stands. - Jonathan M Davis

A workaround is to catch AssertErrors, hook it up with some library code and you get a minimal unit test framework: https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitT ester.d Example of usage: https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d

As an argument for continuing to run tests after one fails, I'm taking a TDD class and the instructor asserted that for unit tests you should generally only have one or two assertions per test method. His reasoning is that when something breaks you immediately know the extent of your breakage by counting the number of failed methods. This argument is pretty convincing to me.

Well, in my library, if an assert error is thrown in a block (passed to the "it" method), the whole block is canceled and it will continue with the next block. So it's up to the user how the asserts should be laid out.

It would be disastrous IMHO to continue to run a unittest block after an assert failed - at least in the general case. Too often further assertions relied on the state assured by the one that failed, so further failures just confuse things and give you too much data to have to sift through. As it stands with the built-in unit testing faciliities, you can put each assertion in its own unittest block if you really want each assertion to run on its own (though until it's fixed so that further unittest blocks within the module run after the first failure in that module, it wouldn't do you any good). - Jonathan M Davis

Agreed--the (rather obscure, sorry) point of my message was that having further unittest blocks run within the module even after one fails is a good idea. Jacob's test frameworks sounds like a good effort in that direction, but built-in would be even better.
Oct 26 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-10-26 21:45, Jonathan M Davis wrote:
 On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:
 Well, in my library, if an assert error is thrown in a block (passed to
 the "it" method), the whole block is canceled and it will continue with
 the next block. So it's up to the user how the asserts should be laid out.

It would be disastrous IMHO to continue to run a unittest block after an assert failed - at least in the general case. Too often further assertions relied on the state assured by the one that failed, so further failures just confuse things and give you too much data to have to sift through. As it stands with the built-in unit testing faciliities, you can put each assertion in its own unittest block if you really want each assertion to run on its own (though until it's fixed so that further unittest blocks within the module run after the first failure in that module, it wouldn't do you any good). - Jonathan M Davis

My solution works now and if you want the whole unit test block to end when an assert error is thrown that is possible too. Just call the "it" method with a block once in a unit test block. You can never convince me that the other solution is better as long as it doesn't work. -- /Jacob Carlborg
Oct 26 2011
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:
 On 2011-10-26 17:40, David Gileadi wrote:
 On 10/25/11 4:04 AM, Jacob Carlborg wrote:
 On 2011-10-24 22:08, Jonathan M Davis wrote:
 On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?

It's a long standing issue that when one unit test fails within a module, no more within that module are run (though fortunately, a while back it was fixed so that other modules' unit tests will still run). As I recall, there had to be a change to the compiler to fix it, but I don't known/remember the details. Certainly, the issue still stands. - Jonathan M Davis

A workaround is to catch AssertErrors, hook it up with some library code and you get a minimal unit test framework: https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitT ester.d Example of usage: https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d

As an argument for continuing to run tests after one fails, I'm taking a TDD class and the instructor asserted that for unit tests you should generally only have one or two assertions per test method. His reasoning is that when something breaks you immediately know the extent of your breakage by counting the number of failed methods. This argument is pretty convincing to me.

Well, in my library, if an assert error is thrown in a block (passed to the "it" method), the whole block is canceled and it will continue with the next block. So it's up to the user how the asserts should be laid out.

It would be disastrous IMHO to continue to run a unittest block after an assert failed - at least in the general case. Too often further assertions relied on the state assured by the one that failed, so further failures just confuse things and give you too much data to have to sift through. As it stands with the built-in unit testing faciliities, you can put each assertion in its own unittest block if you really want each assertion to run on its own (though until it's fixed so that further unittest blocks within the module run after the first failure in that module, it wouldn't do you any good). - Jonathan M Davis
Oct 26 2011
prev sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, October 26, 2011 08:40 David Gileadi wrote:
 On 10/25/11 4:04 AM, Jacob Carlborg wrote:
 On 2011-10-24 22:08, Jonathan M Davis wrote:
 On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?

It's a long standing issue that when one unit test fails within a module, no more within that module are run (though fortunately, a while back it was fixed so that other modules' unit tests will still run). As I recall, there had to be a change to the compiler to fix it, but I don't known/remember the details. Certainly, the issue still stands. - Jonathan M Davis

A workaround is to catch AssertErrors, hook it up with some library code and you get a minimal unit test framework: https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTest er.d Example of usage: https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d

As an argument for continuing to run tests after one fails, I'm taking a TDD class and the instructor asserted that for unit tests you should generally only have one or two assertions per test method. His reasoning is that when something breaks you immediately know the extent of your breakage by counting the number of failed methods. This argument is pretty convincing to me.

The value of that would depend on what exactly the tests are doing - particularly if tests require some setup - but I could see how it would be valuable to essentially list all of the failed assertions rather than just failed unittest blocks. However, I don't think that I'd ever do it that way simply because the code clutter would be far too great. Even if your unit tests consist simply of a bunch of one-liners which are just assertions without any setup, having a unittest block per assertion is going to create a lot of extra code, which will seriously clutter the file. Now, you have the choice to do it either way, which is great, but I don't think that I'd want to be leaning towards one assertion per unittest block. I'd generally favor one unittest block per function, or at least group of related tests. - Jonathan M Davis
Oct 26 2011
prev sibling next sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Mon, 24 Oct 2011 22:08:50 +0200, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:

 On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?

It's a long standing issue that when one unit test fails within a module, no more within that module are run (though fortunately, a while back it was fixed so that other modules' unit tests will still run). As I recall, there had to be a change to the compiler to fix it, but I don't known/remember the details. Certainly, the issue still stands. - Jonathan M Davis

It's also annoying that the runtime quits with a non-zero exit code when you return false from your unit tester. Somehow the boolean return became a flag for fail/success rather than for program continuation. martin
Oct 24 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, October 27, 2011 08:54:12 Jacob Carlborg wrote:
 On 2011-10-26 21:45, Jonathan M Davis wrote:
 On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:
 Well, in my library, if an assert error is thrown in a block (passed
 to
 the "it" method), the whole block is canceled and it will continue
 with
 the next block. So it's up to the user how the asserts should be laid
 out.> 

assert failed - at least in the general case. Too often further assertions relied on the state assured by the one that failed, so further failures just confuse things and give you too much data to have to sift through. As it stands with the built-in unit testing faciliities, you can put each assertion in its own unittest block if you really want each assertion to run on its own (though until it's fixed so that further unittest blocks within the module run after the first failure in that module, it wouldn't do you any good). - Jonathan M Davis

My solution works now and if you want the whole unit test block to end when an assert error is thrown that is possible too. Just call the "it" method with a block once in a unit test block. You can never convince me that the other solution is better as long as it doesn't work.

The current solution works just fine if you want a unittest block to terminate after a single assertion failure (as it should IMHO). It's just the running of subsequent unittest blocks within a module which doesn't work. And while I'd very much like it to work, I don't personally find it to be all that much of a problem. - Jonathan M Davis
Oct 27 2011
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, October 27, 2011 08:54:12 Jacob Carlborg wrote:
 On 2011-10-26 21:45, Jonathan M Davis wrote:
 On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:
 Well, in my library, if an assert error is thrown in a block (passed
 to
 the "it" method), the whole block is canceled and it will continue
 with
 the next block. So it's up to the user how the asserts should be laid
 out.> 

assert failed - at least in the general case. Too often further assertions relied on the state assured by the one that failed, so further failures just confuse things and give you too much data to have to sift through. As it stands with the built-in unit testing faciliities, you can put each assertion in its own unittest block if you really want each assertion to run on its own (though until it's fixed so that further unittest blocks within the module run after the first failure in that module, it wouldn't do you any good). - Jonathan M Davis

My solution works now and if you want the whole unit test block to end when an assert error is thrown that is possible too. Just call the "it" method with a block once in a unit test block. You can never convince me that the other solution is better as long as it doesn't work.

The current solution works just fine if you want a unittest block to terminate after a single assertion failure (as it should IMHO). It's just the running of subsequent unittest blocks within a module which doesn't work. And while I'd very much like it to work, I don't personally find it to be all that much of a problem. - Jonathan M Davis
Oct 27 2011