www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - listing multiple unit test failures

reply Spacen Jasset <spacen yahoo.co.uk> writes:
	I have started some unit tests:

	unittest
	{
		// Check row column indexing is correct
		Matrix	m;
		m[2,0] = 59;
		assert(m.matrix_flat[2]==59, "opIndexAssign failed");
		assert(m.matrix_flat[8]!=59, "opIndexAssign failed");
		
	}

If an assert fails the other assertions in a program do not fire. This 
is inconvient since the "real bug" introduced may be caused by some 
other assertion that has not been evaluated yet.

What I am looking for, therefore is more of a writefln of each failure, 
together with a single assertion to abort the program if any test has 
failed.

If there isn't such a feature then I suppose I need a "Global" class 
which holds a bit flag to be set when an assertion is registered, and 
then in main this class could be made to assert if the flag is set.
Mar 11 2008
next sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Spacen Jasset wrote:
     I have started some unit tests:
 
     unittest
     {
         // Check row column indexing is correct
         Matrix    m;
         m[2,0] = 59;
         assert(m.matrix_flat[2]==59, "opIndexAssign failed");
         assert(m.matrix_flat[8]!=59, "opIndexAssign failed");
        
     }
 
 If an assert fails the other assertions in a program do not fire. This 
 is inconvient since the "real bug" introduced may be caused by some 
 other assertion that has not been evaluated yet.
 
 What I am looking for, therefore is more of a writefln of each failure, 
 together with a single assertion to abort the program if any test has 
 failed.
 
 If there isn't such a feature then I suppose I need a "Global" class 
 which holds a bit flag to be set when an assertion is registered, and 
 then in main this class could be made to assert if the flag is set.

You're looking for DUnit, which hasn't been written yet. I'd like to write such a thing, but a natural, xUnit-like syntax is difficult in D. If you want something that targets D2, that's doable; you could write something like: import dunit.api; class FooTests : TestFixture { mixin(DunitTest); override void setup () {} override void teardown () {} void testSomething() { // ... } } But in D1, there's no way (as far as I know) to iterate over the methods for a type. Well, no, there is the vtbl. But that is a REALLY ugly solution. So you're left with some syntax like: class FooTests : TestFixture { mixin(DunitTest); override void setup () {} override void teardown () {} this () { tests["test for something"] = { // blah } tests["test for something else"] = { // blah } } } I guess that isn't so ugly. I can hack something together tonight.
Mar 11 2008
parent Christopher Wright <dhasenan gmail.com> writes:
Christopher Wright wrote:
 I guess that isn't so ugly. I can hack something together tonight.

That was actually quite easy. I'll add it to Felt and post a link in a few minutes.
Mar 11 2008
prev sibling parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Spacen Jasset wrote:
     I have started some unit tests:
 
     unittest
     {
         // Check row column indexing is correct
         Matrix    m;
         m[2,0] = 59;
         assert(m.matrix_flat[2]==59, "opIndexAssign failed");
         assert(m.matrix_flat[8]!=59, "opIndexAssign failed");
        
     }
 
 If an assert fails the other assertions in a program do not fire. This 
 is inconvient since the "real bug" introduced may be caused by some 
 other assertion that has not been evaluated yet.
 
 What I am looking for, therefore is more of a writefln of each failure, 
 together with a single assertion to abort the program if any test has 
 failed.
 
 If there isn't such a feature then I suppose I need a "Global" class 
 which holds a bit flag to be set when an assertion is registered, and 
 then in main this class could be made to assert if the flag is set.

Not to knock on Mr. Wright's solution, which is probably excellent and more directly addressable to the problem at hand, but I'm working on a solution (designed to use IPC to be used from other applications, etc.) that does just this and a lot more. Check: http://www.dsource.org/projects/descent/browser/trunk/descent.unittest/flute/src/org/dsource/descent/unittests/flute.d And the docs at: http://svn.dsource.org/projects/descent/trunk/descent.unittest/flute/doc/flute.html
Mar 11 2008
next sibling parent Robert Fraser <fraserofthenight gmail.com> writes:
Robert Fraser wrote:
 Spacen Jasset wrote:
     I have started some unit tests:

     unittest
     {
         // Check row column indexing is correct
         Matrix    m;
         m[2,0] = 59;
         assert(m.matrix_flat[2]==59, "opIndexAssign failed");
         assert(m.matrix_flat[8]!=59, "opIndexAssign failed");
            }

 If an assert fails the other assertions in a program do not fire. This 
 is inconvient since the "real bug" introduced may be caused by some 
 other assertion that has not been evaluated yet.

 What I am looking for, therefore is more of a writefln of each 
 failure, together with a single assertion to abort the program if any 
 test has failed.

 If there isn't such a feature then I suppose I need a "Global" class 
 which holds a bit flag to be set when an assertion is registered, and 
 then in main this class could be made to assert if the flag is set.

Not to knock on Mr. Wright's solution, which is probably excellent and more directly addressable to the problem at hand, but I'm working on a solution (designed to use IPC to be used from other applications, etc.) that does just this and a lot more. Check: http://www.dsource.org/projects/descent/browser/trunk/descent.unittest/flute/src/org/dsource/desce t/unittests/flute.d And the docs at: http://svn.dsource.org/projects/descent/trunk/descent.unittest/ lute/doc/flute.html

I should mention flute is in-progress, so unless _really_ don't want to change your unittest { } blocks into DUnit tests, use DUnit instead.
Mar 11 2008
prev sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Robert Fraser wrote:
 Not to knock on Mr. Wright's solution, which is probably excellent and 
 more directly addressable to the problem at hand, but I'm working on a 
 solution (designed to use IPC to be used from other applications, etc.) 
 that does just this and a lot more. Check:
 http://www.dsource.org/projects/descent/browser/trunk/descent.unittest/flute/src/org/dsource/desce
t/unittests/flute.d 
 
 
 And the docs at:
 http://svn.dsource.org/projects/descent/trunk/descent.unittest/
lute/doc/flute.html 
 

Flute seems like a general solution that can easily be adapted to DUnit, and that support would increase the popularity of DUnit. But can I use Flute today? Flute plus standard D unittests will be a smaller commitment than DUnit, and will require somewhat less from programmers. But there's value in requiring organization and separation for your tests, I think. For one, it's easier to abstract out shared setup code, and be certain that that won't be included in your release builds. Your trick for naming unittests is also scary. It works, though, I'll give it that. Against all odds, and without having to run any of the tests.
Mar 12 2008
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Christopher Wright wrote:
 Robert Fraser wrote:
 Not to knock on Mr. Wright's solution, which is probably excellent and 
 more directly addressable to the problem at hand, but I'm working on a 
 solution (designed to use IPC to be used from other applications, 
 etc.) that does just this and a lot more. Check:
 http://www.dsource.org/projects/descent/browser/trunk/descent.unittest/flute/src/org/dsource/desce
t/unittests/flute.d 


 And the docs at:
 http://svn.dsource.org/projects/descent/trunk/descent.unittest/
lute/doc/flute.html 

Flute seems like a general solution that can easily be adapted to DUnit, and that support would increase the popularity of DUnit. But can I use Flute today?

Not unless you're on Windows/Phobos/D1 (I haven't gotten around to updating it for supporting other configs, since it's not officially being released until I get it integrated into Descent, which is a much larger project than Flute itself).
 Flute plus standard D unittests will be a smaller commitment than DUnit, 
 and will require somewhat less from programmers. But there's value in 
 requiring organization and separation for your tests, I think. For one, 
 it's easier to abstract out shared setup code, and be certain that that 
 won't be included in your release builds.

Strongly agreed. xUnit is a much better solution for large-scale development, but sticking mini unit tests in your code is hlpful during development phases.
 Your trick for naming unittests is also scary. It works, though, I'll 
 give it that. Against all odds, and without having to run any of the tests.

Indeed, indeed. It would be very helpful if D allowed unittest(name) { } syntax. Flute is mainly designed to be integrated into frontends, in particular Descent and CruiseControl. It would be brilliant if I could get DUnit tests in there, too, sicne as D moves towards large enterprise-class solutions, stronger unit testing support is certainly needed. Would it be okay if I added DUnit support alongside Flute support to Descent and to my Ant/CruiseControl plugin when I get around to doing that?
Mar 12 2008
parent reply Christopher Wright <dhasenan gmail.com> writes:
== Quote from Robert Fraser (fraserofthenight gmail.com)'s article
 Flute is mainly designed to be integrated into frontends, in particular
 Descent and CruiseControl. It would be brilliant if I could get DUnit
 tests in there, too, sicne as D moves towards large enterprise-class
 solutions, stronger unit testing support is certainly needed. Would it
 be okay if I added DUnit support alongside Flute support to Descent and
 to my Ant/CruiseControl plugin when I get around to doing that?

Have you started such a plugin? If so, I'd love to help out. If not, any suggestions on where to get started? I'd be quite thrilled to have DUnit support in Descent, as well. Let me know if there's anything I can do to make this simpler.
Mar 12 2008
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Christopher Wright wrote:
 Have you started such a plugin? If so, I'd love to help out. If not, any
 suggestions on where to get started?

Nope, it's on my *long* TODO list. I'm not even sure how to go about making it a CC plugin, I'd have to check how the JUnit plugin works. It might end up being an Ant task with some sort of listener/presentation component in CC.
Mar 13 2008
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Robert Fraser wrote:
 Christopher Wright wrote:
 Have you started such a plugin? If so, I'd love to help out. If not, any
 suggestions on where to get started?

Nope, it's on my *long* TODO list. I'm not even sure how to go about making it a CC plugin, I'd have to check how the JUnit plugin works. It might end up being an Ant task with some sort of listener/presentation component in CC.

I took a look into this & it appears that the best solution is to create a tool that outputs unit test result XML in the same format as JUnit. CC can just pick that up & display it as JUnit tests. For full CC integration, adding another builder (such as a DSSS builder) would be a good idea, though. That said, it would be a cool idea to have a unified, configurable D "testing framework" that could run both DUnit tests and built-in unittest { } blocks with a plugable reporter interface for output (i.e. a JUnit-compatible XML for CC plugability, a pretty Web 2.0-ish page for Zen staring sessions, etc.)
Mar 13 2008
parent Christopher Wright <dhasenan gmail.com> writes:
Robert Fraser wrote:
 Robert Fraser wrote:
 Christopher Wright wrote:
 Have you started such a plugin? If so, I'd love to help out. If not, any
 suggestions on where to get started?

Nope, it's on my *long* TODO list. I'm not even sure how to go about making it a CC plugin, I'd have to check how the JUnit plugin works. It might end up being an Ant task with some sort of listener/presentation component in CC.

I took a look into this & it appears that the best solution is to create a tool that outputs unit test result XML in the same format as JUnit. CC can just pick that up & display it as JUnit tests. For full CC integration, adding another builder (such as a DSSS builder) would be a good idea, though.

I came to the same conclusion, though I didn't find an adequate description of the JUnit schema. I'll have to snag some output instead, I guess.
 That said, it would be a cool idea to have a unified, configurable D 
 "testing framework" that could run both DUnit tests and built-in 
 unittest { } blocks with a plugable reporter interface for output (i.e. 
 a JUnit-compatible XML for CC plugability, a pretty Web 2.0-ish page for 
 Zen staring sessions, etc.)

Yep, but that's too much work for me to do.
Mar 13 2008