www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Unit testing with asserts: Why is assertHandler required to throw?

reply Trip Volpe <mraccident gmail.com> writes:
I recently began porting an existing C++ project of mine (a
compiler/interpreter for a dynamic language) to D. In the process I found that
the built-in unit testing support, while an awesome concept, was a little bit
sparse. In particular, assert() is fairly useless for unit tests, since it
throws on a failure and prevents all subsequent tests from running.

Unfortunately, it's the only facility that seems to give you access to the
current file and line number, very important bits of info for unit tests in a
any non-trivial project. I was contemplating just abandoning D and sticking
with C++ and Googletest, but after doing a bit of digging through the source, I
found setAssertHandler() in core.exception. Perfect! I could use my own assert
handler that just records the error and allows future asserts to still be
reached. So I wrote it up and gave it a try, but pretty quickly started getting
access violation errors.

I did a bit of searching, and found a bug report on the issue:

http://d.puremagic.com/issues/show_bug.cgi?id=3208

In the comments it's indicated that this is intentional, that the compiler
expects the assertion handler to throw. So I guess I have a few questions:

1. Why is this the expected behavior? It seems to me that there are relatively
few useful things you can do with a custom assert handler unless it is possible
to refrain from throwing.

2. Is this going to be fixed any time soon? It seems fairly important to me; it
really kills the value of built-in unit tests to be forced to choose between
not having any line number information and being able to recognize only one
assert failure per test run(!!!).

3. Are there any current workarounds for the problem? I could just use my own
assertion function, but 1) I can't call it "assert" because that's a reserved
word, and 2) most importantly, it won't have any way of indicating in its
diagnostic output which source line the failure occurred on. In C++, of course,
I could use preprocessor macros to do that, but D omits a preprocessor (for
very good reasons). Unfortunately, this seems like a case where D hasn't
provided replacement functionality.

But maybe there's a sneaky fix I haven't thought of? I also found
Runtime.moduleUnitTester() in core.runtime, which is very useful in itself, but
provides an _almost_ useful improvement: I can catch the AssertErrors on a
module-by-module basis, which would at least allow the unit tests for all
modules to be run regardless of a failure in one of them. However, a single
module could easily have dozens of tests in it, each of which could contain
many individual asserts. Stopping the test early because one assert failed
makes no sense.

It would be _almost_ acceptable to at least be sure of running all the tests in
each module; is there any way to poke inside or otherwise override ModuleInfo's
unitTest() function?


Thanks in advance for any advice or comments! D is a completely awesome
language, but I found this issue a bit strange.
Jan 31 2010
next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Trip Volpe" <mraccident gmail.com> wrote in message 
news:hk4pql$1ehf$1 digitalmars.com...
I recently began porting an existing C++ project of mine (a 
compiler/interpreter for a dynamic language) to D. In the process I found 
that the built-in unit testing support, while an awesome concept, was a 
little bit sparse. In particular, assert() is fairly useless for unit 
tests, since it throws on a failure and prevents all subsequent tests from 
running.

 Unfortunately, it's the only facility that seems to give you access to the 
 current file and line number, very important bits of info for unit tests 
 in a any non-trivial project. I was contemplating just abandoning D and 
 sticking with C++ and Googletest, but after doing a bit of digging through 
 the source, I found setAssertHandler() in core.exception. Perfect! I could 
 use my own assert handler that just records the error and allows future 
 asserts to still be reached. So I wrote it up and gave it a try, but 
 pretty quickly started getting access violation errors.

 I did a bit of searching, and found a bug report on the issue:

 http://d.puremagic.com/issues/show_bug.cgi?id=3208

 In the comments it's indicated that this is intentional, that the compiler 
 expects the assertion handler to throw. So I guess I have a few questions:

 1. Why is this the expected behavior? It seems to me that there are 
 relatively few useful things you can do with a custom assert handler 
 unless it is possible to refrain from throwing.

 2. Is this going to be fixed any time soon? It seems fairly important to 
 me; it really kills the value of built-in unit tests to be forced to 
 choose between not having any line number information and being able to 
 recognize only one assert failure per test run(!!!).

 3. Are there any current workarounds for the problem? I could just use my 
 own assertion function, but 1) I can't call it "assert" because that's a 
 reserved word, and 2) most importantly, it won't have any way of 
 indicating in its diagnostic output which source line the failure occurred 
 on. In C++, of course, I could use preprocessor macros to do that, but D 
 omits a preprocessor (for very good reasons). Unfortunately, this seems 
 like a case where D hasn't provided replacement functionality.

 But maybe there's a sneaky fix I haven't thought of? I also found 
 Runtime.moduleUnitTester() in core.runtime, which is very useful in 
 itself, but provides an _almost_ useful improvement: I can catch the 
 AssertErrors on a module-by-module basis, which would at least allow the 
 unit tests for all modules to be run regardless of a failure in one of 
 them. However, a single module could easily have dozens of tests in it, 
 each of which could contain many individual asserts. Stopping the test 
 early because one assert failed makes no sense.

 It would be _almost_ acceptable to at least be sure of running all the 
 tests in each module; is there any way to poke inside or otherwise 
 override ModuleInfo's unitTest() function?


 Thanks in advance for any advice or comments! D is a completely awesome 
 language, but I found this issue a bit strange.

The deferAssert module (possible name change and other API improvements pending) of my SemiTwist D Tools library ( http://www.dsource.org/projects/semitwist ) is designed to get around those problems. Here's a sample app with it's own output in the comments at the bottom: http://www.dsource.org/projects/semitwist/browser/trunk/src/semitwist/apps/tests/deferAssertTest/main.d And the closest thing I have to an API reference at this point: http://www.dsource.org/projects/semitwist/browser/trunk/src/semitwist/util/deferAssert.d (BTW, contrary to the names, deferAssert and deferEnsure don't actually defer anything other than throwing the fatal assert exception)
Jan 31 2010
parent reply Trip Volpe <mraccident gmail.com> writes:
Nick Sabalausky Wrote: 
 The deferAssert module (possible name change and other API improvements 
 pending) of my SemiTwist D Tools library ( 
 http://www.dsource.org/projects/semitwist ) is designed to get around those 
 problems.

Good to know I'm not the only one who's had to working around this. :-P Considering how D has unit testing and contract programming built-in already, I'm still a bit surprised it doesn't have assertions and expectations that are actually useful for people doing heavy unit testing. deferAssert is interesting, but it seems to require a lot of boilerplate. The combination of mixin and template syntax is a bit kludgy, heh.
Jan 31 2010
parent "Nick Sabalausky" <a a.a> writes:
"Trip Volpe" <mraccident gmail.com> wrote in message 
news:hk583n$28eb$1 digitalmars.com...
 Nick Sabalausky Wrote:
 The deferAssert module (possible name change and other API improvements
 pending) of my SemiTwist D Tools library (
 http://www.dsource.org/projects/semitwist ) is designed to get around 
 those
 problems.

Good to know I'm not the only one who's had to working around this. :-P Considering how D has unit testing and contract programming built-in already, I'm still a bit surprised it doesn't have assertions and expectations that are actually useful for people doing heavy unit testing. deferAssert is interesting, but it seems to require a lot of boilerplate. The combination of mixin and template syntax is a bit kludgy, heh.

Yea, that's one of the reasons I like to push for a better string mixin syntax whenever the topic comes up. But it's the best I've been able to do in D1 without resorting to JUnit-style garbage like "expectTrue(a.logicalOr(b).logicalAnd(c.greaterThan(d)))" or whatever the actual JUnit gibberish is (and I don't even have to abuse operator overloading either). Though, it's still a lot better than what I'd be able to do in most languages.
Jan 31 2010
prev sibling next sibling parent reply Lutger <lutger.blijdestijn gmail.com> writes:
On 01/31/2010 09:39 PM, Trip Volpe wrote:
 I recently began porting an existing C++ project of mine (a
compiler/interpreter for a dynamic language) to D. In the process I found that
the built-in unit testing support, while an awesome concept, was a little bit
sparse. In particular, assert() is fairly useless for unit tests, since it
throws on a failure and prevents all subsequent tests from running.

 Unfortunately, it's the only facility that seems to give you access to the
current file and line number, very important bits of info for unit tests in a
any non-trivial project. I was contemplating just abandoning D and sticking
with C++ and Googletest, but after doing a bit of digging through the source, I
found setAssertHandler() in core.exception. Perfect! I could use my own assert
handler that just records the error and allows future asserts to still be
reached. So I wrote it up and gave it a try, but pretty quickly started getting
access violation errors.

 I did a bit of searching, and found a bug report on the issue:

 http://d.puremagic.com/issues/show_bug.cgi?id=3208

 In the comments it's indicated that this is intentional, that the compiler
expects the assertion handler to throw. So I guess I have a few questions:

 1. Why is this the expected behavior? It seems to me that there are relatively
few useful things you can do with a custom assert handler unless it is possible
to refrain from throwing.

 2. Is this going to be fixed any time soon? It seems fairly important to me;
it really kills the value of built-in unit tests to be forced to choose between
not having any line number information and being able to recognize only one
assert failure per test run(!!!).

 3. Are there any current workarounds for the problem? I could just use my own
assertion function, but 1) I can't call it "assert" because that's a reserved
word, and 2) most importantly, it won't have any way of indicating in its
diagnostic output which source line the failure occurred on. In C++, of course,
I could use preprocessor macros to do that, but D omits a preprocessor (for
very good reasons). Unfortunately, this seems like a case where D hasn't
provided replacement functionality.

 But maybe there's a sneaky fix I haven't thought of? I also found
Runtime.moduleUnitTester() in core.runtime, which is very useful in itself, but
provides an _almost_ useful improvement: I can catch the AssertErrors on a
module-by-module basis, which would at least allow the unit tests for all
modules to be run regardless of a failure in one of them. However, a single
module could easily have dozens of tests in it, each of which could contain
many individual asserts. Stopping the test early because one assert failed
makes no sense.

 It would be _almost_ acceptable to at least be sure of running all the tests
in each module; is there any way to poke inside or otherwise override
ModuleInfo's unitTest() function?


 Thanks in advance for any advice or comments! D is a completely awesome
language, but I found this issue a bit strange.

You can use line and file info with default arguments, this works (special case). I just hack around the default unittest system, something like this: void test(string testName)(void delegate () testClosure, int line = __LINE__, string file = __FILE__) { // register test start, run testClosure, end test } void expectEquals(A, B)(A a, B b, int line = __LINE__, string file = __FILE__) { // record outcome of assertion } unittest { test!"truth of it"( { expectEquals(true, true); } ); } or slightly abusive: unittest { test!"truth of it" = { expectEquals(true, true); }; }
Jan 31 2010
next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Lutger" <lutger.blijdestijn gmail.com> wrote in message 
news:hk4r35$1hpg$1 digitalmars.com...
 You can use line and file info with default arguments, this works (special 
 case).

That must be a D2 thing, because I know it doesn't work in D1.
Jan 31 2010
parent Lutger <lutger.blijdestijn gmail.com> writes:
On 01/31/2010 10:04 PM, Nick Sabalausky wrote:
 "Lutger"<lutger.blijdestijn gmail.com>  wrote in message
 news:hk4r35$1hpg$1 digitalmars.com...
 You can use line and file info with default arguments, this works (special
 case).

That must be a D2 thing, because I know it doesn't work in D1.

Indeed, here is the change: http://www.digitalmars.com/d/2.0/changelog.html#new2_013
Jan 31 2010
prev sibling parent reply Trip Volpe <mraccident gmail.com> writes:
Lutger Wrote:
 
 You can use line and file info with default arguments, this works 
 (special case). I just hack around the default unittest system, 
 something like this:
 
 void test(string testName)(void delegate () testClosure,
                             int line = __LINE__,
                             string file = __FILE__)
 {
      // register test start, run testClosure, end test
 }
 
 void expectEquals(A, B)(A a, B b,
                          int line = __LINE__,
                          string file = __FILE__)
 {
      // record outcome of assertion
 }

Ah ha! I din't know that D had __LINE__ and __FILE__. I thought those had been done away with along with the preprocessor. It looks like they're a lot smarter than in C as well, since this works just fine: test.d: 1 void main() { 2 printlocation(); // prints "test.d:2" (as opposed to "test.d:5") 3 } 4 5 void printlocation( string file = __FILE__, int line = __LINE__ ) { 6 writefln( "%s:%d", file, line ); 7 } This still feels a little more hacky than I'd like, but it should do. Thanks! :-)
Jan 31 2010
parent bearophile <bearophileHUGS lycos.com> writes:
Trip Volpe:
 Ah ha! I din't know that D had __LINE__ and __FILE__. I thought those had been
done away with along with the preprocessor. It looks like they're a lot smarter
than in C as well, since this works just fine:
 
 test.d:
 1   void main() {
 2      printlocation();    // prints "test.d:2" (as opposed to "test.d:5")
 3   }
 4   
 5   void printlocation( string file = __FILE__, int line = __LINE__ ) {
 6      writefln( "%s:%d", file, line );
 7   }
 
 This still feels a little more hacky than I'd like, but it should do. Thanks!
:-)

It looks like the bad special cased magic you can find in Perl :-) Bye, bearophile
Jan 31 2010
prev sibling parent reply =?UTF-8?B?UGVsbGUgTcOlbnNzb24=?= <pelle.mansson gmail.com> writes:
On 01/31/2010 09:39 PM, Trip Volpe wrote:
 I recently began porting an existing C++ project of mine (a
compiler/interpreter for a dynamic language) to D. In the process I found that
the built-in unit testing support, while an awesome concept, was a little bit
sparse. In particular, assert() is fairly useless for unit tests, since it
throws on a failure and prevents all subsequent tests from running.

 Unfortunately, it's the only facility that seems to give you access to the
current file and line number, very important bits of info for unit tests in a
any non-trivial project. I was contemplating just abandoning D and sticking
with C++ and Googletest, but after doing a bit of digging through the source, I
found setAssertHandler() in core.exception. Perfect! I could use my own assert
handler that just records the error and allows future asserts to still be
reached. So I wrote it up and gave it a try, but pretty quickly started getting
access violation errors.

 I did a bit of searching, and found a bug report on the issue:

 http://d.puremagic.com/issues/show_bug.cgi?id=3208

 In the comments it's indicated that this is intentional, that the compiler
expects the assertion handler to throw. So I guess I have a few questions:

 1. Why is this the expected behavior? It seems to me that there are relatively
few useful things you can do with a custom assert handler unless it is possible
to refrain from throwing.

 2. Is this going to be fixed any time soon? It seems fairly important to me;
it really kills the value of built-in unit tests to be forced to choose between
not having any line number information and being able to recognize only one
assert failure per test run(!!!).

 3. Are there any current workarounds for the problem? I could just use my own
assertion function, but 1) I can't call it "assert" because that's a reserved
word, and 2) most importantly, it won't have any way of indicating in its
diagnostic output which source line the failure occurred on. In C++, of course,
I could use preprocessor macros to do that, but D omits a preprocessor (for
very good reasons). Unfortunately, this seems like a case where D hasn't
provided replacement functionality.

 But maybe there's a sneaky fix I haven't thought of? I also found
Runtime.moduleUnitTester() in core.runtime, which is very useful in itself, but
provides an _almost_ useful improvement: I can catch the AssertErrors on a
module-by-module basis, which would at least allow the unit tests for all
modules to be run regardless of a failure in one of them. However, a single
module could easily have dozens of tests in it, each of which could contain
many individual asserts. Stopping the test early because one assert failed
makes no sense.

 It would be _almost_ acceptable to at least be sure of running all the tests
in each module; is there any way to poke inside or otherwise override
ModuleInfo's unitTest() function?


 Thanks in advance for any advice or comments! D is a completely awesome
language, but I found this issue a bit strange.

I think the reason for this is that assert is used by the compiler for some things, like assert(0) at the end of functions. An assertion handler should be created for use in unittests. Preferably with colorized output. :) I stole and changed slightly from std.contracts.enforce (untested) T test(T, string file=__FILE__, int line=__LINE__) (T value, string message="Test failed") { if (!value) writeln(file, "(", line, "): ", message); return value; } Use just like an assert, but will just write on errors.
Jan 31 2010
parent Trip Volpe <mraccident gmail.com> writes:
Pelle MÃ¥nsson Wrote: 
 
 An assertion handler should be created for use in unittests. Preferably 
 with colorized output. :)
 
 I stole and changed slightly from std.contracts.enforce (untested)
 
 T test(T, string file=__FILE__, int line=__LINE__)
              (T value, string message="Test failed") {
      if (!value) writeln(file, "(", line, "): ", message);
      return value;
 }
 
 Use just like an assert, but will just write on errors.

Heh, yeah, if I had just noticed std.contracts in the first place, I would have realized that I still had __FILE__ and __LINE__ to play around with. Ah well -- D's usual thoughtfulness pulls through again! And I'm a sucker for bright colors too, so I'm building colorized output into my set of unit test augmentations. :-)
Jan 31 2010