www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Helper unit testing functions in Phobos (possible std.unittests)

reply Jonathan M Davis <jmdavisProg gmx.com> writes:
I'm proposing a possible new module for phobos which would be called 
std.unittests. The code and DDoc file can be found here: http://is.gd/gLH9Q

Or you can look at the code here directly (though it has poor highlighting): 
http://ideone.com/EOlod

The module contains a set of helper functions for unit tests which are an 
improvement over simply using assert. I have them to be very helpful in the
unit 
tests in my proposal for std.datetime. My unit tests are more concise, easier
to 
read, and much easier to debug because of them. assertEqual() and 
assertExcThrown() have been particularly useful.

assert is a great tool, but simply telling you that an assertion failed is not 
as useful as telling you _how_ it failed, and a function like assertExcThrown() 
(which, as you may guess, asserts that a particular exception was thrown by a 
particular function call) really reduces the boiler plate code that you have to 
write for testing what it tests. So, I find such helper functions are extremely 
useful.

Because I have used them so extensively in std.datetime, they are pretty much 
either going to have to be added as a module to Phobos or added as private 
helper functions in std.datetime. Personally,  I think that they (or something 
very similar to them) merit being added as a new module in Phobos, which can be 
expanded upon as others come up with particularly useful unit testing functions.

So, I'm requesting that the folks here on the list have a look at my module and 
review it for possible inclusion in Phobos. I don't know what Andrei's position 
on them is (he rightly focused his review of my datetime code on the date/time 
functionality and API rather than its unit tests and didn't mention my unit 
testing functions at all), but I think that such a module would be of great 
worth and would like other's opinions on it.

- Jonathan M Davis
Nov 05 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"Jonathan M Davis" <jmdavisProg gmx.com> wrote in message 
news:mailman.125.1289018234.21107.digitalmars-d puremagic.com...
 I'm proposing a possible new module for phobos which would be called
 std.unittests. The code and DDoc file can be found here: 
 http://is.gd/gLH9Q

 Or you can look at the code here directly (though it has poor 
 highlighting):
 http://ideone.com/EOlod

 The module contains a set of helper functions for unit tests which are an
 improvement over simply using assert. I have them to be very helpful in 
 the unit
 tests in my proposal for std.datetime. My unit tests are more concise, 
 easier to
 read, and much easier to debug because of them. assertEqual() and
 assertExcThrown() have been particularly useful.

 assert is a great tool, but simply telling you that an assertion failed is 
 not
 as useful as telling you _how_ it failed, and a function like 
 assertExcThrown()
 (which, as you may guess, asserts that a particular exception was thrown 
 by a
 particular function call) really reduces the boiler plate code that you 
 have to
 write for testing what it tests. So, I find such helper functions are 
 extremely
 useful.

 Because I have used them so extensively in std.datetime, they are pretty 
 much
 either going to have to be added as a module to Phobos or added as private
 helper functions in std.datetime. Personally,  I think that they (or 
 something
 very similar to them) merit being added as a new module in Phobos, which 
 can be
 expanded upon as others come up with particularly useful unit testing 
 functions.

 So, I'm requesting that the folks here on the list have a look at my 
 module and
 review it for possible inclusion in Phobos. I don't know what Andrei's 
 position
 on them is (he rightly focused his review of my datetime code on the 
 date/time
 functionality and API rather than its unit tests and didn't mention my 
 unit
 testing functions at all), but I think that such a module would be of 
 great
 worth and would like other's opinions on it.

I made a very similar thing in my SemiTwist D Tools library, so I can definitely attest to D's need for all of this. My comments on this one, based on a breif skim of the doc: I would try to change it so this: assertOpCmp!("<")(myfunc(), 7, "My test failed!"); Can be written more like these: assert!("_ < 7", "myfunc()", "My test failed!")(); assert!("_1 < _2", "myfunc()", "7", "My test failed!")(); assert!("_1 < _2", "myfunc()", "somethingElse()", "My test failed!")(); Aside from being more flexible, this way when it fails it can report much more relevent information that can actually compete with JUnit/NUnit. For instance, even with the custom message omitted, it could still give info like: Failed: 'myfunc() < 7' 'myfunc()' was '13' It may require it be used like 'mixin(assert!(blah blah))', but that's a big part of why I've been saying for a loooong time that it needs to be possible to make CTFE's intended for string mixins to be mixed in *without* the "mixin(...)" mess. Because as it is, if you present an interface that requires it be wrapped in "mixin(...)", people will bitch about the extra noise and refuse to use it, rendering string mixins minimally useful. (Last time I suggested this we got the ability to make *template* mixins that are usable without "mixin", which is caparitively useless.) Also, I'd say if file/line info can't be inferred automatically because of varargs, then the interface needs to lose the vararg ability. It'd be nice to have, but not needing to provide line/file info manually is much nicer. Although, we could probably have both by using the suggestion above (ie, make the whole thing generate a string to be mixed-in). Do the 'assertEqual', 'assertOpCmp', etc, all report if an exception was thrown (and if so, what exception)? If not, they should.
Nov 05 2010
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday 05 November 2010 22:54:20 Nick Sabalausky wrote:
 "Jonathan M Davis" <jmdavisProg gmx.com> wrote in message
 news:mailman.125.1289018234.21107.digitalmars-d puremagic.com...
 
 I'm proposing a possible new module for phobos which would be called
 std.unittests. The code and DDoc file can be found here:
 http://is.gd/gLH9Q
 
 Or you can look at the code here directly (though it has poor
 highlighting):
 http://ideone.com/EOlod
 
 The module contains a set of helper functions for unit tests which are an
 improvement over simply using assert. I have them to be very helpful in
 the unit
 tests in my proposal for std.datetime. My unit tests are more concise,
 easier to
 read, and much easier to debug because of them. assertEqual() and
 assertExcThrown() have been particularly useful.
 
 assert is a great tool, but simply telling you that an assertion failed
 is not
 as useful as telling you _how_ it failed, and a function like
 assertExcThrown()
 (which, as you may guess, asserts that a particular exception was thrown
 by a
 particular function call) really reduces the boiler plate code that you
 have to
 write for testing what it tests. So, I find such helper functions are
 extremely
 useful.
 
 Because I have used them so extensively in std.datetime, they are pretty
 much
 either going to have to be added as a module to Phobos or added as
 private helper functions in std.datetime. Personally,  I think that they
 (or something
 very similar to them) merit being added as a new module in Phobos, which
 can be
 expanded upon as others come up with particularly useful unit testing
 functions.
 
 So, I'm requesting that the folks here on the list have a look at my
 module and
 review it for possible inclusion in Phobos. I don't know what Andrei's
 position
 on them is (he rightly focused his review of my datetime code on the
 date/time
 functionality and API rather than its unit tests and didn't mention my
 unit
 testing functions at all), but I think that such a module would be of
 great
 worth and would like other's opinions on it.

I made a very similar thing in my SemiTwist D Tools library, so I can definitely attest to D's need for all of this. My comments on this one, based on a breif skim of the doc: I would try to change it so this: assertOpCmp!("<")(myfunc(), 7, "My test failed!"); Can be written more like these: assert!("_ < 7", "myfunc()", "My test failed!")(); assert!("_1 < _2", "myfunc()", "7", "My test failed!")(); assert!("_1 < _2", "myfunc()", "somethingElse()", "My test failed!")(); Aside from being more flexible, this way when it fails it can report much more relevent information that can actually compete with JUnit/NUnit. For instance, even with the custom message omitted, it could still give info like: Failed: 'myfunc() < 7' 'myfunc()' was '13'

I don't really see the benefit of this. As it is, assertOpComp!() clearly tells you what the actual result was and what the values were. It also reports the file and line number of the failed test. What more do you need? Seeing the exact expressions that resulted in the values to be tested doesn't strike me as necessary. You have the file and line number. You know what the exact result was. You can look at that exact line to see what produced the values that were tested. It seems to me like you have all of the information you need.
 Also, I'd say if file/line info can't be inferred automatically because of
 varargs, then the interface needs to lose the vararg ability. It'd be nice
 to have, but not needing to provide line/file info manually is much nicer.
 Although, we could probably have both by using the suggestion above (ie,
 make the whole thing generate a string to be mixed-in).

We need the ability to call arbitrary functions with some of the unit test functions (the ones where you have to pass LineInfo()). I don't see how you could do that with a string mixin unless you made the test itself a mixin instead of a function call, which strikes me as quite unpleasant. I'm certainly up for suggestions, but what I have is the best solution that I've seen. It is rather annoying to have to pass LineInfo() as the first argument, but it's straightforward. It works. And it's not like you can forget to do it, since it won't compile if you do.
 Do the 'assertEqual', 'assertOpCmp', etc, all report if an exception was
 thrown (and if so, what exception)? If not, they should.

They throw just like assert would. Even assertExcThrown() throws the exception if it's not the type of exception that you were testing for. I see no benefit in having them catch stray exceptions. That's not what they're testing for, and I believe that just like with assert, if a unit test function fails, that should be the end of that unit test. The exception gets reported just as it would be with an assert that threw an exception. So, it seems to me to be the correct behavior. The tests test for exactly what you tell them to and let other exceptions through to be reported by the runtime. - Jonathan M Davis
Nov 06 2010
parent "Nick Sabalausky" <a a.a> writes:
"Jonathan M Davis" <jmdavisProg gmx.com> wrote in message 
news:mailman.129.1289031502.21107.digitalmars-d puremagic.com...
 On Friday 05 November 2010 22:54:20 Nick Sabalausky wrote:
 I made a very similar thing in my SemiTwist D Tools library, so I can
 definitely attest to D's need for all of this.

 My comments on this one, based on a breif skim of the doc:

 I would try to change it so this:

 assertOpCmp!("<")(myfunc(), 7, "My test failed!");

 Can be written more like these:

 assert!("_ < 7", "myfunc()", "My test failed!")();
 assert!("_1 < _2", "myfunc()", "7", "My test failed!")();
 assert!("_1 < _2", "myfunc()", "somethingElse()", "My test failed!")();

 Aside from being more flexible, this way when it fails it can report much
 more relevent information that can actually compete with JUnit/NUnit. For
 instance, even with the custom message omitted, it could still give info
 like:

 Failed: 'myfunc() < 7'
 'myfunc()' was '13'

I don't really see the benefit of this. As it is, assertOpComp!() clearly tells you what the actual result was and what the values were.

I see, I didn't know that.
 Also, I'd say if file/line info can't be inferred automatically because 
 of
 varargs, then the interface needs to lose the vararg ability. It'd be 
 nice
 to have, but not needing to provide line/file info manually is much 
 nicer.
 Although, we could probably have both by using the suggestion above (ie,
 make the whole thing generate a string to be mixed-in).

We need the ability to call arbitrary functions with some of the unit test functions (the ones where you have to pass LineInfo()). I don't see how you could do that with a string mixin unless you made the test itself a mixin instead of a function call, which strikes me as quite unpleasant.

No more unpleasant than LineInfo(), and it also opens the door for other improvements.
 Do the 'assertEqual', 'assertOpCmp', etc, all report if an exception was
 thrown (and if so, what exception)? If not, they should.

They throw just like assert would. Even assertExcThrown() throws the exception if it's not the type of exception that you were testing for. I see no benefit in having them catch stray exceptions. That's not what they're testing for, and I believe that just like with assert, if a unit test function fails, that should be the end of that unit test.

STRONGLY disagree here. (Mentioned more about it in a different branch of this thread.)
Nov 06 2010
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2010-11-06 05:37, Jonathan M Davis wrote:
 I'm proposing a possible new module for phobos which would be called
 std.unittests. The code and DDoc file can be found here: http://is.gd/gLH9Q

 Or you can look at the code here directly (though it has poor highlighting):
 http://ideone.com/EOlod

 The module contains a set of helper functions for unit tests which are an
 improvement over simply using assert. I have them to be very helpful in the
unit
 tests in my proposal for std.datetime. My unit tests are more concise, easier
to
 read, and much easier to debug because of them. assertEqual() and
 assertExcThrown() have been particularly useful.

 assert is a great tool, but simply telling you that an assertion failed is not
 as useful as telling you _how_ it failed, and a function like assertExcThrown()
 (which, as you may guess, asserts that a particular exception was thrown by a
 particular function call) really reduces the boiler plate code that you have to
 write for testing what it tests. So, I find such helper functions are extremely
 useful.

 Because I have used them so extensively in std.datetime, they are pretty much
 either going to have to be added as a module to Phobos or added as private
 helper functions in std.datetime. Personally,  I think that they (or something
 very similar to them) merit being added as a new module in Phobos, which can be
 expanded upon as others come up with particularly useful unit testing
functions.

 So, I'm requesting that the folks here on the list have a look at my module and
 review it for possible inclusion in Phobos. I don't know what Andrei's position
 on them is (he rightly focused his review of my datetime code on the date/time
 functionality and API rather than its unit tests and didn't mention my unit
 testing functions at all), but I think that such a module would be of great
 worth and would like other's opinions on it.

 - Jonathan M Davis

I think that the problems that this module tries to fix should instead properly be fixed in the compiler. An assert error should be able to show what expression failed and one failing unit test in a module should now stop the renaming unit tests. The failing unit test would of course stop running but not the renaming ones. An idea is to have a function that takes a delegate and wrap the whole test in the delegate, like this: unittest { test("possibly a message here as well", { assert(/*some condition*/); }); } "test" would catch all AssertErrors, collect them and then at the end print out the result of all the tests. If one assert throws in a delegate the whole test (delegate) will stop running but the renaming tests in the unit test will continue to run. Since the assert error will contain the file and line number one could also read in the file and print out the failing line and possibly a couple of surrounding lines. Of course it would want the compiler to do this automatically instead of inventing a library solution. -- /Jacob Carlborg
Nov 06 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Sat, 06 Nov 2010 01:54:20 -0400, Nick Sabalausky wrote:

 It may require it be used like 'mixin(assert!(blah blah))', but that's a
 big part of why I've been saying for a loooong time that it needs to be
 possible to make CTFE's intended for string mixins to be mixed in
 *without* the "mixin(...)" mess. Because as it is, if you present an
 interface that requires it be wrapped in "mixin(...)", people will bitch
 about the extra noise and refuse to use it, rendering string mixins
 minimally useful. (Last time I suggested this we got the ability to make
 *template* mixins that are usable without "mixin", which is caparitively
 useless.)

I thought the keyword 'macro' was reserved for these purposes. Just wait for D3.. It's possible then.
Nov 06 2010
prev sibling next sibling parent "Masahiro Nakagawa" <repeatedly gmail.com> writes:
On Sat, 06 Nov 2010 13:37:02 +0900, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:

 I'm proposing a possible new module for phobos which would be called
 std.unittests. The code and DDoc file can be found here:  
 http://is.gd/gLH9Q

 Or you can look at the code here directly (though it has poor  
 highlighting):
 http://ideone.com/EOlod

 The module contains a set of helper functions for unit tests which are an
 improvement over simply using assert. I have them to be very helpful in  
 the unit
 tests in my proposal for std.datetime. My unit tests are more concise,  
 easier to
 read, and much easier to debug because of them. assertEqual() and
 assertExcThrown() have been particularly useful.

 assert is a great tool, but simply telling you that an assertion failed  
 is not
 as useful as telling you _how_ it failed, and a function like  
 assertExcThrown()
 (which, as you may guess, asserts that a particular exception was thrown  
 by a
 particular function call) really reduces the boiler plate code that you  
 have to
 write for testing what it tests. So, I find such helper functions are  
 extremely
 useful.

 Because I have used them so extensively in std.datetime, they are pretty  
 much
 either going to have to be added as a module to Phobos or added as  
 private
 helper functions in std.datetime. Personally,  I think that they (or  
 something
 very similar to them) merit being added as a new module in Phobos, which  
 can be
 expanded upon as others come up with particularly useful unit testing  
 functions.

 So, I'm requesting that the folks here on the list have a look at my  
 module and
 review it for possible inclusion in Phobos. I don't know what Andrei's  
 position
 on them is (he rightly focused his review of my datetime code on the  
 date/time
 functionality and API rather than its unit tests and didn't mention my  
 unit
 testing functions at all), but I think that such a module would be of  
 great
 worth and would like other's opinions on it.

 - Jonathan M Davis

I don't know this module's usability, but is getExcMsg useful? I can't imagine useful situation. In addition, I grepped your datetime.d and getExcMsg not found :( If possible, I want the mock, stub and related features. Masahiro
Nov 06 2010
prev sibling next sibling parent Tomek =?UTF-8?B?U293acWEc2tp?= <just ask.me> writes:
Jonathan M Davis napisaƂ:

 I'm proposing a possible new module for phobos which would be called
 std.unittests. The code and DDoc file can be found here:
 http://is.gd/gLH9Q
 
 Or you can look at the code here directly (though it has poor
 highlighting): http://ideone.com/EOlod
 
 The module contains a set of helper functions for unit tests which are an
 improvement over simply using assert. I have them to be very helpful in
 the unit tests in my proposal for std.datetime. My unit tests are more
 concise, easier to read, and much easier to debug because of them.
 assertEqual() and assertExcThrown() have been particularly useful.

If the language can't help, helpers are definitely of value. In Java I rarely use the built-in assert statement and resort to JUnit functions where possible.
 assert is a great tool, but simply telling you that an assertion failed is
 not as useful as telling you _how_ it failed, and a function like
 assertExcThrown() (which, as you may guess, asserts that a particular
 exception was thrown by a particular function call) really reduces the
 boiler plate code that you have to write for testing what it tests. So, I
 find such helper functions are extremely useful.

From a different angle: if assert is already part of the language, and the compiler has access to the AST of the tested expression, perhaps it could also automatically generate code that pretty-prints the expression? -- Tomek
Nov 06 2010
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday 06 November 2010 12:28:02 Masahiro Nakagawa wrote:
 I don't know this module's usability, but is getExcMsg useful?
 I can't imagine useful situation.
 In addition, I grepped your datetime.d and getExcMsg not found :(
 
 If possible, I want the mock, stub and related features.

It's used to test unittests.d itself, but I don't think that I'm using it in datetime.d at the moment. The idea is to use it if you want to verify that an exception says exactly what you want it to say (especially if you could get multiple exceptions of the same type but with different messages from a function). I will grant you, however, that it's one of the less useful functions. - Jonathan M Davis
Nov 06 2010
parent "Nick Sabalausky" <a a.a> writes:
"Jonathan M Davis" <jmdavisProg gmx.com> wrote in message 
news:mailman.138.1289084125.21107.digitalmars-d puremagic.com...
 On Saturday 06 November 2010 12:28:02 Masahiro Nakagawa wrote:
 I don't know this module's usability, but is getExcMsg useful?
 I can't imagine useful situation.
 In addition, I grepped your datetime.d and getExcMsg not found :(

 If possible, I want the mock, stub and related features.

It's used to test unittests.d itself, but I don't think that I'm using it in datetime.d at the moment. The idea is to use it if you want to verify that an exception says exactly what you want it to say (especially if you could get multiple exceptions of the same type but with different messages from a function). I will grant you, however, that it's one of the less useful functions.

I'm not suggesting getExcMsg be removed, but if you ever need to verify an exception's message is exactly matches a specific string, then you're almost certainly using exceptions the wrong way to begin with.
Nov 06 2010
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday 06 November 2010 18:35:29 Nick Sabalausky wrote:
 "Jonathan M Davis" <jmdavisProg gmx.com> wrote in message
 news:mailman.138.1289084125.21107.digitalmars-d puremagic.com...
 
 On Saturday 06 November 2010 12:28:02 Masahiro Nakagawa wrote:
 I don't know this module's usability, but is getExcMsg useful?
 I can't imagine useful situation.
 In addition, I grepped your datetime.d and getExcMsg not found :(
 
 If possible, I want the mock, stub and related features.

It's used to test unittests.d itself, but I don't think that I'm using it in datetime.d at the moment. The idea is to use it if you want to verify that an exception says exactly what you want it to say (especially if you could get multiple exceptions of the same type but with different messages from a function). I will grant you, however, that it's one of the less useful functions.

I'm not suggesting getExcMsg be removed, but if you ever need to verify an exception's message is exactly matches a specific string, then you're almost certainly using exceptions the wrong way to begin with.

The reason I created it originally IIRC was to verify stuff like the fact that passing an invalid hour to TimeOfDay complained about the hour being incorrect rather than the minute or the second. It _is_ arguably error-prone, because it's testing the _exact_ error message, but if you're trying to verify that the correct exception was thrown and multiple exceptions of the same type can be thrown from a function, I don't really see any other way to do it. I ended up not using it in std.datetime (probably because I consolidated most of the code which threw DateTimeExceptions, so there was less need for it), but I do think that it's at least potentially useful if you want to verify that your exceptions have the correct error messages. - Jonathan M Davis
Nov 06 2010