www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Unit tests in D

reply bearophile <bearophileHUGS lycos.com> writes:
dmd D 2.045 improves the built-in unit tests resuming their run when they fail
(it reports only the first failed assert for each unit test).

There are many features that today a professional unittest system is expected
to offer, I can write a long list. But in the past I have explained that it's a
wrong idea to try to implement all those things in dmd.

So a good solution that has all the advantages is:
- To add dmd the "core" features that are both important and hard to implement
nicely in an external library or IDE (or make D more flexible so writing such
libs is possible, but this can be not easy).
- To add dmd the compile-time reflection, run-time reflection or hooks that
external unittest libs/IDEs can use to extend the built-in unit testing
functionality.

It's not easy to find such core features (that can be used by an IDE, but are
usable from the normal command line too), this is my first try, and I can be
wrong. Feel free to add items you think are necessary, or to remove items you
know can be implemented nicely in an external library. Later I can write an
enhancement request.

---------------------

1) It's very useful to have a way to catch static asserts too, because
templates and other things can contain static asserts too, that for example can
be used to test if input types or constants are correct. When I write unittests
for those templates I want to test that they actually statically asserts when I
use them in a wrong way.

A possible syntax (this has to act like static assert):
static throws(foo(10), StaticException1);
static throws(foo(10), StaticException1, StaticException2, ...);

A version that catches run-time asserts (this has to act like asserts):
throws(foo(10), Exception1);
throws(foo(10), Exception1, Exception2, ...);


There are ways to partially implement this for run-time asserts, but badly:

void throws(Exceptions, TCallable, string filename=__FILE__, int line=__LINE__)
           (lazy TCallable callable) {
    try
        callable();
    catch (Exception e) {
        if (cast(Exceptions)e !is null)
            return;
    }

    assert(0, text(filename, "(", line, "): doesn't throw any of the specified
exceptions."));
}


If that syntax is asking too much, an intermediate solution can be acceptable,
like (but the point of this list is to enumerate important things that are not
easy to implement in an external library):

static throws!(StaticException1)(foo(10));
static throws!(StaticException1, StaticException2)(foo(10));

throws!(Exception1)(foo(10));
throws!(Exception1, Exception2(foo(10));

---------------------

2) Names for unittests. Giving names to things in the universe is a first
essential step if you want to try to understand some part of it. The compiler
makes sure in each module two unittest tests don't share the same name. An
example:

int sqr(int x) { return 2 * x; }

/// asserts that it doesn't return a negative value
unittest(sqr) {
    assert(sqr(10) >= 0);
    assert(sqr(-10) >= 0);
}

---------------------

3) Each unittest error has to say the (optional) name of the unit tests it is
contained into. For example:

test4(sqr,6): unittest failure

---------------------

4) The dmd JSON output has to list all the unitttests, with their optional name
(because the IDE can use this information to do many important things).

---------------------

5) Optional ddoc text for unittests (to allow IDEs to answer the programmer the
purpose of a  specific test that has failed).

Unittest ddocs don't show up inside the HTML generated with -D because the user
of the module doesn't need to know the purpose of its unittests. So maybe they
appear only inside the JSON output.

---------------------

6a) A way to enable only unittests of a module. Because in a project there are
several modules, and when I work on a module I often want to run only its
unittests. In general it's quite useful to be able to disable unittests.

6b) A way to define groups of unittests (that cross modules too): because you
sometimes want to unittest for a feature spread in more than one module, or you
want to tell apart quick and slow unittests, to run fast unittests often and
slow ones only once in a while.


One way to support unittest groups is to allow for tag names after the unittest
name that define overlapping groups:

unittest(foo_name, group1_tag_name, group2_tag_name, ...) {...}


One way to extend the unittest compiler switch syntax to use those tags:
allowing multiple unittest switches in a command line, allowing a = that can be
used to specify a module name to run the unittests of, or a =tag: to specify
one tag name:

-unittest=module_foo -unittest=tag:group1_tag_name

This is just a first idea for unittest groups management, better designs are
possible.

====================================


Three more half-backed things, if you know how to improve/design this ideas you
can tell me:

A) Serious unittest system needs a way to allow sharing of setup and shutdown
code for tests.

From Python unittest: a test fixture is the preparation needed to perform one
or more tests, and any associate cleanup actions. This may involve, for
example, creating temporary or proxy databases, directories, or starting a
server process, etc.

Fixtures can be supported at the package, module, class and function level.
Setup always runs before any test (or groups of tests).

setUp(): Method called to prepare the test fixture.
tearDown(): Method called immediately after the test method has been called and
the result recorded.

---------------------

B) There are situations when you don't want to count how many unittests have
failed, but you want to fix a bug, with a debugger. For this it can be useful a
command line switch to turn unittest asserts back into normal asserts.

---------------------

C) I'd like a way to associate a specific unittest to a function, class or
module, or something else that is being tested, because this can be useful in
several different ways. But this seems not easy to design. Keep in mind that
tests can be in a different module. I don't know how to design this, or if it
can be designed well.

---------------------

Bye,
bearophile
May 04 2010
next sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-05-04 21:24:50 -0400, bearophile <bearophileHUGS lycos.com> said:

 There are ways to partially implement this for run-time asserts, but badly:
 
 void throws(Exceptions, TCallable, string filename=__FILE__, int line=__LINE__)
            (lazy TCallable callable) {
     try
         callable();
     catch (Exception e) {
         if (cast(Exceptions)e !is null)
             return;
     }
 
     assert(0, text(filename, "(", line, "): doesn't throw any of the 
 specified exceptions."));
 }

Your 'throws' template seems good. Should create std.testing and include it there. Also, perhaps it'd work to use a double-template for this: template throws(Exceptions...) { void throws(TCallable, string filename=__FILE__, int line=__LINE__) (lazy TCallable callable) { ... } } I know this trick was working in D1, but I'm not sure if it wasn't broken in D2. -- Michel Fortin michel.fortin michelf.com http://michelf.com/
May 04 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
(it reports only the first failed assert for each unit test).<

I was wrong (or the behaviour on this is mutable). Michel Fortin:
 Your 'throws' template seems good. Should create std.testing and 
 include it there.
 Also, perhaps it'd work to use a double-template for this:
 	template throws(Exceptions...) {
 		void throws(TCallable, string filename=__FILE__, int line=__LINE__)

Yes, thank you, with this suggestion of yours it works in D2 too: class FooException : Exception { this(string msg) { super(msg); } } class OtherException : Exception { this(string msg) { super(msg); } } int sqr(int x) { if (x < 0) throw new FooException(""); return x * 2; } template throws(Exceptions...) { bool throws(TCallable)(lazy TCallable callable) { try callable(); catch (Exception e) { /*static*/ foreach (Exc; Exceptions) if (cast(Exc)e !is null) return true; return false; } return !Exceptions.length; } } unittest { assert(throws!(OtherException)(sqr(-5))); assert(throws!(OtherException)( sqr(-5) )); } void main() {} But I have to wrap it into another assert like that if I want it to behave as an assert inside the unittests. With a bit more compiler support it can be possible to write that in a library. While the static throws can require more compiler support. Bye, bearophile
May 05 2010
parent Michel Fortin <michel.fortin michelf.com> writes:
On 2010-05-05 05:10:05 -0400, bearophile <bearophileHUGS lycos.com> said:

 unittest {
     assert(throws!(OtherException)(sqr(-5)));
     assert(throws!(OtherException)( sqr(-5) ));
 }

 While the static throws can require more compiler support.

Can't you do: static assert(throws!OtherException(sqrt(-5))); ? -- Michel Fortin michel.fortin michelf.com http://michelf.com/
May 05 2010
prev sibling next sibling parent reply Don <nospam nospam.com> writes:
bearophile wrote:
 dmd D 2.045 improves the built-in unit tests resuming their run when they fail
(it reports only the first failed assert for each unit test).
 
 There are many features that today a professional unittest system is expected
to offer, I can write a long list. But in the past I have explained that it's a
wrong idea to try to implement all those things in dmd.
 
 So a good solution that has all the advantages is:
 - To add dmd the "core" features that are both important and hard to implement
nicely in an external library or IDE (or make D more flexible so writing such
libs is possible, but this can be not easy).
 - To add dmd the compile-time reflection, run-time reflection or hooks that
external unittest libs/IDEs can use to extend the built-in unit testing
functionality.
 
 It's not easy to find such core features (that can be used by an IDE, but are
usable from the normal command line too), this is my first try, and I can be
wrong. Feel free to add items you think are necessary, or to remove items you
know can be implemented nicely in an external library. Later I can write an
enhancement request.

I think the majority of the items in your list can already be done fairly well (or easily enough by a library), except for giving names to unit tests. One thing which you've not mentioned is in unittests for interfaces (and derived classes). I would like to be able to check that *all* implementations of an interface satisfy a sequence of tests.
May 05 2010
next sibling parent Lutger <lutger.blijdestijn gmail.com> writes:
Don wrote:

 bearophile wrote:
 dmd D 2.045 improves the built-in unit tests resuming their run when they
 fail (it reports only the first failed assert for each unit test).
 
 There are many features that today a professional unittest system is
 expected to offer, I can write a long list. But in the past I have
 explained that it's a wrong idea to try to implement all those things in
 dmd.
 
 So a good solution that has all the advantages is:
 - To add dmd the "core" features that are both important and hard to
 implement nicely in an external library or IDE (or make D more flexible so
 writing such libs is possible, but this can be not easy). - To add dmd the
 compile-time reflection, run-time reflection or hooks that external
 unittest libs/IDEs can use to extend the built-in unit testing
 functionality.
 
 It's not easy to find such core features (that can be used by an IDE, but
 are usable from the normal command line too), this is my first try, and I
 can be wrong. Feel free to add items you think are necessary, or to remove
 items you know can be implemented nicely in an external library. Later I
 can write an enhancement request.

I think the majority of the items in your list can already be done fairly well (or easily enough by a library), except for giving names to unit tests. One thing which you've not mentioned is in unittests for interfaces (and derived classes). I would like to be able to check that *all* implementations of an interface satisfy a sequence of tests.

Do you mean something like parameters for unittests?
May 05 2010
prev sibling next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Don:

I think the majority of the items in your list can already be done fairly well
(or easily enough by a library), except for giving names to unit tests.<

Let's see the points: 1) The following syntax is compatibile with D2 to implement the two parts of point 1: assert(throws!(Ex1, Ex2)(foo(10))); static assert(!__traits(compiles, foo(10))); This works well enough, but the second syntax is both common enough in my code and ugly enough (ugly also means bug-prone) that some syntax improvement can be good. Future syntax like this can improve the situation a bit: static assert(!meta.compiles(foo(10))); 3) Currently unittest errors report only the module name and line number. If unittests get a name, but such name isn't shown up in the unittest errors, the IDE has to parse the source code to know what's the name of the failed unittest (if the IDE can't know this name, then there is little point in giving names to unittests in the first place). This is bad, so I think this tiny feature has to be built-in. 4) I think that currently the JSON ignores unittests (http://dsource.org/projects/dmd/changeset/453 ). For the IDE it's useful to know what unittests are present in a module, for example to compute the percentage of failed unittests, etc. If the JSON doesn't list them, then the IDE has to parse the code to find them by itself. This is doable, but the JSON was invented to avoid this. So I think this little feature has to be built-in. 5) This is not so important, the compiler can just ignore the unittest ddocs. The IDE can find them (knowing the starting line of the unittests) and use them as necessary to answer the user questions. Putting such unittest ddocs in the JSON can simplify the IDE a little, but this is not a vital thing. I think this is simple enough to implement that it's better for this to be built-in. 6) All serious unit test systems have several ways to disable unittests, this is a very common need. And a way to group them is often a need. D language has version(){} statements, and debug(){} too, they can be used to enable and disable unittests with no need of extra syntax support. With enough code to juggle defined and undefined symbols you can even support groups of unittests. But this is true for the debug() too, you can emulate the purpose of debug() given version() or version() given debug(). Some syntax support can turn something ad hoc and hairy in something simpler and more elegant. So some support for disabling unittests and grouping them can be useful. The three half-baked things need more thinking, I invite people to give opinions and suggestions. B) I don't know if this is necessary. C) This is nice, but I don't know if this is worth the complexity to implement it. Opinions welcome.
One thing which you've not mentioned is in unittests for interfaces (and
derived classes). I would like to be able to check that *all* implementations
of an interface satisfy a sequence of tests.<

A "sequence of tests" can mean a group of tests, see point 6. So what you ask for can be seen as related to the point C, to associate a group of tests to a interface (and all its implementation tree). A possible way to implement this is with an attribute: dotest(group1) interface Foo {...} // association interface <=> unittest group dotest(test1) int bar(int x) {...} // association function <=> unittest I am not sure if this is a good way to implement this idea. Bye, bearophile
May 05 2010
parent reply Don <nospam nospam.com> writes:
bearophile wrote:
 Don:
 One thing which you've not mentioned is in unittests for interfaces (and
derived classes). I would like to be able to check that *all* implementations
of an interface satisfy a sequence of tests.<

A "sequence of tests" can mean a group of tests, see point 6. So what you ask for can be seen as related to the point C, to associate a group of tests to a interface (and all its implementation tree). A possible way to implement this is with an attribute:

No, it's not related to point C in any way. The "sequence of tests" was not important. I should have said "one or more tests". Also an attribute is a terrible way to implement almost anything -- you need a really good reason to add a new attribute. My point could be solved with some level of run-time reflection. Or perhaps by allowing unittest{} as a (static) interface member.
May 05 2010
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Don:

I didn't mean to put my words in your mouth, sorry. I'll try to be more careful.

Also an attribute is a terrible way to implement almost anything -- you need a
really good reason to add a new attribute.<

You have noticed that some of the things I have proposed in the last months use attributes. I don't know why attributes are so bad, I like the idea of user-defined attributes too (for D3), in C# they are useful. Bye, bearophile
May 06 2010
parent Don <nospam nospam.com> writes:
bearophile wrote:
 Don:
 
 I didn't mean to put my words in your mouth, sorry. I'll try to be more
careful.
 
 Also an attribute is a terrible way to implement almost anything -- you need a
really good reason to add a new attribute.<

You have noticed that some of the things I have proposed in the last months use attributes. I don't know why attributes are so bad, I like the idea of user-defined attributes too (for D3), in C# they are useful. Bye, bearophile

The thing is, they're not really any different to keywords in how difficult they make the language to learn, in how much complexity they add to the compiler, and in how much complexity they add to the spec. We could even add an to the start of every existing keyword, and then argue that the language has no keywords at all! So we still need to be reluctant to add any new ones.
May 06 2010
prev sibling parent BCS <none anon.com> writes:
Hello Don,

 My point could be solved with some level of run-time reflection. Or
 perhaps by allowing unittest{} as a (static) interface member.

Might something ad-hoc like this work? interface Foo { ... } class TestFoo(T) if(is(T : Foo)) { unittest { } } -- ... <IXOYE><
May 06 2010
prev sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
Don <nospam nospam.com> wrote:
 bearophile wrote:
 dmd D 2.045 improves the built-in unit tests resuming their run when
 they fail (it reports only the first failed assert for each unit
 test).
 There are many features that today a professional unittest system
 is expected to offer, I can write a long list. But in the past I
 have explained that it's a wrong idea to try to implement all
 those things in dmd.
 So a good solution that has all the advantages is:

implement nicely in an external library or IDE (or make D more flexible so writing such libs is possible, but this can be not easy). - To add dmd the compile-time reflection, run-time reflection or hooks that external unittest libs/IDEs can use to extend the built-in unit testing functionality.
 It's not easy to find such core features (that can be used by an
 IDE, but are usable from the normal command line too), this is my
 first try, and I can be wrong. Feel free to add items you think
 are necessary, or to remove items you know can be implemented
 nicely in an external library. Later I can write an enhancement
 request.


I think the majority of the items in your list can already be done fairly well (or easily enough by a library), except for giving names to unit tests.

A while back I proposed: unittest("name") {} As an optional way to name a unittest. It's the last function-like thing without parens anyway.
May 08 2010
next sibling parent Jason House <jason.james.house gmail.com> writes:
Sean Kelly Wrote:

 Don <nospam nospam.com> wrote:
 bearophile wrote:
 dmd D 2.045 improves the built-in unit tests resuming their run when
 they fail (it reports only the first failed assert for each unit
 test).
 There are many features that today a professional unittest system
 is expected to offer, I can write a long list. But in the past I
 have explained that it's a wrong idea to try to implement all
 those things in dmd.
 So a good solution that has all the advantages is:

implement nicely in an external library or IDE (or make D more flexible so writing such libs is possible, but this can be not easy). - To add dmd the compile-time reflection, run-time reflection or hooks that external unittest libs/IDEs can use to extend the built-in unit testing functionality.
 It's not easy to find such core features (that can be used by an
 IDE, but are usable from the normal command line too), this is my
 first try, and I can be wrong. Feel free to add items you think
 are necessary, or to remove items you know can be implemented
 nicely in an external library. Later I can write an enhancement
 request.


I think the majority of the items in your list can already be done fairly well (or easily enough by a library), except for giving names to unit tests.

A while back I proposed: unittest("name") {} As an optional way to name a unittest. It's the last function-like thing without parens anyway.

I think it's been repeatedly proposed. One contributor to my open source project created a function test(string name, void delegate() what)... It would allow the program to continue after a failed unit test, but would print out the names of the failing tests.
May 08 2010
prev sibling parent bearophile <bearophileHUGS lycos.com> writes:
Sean Kelly:
 A while back I proposed:
 unittest("name") {}

Now I think this syntax is a little better: unittest name {} Bye, bearophile
May 08 2010
prev sibling next sibling parent reply =?iso-8859-2?B?VG9tZWsgU293afFza2k=?= <just ask.me> writes:
Dnia 05-05-2010 o 03:24:50 bearophile <bearophileHUGS lycos.com>  =

napisa=B3(a):

[snip]
 2) Names for unittests. Giving names to things in the universe is a  =

 first essential step if you want to try to understand some part of it.=

 The compiler makes sure in each module two unittest tests don't share =

 the same name. An example:

 int sqr(int x) { return 2 * x; }

 /// asserts that it doesn't return a negative value
 unittest(sqr) {
     assert(sqr(10) >=3D 0);
     assert(sqr(-10) >=3D 0);
 }

I think it's achievable through mixin templates: mixin Unittest!("sqr", { assert(sqr(10) >=3D 0); assert(sqr(-10) >=3D 0); }); and even doesn't poke your eyes like most mixin code:)
 3) Each unittest error has to say the (optional) name of the unit test=

 it is contained into. For example:

 test4(sqr,6): unittest failure

If Unittest template** above would be something like: mixin template Unittest(string name, alias fun) { mixin(" void "~name~"() { fun(); } unittest { writeln(\"Testing "~name~"...\"); "~name~"(); } "); } then you would see the name of the unit test in the stack trace.
 4) The dmd JSON output has to list all the unitttests, with their  =

 optional name (because the IDE can use this information to do many  =

 important things).

This is interesting. How the IDE may help knowing the unittest name?
 5) Optional ddoc text for unittests (to allow IDEs to answer the  =

 programmer the purpose of a  specific test that has failed).

When a test has failed I go to the testing code to see what's up. So I'l= l = see any non-ddoc comments someone left there.
 6a) A way to enable only unittests of a module. Because in a project  =

 there are several modules, and when I work on a module I often want to=

 run only its unittests. In general it's quite useful to be able to  =

 disable unittests.

... or in a package, or one folder above the package level, or ... ;) Versioning and conditional compilation are the right tools for the job.
 6b) A way to define groups of unittests (that cross modules too):  =

 because you sometimes want to unittest for a feature spread in more th=

 one module, or you want to tell apart quick and slow unittests, to run=

 fast unittests often and slow ones only once in a while.

 One way to support unittest groups is to allow for tag names after the=

 unittest name that define overlapping groups:

 unittest(foo_name, group1_tag_name, group2_tag_name, ...) {...}

It's more of a version statement issue: can't have many versions or'ed = together in a single condition. Then again, this is possible now: template isVersion(string name) { enum bool isVersion =3D !is(typeof({ mixin("version("~name~") { static assert(false); }"); })); } static if (isVersion!"group1_tag_name" || isVersion!"group2_tag_name" ||= = ... ) mixin Unittest!("foo_name", {...}); [snip]
 A) Serious unittest system needs a way to allow sharing of setup and  =

 shutdown code for tests.

How about: unittest { setUp(); scope(exit) tearDown(); ... };
 Fixtures can be supported at the package, module, class and function  =

 level. Setup always runs before any test (or groups of tests).

Unittests are executed in declaration order, so: unittest { setUp(); } unittest { ... } unittest { ... } unittest { tearDown(); } In a nutshell, I agree with you that unittests would use more features b= ut = we should explore a (standard?) library solution first, before putting = more metal into the language/compiler. Tomek ** Actually, it doesn't compile now: mixin Unittest!("my_test", { assert (2 + 2 =3D=3D 4); }); Error: delegate test.__dgliteral2 is a nested function and cannot be = accessed from my_test I'd appreciate if someone elaborated on why exactly it cannot be accesse= d = from my_test. To me, if a delegate literal is defined in the global sco= pe, = then, with respect to nesting, it's not much different than a normal = function, no?
May 05 2010
parent bearophile <bearophileHUGS lycos.com> writes:
Tomek S.:

and even doesn't poke your eyes like most mixin code:)<

Thanks, but no thanks. String mixin hacks don't help, they make the situation worse.
then you would see the name of the unit test in the stack trace.<

I'd love to see the stack trace with stock dmd :-) I don't know/remember why standard dmd has never adopted the old "phobos hack" to show the stack trace.
This is interesting. How the IDE may help knowing the unittest name?<

What's the point in giving a name to a unittest? If the unittests have a name, and unittest failures show names too, but the IDE doesn't know all the names of the unittests of a module, then the IDE can't use those names well: - If the IDE knows the names of the unittests of a module it also knows how many unittests are present, so it can give the percentage of failed unittests. - IDEs show lists of unittests, with red and green symbols beside them. If the IDE doesn't know the all names of the unittests, it can show only the unittests with red symbols :-) - If the IDE doesn't knows the names of the unittests of a module, it can't compute other statistics commonly useful for higher level management of tests, like the test coverage of a project, percentage of tests written by a programmer, and so on.
When a test has failed I go to the testing code to see what's up. So I'll see
any non-ddoc comments someone left there.<

Of course that's a possibility. But IDEs usually show a Window that lists the unittests, the failed and passed ones, etc. You can move the mouse cursor on a unittest and the IDE can show you its ddoc, no need to jump to the code. As I have said this is not an essential feature, but nothing inside the JSON files is essential to program in D, it's a possible help for external tools.
... or in a package, or one folder above the package level, or ... ;)<

Right. The three main use cases are probably: - to disable an unittetest (it can be done with disable applied on an unitttes); - activating unittests only of a module (the one currently worked on); - activating only a group of unittests.
Versioning and conditional compilation are the right tools for the job.<

This works, but: - The code to do it can be a bit messy. (If I have to do something many times always in the same way, then I prefer clean code that clearly shows my semantics. This is why D has both version() and debug() in the first place). - If you are _not_ using an IDE it can become not handy to manage it. Let's say I want to run the unittests only of the module foobar. I have to put before all unittests code like: version(modulename) unittest(foo) {...} Then if I change the module name I have to change them all.
 static if (isVersion!"group1_tag_name" || isVersion!"group2_tag_name" ||  
 ... )
 mixin Unittest!("foo_name", {...});

That's a bad solution.
In a nutshell, I agree with you that unittests would use more features but we
should explore a (standard?) library solution first, before putting more metal
into the language/compiler.<

In that list I have not asked for very complex features. One of the main points of my post was to see if those things can be implemented well in the current language. Bye and thank you for your answers, bearophile
May 05 2010
prev sibling parent reply Lutger <lutger.blijdestijn gmail.com> writes:
bearophile wrote:

 dmd D 2.045 improves the built-in unit tests resuming their run when they
 fail (it reports only the first failed assert for each unit test).
 
 There are many features that today a professional unittest system is
 expected to offer, I can write a long list. But in the past I have
 explained that it's a wrong idea to try to implement all those things in
 dmd.
 
 So a good solution that has all the advantages is:
 - To add dmd the "core" features that are both important and hard to
 implement nicely in an external library or IDE (or make D more flexible so
 writing such libs is possible, but this can be not easy). - To add dmd the
 compile-time reflection, run-time reflection or hooks that external
 unittest libs/IDEs can use to extend the built-in unit testing
 functionality.
 
 It's not easy to find such core features (that can be used by an IDE, but
 are usable from the normal command line too), this is my first try, and I
 can be wrong. Feel free to add items you think are necessary, or to remove
 items you know can be implemented nicely in an external library. Later I
 can write an enhancement request.

I think that most of the features you mention can be implemented in a library, but at some cost. For example, tests get more verbose. Or for example with the string mixin syntax with unaryFun and binaryFun you are limited to the parameters and symbols from a select few phobos modules. There is something to be said for the simplicity of how D natively handles unittesting I think. Perhaps some issues with local template instantiation and / or mixin visibilty can be sorted out to improve this, I'm not sure - it's a bit above my head. Anyway I think if unittests can get a name as a parameter and dmd let's the user set the AssertError handler, that will be a sufficient hook to provide some useful user extension of the unittest system. I'll post some more specific comments below:
 ---------------------
 
 1) It's very useful to have a way to catch static asserts too, because
 templates and other things can contain static asserts too, that for example
 can be used to test if input types or constants are correct. When I write
 unittests for those templates I want to test that they actually statically
 asserts when I use them in a wrong way.
 
 A possible syntax (this has to act like static assert):
 static throws(foo(10), StaticException1);
 static throws(foo(10), StaticException1, StaticException2, ...);
 
 A version that catches run-time asserts (this has to act like asserts):
 throws(foo(10), Exception1);
 throws(foo(10), Exception1, Exception2, ...);
 
 
 There are ways to partially implement this for run-time asserts, but badly:
 
 void throws(Exceptions, TCallable, string filename=__FILE__, int
 line=__LINE__)
            (lazy TCallable callable) {
     try
         callable();
     catch (Exception e) {
         if (cast(Exceptions)e !is null)
             return;
     }
 
     assert(0, text(filename, "(", line, "): doesn't throw any of the
     specified exceptions."));
 }

I have set up a unittesting system in a hacky way that does something like this, but instead of asserting it just prints the error and collects the test result in a global storage. When the program ends this gets written to a json file. Seems to work well enough, what do you think the above lacks? I have wrapped it something like this: expectEx!SomeException(foo(10)); I have also done this for compile time assertions, but it is more limited: int a; expectCompileError!(isSorted, a); or: expectCompileError!(q{ isSorted(a) }, a); One cool thing of D2 however is that you can get the exact name and value of an alias parameter by local instantiation, for example this test: int[] numbers = [3,2,1,7]; expect!( isSorted, numbers ); prints: test.d(99) numbers failed: isSorted(alias less = "a < b",Range) if (isForwardRange!(Range)) ( numbers ) :: numbers == [3 2 1 7]
 2) Names for unittests. Giving names to things in the universe is a first
 essential step if you want to try to understand some part of it. The
 compiler makes sure in each module two unittest tests don't share the same
 name. An example:
 
 int sqr(int x) { return 2 * x; }
 
 /// asserts that it doesn't return a negative value
 unittest(sqr) {
     assert(sqr(10) >= 0);
     assert(sqr(-10) >= 0);
 }

I agree, as mentioned I have done this by writing to some global state which records the currently running test, then are 'assertions' write to the same state. If only one thing could improve in the native system I think this should be it.
 ---------------------
 
 3) Each unittest error has to say the (optional) name of the unit tests it
 is contained into. For example:
 
 test4(sqr,6): unittest failure

One tiny tip: test.d(6): unittest sqr failed This way ide's and editors can parse it like regular D errors and jump to the failed test.
 ---------------------
 
 4) The dmd JSON output has to list all the unitttests, with their optional
 name (because the IDE can use this information to do many important
 things).
 
 ---------------------
 
 5) Optional ddoc text for unittests (to allow IDEs to answer the programmer
 the purpose of a  specific test that has failed).
 
 Unittest ddocs don't show up inside the HTML generated with -D because the
 user of the module doesn't need to know the purpose of its unittests. So
 maybe they appear only inside the JSON output.

I think there's a report in bugzilla from Andrei requesting that the unittests themselves can be turned into documentation. Together with preconditions in ddoc, that would seem very useful.
 ---------------------
 
 6a) A way to enable only unittests of a module. Because in a project there
 are several modules, and when I work on a module I often want to run only
 its unittests. In general it's quite useful to be able to disable
 unittests.

Shouldn't this be part of a tool that compiles and runs tests of each module? rdmd has the option --main, together with -unittest you can easily do this. ...
 
 
 Three more half-backed things, if you know how to improve/design this ideas
 you can tell me:
 
 A) Serious unittest system needs a way to allow sharing of setup and
 shutdown code for tests.
 
 From Python unittest: a test fixture is the preparation needed to perform
 one or more tests, and any associate cleanup actions. This may involve, for
 example, creating temporary or proxy databases, directories, or starting a
 server process, etc.
 
 Fixtures can be supported at the package, module, class and function level.
 Setup always runs before any test (or groups of tests).
 
 setUp(): Method called to prepare the test fixture.
 tearDown(): Method called immediately after the test method has been called
 and the result recorded.

I think this is standard xUnit. I have this: class TestSuiteA : Fixture { void setup() { /* initialize */} void teardown() { /* cleanup or restore */ } void test_foo() { /* test ...*/} void test_bar() { /* test ...*/} } // running setup, test_foo, test_bar and finally teardown: unittest { runSuite!TestSuite(); } But it seems like a good idea to separate the fixture from the test suite, so they can easily be reused.
 ---------------------
 
 B) There are situations when you don't want to count how many unittests
 have failed, but you want to fix a bug, with a debugger. For this it can be
 useful a command line switch to turn unittest asserts back into normal
 asserts.

This is possible if you replace the assert statement with your own or perhaps hook up an assertHandler. At the moment this is a bit weird, perhaps it is a bug? unittest { void inner() { assert( false, "bar" ); } assert( false, "foo" ); inner(); assert( false, "baz" ); } This unittest halts with an AssertError inside inner().
 ---------------------
 
 C) I'd like a way to associate a specific unittest to a function, class or
 module, or something else that is being tested, because this can be useful
 in several different ways. But this seems not easy to design. Keep in mind
 that tests can be in a different module. I don't know how to design this,
 or if it can be designed well.

Looks hard to me too. I think I like the idea that unittests should be close to the code it tests, otherwise it is a different kind of test. Perhaps we should consider ddoc, contracts and unittest as much part of the code as the code itself?
 
 Bye,
 bearophile

May 05 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Lutger:

Anyway I think if unittests can get a name as a parameter and dmd let's the
user set the AssertError handler, that will be a sufficient hook to provide
some useful user extension of the unittest system.<

Ah, I didn't think about an AssertError handler. Among other things, it solves my point B. Digging a little, I have seen core.exceptions has already a setAssertHandler, but it's not useful yet: http://d.puremagic.com/issues/show_bug.cgi?id=3208
 One tiny tip: test.d(6): unittest sqr failed
 This way ide's and editors can parse it like regular D errors and jump to the
failed test.

Good.
I think there's a report in bugzilla from Andrei requesting that the unittests
themselves can be turned into documentation.<

Given the size of my unittests I don't think this is good idea in general. Maybe Andrei was trying to add something like python doctests to D. I think to do this well it can be necessary a way to tell apart "documentation doctests" from normal (and often boring or large) unittests.
Shouldn't this be part of a tool that compiles and runs tests of each module?<

Most of the features I have listed are meant for an IDE, while this one is for programmer that uses just the command line with no tools :-) And to enable/disable test groups things gets a little more complex.
rdmd has the option --main, together with -unittest you can easily do this.<

This page doesn't list that option: http://www.digitalmars.com/d/2.0/rdmd.html rdmd prints: --eval=code evaluate code + la perl -e (multiple --eval allowed) --loop assume "foreach (line; stdin.byLine()) { ... }" for eval --main add a stub main program to the mix (e.g. for unittesting) I don't understand how to use those three switches. Do you know where I can find usage examples for those three?
This unittest halts with an AssertError inside inner().<

That's why my throws!()() has to return a boolean instead of just asserting :-( I don't know if this can be seen as a dmd bug.
I think I like the idea that unittests should be close to the code it tests,
otherwise it is a different kind of test.<

There are strong opionions on this. Some people love this, other people ate it. Some people can appreciate to move large amounts of unittests in separate modules. The best thing is to leave the programmers to put the unittests where they want.
Perhaps we should consider ddoc, contracts and unittest as much part of the
code as the code itself?<

Of course :-) Bye and thank you for your answers and ideas, bearophile
May 05 2010
parent Lutger <lutger.blijdestijn gmail.com> writes:
bearophile wrote:

 Lutger:

rdmd has the option --main, together with -unittest you can easily do
this.<

This page doesn't list that option: http://www.digitalmars.com/d/2.0/rdmd.html rdmd prints: --eval=code evaluate code +� la perl -e (multiple --eval allowed) --loop assume "foreach (line; stdin.byLine()) { ... }" for eval --main add a stub main program to the mix (e.g. for unittesting) I don't understand how to use those three switches. Do you know where I can find usage examples for those three?

I don't know. This is how you can unittest a single module: rdmd --main -unittest test.d it just links in a predefined .d file with a main() routine, that's all.
May 05 2010