www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - 'live' testing style

reply spir <denis.spir gmail.com> writes:
Hello,


Back to the subject of using unittest and assert. I'll try to illustrate a 
testing style that seems to be rare in the D community, is not properly 
supported by D builtin tools --while this would require only minimal 
improvements--, and is imo rather sensible, practicle and efficient. Using an 
example of a current project of mine (a toy homoiconic language, kind of 
special Lisp).

Below is a copy of part of the parser's test suite. Maybe have a quick look 
before reading further [Note: I don't mean the example is a model of any sort, 
not even it's good code. It is just illustration of some testing practice.] The 
testList func tests the match func for List literal notations. Some comments, 
in form of question-answer:

* Why isn't testList a unittest block?

Using named funcs, I can switch on & off specific test suites by (un)commenting 
their call from the main and unique unittest block. Else, either they all run, 
or none. During development, I only keep active the test func(s) relative to 
the feature I'm currently working on.
Remedy: named unittests.

* Why those write(f)(ln) in unitests?

For me unittests are not only regression tests, to be used /after/ development 
for maintenance purpose. Instead, they are the #1 and necessary source of 
information about what the code actually does, how, and why. I constantly run 
tests, for feedback, just to check all runs fine, or diagnose what goes wrong.
Thus, the first method I write for a type if often toString; I give it a form 
which tells me all I need about an object; commonly, it's just the notation to 
re-create it. This, and having test in mind for writing toString, also helps in 
properly defining the type itself.
For diagnosis, this information is of primordial value. And, against common 
sense maybe, data about what goes right, or seems to, is commonly even more 
important than data about what goes wrong, to determine the cause(s). (*)
Remedy: verbose asserts on request.

* What are those '// for diagnose' statement?

Since the test func has a form of loop over a test suite, a failing assert 
would fail (!) to tell which case is in cause, instead miserably spitting out a 
line number only. To get the proper info, I just need to comment out those few 
lines of code, which would output for instance (artificial failure case):
...
`[nil false  true]` --> [nil false true]
`["a"  "ab"   "abc"]` --> ["a" "ab" "abc"]
`[[3] [3 2] [3 1 2]]` --> [[3] [3 2] [3 1 2]]
*** error *** expected: [[3] [32] [3 1 2]]
Remedy: improved assertion failure message form.

* Why string'ed results?

In this case, like in many others due to various reasons, it's uneasy to write 
correct result objects; instead I would spend much stupid time in trying to 
construct by hand portions of ASTs. But I have just what I need already: normal 
runs of unitests in verbose mode write them out for me. Once I'm convinced, as 
much as possible, all works fine, I just copy paste result /strings/ into 
assert statements.
Then only I activate said assertions and switch verbose mode off, thus getting 
a regression test suite for free. Even better, this test suite holds 
information providing code, ready to use in case of failure.
Remedy: assert converts test result to string if needed.



Summary:

1. Named unittests allowing test suites in the form of (just an example):

unittest test1 {
     ...
}
unittest test2 {
     ...
}
unittest test3 {
     ...
}
unittest {
     test1;
     test2;
     test3;
}

/Unnamed/ unittests are run with --unittest. Named ones are intended to be 
called from unnamed ones. Backward compatible change.


2. A variant of assert(), or better a distinct check() statement, in the form
of:

     check(expression, expectation)

Separating arguments allows writing out messages like (example formats):

* case success (outcome == expectation):
     <expression> --> <outcome>
bringing valuable feedback during development.

* case failure
     *** test failure **********
     expression  : <expression>
     outcome     : <outcome>
     expectation : <expectation>
     ***************************
In silent mode, check() writes only in case of failure.

If the expected result is a separated argument, it is trivial for check to call 
to!string on the outcome when it's not a string and the expected result is a 
string; then only compare.

I guess both changes require compiler support (but may very well be wrong).
As illustrated by the example, D does not prevent one to use such testing 
practices; instead, its very nice string and literal features make it far 
better suited for that than any other static language I know.
Still, like Walter, I think it's important for (good) testing practices to be 
supported by the language, so-to-say officially. 'unittest' itself does not 
bring any feature: one could trivially have a 'test' control func intended to 
be (un)commented. But the fact it's a builtin feature makes a decisive 
difference; if only because it then becomes part of the D community's shared 
culture.


Denis


(*) Please don't argue on this point, I have been a maintenance engineer for 
five years, diagnosing broken machines everyday.


=== sample part of test code ====================
void testList () {
     writeln("=== List ===============================================");
     List list;
     LexemeStream lexemes;

     // test data
     string[] sources = [
         `[  ]` ,
         `[ 3    ]` ,
         `[ 3   1  2 ]` ,
         `[zxy  x   zx]` ,
         `[nil false  true]` ,
         `["a"  "ab"   "abc"]` ,
         `[[3] [3 2] [3 1 2]]` ,
         `[3 "abc" zxy nil [3 1 2]]` ,
     ];
     string[] correctResultStrings = [
         `[]` ,
         `[3]` ,
         `[3 1 2]` ,
         `[zxy x zx]` ,
         `[nil false true]` ,
         `["a" "ab" "abc"]` ,
         `[[3] [3 2] [3 1 2]]` ,
         `[3 "abc" zxy nil [3 1 2]]` ,
     ];

     // test run
     foreach (i,source ; sources) {
         lexemes = claroLexer.lexemes(source);
         list = cast(List)matchList(lexemes);
         // output
         writefln("`%s` --> %s", source,list);
         // for diagnose
         if ( to!string(list) != correctResultStrings[i] ) {
             writefln("*** error *** expected: %s", correctResultStrings[i]);
             stop(); // core.stdc.stdlib.exit(0)
         }
         // regression test assertion
         assert ( to!string(list) == correctResultStrings[i] );
     }
     writeln();
}
...
unittest {
     testList();
//~     testUnit();
//~     testStatement();
//~     ...
}
=================================================
=== success output in 'verbose' mode ============
`[  ]` --> []
`[ 3    ]` --> [3]
`[ 3   1  2 ]` --> [3 1 2]
`[zxy  x   zx]` --> [zxy x zx]
`[nil false  true]` --> [nil false true]
`["a"  "ab"   "abc"]` --> ["a" "ab" "abc"]
`[[3] [3 2] [3 1 2]]` --> [[3] [3 2] [3 1 2]]
`[3 "abc" zxy nil [3 1 2]]` --> [3 "abc" zxy nil [3 1 2]]
=================================================

-- 
_________________
vita es estrany
spir.wikidot.com
Feb 13 2011
next sibling parent reply Tomek =?ISO-8859-2?Q?Sowi=F1ski?= <just ask.me> writes:
spir napisa=B3:

 * Why isn't testList a unittest block?
=20
 Using named funcs, I can switch on & off specific test suites by (un)comm=

 their call from the main and unique unittest block. Else, either they all=

 or none. During development, I only keep active the test func(s) relative=

 the feature I'm currently working on.
 Remedy: named unittests.

The interesting thing about named unit tests is that their names aren't int= eresting at all. They are usually dull and forced; testing filterFoo will b= e called testFilterFoo, etc. Their only purpose is to suppress running of u= nrelated tests. Now, there is a seemingly unrelated proposal to include every ddoc'ed unit = test in the preceding declaration as an example. This is great because it i= mplies ownership -- a unit test is 'owned' by the symbol above. Going furth= er, it can also be named after its owner. module ooh; void foo(); unittest { test foo... } Compiling with --unittest=3Dooh.foo runs this unittest only. Nested control= as a bonus: compiling with --unittest=3Dooh runs only the tests in module = ooh. So there you go, named unit tests without naming. --=20 Tomek
Feb 14 2011
parent reply Don <nospam nospam.com> writes:
Jonathan M Davis wrote:
 On Monday, February 14, 2011 12:49:11 Tomek Sowiński wrote:
 spir napisał:
 * Why isn't testList a unittest block?

 Using named funcs, I can switch on & off specific test suites by
 (un)commenting their call from the main and unique unittest block. Else,
 either they all run, or none. During development, I only keep active the
 test func(s) relative to the feature I'm currently working on.
 Remedy: named unittests.

interesting at all. They are usually dull and forced; testing filterFoo will be called testFilterFoo, etc. Their only purpose is to suppress running of unrelated tests.


 Now, there is a seemingly unrelated proposal to include every ddoc'ed unit
 test in the preceding declaration as an example. This is great because it
 implies ownership -- a unit test is 'owned' by the symbol above. Going
 further, it can also be named after its owner.

 module ooh;

 void foo();

 unittest { test foo... }

 Compiling with --unittest=ooh.foo runs this unittest only. Nested control
 as a bonus: compiling with --unittest=ooh runs only the tests in module
 ooh.

 So there you go, named unit tests without naming.

The number one reason that I want named unit tests is stack traces. unittest428 or whatever a unittest block gets named to is pretty useless. So, while an assert within the unit test is fine, since it gives a file and line number which is in the unittest block, exceptions thrown from stuff called by the unit test become hard to track down, since you don't know which unit test was being run.

Sounds as though a solution would be to have unittests automatically named as "_unittest" ~ __LINE__ rather than an arbitrary integer. Wouldn't work in the case where there are two unittests on the same line: unittest { assert(true); } unittest { assert(true); } But we can deal with that.
Feb 14 2011
parent spir <denis.spir gmail.com> writes:
On 02/15/2011 02:42 AM, Jonathan M Davis wrote:
 On Monday, February 14, 2011 17:26:26 Don wrote:
 Jonathan M Davis wrote:
 On Monday, February 14, 2011 12:49:11 Tomek SowiƄski wrote:
 spir napisaƂ:
 * Why isn't testList a unittest block?

 Using named funcs, I can switch on&  off specific test suites by
 (un)commenting their call from the main and unique unittest block.
 Else, either they all run, or none. During development, I only keep
 active the test func(s) relative to the feature I'm currently working
 on. Remedy: named unittests.

The interesting thing about named unit tests is that their names aren't interesting at all. They are usually dull and forced; testing filterFoo will be called testFilterFoo, etc. Their only purpose is to suppress running of unrelated tests. Now, there is a seemingly unrelated proposal to include every ddoc'ed unit test in the preceding declaration as an example. This is great because it implies ownership -- a unit test is 'owned' by the symbol above. Going further, it can also be named after its owner. module ooh; void foo(); unittest { test foo... } Compiling with --unittest=ooh.foo runs this unittest only. Nested control as a bonus: compiling with --unittest=ooh runs only the tests in module ooh. So there you go, named unit tests without naming.

The number one reason that I want named unit tests is stack traces. unittest428 or whatever a unittest block gets named to is pretty useless. So, while an assert within the unit test is fine, since it gives a file and line number which is in the unittest block, exceptions thrown from stuff called by the unit test become hard to track down, since you don't know which unit test was being run.

Sounds as though a solution would be to have unittests automatically named as "_unittest" ~ __LINE__ rather than an arbitrary integer. Wouldn't work in the case where there are two unittests on the same line: unittest { assert(true); } unittest { assert(true); } But we can deal with that.

I still think that having named unit tests is something that we should do at some point (and it would be nicely backwards compatible), but naming the unit test function after the line number that it's on would already be a big improvement. I have no idea how they're named/numbered now. I haven't been able to discern any obvious pattern, and with a module like std.datetime, which has lots of unittest blocks, even if it were clearly unittest1, unittest2, unittest3, etc. it would still be quite hard to figure out which unittest block was running when the exception threw, since you'd have to count them all. Using __LINE__ would be a big improvement - though unless we figure out some way of advertising that, most people probably wouldn't have a clue, and it wouldn't help them much. It would definitely be a positive change though. And since if someone is bizarre enough to put more than one unittest block on one line, well then they suffer for what is almost certainly a poor choice. But it would still narrow it down for them.

The only /meaningful/ choice for a programmer, in the general case, is programmer-chosen names. Don't you think? One can then even nicely use those names like eg "see example use in unittest ApplicationToArithmetics". If people are happy with naming unittests using ordinals (1 2 3...) or line numbers (in realease), well... let them free, after all it's their choice :-) Denis -- _________________ vita es estrany spir.wikidot.com
Feb 15 2011
prev sibling next sibling parent spir <denis.spir gmail.com> writes:
On 02/14/2011 09:49 PM, Tomek SowiƄski wrote:
 spir napisaƂ:

 * Why isn't testList a unittest block?

 Using named funcs, I can switch on&  off specific test suites by (un)commenting
 their call from the main and unique unittest block. Else, either they all run,
 or none. During development, I only keep active the test func(s) relative to
 the feature I'm currently working on.
 Remedy: named unittests.

The interesting thing about named unit tests is that their names aren't interesting at all. They are usually dull and forced; testing filterFoo will be called testFilterFoo, etc. Their only purpose is to suppress running of unrelated tests. Now, there is a seemingly unrelated proposal to include every ddoc'ed unit test in the preceding declaration as an example. This is great because it implies ownership -- a unit test is 'owned' by the symbol above. Going further, it can also be named after its owner. module ooh; void foo(); unittest { test foo... } Compiling with --unittest=ooh.foo runs this unittest only. Nested control as a bonus: compiling with --unittest=ooh runs only the tests in module ooh. So there you go, named unit tests without naming.

I have thought at something similar (alse after the related thread about ddoc), but let it down for various reasons: * Completely implicit. In particular, one cannot learn the feature by reading other people's code; actually, one cannot even guess there is a feature there... * Imposed placement, order. Sometimes, placing the unittest immediately after what it test is not the best. Sometimes, one wants to group part ot all unittests. Sometimes, there are tests relative several funcs / types, etdc. * What about several unittests for a given feature. They would all run, or only the first one, or what? * Naming is a clue for the reader. Esp when there are several tests, or when the test check a particular point. * Possible extension: params. (I thinkin particular at benchmarks, for instance, but there would certainly be numerous other cases, including getting args such as data file names from the command line.) unittest testScan (string filename) { ... } or even unittest testScan (string source, string result) { ... } [Well, actually, i'm not fan of such an extension, prefere to keep things simple, and it's already trivial to factor out a piece of test.] Denis -- _________________ vita es estrany spir.wikidot.com
Feb 14 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, February 14, 2011 12:49:11 Tomek Sowi=F1ski wrote:
 spir napisa=B3:
 * Why isn't testList a unittest block?
=20
 Using named funcs, I can switch on & off specific test suites by
 (un)commenting their call from the main and unique unittest block. Else,
 either they all run, or none. During development, I only keep active the
 test func(s) relative to the feature I'm currently working on.
 Remedy: named unittests.

The interesting thing about named unit tests is that their names aren't interesting at all. They are usually dull and forced; testing filterFoo will be called testFilterFoo, etc. Their only purpose is to suppress running of unrelated tests. =20 Now, there is a seemingly unrelated proposal to include every ddoc'ed unit test in the preceding declaration as an example. This is great because it implies ownership -- a unit test is 'owned' by the symbol above. Going further, it can also be named after its owner. =20 module ooh; =20 void foo(); =20 unittest { test foo... } =20 Compiling with --unittest=3Dooh.foo runs this unittest only. Nested contr=

 as a bonus: compiling with --unittest=3Dooh runs only the tests in module
 ooh.
=20
 So there you go, named unit tests without naming.

The number one reason that I want named unit tests is stack traces. unittes= t428=20 or whatever a unittest block gets named to is pretty useless. So, while an= =20 assert within the unit test is fine, since it gives a file and line number = which=20 is in the unittest block, exceptions thrown from stuff called by the unit t= est=20 become hard to track down, since you don't know which unit test was being r= un.=20 Named unit tests would fix the problem. I believe that the best proposed sy= ntax=20 for that is unittest(testName) { } That way, something like unittest_testName would end up in the stack trace,= and=20 it would be easy to figure out where exception came from. =2D Jonathan M Davis
Feb 14 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, February 14, 2011 17:26:26 Don wrote:
 Jonathan M Davis wrote:
 On Monday, February 14, 2011 12:49:11 Tomek Sowi=F1ski wrote:
 spir napisa=B3:
 * Why isn't testList a unittest block?
=20
 Using named funcs, I can switch on & off specific test suites by
 (un)commenting their call from the main and unique unittest block.
 Else, either they all run, or none. During development, I only keep
 active the test func(s) relative to the feature I'm currently working
 on. Remedy: named unittests.

The interesting thing about named unit tests is that their names aren't interesting at all. They are usually dull and forced; testing filterFoo will be called testFilterFoo, etc. Their only purpose is to suppress running of unrelated tests. =20 =20 Now, there is a seemingly unrelated proposal to include every ddoc'ed unit test in the preceding declaration as an example. This is great because it implies ownership -- a unit test is 'owned' by the symbol above. Going further, it can also be named after its owner. =20 module ooh; =20 void foo(); =20 unittest { test foo... } =20 Compiling with --unittest=3Dooh.foo runs this unittest only. Nested control as a bonus: compiling with --unittest=3Dooh runs only the tests in module ooh. =20 So there you go, named unit tests without naming.

The number one reason that I want named unit tests is stack traces. unittest428 or whatever a unittest block gets named to is pretty useless. So, while an assert within the unit test is fine, since it gives a file and line number which is in the unittest block, exceptions thrown from stuff called by the unit test become hard to track down, since you don't know which unit test was being run.

Sounds as though a solution would be to have unittests automatically named as "_unittest" ~ __LINE__ rather than an arbitrary integer. Wouldn't work in the case where there are two unittests on the same line: unittest { assert(true); } unittest { assert(true); } But we can deal with that.

I still think that having named unit tests is something that we should do a= t=20 some point (and it would be nicely backwards compatible), but naming the un= it=20 test function after the line number that it's on would already be a big=20 improvement. I have no idea how they're named/numbered now. I haven't been able to disce= rn=20 any obvious pattern, and with a module like std.datetime, which has lots of= =20 unittest blocks, even if it were clearly unittest1, unittest2, unittest3, e= tc.=20 it would still be quite hard to figure out which unittest block was running= when=20 the exception threw, since you'd have to count them all. Using __LINE__ would be a big improvement - though unless we figure out som= e way=20 of advertising that, most people probably wouldn't have a clue, and it woul= dn't=20 help them much. It would definitely be a positive change though. And since if someone is bizarre enough to put more than one unittest block = on=20 one line, well then they suffer for what is almost certainly a poor choice.= But=20 it would still narrow it down for them. =2D Jonathan M Davis
Feb 14 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday 15 February 2011 00:46:25 spir wrote:
 On 02/15/2011 02:42 AM, Jonathan M Davis wrote:
 On Monday, February 14, 2011 17:26:26 Don wrote:
 Jonathan M Davis wrote:
 On Monday, February 14, 2011 12:49:11 Tomek Sowi=C5=84ski wrote:
 spir napisa=C5=82:
 * Why isn't testList a unittest block?
=20
 Using named funcs, I can switch on&  off specific test suites by
 (un)commenting their call from the main and unique unittest block.
 Else, either they all run, or none. During development, I only keep
 active the test func(s) relative to the feature I'm currently worki=






 on. Remedy: named unittests.

The interesting thing about named unit tests is that their names aren't interesting at all. They are usually dull and forced; testing filterFoo will be called testFilterFoo, etc. Their only purpose is to suppress running of unrelated tests. =20 =20 Now, there is a seemingly unrelated proposal to include every ddoc'ed unit test in the preceding declaration as an example. This is great because it implies ownership -- a unit test is 'owned' by the symbol above. Going further, it can also be named after its owner. =20 module ooh; =20 void foo(); =20 unittest { test foo... } =20 Compiling with --unittest=3Dooh.foo runs this unittest only. Nested control as a bonus: compiling with --unittest=3Dooh runs only the te=





 in module ooh.
=20
 So there you go, named unit tests without naming.

The number one reason that I want named unit tests is stack traces. unittest428 or whatever a unittest block gets named to is pretty useless. So, while an assert within the unit test is fine, since it gives a file and line number which is in the unittest block, exceptio=




 thrown from stuff called by the unit test become hard to track down,
 since you don't know which unit test was being run.

Sounds as though a solution would be to have unittests automatically named as "_unittest" ~ __LINE__ rather than an arbitrary integer. Wouldn't work in the case where there are two unittests on the same line: unittest { assert(true); } unittest { assert(true); } But we can deal with that.

I still think that having named unit tests is something that we should =


 at some point (and it would be nicely backwards compatible), but naming
 the unit test function after the line number that it's on would already
 be a big improvement.
=20
 I have no idea how they're named/numbered now. I haven't been able to
 discern any obvious pattern, and with a module like std.datetime, which
 has lots of unittest blocks, even if it were clearly unittest1,
 unittest2, unittest3, etc. it would still be quite hard to figure out
 which unittest block was running when the exception threw, since you'd
 have to count them all.
=20
 Using __LINE__ would be a big improvement - though unless we figure out
 some way of advertising that, most people probably wouldn't have a clue,
 and it wouldn't help them much. It would definitely be a positive change
 though.
=20
 And since if someone is bizarre enough to put more than one unittest
 block on one line, well then they suffer for what is almost certainly a
 poor choice. But it would still narrow it down for them.

The only /meaningful/ choice for a programmer, in the general case, is programmer-chosen names. Don't you think? One can then even nicely use those names like eg "see example use in unittest ApplicationToArithmetics". If people are happy with naming unittests using ordinals (1 2 3...) or line numbers (in realease), well... let them free, after all it's their choice :-)

I'm looking for usable stack traces from unit test blocks. Ideally, that wo= uld=20 be with named unit tests, but that's a language change which may not happen= for=20 a while, if ever. Having the unittest block functions named with __LINE__ b= eing=20 used to generate the numeric part of their name would at least give me the= =20 information that I need to use stack traces from exceptions that went throu= gh a=20 unit test block. So, while named unit tests would be nice, the __LINE__ bit= =20 would be a big help. =2D Jonathan M Davis
Feb 15 2011
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.NOSPAMnet> writes:
On Sun, 13 Feb 2011 23:03:06 +0100, spir wrote:

 1. Named unittests allowing test suites in the form of (just an
 example):
 
 unittest test1 {
      ...
 }
 unittest test2 {
      ...
 }
 unittest test3 {
      ...
 }
 unittest {
      test1;
      test2;
      test3;
 }
 
 /Unnamed/ unittests are run with --unittest. Named ones are intended to
 be called from unnamed ones. Backward compatible change.

Works now: version(unittest) void test1() { ... } version(unittest) void test2() { ... } version(unittest) void test3() { ... } unittest { test1(); test2(); test3(); } -Lars
Feb 15 2011
prev sibling parent spir <denis.spir gmail.com> writes:
On 02/15/2011 10:00 AM, Lars T. Kyllingstad wrote:
 On Sun, 13 Feb 2011 23:03:06 +0100, spir wrote:

 1. Named unittests allowing test suites in the form of (just an
 example):

 unittest test1 {
       ...
 }
 unittest test2 {
       ...
 }
 unittest test3 {
       ...
 }
 unittest {
       test1;
       test2;
       test3;
 }

 /Unnamed/ unittests are run with --unittest. Named ones are intended to
 be called from unnamed ones. Backward compatible change.

Works now: version(unittest) void test1() { ... } version(unittest) void test2() { ... } version(unittest) void test3() { ... } unittest { test1(); test2(); test3(); }

Agreed, that's about what I do. The whole point is to have it as a standard feature of the language. You don't need the current 'unittest' feature to write unittests, do you? I doesn't even solve typing (or so few?). But as stated by Walter, having it as a builtin feature makes the difference in D code beeing commonly unit-tested. denis -- _________________ vita es estrany spir.wikidot.com
Feb 15 2011