www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Hack to name unit tests?

reply Robert Fraser <fraserofthenight gmail.com> writes:
I'm creating a Flectioned-based way to run unit tests individually 
(it'll be hooked into a front-end, so you can run test suites, repeat 
only failed tests after a code change, etc.)

However, I have run into a problem: unit tests can't be named. Right now 
I have a signature system, but I was wondering if there's any hackish 
way to predictably insert a string into the generated object file in a 
way that users can name unittests. That is to say, given a function 
pointer, is there any way I can scan the code to see if a name has been 
given to the unit test, and if so extract that name at runtime?

Ideally, it'd be as easy for the user as something like:

unittest
{
    assert(":testName:");
    // Rest of test goes here...
}

Right now, I have the associations being done in the front-end I'm 
working on, that scans the code for comments in a particular format and 
associates those with the unit tests. However, that ties the unit test 
executor back-end to the code analysis front-end, so I was hoping 
there's a better way.

Thanks!
Nov 28 2007
next sibling parent reply Dan <murpsoft hotmail.com> writes:
Robert Fraser Wrote:

 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)

I would have recommended you approach Walter and his team (whomever they may be nowadays) with the idea.
 Ideally, it'd be as easy for the user as something like:
 
 unittest
 {
     assert(":testName:");
     // Rest of test goes here...
 }

I'd like to see: unittest testName { // rest of text } I'd then like to be able to specifically execute only specific unit test(s) so that for instance I can debug the relationship between my x and y modules without triggering off a test of the whole freakin' alphabet. I would then like an improvement of unittests so that one can do more than merely assert(). Complete self-examination code should be possible - which means being able to examine program flow and trace variables. This would ultimately lead to *being able to* generate proofs on the correctness of the code. Someone could then just write a library.
Nov 28 2007
next sibling parent Leandro Lucarella <llucax gmail.com> writes:
Dan, el 28 de noviembre a las 18:20 me escribiste:
 Robert Fraser Wrote:
 
 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)

I would have recommended you approach Walter and his team (whomever they may be nowadays) with the idea.
 Ideally, it'd be as easy for the user as something like:
 
 unittest
 {
     assert(":testName:");
     // Rest of test goes here...
 }

I'd like to see: unittest testName { // rest of text }

Maybe something like unittest "test name" { // rest of the test } could be more flexible about naming =) -- Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/ ---------------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------------- You should've seen her face. It was the exact same look my father gave me when I told him I wanted to be a ventriloquist. -- George Constanza
Nov 28 2007
prev sibling next sibling parent Robert Fraser <fraserofthenight gmail.com> writes:
Dan wrote:
 I'd then like to be able to specifically execute only specific unit test(s) so
that for instance I can debug the relationship between my x and y modules
without triggering off a test of the whole freakin' alphabet.
 

Besides the naming, all that is possible right now, which is what my tool is doing.
 I would then like an improvement of unittests so that one can do more than
merely assert().  Complete self-examination code should be possible - which
means being able to examine program flow and trace variables.  This would
ultimately lead to *being able to* generate proofs on the correctness of the
code.
 
 Someone could then just write a library.

Hmmm... that sounds a bit complex for an average tester. Might be a good idea in a pure-functional language, but I can't see it being much use in D.
Nov 28 2007
prev sibling parent =?iso-8859-15?Q?Jari-Matti_M=E4kel=E4?= <jmjmak utu.fi.invalid> writes:
On Wed, 28 Nov 2007, Robert Fraser wrote:

 Dan wrote:
 I'd then like to be able to specifically execute only specific unit test(s) 
 so that for instance I can debug the relationship between my x and y 
 modules without triggering off a test of the whole freakin' alphabet.
 

Besides the naming, all that is possible right now, which is what my tool is doing.
 I would then like an improvement of unittests so that one can do more than 
 merely assert().  Complete self-examination code should be possible - which 
 means being able to examine program flow and trace variables.  This would 
 ultimately lead to *being able to* generate proofs on the correctness of 
 the code.
 
 Someone could then just write a library.

Hmmm... that sounds a bit complex for an average tester. Might be a good idea in a pure-functional language, but I can't see it being much use in D.

Yep, it's a bit difficult to prove the correctness with unittests because of side effects. Also 100% test coverage won't mean it works perfectly. Hmm, what about unittest as an attribute / annotation for functions. This works pretty well in Java and is more flexible than the one in D.
Nov 29 2007
prev sibling next sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Robert Fraser wrote:
 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)
 
 However, I have run into a problem: unit tests can't be named. Right now 
 I have a signature system, but I was wondering if there's any hackish 
 way to predictably insert a string into the generated object file in a 
 way that users can name unittests. That is to say, given a function 
 pointer, is there any way I can scan the code to see if a name has been 
 given to the unit test, and if so extract that name at runtime?
 
 Ideally, it'd be as easy for the user as something like:
 
 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }
 
 Right now, I have the associations being done in the front-end I'm 
 working on, that scans the code for comments in a particular format and 
 associates those with the unit tests. However, that ties the unit test 
 executor back-end to the code analysis front-end, so I was hoping 
 there's a better way.
 
 Thanks!

class Unittest (string _name, alias _dg) : IUnittest { static string name = _name; static void delegate() test = _dg; } Usage: Unittest!("my test for wossname", { assert (false, "haven't gotten around to implementing this yet"); }); Not guaranteed to work. Another tactic would be something like: class UnittestManager { static void registerCurrentTest (string name) {} static void endTest (bool success) {} static bool performTest () {} } template Unittest (string _name, alias _dg) { unittest { UnittestManager.registerCurrentTest(_name); // This lets us skip the test if we're just trying // to find out what tests there are currently. if (UnittestManager.performTest) { scope(success) UnittestManager.endTest(true); scope(failure) UnittestManager.endTest(false); _dg(); } } } mixin Unittest!("my test name", { assert (false, "not yet implemented"); });
Nov 28 2007
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Christopher Wright wrote:
 Robert Fraser wrote:
 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. Right 
 now I have a signature system, but I was wondering if there's any 
 hackish way to predictably insert a string into the generated object 
 file in a way that users can name unittests. That is to say, given a 
 function pointer, is there any way I can scan the code to see if a 
 name has been given to the unit test, and if so extract that name at 
 runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm 
 working on, that scans the code for comments in a particular format 
 and associates those with the unit tests. However, that ties the unit 
 test executor back-end to the code analysis front-end, so I was hoping 
 there's a better way.

 Thanks!

class Unittest (string _name, alias _dg) : IUnittest { static string name = _name; static void delegate() test = _dg; } Usage: Unittest!("my test for wossname", { assert (false, "haven't gotten around to implementing this yet"); }); Not guaranteed to work. Another tactic would be something like: class UnittestManager { static void registerCurrentTest (string name) {} static void endTest (bool success) {} static bool performTest () {} } template Unittest (string _name, alias _dg) { unittest { UnittestManager.registerCurrentTest(_name); // This lets us skip the test if we're just trying // to find out what tests there are currently. if (UnittestManager.performTest) { scope(success) UnittestManager.endTest(true); scope(failure) UnittestManager.endTest(false); _dg(); } } } mixin Unittest!("my test name", { assert (false, "not yet implemented"); });

Thanks! That's a good idea, but I want something compatible with current unittest {} declarations.
Nov 28 2007
parent reply Christopher Wright <dhasenan gmail.com> writes:
Robert Fraser wrote:
 Christopher Wright wrote:
 Robert Fraser wrote:
 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. Right 
 now I have a signature system, but I was wondering if there's any 
 hackish way to predictably insert a string into the generated object 
 file in a way that users can name unittests. That is to say, given a 
 function pointer, is there any way I can scan the code to see if a 
 name has been given to the unit test, and if so extract that name at 
 runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm 
 working on, that scans the code for comments in a particular format 
 and associates those with the unit tests. However, that ties the unit 
 test executor back-end to the code analysis front-end, so I was 
 hoping there's a better way.

 Thanks!

class Unittest (string _name, alias _dg) : IUnittest { static string name = _name; static void delegate() test = _dg; } Usage: Unittest!("my test for wossname", { assert (false, "haven't gotten around to implementing this yet"); }); Not guaranteed to work. Another tactic would be something like: class UnittestManager { static void registerCurrentTest (string name) {} static void endTest (bool success) {} static bool performTest () {} } template Unittest (string _name, alias _dg) { unittest { UnittestManager.registerCurrentTest(_name); // This lets us skip the test if we're just trying // to find out what tests there are currently. if (UnittestManager.performTest) { scope(success) UnittestManager.endTest(true); scope(failure) UnittestManager.endTest(false); _dg(); } } } mixin Unittest!("my test name", { assert (false, "not yet implemented"); });

Thanks! That's a good idea, but I want something compatible with current unittest {} declarations.

You already have to modify the tests to insert the name, but you want something that works as a decorator inside the unittest. Can your unittest runner do a try/catch on a unittest? Then you could have something like: void testName(string name) { if (collectingNames) throw new UnittestNameException(name); } struct NamedTest { Unittest test; string name; } ... void main () { NamedTest[] tests; collectingNames = true; foreach (test; getUnittests) { try { test(); } catch (UnittestNameException e) { tests ~= NamedTest(test, e.name); } } collectingNames = false; } unittest { testName("blargh"); assert (false, "still working on this"); }
Nov 29 2007
next sibling parent reply Dan <murpsoft hotmail.com> writes:
Christopher Wright Wrote:

 Robert Fraser wrote:
 Christopher Wright wrote:
 Robert Fraser wrote:
 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. Right 
 now I have a signature system, but I was wondering if there's any 
 hackish way to predictably insert a string into the generated object 
 file in a way that users can name unittests. That is to say, given a 
 function pointer, is there any way I can scan the code to see if a 
 name has been given to the unit test, and if so extract that name at 
 runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm 
 working on, that scans the code for comments in a particular format 
 and associates those with the unit tests. However, that ties the unit 
 test executor back-end to the code analysis front-end, so I was 
 hoping there's a better way.

 Thanks!

class Unittest (string _name, alias _dg) : IUnittest { static string name = _name; static void delegate() test = _dg; } Usage: Unittest!("my test for wossname", { assert (false, "haven't gotten around to implementing this yet"); }); Not guaranteed to work. Another tactic would be something like: class UnittestManager { static void registerCurrentTest (string name) {} static void endTest (bool success) {} static bool performTest () {} } template Unittest (string _name, alias _dg) { unittest { UnittestManager.registerCurrentTest(_name); // This lets us skip the test if we're just trying // to find out what tests there are currently. if (UnittestManager.performTest) { scope(success) UnittestManager.endTest(true); scope(failure) UnittestManager.endTest(false); _dg(); } } } mixin Unittest!("my test name", { assert (false, "not yet implemented"); });

Thanks! That's a good idea, but I want something compatible with current unittest {} declarations.

You already have to modify the tests to insert the name, but you want something that works as a decorator inside the unittest. Can your unittest runner do a try/catch on a unittest? Then you could have something like: void testName(string name) { if (collectingNames) throw new UnittestNameException(name); } struct NamedTest { Unittest test; string name; } ... void main () { NamedTest[] tests; collectingNames = true; foreach (test; getUnittests) { try { test(); } catch (UnittestNameException e) { tests ~= NamedTest(test, e.name); } } collectingNames = false; } unittest { testName("blargh"); assert (false, "still working on this"); }

Eww... no. You guys are suggesting that code provability is useless in D because it's "too complicated" and then go on to suggest it can be used as a heavily over-classed debug statement. unittest is currently used more or less as a place to put a barrage of random sample tests. I was suggesting (beyond the obvious improvement of naming and only executing select unittests) that we should be able to put hooks to a provability program here. The provability program would be a separate utility that would go through and show things like bit overflow potential in numbers, potential infinities or potential bad memory accesses. It would do these things by following variables throughout their lifetimes to show all the different values they could possibly be at each time they're used. Combined with contract programming (in/out and invariant) this kind of thing exposes code in a way that running it with test values simply can't. For example: In libc, in qsort(), there was a bug because: median = (low + high) >> 1; The problem here is that in a very large array, low + high will eventually be > MAX_INT. This kind of bug isn't detected by simply doing a dumb random test on an array of 10. You need to be aware of the full range of possible values and use inductive logic to predict the bug without testing every last case. Regards, Dan
Nov 29 2007
next sibling parent Christopher Wright <dhasenan gmail.com> writes:
Dan wrote:
 Eww... no.  You guys are suggesting that code provability is useless in D
because it's "too complicated" and then go on to suggest it can be used as a
heavily over-classed debug statement.

I don't know what thread you meant to reply to. This is only about naming tests somehow, not about proving anything about test coverage. Or am I confused about something?
Nov 29 2007
prev sibling parent Robert Fraser <fraserofthenight gmail.com> writes:
Dan wrote:
 Christopher Wright Wrote:
 
 Robert Fraser wrote:
 Christopher Wright wrote:
 Robert Fraser wrote:
 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. Right 
 now I have a signature system, but I was wondering if there's any 
 hackish way to predictably insert a string into the generated object 
 file in a way that users can name unittests. That is to say, given a 
 function pointer, is there any way I can scan the code to see if a 
 name has been given to the unit test, and if so extract that name at 
 runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm 
 working on, that scans the code for comments in a particular format 
 and associates those with the unit tests. However, that ties the unit 
 test executor back-end to the code analysis front-end, so I was 
 hoping there's a better way.

 Thanks!

static string name = _name; static void delegate() test = _dg; } Usage: Unittest!("my test for wossname", { assert (false, "haven't gotten around to implementing this yet"); }); Not guaranteed to work. Another tactic would be something like: class UnittestManager { static void registerCurrentTest (string name) {} static void endTest (bool success) {} static bool performTest () {} } template Unittest (string _name, alias _dg) { unittest { UnittestManager.registerCurrentTest(_name); // This lets us skip the test if we're just trying // to find out what tests there are currently. if (UnittestManager.performTest) { scope(success) UnittestManager.endTest(true); scope(failure) UnittestManager.endTest(false); _dg(); } } } mixin Unittest!("my test name", { assert (false, "not yet implemented"); });

unittest {} declarations.

something that works as a decorator inside the unittest. Can your unittest runner do a try/catch on a unittest? Then you could have something like: void testName(string name) { if (collectingNames) throw new UnittestNameException(name); } struct NamedTest { Unittest test; string name; } ... void main () { NamedTest[] tests; collectingNames = true; foreach (test; getUnittests) { try { test(); } catch (UnittestNameException e) { tests ~= NamedTest(test, e.name); } } collectingNames = false; } unittest { testName("blargh"); assert (false, "still working on this"); }

Eww... no. You guys are suggesting that code provability is useless in D because it's "too complicated" and then go on to suggest it can be used as a heavily over-classed debug statement. unittest is currently used more or less as a place to put a barrage of random sample tests. I was suggesting (beyond the obvious improvement of naming and only executing select unittests) that we should be able to put hooks to a provability program here. The provability program would be a separate utility that would go through and show things like bit overflow potential in numbers, potential infinities or potential bad memory accesses. It would do these things by following variables throughout their lifetimes to show all the different values they could possibly be at each time they're used. Combined with contract programming (in/out and invariant) this kind of thing exposes code in a way that running it with test values simply can't. For example: In libc, in qsort(), there was a bug because: median = (low + high) >> 1; The problem here is that in a very large array, low + high will eventually be > MAX_INT. This kind of bug isn't detected by simply doing a dumb random test on an array of 10. You need to be aware of the full range of possible values and use inductive logic to predict the bug without testing every last case. Regards, Dan

It's not a bad idea, especially for smaller algorithms, but from my experience, using Coverity on a ~18 million LOC Java project fixed about 10 bugs, while the unit tests were constantly auditing check-ins, and prevented me from checking in at least that many of my own.
Nov 29 2007
prev sibling parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Christopher Wright wrote:
 Robert Fraser wrote:
 Christopher Wright wrote:
 Robert Fraser wrote:
 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, 
 repeat only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. Right 
 now I have a signature system, but I was wondering if there's any 
 hackish way to predictably insert a string into the generated object 
 file in a way that users can name unittests. That is to say, given a 
 function pointer, is there any way I can scan the code to see if a 
 name has been given to the unit test, and if so extract that name at 
 runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm 
 working on, that scans the code for comments in a particular format 
 and associates those with the unit tests. However, that ties the 
 unit test executor back-end to the code analysis front-end, so I was 
 hoping there's a better way.

 Thanks!

class Unittest (string _name, alias _dg) : IUnittest { static string name = _name; static void delegate() test = _dg; } Usage: Unittest!("my test for wossname", { assert (false, "haven't gotten around to implementing this yet"); }); Not guaranteed to work. Another tactic would be something like: class UnittestManager { static void registerCurrentTest (string name) {} static void endTest (bool success) {} static bool performTest () {} } template Unittest (string _name, alias _dg) { unittest { UnittestManager.registerCurrentTest(_name); // This lets us skip the test if we're just trying // to find out what tests there are currently. if (UnittestManager.performTest) { scope(success) UnittestManager.endTest(true); scope(failure) UnittestManager.endTest(false); _dg(); } } } mixin Unittest!("my test name", { assert (false, "not yet implemented"); });

Thanks! That's a good idea, but I want something compatible with current unittest {} declarations.

You already have to modify the tests to insert the name, but you want something that works as a decorator inside the unittest. Can your unittest runner do a try/catch on a unittest? Then you could have something like: void testName(string name) { if (collectingNames) throw new UnittestNameException(name); } struct NamedTest { Unittest test; string name; } ... void main () { NamedTest[] tests; collectingNames = true; foreach (test; getUnittests) { try { test(); } catch (UnittestNameException e) { tests ~= NamedTest(test, e.name); } } collectingNames = false; } unittest { testName("blargh"); assert (false, "still working on this"); }

Nov 29 2007
parent reply Christopher Wright <dhasenan gmail.com> writes:
Robert Fraser wrote:
 Christopher Wright wrote:
 Robert Fraser wrote:
 Christopher Wright wrote:
 Robert Fraser wrote:
 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, 
 repeat only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. 
 Right now I have a signature system, but I was wondering if there's 
 any hackish way to predictably insert a string into the generated 
 object file in a way that users can name unittests. That is to say, 
 given a function pointer, is there any way I can scan the code to 
 see if a name has been given to the unit test, and if so extract 
 that name at runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm 
 working on, that scans the code for comments in a particular format 
 and associates those with the unit tests. However, that ties the 
 unit test executor back-end to the code analysis front-end, so I 
 was hoping there's a better way.

 Thanks!

class Unittest (string _name, alias _dg) : IUnittest { static string name = _name; static void delegate() test = _dg; } Usage: Unittest!("my test for wossname", { assert (false, "haven't gotten around to implementing this yet"); }); Not guaranteed to work. Another tactic would be something like: class UnittestManager { static void registerCurrentTest (string name) {} static void endTest (bool success) {} static bool performTest () {} } template Unittest (string _name, alias _dg) { unittest { UnittestManager.registerCurrentTest(_name); // This lets us skip the test if we're just trying // to find out what tests there are currently. if (UnittestManager.performTest) { scope(success) UnittestManager.endTest(true); scope(failure) UnittestManager.endTest(false); _dg(); } } } mixin Unittest!("my test name", { assert (false, "not yet implemented"); });

Thanks! That's a good idea, but I want something compatible with current unittest {} declarations.

You already have to modify the tests to insert the name, but you want something that works as a decorator inside the unittest. Can your unittest runner do a try/catch on a unittest? Then you could have something like: void testName(string name) { if (collectingNames) throw new UnittestNameException(name); } struct NamedTest { Unittest test; string name; } ... void main () { NamedTest[] tests; collectingNames = true; foreach (test; getUnittests) { try { test(); } catch (UnittestNameException e) { tests ~= NamedTest(test, e.name); } } collectingNames = false; } unittest { testName("blargh"); assert (false, "still working on this"); }


No worries. It's pretty much identical in principle to mocked methods in dmocks, except for using exceptions to pass the name around and abort the method.
Nov 29 2007
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Christopher Wright wrote:
 Robert Fraser wrote:
 Christopher Wright wrote:
 Robert Fraser wrote:
 Christopher Wright wrote:
 Robert Fraser wrote:
 I'm creating a Flectioned-based way to run unit tests individually
 (it'll be hooked into a front-end, so you can run test suites,
 repeat only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named.
 Right now I have a signature system, but I was wondering if
 there's any hackish way to predictably insert a string into the
 generated object file in a way that users can name unittests. That
 is to say, given a function pointer, is there any way I can scan
 the code to see if a name has been given to the unit test, and if
 so extract that name at runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm
 working on, that scans the code for comments in a particular
 format and associates those with the unit tests. However, that
 ties the unit test executor back-end to the code analysis
 front-end, so I was hoping there's a better way.

 Thanks!

class Unittest (string _name, alias _dg) : IUnittest { static string name = _name; static void delegate() test = _dg; } Usage: Unittest!("my test for wossname", { assert (false, "haven't gotten around to implementing this yet"); }); Not guaranteed to work. Another tactic would be something like: class UnittestManager { static void registerCurrentTest (string name) {} static void endTest (bool success) {} static bool performTest () {} } template Unittest (string _name, alias _dg) { unittest { UnittestManager.registerCurrentTest(_name); // This lets us skip the test if we're just trying // to find out what tests there are currently. if (UnittestManager.performTest) { scope(success) UnittestManager.endTest(true); scope(failure) UnittestManager.endTest(false); _dg(); } } } mixin Unittest!("my test name", { assert (false, "not yet implemented"); });

Thanks! That's a good idea, but I want something compatible with current unittest {} declarations.

You already have to modify the tests to insert the name, but you want something that works as a decorator inside the unittest. Can your unittest runner do a try/catch on a unittest? Then you could have something like: void testName(string name) { if (collectingNames) throw new UnittestNameException(name); } struct NamedTest { Unittest test; string name; } ... void main () { NamedTest[] tests; collectingNames = true; foreach (test; getUnittests) { try { test(); } catch (UnittestNameException e) { tests ~= NamedTest(test, e.name); } } collectingNames = false; } unittest { testName("blargh"); assert (false, "still working on this"); }


No worries. It's pretty much identical in principle to mocked methods in dmocks, except for using exceptions to pass the name around and abort the method.

Wouldn't it be simpler to provide an exern method that unittests could call into? extern(C) void testName(char[] name); unittest { testName("blargh"); assert(false, "too lazy to write a real unit test..."); } You then provide testName somewhere in your runner program. If you want to compile without the unittest frontend, just create a stub object file with this in it: extern(C) void testName(char[] name) {} Should be a hell of a lot faster than tossing exceptions willy-nilly. -- Daniel
Nov 29 2007
parent reply Christopher Wright <dhasenan gmail.com> writes:
Daniel Keep wrote:
 
 Christopher Wright wrote:
 Robert Fraser wrote:
 Christopher Wright wrote:
 Robert Fraser wrote:
 Christopher Wright wrote:
 Robert Fraser wrote:
 I'm creating a Flectioned-based way to run unit tests individually
 (it'll be hooked into a front-end, so you can run test suites,
 repeat only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named.
 Right now I have a signature system, but I was wondering if
 there's any hackish way to predictably insert a string into the
 generated object file in a way that users can name unittests. That
 is to say, given a function pointer, is there any way I can scan
 the code to see if a name has been given to the unit test, and if
 so extract that name at runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

 Right now, I have the associations being done in the front-end I'm
 working on, that scans the code for comments in a particular
 format and associates those with the unit tests. However, that
 ties the unit test executor back-end to the code analysis
 front-end, so I was hoping there's a better way.

 Thanks!

static string name = _name; static void delegate() test = _dg; } Usage: Unittest!("my test for wossname", { assert (false, "haven't gotten around to implementing this yet"); }); Not guaranteed to work. Another tactic would be something like: class UnittestManager { static void registerCurrentTest (string name) {} static void endTest (bool success) {} static bool performTest () {} } template Unittest (string _name, alias _dg) { unittest { UnittestManager.registerCurrentTest(_name); // This lets us skip the test if we're just trying // to find out what tests there are currently. if (UnittestManager.performTest) { scope(success) UnittestManager.endTest(true); scope(failure) UnittestManager.endTest(false); _dg(); } } } mixin Unittest!("my test name", { assert (false, "not yet implemented"); });

current unittest {} declarations.

something that works as a decorator inside the unittest. Can your unittest runner do a try/catch on a unittest? Then you could have something like: void testName(string name) { if (collectingNames) throw new UnittestNameException(name); } struct NamedTest { Unittest test; string name; } ... void main () { NamedTest[] tests; collectingNames = true; foreach (test; getUnittests) { try { test(); } catch (UnittestNameException e) { tests ~= NamedTest(test, e.name); } } collectingNames = false; } unittest { testName("blargh"); assert (false, "still working on this"); }


dmocks, except for using exceptions to pass the name around and abort the method.

Wouldn't it be simpler to provide an exern method that unittests could call into? extern(C) void testName(char[] name); unittest { testName("blargh"); assert(false, "too lazy to write a real unit test..."); } You then provide testName somewhere in your runner program. If you want to compile without the unittest frontend, just create a stub object file with this in it: extern(C) void testName(char[] name) {} Should be a hell of a lot faster than tossing exceptions willy-nilly. -- Daniel

There are advantages to throwing an exception when registering the name: unittest { testName("this is the test that doesn't end"); while (true) {} } I was thinking that AssertionErrors couldn't be caught, but that was an error on my part.
Nov 29 2007
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Christopher Wright wrote:
 ...
 There are advantages to throwing an exception when registering the name:
 
 unittest {
    testName("this is the test that doesn't end");
    while (true) {}
 }
 
 I was thinking that AssertionErrors couldn't be caught, but that was an
 error on my part.

How is an exception an advantage in that case? When you run the test, it's still going to spinlock, no matter how you've obtained the name. If what you're trying to get at is that you can build a list of all the names without having to run the actual test itself, then yes, I can see that being an advantage. -- Daniel
Nov 29 2007
parent reply Christopher Wright <dhasenan gmail.com> writes:
Daniel Keep wrote:
 
 Christopher Wright wrote:
 ...
 There are advantages to throwing an exception when registering the name:

 unittest {
    testName("this is the test that doesn't end");
    while (true) {}
 }

 I was thinking that AssertionErrors couldn't be caught, but that was an
 error on my part.

How is an exception an advantage in that case? When you run the test, it's still going to spinlock, no matter how you've obtained the name. If what you're trying to get at is that you can build a list of all the names without having to run the actual test itself, then yes, I can see that being an advantage.

Exactly, exactly! Precisely what I told Miss Chow. Tissue augmentation, it's not a matter of twiddling bits. On a more serious note, tools like the XUnit test runners display a list of names without running the tests. That's convenient, I think. The NUnit gui takes about twenty seconds to find all the unit tests we have at work, and it takes two or three minutes to run the tests. If I want to check a single test, two or three minutes is a long time.
 	-- Daniel

Nov 30 2007
parent Robert Fraser <fraserofthenight gmail.com> writes:
Christopher Wright wrote:
 Daniel Keep wrote:
 Christopher Wright wrote:
 ...
 There are advantages to throwing an exception when registering the name:

 unittest {
    testName("this is the test that doesn't end");
    while (true) {}
 }

 I was thinking that AssertionErrors couldn't be caught, but that was an
 error on my part.

How is an exception an advantage in that case? When you run the test, it's still going to spinlock, no matter how you've obtained the name. If what you're trying to get at is that you can build a list of all the names without having to run the actual test itself, then yes, I can see that being an advantage.

Exactly, exactly! Precisely what I told Miss Chow. Tissue augmentation, it's not a matter of twiddling bits. On a more serious note, tools like the XUnit test runners display a list of names without running the tests. That's convenient, I think. The NUnit gui takes about twenty seconds to find all the unit tests we have at work, and it takes two or three minutes to run the tests. If I want to check a single test, two or three minutes is a long time.
     -- Daniel


Check Don's post in this topic for a way to get test names w/o runnning them at all (and depending on them throwing the exception; that is, unnamed tests will still work correctly). I'm going to be implementing that... assuming I can get the free time & sleep required for that.
Nov 30 2007
prev sibling parent reply Don Clugston <dac nospam.com.au> writes:
Robert Fraser wrote:
 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)
 
 However, I have run into a problem: unit tests can't be named. Right now 
 I have a signature system, but I was wondering if there's any hackish 
 way to predictably insert a string into the generated object file in a 
 way that users can name unittests. That is to say, given a function 
 pointer, is there any way I can scan the code to see if a name has been 
 given to the unit test, and if so extract that name at runtime?
 
 Ideally, it'd be as easy for the user as something like:
 
 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

I think this is possible, but tricky. If your unit test has a local variable, and you instantiate a template using that local variable as an alias parameter, the module and the number of the unit test gets included into the mangled name. This means that it's possible for code to know which unit test it is inside. By making the template a class, you could to walk the RTTI list and make a lookup table to associate test numbers with their names. If you can get the list of unittest numbers from flectioned, you can join it BUT... that's all pretty hairy. Maybe there's a less convoluted way of doing it.
Nov 29 2007
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Don Clugston wrote:
 Robert Fraser wrote:
 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. Right 
 now I have a signature system, but I was wondering if there's any 
 hackish way to predictably insert a string into the generated object 
 file in a way that users can name unittests. That is to say, given a 
 function pointer, is there any way I can scan the code to see if a 
 name has been given to the unit test, and if so extract that name at 
 runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

I think this is possible, but tricky. If your unit test has a local variable, and you instantiate a template using that local variable as an alias parameter, the module and the number of the unit test gets included into the mangled name. This means that it's possible for code to know which unit test it is inside. By making the template a class, you could to walk the RTTI list and make a lookup table to associate test numbers with their names. If you can get the list of unittest numbers from flectioned, you can join it BUT... that's all pretty hairy. Maybe there's a less convoluted way of doing it.

run that way... time to get hacking!
Nov 30 2007
parent Don Clugston <dac nospam.com.au> writes:
Robert Fraser wrote:
 Don Clugston wrote:
 Robert Fraser wrote:
 I'm creating a Flectioned-based way to run unit tests individually 
 (it'll be hooked into a front-end, so you can run test suites, repeat 
 only failed tests after a code change, etc.)

 However, I have run into a problem: unit tests can't be named. Right 
 now I have a signature system, but I was wondering if there's any 
 hackish way to predictably insert a string into the generated object 
 file in a way that users can name unittests. That is to say, given a 
 function pointer, is there any way I can scan the code to see if a 
 name has been given to the unit test, and if so extract that name at 
 runtime?

 Ideally, it'd be as easy for the user as something like:

 unittest
 {
    assert(":testName:");
    // Rest of test goes here...
 }

I think this is possible, but tricky. If your unit test has a local variable, and you instantiate a template using that local variable as an alias parameter, the module and the number of the unit test gets included into the mangled name. This means that it's possible for code to know which unit test it is inside. By making the template a class, you could to walk the RTTI list and make a lookup table to associate test numbers with their names. If you can get the list of unittest numbers from flectioned, you can join it BUT... that's all pretty hairy. Maybe there's a less convoluted way of doing it.

run that way... time to get hacking!

Unfortunately, templated class don't show up in the normal class list accessable through moduleinit.d, so it's you'll need to use flectioned for that too. Here's the simplest example I came up with which displays the module and unit test number (taken from my now-defunct NameOf module -- hmmm, might be worth rescuscitating a couple of ideas from it; it's much easier now we have CTFE). It only works if you've defined a local variable to use as an alias, which is a bit annoying. --- template testname(alias F) { enum testname { ignore } } template TESTNAME(alias K) { typedef void function(testname!(K)) TESTNAME; } unittest { int foo; pragma(msg, TESTNAME!(foo).mangleof[3..$-13]); } ----------
Dec 01 2007