www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - unittests for inhereted classes

reply John R <John_member pathlink.com> writes:
Hello, I, like every good little java programmer, was raised on contracts, rep
invariants, and abstraction barriers, so naturally I like D. I've been playing
with it a bit and one of my unittests seems to be doing strange things. It is of
the form:

module FooClass;

public class FooClass {

unittest {
assert(FooClass.tick() == 0);
assert(FooClass.tick() == 1);
}

public static int count = 0;

public static int tick() {
return(FooClass.count++);
}

}

public class BarClass : FooClass{

unittest {
assert(BarClass.tick() == 0);		
}

}

The FooClass unittest runs fine, of course, but the BarClass test fails because
the variable has alread been incremented and is 2 when it gets to the Bar
unittest. If you move BarClass to the top of the file its unittest runs ok and
FooClass' fails.

I can only conclude that this implementation of unittest does not fully capture
the concept behind unit testing. My understanding has always been that a unit
test must not be dependent on other unittests. If it were, then a unittest of a
child class would not, in fact, be testing the child class, but the child class
with some arbitrary test code already run upon it. Then the child test may pass,
while giving no guarantee that the test code will work by itself in actual
program code.

I looked for other posts on this but I couldn't find any. Is this meant to be
this way? If this was a design choice, I might suggest that when class
hierarchies get large (I only had 4 levels of classes and I banged my head on
this bug for a while before I realized what was happening) independent unittests
get more important.

I haven't looked at the implementation (I'm not quite that brave), but couldn't
the program just sort of start with a clean slate for each unittest and not run
into this issue? Just a thought.

Thanks,
John R.
Feb 27 2005
next sibling parent Manfred Nowak <svv1999 hotmail.com> writes:
John R <John_member pathlink.com> wrote:

[...]
 My understanding has always been that a unit test must not be
 dependent on other unittests. 
[...] `unittest' is meant to contain code that tests the unit they are contained in. But I do not know of any semantically constraint that hinders the coder to test something completely different. In your example the code of the unittests depends on a public static variable, which can be changed at runtime from everywhere, and your testcase in the `unittest' is choosen in a way that points this out. So you can be lucky, that you had such golden fingers to choose your testcase right to convince you, that your code is vulnerable. Only your requirement that `unitest' should not point out dangerous code seems to be wrong. -manfred
Feb 27 2005
prev sibling parent reply John Demme <me teqdruid.com> writes:
Since you're using a static class variable, I'm relatively sure that a 
test like this would fail in Java using JUnit.  This is not a behavior 
that I think the compiler can help.  It would have to keep track of all 
the changes that happen as a result of the unittests, then reset them 
afterall.  I don't see it happening.

John (the other John)

John R wrote:
 Hello, I, like every good little java programmer, was raised on contracts, rep
 invariants, and abstraction barriers, so naturally I like D. I've been playing
 with it a bit and one of my unittests seems to be doing strange things. It is
of
 the form:
 
 module FooClass;
 
 public class FooClass {
 
 unittest {
 assert(FooClass.tick() == 0);
 assert(FooClass.tick() == 1);
 }
 
 public static int count = 0;
 
 public static int tick() {
 return(FooClass.count++);
 }
 
 }
 
 public class BarClass : FooClass{
 
 unittest {
 assert(BarClass.tick() == 0);		
 }
 
 }
 
 The FooClass unittest runs fine, of course, but the BarClass test fails because
 the variable has alread been incremented and is 2 when it gets to the Bar
 unittest. If you move BarClass to the top of the file its unittest runs ok and
 FooClass' fails.
 
 I can only conclude that this implementation of unittest does not fully capture
 the concept behind unit testing. My understanding has always been that a unit
 test must not be dependent on other unittests. If it were, then a unittest of a
 child class would not, in fact, be testing the child class, but the child class
 with some arbitrary test code already run upon it. Then the child test may
pass,
 while giving no guarantee that the test code will work by itself in actual
 program code.
 
 I looked for other posts on this but I couldn't find any. Is this meant to be
 this way? If this was a design choice, I might suggest that when class
 hierarchies get large (I only had 4 levels of classes and I banged my head on
 this bug for a while before I realized what was happening) independent
unittests
 get more important.
 
 I haven't looked at the implementation (I'm not quite that brave), but couldn't
 the program just sort of start with a clean slate for each unittest and not run
 into this issue? Just a thought.
 
 Thanks,
 John R.
 
 
Feb 27 2005
next sibling parent reply John Reimer <brk_6502 yahoo.com> writes:
You know what?  That post wasn't by me!

Darn it.  This name is just too common. :-(

- John R. (the other John R.!)


John Demme wrote:
 Since you're using a static class variable, I'm relatively sure that a 
 test like this would fail in Java using JUnit.  This is not a behavior 
 that I think the compiler can help.  It would have to keep track of all 
 the changes that happen as a result of the unittests, then reset them 
 afterall.  I don't see it happening.
 
 John (the other John)
 
 John R wrote:
 
 Hello, I, like every good little java programmer, was raised on 
 contracts, rep
 invariants, and abstraction barriers, so naturally I like D. I've been 
 playing
 with it a bit and one of my unittests seems to be doing strange 
 things. It is of
 the form:

 module FooClass;

 public class FooClass {

 unittest {
 assert(FooClass.tick() == 0);
 assert(FooClass.tick() == 1);
 }

 public static int count = 0;

 public static int tick() {
 return(FooClass.count++);
 }

 }

 public class BarClass : FooClass{

 unittest {
 assert(BarClass.tick() == 0);       
 }

 }

 The FooClass unittest runs fine, of course, but the BarClass test 
 fails because
 the variable has alread been incremented and is 2 when it gets to the Bar
 unittest. If you move BarClass to the top of the file its unittest 
 runs ok and
 FooClass' fails.

 I can only conclude that this implementation of unittest does not 
 fully capture
 the concept behind unit testing. My understanding has always been that 
 a unit
 test must not be dependent on other unittests. If it were, then a 
 unittest of a
 child class would not, in fact, be testing the child class, but the 
 child class
 with some arbitrary test code already run upon it. Then the child test 
 may pass,
 while giving no guarantee that the test code will work by itself in 
 actual
 program code.

 I looked for other posts on this but I couldn't find any. Is this 
 meant to be
 this way? If this was a design choice, I might suggest that when class
 hierarchies get large (I only had 4 levels of classes and I banged my 
 head on
 this bug for a while before I realized what was happening) independent 
 unittests
 get more important.

 I haven't looked at the implementation (I'm not quite that brave), but 
 couldn't
 the program just sort of start with a clean slate for each unittest 
 and not run
 into this issue? Just a thought.

 Thanks,
 John R.
Feb 27 2005
parent reply John Demme <me teqdruid.com> writes:
Heh.  OK... so from now on, to prevent confusion, we all must sign our 
messages as so:

Sincerely,
<Last Name>, <First Name> <Middle Name>
<Social Security Number, or non-USA equivalent>
<PGP Key Fingerprint>
<Email Address>

Or maybe it would be easier to just sign our messages via PGP.

John Demme

John Reimer wrote:
 You know what?  That post wasn't by me!
 
 Darn it.  This name is just too common. :-(
 
 - John R. (the other John R.!)
 
 
 John Demme wrote:
 
 Since you're using a static class variable, I'm relatively sure that a 
 test like this would fail in Java using JUnit.  This is not a behavior 
 that I think the compiler can help.  It would have to keep track of 
 all the changes that happen as a result of the unittests, then reset 
 them afterall.  I don't see it happening.

 John (the other John)

 John R wrote:

 Hello, I, like every good little java programmer, was raised on 
 contracts, rep
 invariants, and abstraction barriers, so naturally I like D. I've 
 been playing
 with it a bit and one of my unittests seems to be doing strange 
 things. It is of
 the form:

 module FooClass;

 public class FooClass {

 unittest {
 assert(FooClass.tick() == 0);
 assert(FooClass.tick() == 1);
 }

 public static int count = 0;

 public static int tick() {
 return(FooClass.count++);
 }

 }

 public class BarClass : FooClass{

 unittest {
 assert(BarClass.tick() == 0);       }

 }

 The FooClass unittest runs fine, of course, but the BarClass test 
 fails because
 the variable has alread been incremented and is 2 when it gets to the 
 Bar
 unittest. If you move BarClass to the top of the file its unittest 
 runs ok and
 FooClass' fails.

 I can only conclude that this implementation of unittest does not 
 fully capture
 the concept behind unit testing. My understanding has always been 
 that a unit
 test must not be dependent on other unittests. If it were, then a 
 unittest of a
 child class would not, in fact, be testing the child class, but the 
 child class
 with some arbitrary test code already run upon it. Then the child 
 test may pass,
 while giving no guarantee that the test code will work by itself in 
 actual
 program code.

 I looked for other posts on this but I couldn't find any. Is this 
 meant to be
 this way? If this was a design choice, I might suggest that when class
 hierarchies get large (I only had 4 levels of classes and I banged my 
 head on
 this bug for a while before I realized what was happening) 
 independent unittests
 get more important.

 I haven't looked at the implementation (I'm not quite that brave), 
 but couldn't
 the program just sort of start with a clean slate for each unittest 
 and not run
 into this issue? Just a thought.

 Thanks,
 John R.
Feb 27 2005
parent reply John Reimer <brk_6502 yahoo.com> writes:
He he... no SSN nor SIN (Canadian) numbers going on this list! ... 
although I have both.

I haven't used PGP before... probably a good idea.  For now, I think I 
might just use my initials or my full name :-P. I'll experiment.

It's kinda funny.  At my station, there's two of us with this name, 
although the other guy goes by the "Jon" variant (I'm always careful to 
emphasize that I'm "John" with an "h" :-) ), but when we do calls 
together he usually introduces himself to the patient as "little" Jon 
and refers to me as "big" John.  The naming convention has stuck even in 
the hospital. That's what you get for being 6'7". :-D

-JJR

John Demme wrote:
 Heh.  OK... so from now on, to prevent confusion, we all must sign our 
 messages as so:
 
 Sincerely,
 <Last Name>, <First Name> <Middle Name>
 <Social Security Number, or non-USA equivalent>
 <PGP Key Fingerprint>
 <Email Address>
 
 Or maybe it would be easier to just sign our messages via PGP.
 
 John Demme
 
 John Reimer wrote:
 
 You know what?  That post wasn't by me!

 Darn it.  This name is just too common. :-(

 - John R. (the other John R.!)


 John Demme wrote:

 Since you're using a static class variable, I'm relatively sure that 
 a test like this would fail in Java using JUnit.  This is not a 
 behavior that I think the compiler can help.  It would have to keep 
 track of all the changes that happen as a result of the unittests, 
 then reset them afterall.  I don't see it happening.

 John (the other John)
Feb 27 2005
parent reply John Demme <me teqdruid.com> writes:
I'm in a Rocky Horror cast with two Johns.  The other one is Good John, 
and I'm Bad John.  It's fun being bad.

Bad John

John Reimer wrote:
 He he... no SSN nor SIN (Canadian) numbers going on this list! ... 
 although I have both.
 
 I haven't used PGP before... probably a good idea.  For now, I think I 
 might just use my initials or my full name :-P. I'll experiment.
 
 It's kinda funny.  At my station, there's two of us with this name, 
 although the other guy goes by the "Jon" variant (I'm always careful to 
 emphasize that I'm "John" with an "h" :-) ), but when we do calls 
 together he usually introduces himself to the patient as "little" Jon 
 and refers to me as "big" John.  The naming convention has stuck even in 
 the hospital. That's what you get for being 6'7". :-D
 
 -JJR
 
 John Demme wrote:
 
 Heh.  OK... so from now on, to prevent confusion, we all must sign our 
 messages as so:

 Sincerely,
 <Last Name>, <First Name> <Middle Name>
 <Social Security Number, or non-USA equivalent>
 <PGP Key Fingerprint>
 <Email Address>

 Or maybe it would be easier to just sign our messages via PGP.

 John Demme

 John Reimer wrote:

 You know what?  That post wasn't by me!

 Darn it.  This name is just too common. :-(

 - John R. (the other John R.!)


 John Demme wrote:

 Since you're using a static class variable, I'm relatively sure that 
 a test like this would fail in Java using JUnit.  This is not a 
 behavior that I think the compiler can help.  It would have to keep 
 track of all the changes that happen as a result of the unittests, 
 then reset them afterall.  I don't see it happening.

 John (the other John)
Feb 27 2005
parent John Reimer <brk_6502 yahoo.com> writes:
John Demme wrote:
 I'm in a Rocky Horror cast with two Johns.  The other one is Good John, 
 and I'm Bad John.  It's fun being bad.
 
 Bad John
Ha! That paints a ...um... picture. Now I'm reminded of the classic song "big bad John". I like that song. The combination is striking. :-) Big John
Feb 28 2005
prev sibling parent John R <John_member pathlink.com> writes:
In article <cvt9is$nnh$1 digitaldaemon.com>, John Demme says...
Since you're using a static class variable, I'm relatively sure that a 
test like this would fail in Java using JUnit.  This is not a behavior 
that I think the compiler can help.  It would have to keep track of all 
the changes that happen as a result of the unittests, then reset them 
afterall.  I don't see it happening.

John (the other John)
I did a JUnit test of this: public class ClassFoo { public static int count = 0; public static int tick() { return(ClassFoo.count++); } } public class ClassBar extends ClassFoo { } public class FooTest extends TestCase { public static Test suite() { return new TestSuite(FooTest.class); } protected void setUp() { //ClassFoo.count = 0; } public void testTick() { assertTrue(ClassFoo.tick() == 0); assertTrue(ClassFoo.tick() == 1); } public void testTickAgain() { assertTrue(ClassFoo.tick() == 0); assertTrue(ClassFoo.tick() == 1); } } public class BarExam extends TestCase { public static Test suite() { return new TestSuite(BarExam.class); } public void testInheretedTick() { assertTrue(ClassBar.tick() == 0); } } and found that the problem with the d unit test (that the inhereted FooClass' unittest inhereted a modified object) did not occur. That is, the BarExam unit test passes after having run FooTest. That being said, the tests were run as seperate junit tests, which means the two executions were independent. Now, the FooTest exam fails as is, you have to uncomment the line in the initializing function setUp() to get it to work. I realize that you could implement this setUp() function in D and then call it at the end of the unittest, and so be more or less assured that the unittest does not affect the subclass' unittest, but I think this might be a less than ideal solution. It still leaves all the subclass' unittests dependent on you having correctly reinitialized every class in the chain. There is also no test for correct reinitialization, short of the framework keeping track of all changes to the class (which I agree is an unreasonable expectation). This could lead to problems like: Adding in a new static variable to a base class, unittesting it, and then forgetting to reinitialize at the end of the unittest. This could lead to bizarre behavior in later unittests which would be hard to track down. Or you might want to extend a class that you wrote a long time ago or someone else wrote. If you run the unittest for it you have no guarantee that it has been properly reinitialized and it could mess up the subclass' and all subsequent unittests. Just as an idea for the future of unittests, perhaps they could be somewhat like the JUnit tests in the sense that each module has its unittest run seperately. I've seen suggestions that the unittests be run automaticaly after compiling and linking. If the tests were run then a seperate (temporary) executable could be linked and run for each module's unittests. Therefore you would not have the problem of subclass unittest dependence because each execution would be clean. Walter, I believe you run this show, any thoughts in this direction? John-Matt (the poster formerly known as John R.)
John R wrote:
 Hello, I, like every good little java programmer, was raised on contracts, rep
 invariants, and abstraction barriers, so naturally I like D. I've been playing
 with it a bit and one of my unittests seems to be doing strange things. It is
of
 the form:
 
 module FooClass;
 
 public class FooClass {
 
 unittest {
 assert(FooClass.tick() == 0);
 assert(FooClass.tick() == 1);
 }
 
 public static int count = 0;
 
 public static int tick() {
 return(FooClass.count++);
 }
 
 }
 
 public class BarClass : FooClass{
 
 unittest {
 assert(BarClass.tick() == 0);		
 }
 
 }
 
 The FooClass unittest runs fine, of course, but the BarClass test fails because
 the variable has alread been incremented and is 2 when it gets to the Bar
 unittest. If you move BarClass to the top of the file its unittest runs ok and
 FooClass' fails.
 
 I can only conclude that this implementation of unittest does not fully capture
 the concept behind unit testing. My understanding has always been that a unit
 test must not be dependent on other unittests. If it were, then a unittest of a
 child class would not, in fact, be testing the child class, but the child class
 with some arbitrary test code already run upon it. Then the child test may
pass,
 while giving no guarantee that the test code will work by itself in actual
 program code.
 
 I looked for other posts on this but I couldn't find any. Is this meant to be
 this way? If this was a design choice, I might suggest that when class
 hierarchies get large (I only had 4 levels of classes and I banged my head on
 this bug for a while before I realized what was happening) independent
unittests
 get more important.
 
 I haven't looked at the implementation (I'm not quite that brave), but couldn't
 the program just sort of start with a clean slate for each unittest and not run
 into this issue? Just a thought.
 
 Thanks,
 John R.
 
 
Feb 28 2005