www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - Stress Suite codes

reply Derek Parnell <derek psych.ward> writes:
 I still don't understand xpass, fail, xfail etc. What does the 'x' mean?

pass test case was expected to pass, and it did xpass test case was expected to fail, but passed fail test case was expected to pass, but failed xfail test case was expected to fail, and it did

It is probably too late but could you use something based on this idea ... goodpass test case was expected to pass, and it did badpass test case was expected to fail, but passed badfail test case was expected to pass, but failed goodfail test case was expected to fail, and it did -- Derek Melbourne, Australia 14/02/2005 10:39:07 AM
Feb 13 2005
next sibling parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
I think that's my confusion as well.  To me it looks like this, by the 
way I reckon it:

        pass      fail
no x   goodpass  badfail
x      badpass   goodfail

It would make more sense to me if it was simply:


        pass      fail
no x   goodpass  goodfail
x      badpass   badfail

In other words, pass and fail would mean it did as expected, and xpass 
and xfail would mean it did what it says but shouldn't have (thus the x 
would be a modifier meaning "bad".)

Doesn't really matter, though, and I think the tests are wonderful. 
Thanks for maintaining them, Thomas Kühne.

(let's just hope Thunderbird properly preserves my whitespace...)

-[Unknown]


I still don't understand xpass, fail, xfail etc. What does the 'x' mean?

pass test case was expected to pass, and it did xpass test case was expected to fail, but passed fail test case was expected to pass, but failed xfail test case was expected to fail, and it did

It is probably too late but could you use something based on this idea ... goodpass test case was expected to pass, and it did badpass test case was expected to fail, but passed badfail test case was expected to pass, but failed goodfail test case was expected to fail, and it did

Feb 13 2005
parent reply =?ISO-8859-1?Q?Thomas_K=FChne?= <thomas-dloop kuehne.THISISSPAM.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

|>>> I still don't understand xpass, fail, xfail etc. What does the 'x'
|>>> mean?
|>>
|>>
|>> pass    test case was expected to pass, and it did
|>> xpass    test case was expected to fail, but passed
|>> fail    test case was expected to pass, but failed
|>> xfail    test case was expected to fail, and it did
|>
|>
|>
|> It is probably too late but could you use something based on this
|> idea ...
|>
|> goodpass  test case was expected to pass, and it did
|> badpass   test case was expected to fail, but passed
|> badfail   test case was expected to pass, but failed
|> goodfail  test case was expected to fail, and it did

| I think that's my confusion as well.  To me it looks like this, by the
| way I reckon it:
|
|        pass      fail
| no x   goodpass  badfail
| x      badpass   goodfail
|
| It would make more sense to me if it was simply:
|
|
|        pass      fail
| no x   goodpass  goodfail
| x      badpass   badfail
|
| In other words, pass and fail would mean it did as expected, and xpass
| and xfail would mean it did what it says but shouldn't have (thus the
| x would be a modifier meaning "bad".)

I'm not trying to create my own language but rather to use the general
testframe words and meanings.
http://gcc.gnu.org/install/test.html#TOC3
http://www.gnu.org/software/greg/greg.html
http://www.gnu.org/software/dejagnu/dejagnu.html

I agree that "xpass" and "xfail" used as "unexpected pass" and "expected
failure" pose some linguistic difficulties.

In your example above "fail" would be an expected - good - behavior.
That's somehow troublesome too.

| Doesn't really matter, though, and I think the tests are wonderful.
Thanks

Thomas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (MingW32)

iD8DBQFCEAXE3w+/yD4P9tIRAnIoAJ9wzFkXWOcAGqu0rFp5P5oAueG/ZgCeJ9kM
7NpcbjNJKRp2hv76PaaF/Hs=
=+Ot/
-----END PGP SIGNATURE-----
Feb 13 2005
next sibling parent "Regan Heath" <regan netwin.co.nz> writes:
On Mon, 14 Feb 2005 02:58:28 +0100, Thomas Kühne  
<thomas-dloop kuehne.THISISSPAM.cn> wrote:
 I'm not trying to create my own language but rather to use the general
 testframe words and meanings.
 http://gcc.gnu.org/install/test.html#TOC3
 http://www.gnu.org/software/greg/greg.html
 http://www.gnu.org/software/dejagnu/dejagnu.html

 I agree that "xpass" and "xfail" used as "unexpected pass" and "expected
 failure" pose some linguistic difficulties.

It does, not only is one good, and one bad, but the 'x' stands for something different in each case.
 In your example above "fail" would be an expected - good - behavior.
 That's somehow troublesome too.

I agree. Ok, I think the difficulty comes because some tests are expected to compile and some aren't, and further, some are also expected to produce some sort of result. I suggest a table like so: Test Name | Compile? | Result? | -------------------------------- foo | yes | NA | bar | no | pass | baz | yes | NA | bob | no | pass | We then colour the compile column entries green if they match the expected outcome, and red otherwise.
 | Doesn't really matter, though, and I think the tests are wonderful.
 Thanks

I agree, you're doing a great job. Regan
Feb 13 2005
prev sibling next sibling parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
Ick.  Well, sorry for assuming you came up with it.... but I suppose if 
it's used for gcc, etc. it's a good idea to use it for D as well.

If you want my personal opinion, though, I wouldn't use the "x" at all. 
  Probably more like "+pass", "+fail", "-pass", "-fail", and "fault".... 
which imho are more logical (+ means good, - means bad) and get the 
added bonus of all being the same length.

-[Unknown]


 I'm not trying to create my own language but rather to use the general
 testframe words and meanings.
 http://gcc.gnu.org/install/test.html#TOC3
 http://www.gnu.org/software/greg/greg.html
 http://www.gnu.org/software/dejagnu/dejagnu.html

Feb 13 2005
parent zwang <nehzgnaw gmail.com> writes:
Unknown W. Brackets wrote:
 Ick.  Well, sorry for assuming you came up with it.... but I suppose if 
 it's used for gcc, etc. it's a good idea to use it for D as well.
 
 If you want my personal opinion, though, I wouldn't use the "x" at all. 
  Probably more like "+pass", "+fail", "-pass", "-fail", and "fault".... 
 which imho are more logical (+ means good, - means bad) and get the 
 added bonus of all being the same length.
 
 -[Unknown]
 
 
 I'm not trying to create my own language but rather to use the general
 testframe words and meanings.
 http://gcc.gnu.org/install/test.html#TOC3
 http://www.gnu.org/software/greg/greg.html
 http://www.gnu.org/software/dejagnu/dejagnu.html


IMO, "fail"/"xfail" are as clear as "+fail"/"-fail", if not clearer. Anyway, this is trivial. Get used to the codes:) BTW, I have to say Thomas is doing a really great job. Kudos!
Feb 13 2005
prev sibling parent reply Dave <Dave_member pathlink.com> writes:
In article <cup0kv$2cjj$1 digitaldaemon.com>, =?ISO-8859-1?Q?Thomas_K=FChne?=
says...

| Doesn't really matter, though, and I think the tests are wonderful.
Thanks

Thomas

Yes, they are great Thomas - Thanks! Just a couple of thoughts - a total # of tests for each run in the summary and maybe even a ratio next to the raw number for each result, i.e.: -------------- 687 (51.8%) | -------------- - Dave
Feb 14 2005
parent =?UTF-8?B?VGhvbWFzIEvDvGhuZQ==?= writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dave wrote:

| Just a couple of thoughts - a total # of tests for each run in the
| summary and maybe even a ratio next to the raw number for each
| result, i.e.:
|
| --------------
|  687 (51.8%) |
| --------------

Presenting absolute and relative numbers in on cell will blast the table.

Maybe I'll take Graf, GnuPlot or co. to create some diagrams.

Just in case: Please keep in mind that there is no relation like:
n test cases -> m bugs
2m bugs -> 2n test cases

Thomas

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (MingW32)

iD8DBQFCFnQ/3w+/yD4P9tIRAovGAJ4lo6iABenwb38RP3PiINF7y854tQCgkmXR
z9NU86eChOtQgnZ5CBTxHLM=
=ZUTD
-----END PGP SIGNATURE-----
Feb 18 2005
prev sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Derek Parnell wrote:
 I still don't understand xpass, fail, xfail etc. What does the 
 'x' mean?

pass test case was expected to pass, and it did xpass test case was expected to fail, but passed fail test case was expected to pass, but failed xfail test case was expected to fail, and it did

It is probably too late but could you use something based on this idea ... goodpass test case was expected to pass, and it did badpass test case was expected to fail, but passed badfail test case was expected to pass, but failed goodfail test case was expected to fail, and it did

Guess you're right, but I also guess that those names would make the results table too wide. But I think many of us now know what the current names mean - can we really change it now without confusing more people? A few more of my thoughts: 1. What about issues where the spec is unclear about whether something should work or not? Maybe the results could be called "?pass" and "?fail". Or are these classified by some opinion on whether it should be valid? 2. Problems with the abstract keyword are somewhat underrepresented in DStress: digitalmars.D.bugs/1940 3. Two other things that don't currently seem to fit into DStress, but could do with tracking somehow: (a) issues where code correctly fails to compile, but the error message is meaningless or hopelessly inadequate (but doesn't constitute an error result) (b) certain problems that depend on I/O operations, such as digitalmars.D.bugs/2001 Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 14 2005
parent reply =?ISO-8859-1?Q?Thomas_K=FChne?= <thomas-dloop kuehne.THISISSPAM.cn> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stewart Gordon wrote:

| A few more of my thoughts:
|
| 1. What about issues where the spec is unclear about whether
| something should work or not?  Maybe the results could be called
| "?pass" and "?fail".  Or are these classified by some opinion on
| whether it should be valid?

Example?

| 2. Problems with the abstract keyword are somewhat underrepresented
| in DStress:
|
| digitalmars.D.bugs/1940

I'll have a look.

| 3. Two other things that don't currently seem to fit into DStress,
| but could do with tracking somehow:
|
| (a) issues where code correctly fails to compile, but the error
| message is meaningless or hopelessly inadequate (but doesn't
| constitute an error result)

That's a tricky one. Inadequate error messages could be black listed for
a specific test case, but I'm not sure how to properly white list error
messages.

I plan to add a check for correct line numbers in the error messages.
This might detect some of the botched error messages.

| (b) certain problems that depend on I/O operations, such as
|
| digitalmars.D.bugs/2001

This isn't a bug, it is an unconventional style.

# If not, the default Error handler is run, which displays the message
# and terminates the program.

The documentation simply states "displays" without any limitations on
the howto, could be a pop-up, stderr, stdout, the keyboard leds...

A simple fix.
1) Open internal/dmain2.d

2) Change from:
#	printf("Error: ");
#	o.print();

3) Change to:
#	fprintf(stderr,"Error: %.*s", o.toString());

You'll notice that SomeError.toString()!=SomeError.print() arg...

Thomas


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (MingW32)

iD8DBQFCFnIp3w+/yD4P9tIRAsyOAJ9bNOAQfuFaVLdbpn4v4FmCt3BwjACfSx4M
LF5Me9fgTA5e1np+bl0+LjI=
=WArs
-----END PGP SIGNATURE-----
Feb 18 2005
parent Stewart Gordon <smjg_1998 yahoo.com> writes:
Thomas Kühne wrote:
 Stewart Gordon wrote:
 
 A few more of my thoughts:
 
 1. What about issues where the spec is unclear about whether 
 something should work or not?  Maybe the results could be called 
 "?pass" and "?fail".  Or are these classified by some opinion on 
 whether it should be valid?

Example?

Any test of the layout of bits within a struct. Theoretically, anything related to an undocumented language feature (such as the abstract keyword, though deciding how this should work is fairly straightforward). No doubt others. <snip>
 I plan to add a check for correct line numbers in the error 
 messages. This might detect some of the botched error messages.

For that matter, should error messages with no line number at all be considered an error result, considering that such things aren't supposed to be part of DMD? <snip>
 # If not, the default Error handler is run, which displays the
 # message and terminates the program.
 
 The documentation simply states "displays" without any limitations 
 on the howto, could be a pop-up, stderr, stdout, the keyboard 
 leds...

And as I said, the documentation states that the program "gracefully exits".
 A simple fix.
 1) Open internal/dmain2.d
 
 2) Change from:
 #    printf("Error: ");
 #    o.print();
 
 3) Change to:
 #    fprintf(stderr,"Error: %.*s", o.toString());

You mean fprintf(stderr,"Error: %.*s\n", o.toString()); ?
 You'll notice that SomeError.toString()!=SomeError.print() arg...

Indeed. Is Object.print supposed to be pure syntactic sugar, or is there some other hidden motive for its existence? Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 21 2005