www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.internals - Automatic finding of sporadic autotester failures

Since my original post got no interest in the general section 
[1], I'm going to rehash here:

Sporadic auto-tester failures often go unreported with the 
general attitude being "just restart the test."

I fixed the failures I found while simply looking at open PRs 
Then I put a together a proof of concept which:
  - scrapes the auto-tester website and finds "random" failures
  - searches the log files for the test that failed
  - groups similar failures and outputs a report

I've posted the code in [5]. I pipe the python output to a file 
and pass the filename as argument to the D program.

Shouldn't something similar be integrated into to the auto-tester?

It's important to report "random" failures so that they can be 
While many are caused by bad tests, they can have underlying 
critical causes (just look at [2].)
They can also make contributing a sour experience.


P.S. PR [4] is sitting in review limbo due to codecov's red X.

https://forum.dlang.org/thread/hjkkpiavwqjlmnaskqfv forum.dlang.org

[2] https://github.com/dlang/phobos/pull/4988

[3] https://github.com/dlang/phobos/pull/4997

[4] https://github.com/dlang/phobos/pull/5004

Jan 16 2017