www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - understanding Auto-Test

reply John Belmonte <john neggie.net> writes:
I'm trying to understand why my pull request was queued in D2 
Auto-Test for only 2 of 8 tests, with the remaining left in 
pending state.

   
https://auto-tester.puremagic.com/pull-history.ghtml?projectid=1&repoid=1&pullid=8051

Since there are pending tests, I'd expect it to appear in the 
standard priority pull queue under "Has passes".

   https://auto-tester.puremagic.com/pulls.ghtml?projectid=1

What I'm I overlooking?

Kindly,
--John
Mar 21 2018
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 22 March 2018 at 01:46:08 UTC, John Belmonte wrote:

 I'm trying to understand why my pull request was queued in D2 
 Auto-Test for only 2 of 8 tests, with the remaining left in 
 pending state.

   
 https://auto-tester.puremagic.com/pull-history.ghtml?projectid=1&repoid=1&pullid=8051

 Since there are pending tests, I'd expect it to appear in the 
 standard priority pull queue under "Has passes".

   https://auto-tester.puremagic.com/pulls.ghtml?projectid=1

 What I'm I overlooking?

 Kindly,
 --John


I believe what happened is a different PR was merged.  When a PR 
is merged, all tests are invalidated, and the autotester begins 
testing them again.

There is also a priority affecting which PRs get tested first.  
Those that are labeled with "AutoMerge" are given a higher 
priority.

Mike
Mar 21 2018
parent reply John Belmonte <john neggie.net> writes:
On Thursday, 22 March 2018 at 01:51:49 UTC, Mike Franklin wrote:

 I believe what happened is a different PR was merged.  When a 
 PR is merged, all tests are invalidated, and the autotester 
 begins testing them again.

 There is also a priority affecting which PRs get tested first.  
 Those that are labeled with "AutoMerge" are given a higher 
 priority.


I'm still rather puzzled.  My pull request remains with 8 tests 
pending after several hours.  I can't find any confirmation on 
the pulls display 
https://auto-tester.puremagic.com/pulls.ghtml?projectid=1 that it 
intends to run the tests.   Surely being listed in the "Old 
results" table doesn't imply pending runs against a new SHA, 
otherwise why would there be entries there dated 2017?
Mar 21 2018
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 22 March 2018 at 04:12:00 UTC, John Belmonte wrote:


 I'm still rather puzzled.  My pull request remains with 8 tests 
 pending after several hours.  I can't find any confirmation on 
 the pulls display 
 https://auto-tester.puremagic.com/pulls.ghtml?projectid=1 that 
 it intends to run the tests.   Surely being listed in the "Old 
 results" table doesn't imply pending runs against a new SHA, 
 otherwise why would there be entries there dated 2017?


I see.  It appears GitHub didn't pick of the latest status (or 
the auto-tester didn't notify GitHub; I'm not which way the data 
flows).

Let's see what happens after the auto-tester starts testing it 
again.  I'll keep an eye on it.

Mike
Mar 21 2018
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 22 March 2018 at 04:17:52 UTC, Mike Franklin wrote:


 Let's see what happens after the auto-tester starts testing it 
 again.


Note that it could be a while, as there are PRs that will be 
getting merged in the next 24 hours that will restart the test 
queue.

Mike
Mar 21 2018
prev sibling parent Seb <seb wilzba.ch> writes:
On Thursday, 22 March 2018 at 04:17:52 UTC, Mike Franklin wrote:

 On Thursday, 22 March 2018 at 04:12:00 UTC, John Belmonte wrote:


 I'm still rather puzzled.  My pull request remains with 8 
 tests pending after several hours.  I can't find any 
 confirmation on the pulls display 
 https://auto-tester.puremagic.com/pulls.ghtml?projectid=1 that 
 it intends to run the tests.   Surely being listed in the "Old 
 results" table doesn't imply pending runs against a new SHA, 
 otherwise why would there be entries there dated 2017?


 I see.  It appears GitHub didn't pick of the latest status (or 
 the auto-tester didn't notify GitHub;


Everything is working as expected. Every time a PR gets merged, 
ALL pull requests get invalidated and are moved to "Old results" 
as their results have been calculated against an old head.
Then auto-tester starts to reduce the "Old results" list by 
building them again.
It's a constant fight.


 I'm not which way the  data flows).


GitHub -> CIs (via hooks)

(though especially auto-tester queries back a lot)


 Let's see what happens after the auto-tester starts testing it 
 again.  I'll keep an eye on it.


No need to keep an eye on this. auto-tester will constantly 
rebuild the PR and invalidate it.
The only way interaction is possible is via:

- manually invalidating builds (called "delete" at the 
auto-tester) - not really useful as builds are automatically 
invalidated
- "auto-merge" priority builds. The PR has super-priority in the 
build queue.

That's happening currently with the four builds at the top:

https://imgur.com/a/29Ohn
Mar 21 2018
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Thursday, 22 March 2018 at 04:12:00 UTC, John Belmonte wrote:

 On Thursday, 22 March 2018 at 01:51:49 UTC, Mike Franklin wrote:

 I believe what happened is a different PR was merged.  When a 
 PR is merged, all tests are invalidated, and the autotester 
 begins testing them again.

 There is also a priority affecting which PRs get tested first.
  Those that are labeled with "AutoMerge" are given a higher 
 priority.


 I'm still rather puzzled.  My pull request remains with 8 tests 
 pending after several hours.  I can't find any confirmation on 
 the pulls display 
 https://auto-tester.puremagic.com/pulls.ghtml?projectid=1 that 
 it intends to run the tests.   Surely being listed in the "Old 
 results" table doesn't imply pending runs against a new SHA, 
 otherwise why would there be entries there dated 2017?


I give it a shot:

- Every time a PR is merged at dmd, druntime or phobos ALL 
auto-tester results get invalidated
- The auto-tester constantly tries to build PRs which have now 
results based on his queue (it's sorted after merge priority + 
last pushed timestamp)

The traffic is usually so high that your PR will always be at 
pending. That's really not a big deal. On the contrary, once your 
PR has been approved and gets on the merge queue, it will receive 
higher priority and usually will be merged within 1-3 hours.

Also all reviewers are familiar with auto-tester and pending 
tests have zero effect on the reviewers. All the other CIs 
already give a good feedback and if they want to, they can check 
the history of auto-tester for permanent failures


 I can't find any confirmation on the pulls display that it 
 intends to run the tests.


There's none. It _constantly_ reruns your PR.


  Surely being listed in the "Old results" table doesn't imply 
 pending runs against a new SHA, otherwise why would there be 
 entries there dated 2017?


The timestamp you see on pulls.ghtml?projectid=1 is GitHub's last 
modified timestamp which essentially is timestamp of the last 
push to the respective branch.

And yep there are a few old PRs, but the auto-tester newer tests 
them as its working queue order is sorted by the "Last Commited" 
timestamp.

Did this clarify things for you?
Mar 21 2018
parent reply John Belmonte <john neggie.net> writes:
On Thursday, 22 March 2018 at 04:25:00 UTC, Seb wrote:

 - Every time a PR is merged at dmd, druntime or phobos ALL 
 auto-tester results get invalidated


If a change on the destination branch causes a PR to require an 
automatic merge, certainly build and tests should be rerun.  But 
if the two changes are unrelated do the results of all pending 
PR's need to be invalidated?


 Did this clarify things for you?


Yes, thank you for taking the time to explain the system.
Mar 21 2018
parent Seb <seb wilzba.ch> writes:
On Thursday, 22 March 2018 at 05:00:07 UTC, John Belmonte wrote:

 On Thursday, 22 March 2018 at 04:25:00 UTC, Seb wrote:

 - Every time a PR is merged at dmd, druntime or phobos ALL 
 auto-tester results get invalidated


 If a change on the destination branch causes a PR to require an 
 automatic merge, certainly build and tests should be rerun.  
 But if the two changes are unrelated do the results of all 
 pending PR's need to be invalidated?


Well, auto-tester is in place for a few years already. Its git 
history [1] goes back to 2010, but I assume the auto-tester to be 
even older.
Back in these old, dark ages, a commit in another PR (even if 
it's a different repo), was a bit more likely to break the 
current PR.
Nowadays, that's a lot less common, but still happens from time 
to time (e.g. you make a PR to deprecate sth., but in the 
meanwhile a new usage got added to Phobos or a new  safe-ty 
violation got added).

Anyhow, the idea is to guarantee that `master` is always 
buildable and passes all tests no matter what and this constant 
invalidation is the only way to deal with this as there's no way 
to automatically determine whether the new changes in master are 
really unrelated.

Note that what the auto-tester does for you automatically, is 
very similar to GitHub's "Require branches to be up to date 
before merging" branch setting, except that instead of manually 
needing to rebase the branch all the time, auto-tester does the 
rebases automatically and is "cross-repository aware".

[1] https://github.com/braddr/at-client
Mar 21 2018