www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Auto-testing of GitHub pull requests

reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
Hello all,

GitHub very nicely ensures that all pull requests to dmd, druntime and phobos 
are sent to the auto-tester and that success/failure reports are sent back.

But ... how is this done?  Is the auto-tester entirely custom written or does
it 
use some standard tools?  And how is it hooked into GitHub?

Thanks & best wishes,

     -- Joe
Sep 24 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-09-24 12:25, Joseph Rushton Wakeling wrote:
 Hello all,

 GitHub very nicely ensures that all pull requests to dmd, druntime and
 phobos are sent to the auto-tester and that success/failure reports are
 sent back.

 But ... how is this done?  Is the auto-tester entirely custom written or
 does it use some standard tools?  And how is it hooked into GitHub?
It's custom. I would guess it either uses a git hook or it uses some API provided by Github. -- /Jacob Carlborg
Sep 24 2013
next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 24/09/13 13:01, Jacob Carlborg wrote:
 It's custom. I would guess it either uses a git hook or it uses some API
 provided by Github.
Ahh, OK. I was wondering if it used some standard tool to manage starting the test jobs and reporting back the tests -- something like Gerrit, Jenkins, etc. I don't have any personal experience of these so couldn't tell on the basis of the test website. Thanks for the info! :-)
Sep 24 2013
prev sibling next sibling parent reply Brad Roberts <braddr puremagic.com> writes:
On 9/24/13 4:29 AM, Joseph Rushton Wakeling wrote:
 On 24/09/13 13:01, Jacob Carlborg wrote:
 It's custom. I would guess it either uses a git hook or it uses some API
 provided by Github.
Ahh, OK. I was wondering if it used some standard tool to manage starting the test jobs and reporting back the tests -- something like Gerrit, Jenkins, etc. I don't have any personal experience of these so couldn't tell on the basis of the test website. Thanks for the info! :-)
https://github.com/braddr/d-tester I'd use a standard tool (or would have, silly to switch when this one does the job) but none of them support the combination of features used. Primarily: github, testing on multiple platforms, testing pulls as merges into the target branch (rather than just the pull itself), multiple repositories in the same build. There's probably more requirements, it's been a while since I researched the state of the ecosystem and exactly what the gaps are. But now that it's written, it's just fundamentally not that much code, nor is it actually difficult code.
Sep 24 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-09-24 18:25, Brad Roberts wrote:

 https://github.com/braddr/d-tester
Could you add a link to this on the test results page? I always forget where this code is located. -- /Jacob Carlborg
Sep 24 2013
parent reply "simendsjo" <simendsjo gmail.com> writes:
On Wednesday, 25 September 2013 at 06:51:44 UTC, Jacob Carlborg 
wrote:
 On 2013-09-24 18:25, Brad Roberts wrote:

 https://github.com/braddr/d-tester
Could you add a link to this on the test results page? I always forget where this code is located.
http://d.puremagic.com/test-results/
Sep 25 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-09-25 09:30, simendsjo wrote:

 http://d.puremagic.com/test-results/
I know where the test results are. But, what I'm forgetting is where the code is which implements these test results. I want this page: http://d.puremagic.com/test-results/ To have a link to this: https://github.com/braddr/d-tester -- /Jacob Carlborg
Sep 25 2013
parent "simendsjo" <simendsjo gmail.com> writes:
On Wednesday, 25 September 2013 at 11:29:16 UTC, Jacob Carlborg 
wrote:
 On 2013-09-25 09:30, simendsjo wrote:

 http://d.puremagic.com/test-results/
I know where the test results are. But, what I'm forgetting is where the code is which implements these test results. I want this page: http://d.puremagic.com/test-results/ To have a link to this: https://github.com/braddr/d-tester
Seems like I didn't actually read you post :)
Sep 25 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 24/09/13 18:25, Brad Roberts wrote:
 I'd use a standard tool (or would have, silly to switch when this one does the
 job) but none of them support the combination of features used.  Primarily:
 github, testing on multiple platforms, testing pulls as merges into the target
 branch (rather than just the pull itself), multiple repositories in the same
 build.  There's probably more requirements, it's been a while since I
researched
 the state of the ecosystem and exactly what the gaps are.

 But now that it's written, it's just fundamentally not that much code, nor is
it
 actually difficult code.
Thanks very much for pointing me to that. I'm looking into auto-testing stuff that would work with a project hosted on Gitorious (GitHub is unsatisfactory for political reasons) -- I'm working on the assumption that one of the standard testing frameworks will play nice with that (I've seen a writeup on Gerrit + Gitorious), but assuming anything custom is required, this could be a very useful template.
Sep 25 2013
prev sibling parent Brad Roberts <braddr puremagic.com> writes:
On 9/25/13 2:58 PM, Joseph Rushton Wakeling wrote:
 On 24/09/13 18:25, Brad Roberts wrote:
 I'd use a standard tool (or would have, silly to switch when this one does the
 job) but none of them support the combination of features used.  Primarily:
 github, testing on multiple platforms, testing pulls as merges into the target
 branch (rather than just the pull itself), multiple repositories in the same
 build.  There's probably more requirements, it's been a while since I
researched
 the state of the ecosystem and exactly what the gaps are.

 But now that it's written, it's just fundamentally not that much code, nor is
it
 actually difficult code.
Thanks very much for pointing me to that. I'm looking into auto-testing stuff that would work with a project hosted on Gitorious (GitHub is unsatisfactory for political reasons) -- I'm working on the assumption that one of the standard testing frameworks will play nice with that (I've seen a writeup on Gerrit + Gitorious), but assuming anything custom is required, this could be a very useful template.
The only parts that really hook into github are: 1) serverd/update_pulls.d -- That's a cronjob that polls github ever few minutes for the current set of pull requests. It's pretty ghetto due to evolution and lack of time investment. It really ought to be replaced with a webhook that github posts to and just left around as a backup in case a hook call gets lost. 2) server/add/github-post.ghtml -- That's called by a github web hook when the master branches are checked into. Ignore the language it's written in. The gist is easy to understand: grab what's been updated, flush old state in the db. Unless I'm forgetting something, all the rest of the code is the tester itself: a) client -- the code that executes on the clients b) server -- the web ui, written in an ancient scripting language that I still use from a previous job. Don't use as is unless you enjoy reverse engineering interpreters. c) serverd -- the apis that the client code calls to ask about work and report status. And there's a massive lack of ui surrounding the configuration tables. I keep putting off working on them since it's awfully rare that any of them need to be updated, but I always hate running sql directly against a live data store. In case stats are of interesting to anyone (they are to me): # of builds/tests executed so far 741000 pull (first one on Dec 14, 2011) 51000 master branch (first one on Sept 5th, 2010 -- just over 3 years ago) Users with 100 or more closed pull requests (note, data doesn't cover a small number pull requests that pre-dates the tester) +---------+----------------+----------+ | user_id | username | count(*) | +---------+----------------+----------+ | 254394 | 9rnsr | 1068 | | 288976 | dawgfoto | 328 | | 580100 | yebblies | 306 | | 567837 | donc | 283 | | 568298 | WalterBright | 229 | | 360179 | AndrejMitrovic | 226 | | 579956 | jmdavis | 167 | | 44076 | alexrp | 152 | | 1911406 | monarchdodra | 129 | | 40242 | braddr | 123 | +---------+----------------+----------+
Sep 25 2013
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 24 September 2013 at 10:25:21 UTC, Joseph Rushton 
Wakeling wrote:
 Hello all,

 GitHub very nicely ensures that all pull requests to dmd, 
 druntime and phobos are sent to the auto-tester and that 
 success/failure reports are sent back.

 But ... how is this done?  Is the auto-tester entirely custom 
 written or does it use some standard tools?  And how is it 
 hooked into GitHub?

 Thanks & best wishes,

     -- Joe
D one is custom but most major CI tools have such functionality (I personally have used Jenkins). It uses GitHub application API and needs to be explicitly allowed by repository owner.
Sep 24 2013