www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Bug Prediction at Google

reply Robert Clipsham <robert octarineparrot.com> writes:
I just read this pretty interesting article on the Google Engineering 
Tools website, thought it might interest some people here:

http://google-engtools.blogspot.com/2011/12/bug-prediction-at-google.html

( http://goo.gl/2O6YT <= a short link in case the above one gets wrapped)

It basically describes a new process in place at Google whereby each 
file within a project is assigned a rating saying how likely the given 
file is to have a bug in it compared to everything else. This can be 
used by code reviewers so they can take extra care when reviewing 
certain changes.

I really think github needs built in review tools (more advanced than 
just pull requests) to allow things like the auto-tester to be run, or 
algorithms like this to be used for manual review and so on.

-- 
Robert
http://octarineparrot.com/
Dec 15 2011
next sibling parent Andrew Wiley <wiley.andrew.j gmail.com> writes:
On Thu, Dec 15, 2011 at 6:04 PM, Robert Clipsham
<robert octarineparrot.com> wrote:
 I just read this pretty interesting article on the Google Engineering Tools
 website, thought it might interest some people here:

 http://google-engtools.blogspot.com/2011/12/bug-prediction-at-google.html

 ( http://goo.gl/2O6YT <= a short link in case the above one gets wrapped)

 It basically describes a new process in place at Google whereby each file
 within a project is assigned a rating saying how likely the given file is to
 have a bug in it compared to everything else. This can be used by code
 reviewers so they can take extra care when reviewing certain changes.

 I really think github needs built in review tools (more advanced than just
 pull requests) to allow things like the auto-tester to be run, or algorithms
 like this to be used for manual review and so on.

Well, Github does have an API to allow that sort of thing to happen. In theory, a bot could examine a pull request, merge it and run tests, and post back the results.
Dec 15 2011
prev sibling next sibling parent Brad Roberts <braddr puremagic.com> writes:
On Thu, 15 Dec 2011, Andrew Wiley wrote:

 On Thu, Dec 15, 2011 at 6:04 PM, Robert Clipsham
 <robert octarineparrot.com> wrote:
 I really think github needs built in review tools (more advanced than just
 pull requests) to allow things like the auto-tester to be run, or algorithms
 like this to be used for manual review and so on.

Well, Github does have an API to allow that sort of thing to happen. In theory, a bot could examine a pull request, merge it and run tests, and post back the results.

I'm 90% done with adding pull testing to the existing auto-tester fleet. It's based on the work that Daniel did. The basic overview: server: every 10 minutes, check github for changes to pull requests (if they had notifications for pull changes, I'd use it instead) every time a trunk commit notification is received, check github for changes to pull requests client: forever { check for trunk build if yes { build trunk; continue } check for pull build if yes { build pull; continue } sleep 1m } Left to do: 1) deploy changes to the tester hosts (it's on 2 already) 2) finish the ui 3) trigger pull rebuilds when trunk is updated 4) add back in support for related pull requests (ie, two pulls that separately fail but together succeed) 5) consider adding updating the pull request on github with tester results. This one needs to be very carefully done to not spam the report every time a build fails or succeeds. 6) update the auto-tester grease monkey script to integrate the pull tester results with github's ui. I'll hopefully finish 1 and 2 tonight. I can do 3 manually until it's automated. I'm not sure about the ordering of 4-6. They're nice to haves rather than must haves. All these extra builds are going to cost a lot of time. There's about 100 open pull requests right now. The fastest runs are on the order of 10 minutes. That's 6 per hour or roughly 17 hours. The slowest are closer to an hour. So, obviously there's some growing pains to deal with. I'll probably add a way for github committers to prioritize pull requests so they build first. Luckily this stuff is trivial to throw hardware at.. it's super parallelizeable. Also the hardware I have for those long runs is super old. I think the freebsd/32 box is p4 era box. The win/32 is my asus eee atom based netbook. If anyone wants to volunteer build hardware, particularly for the non-linux platforms, please contact me via email (let's not clutter up the newsgroup with that chatter). Requirements: I need to be able to access it remotely. It needs to be have reliable connectivity (bandwidth is pretty much a non-issue.. it doesn't need much at all). It needs to be hardware you're willing to have hammered fairly hard at random times. I'll almost certainly write some code during the holiday to fire up and use ec2 nodes for windows and linux builds. With the application of just a little money, all of those runs could be done fully parallel.. which would just be sweet to see. Ok, I admit it.. I love working at amazon on ec2.. and I'm happy to finally have a project that could actually use it. Later, Brad
Dec 15 2011
prev sibling next sibling parent Brad Anderson <eco gnuk.net> writes:
--f46d0408d57930a64604b43c4a7d
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Dec 15, 2011 at 6:43 PM, Brad Roberts <braddr puremagic.com> wrote:

 On Thu, 15 Dec 2011, Andrew Wiley wrote:

 On Thu, Dec 15, 2011 at 6:04 PM, Robert Clipsham
 <robert octarineparrot.com> wrote:
 I really think github needs built in review tools (more advanced than


 pull requests) to allow things like the auto-tester to be run, or


 like this to be used for manual review and so on.

Well, Github does have an API to allow that sort of thing to happen. In theory, a bot could examine a pull request, merge it and run tests, and post back the results.

I'm 90% done with adding pull testing to the existing auto-tester fleet. It's based on the work that Daniel did. The basic overview: server: every 10 minutes, check github for changes to pull requests (if they had notifications for pull changes, I'd use it instead) every time a trunk commit notification is received, check github for changes to pull requests client: forever { check for trunk build if yes { build trunk; continue } check for pull build if yes { build pull; continue } sleep 1m } Left to do: 1) deploy changes to the tester hosts (it's on 2 already) 2) finish the ui 3) trigger pull rebuilds when trunk is updated 4) add back in support for related pull requests (ie, two pulls that separately fail but together succeed) 5) consider adding updating the pull request on github with tester results. This one needs to be very carefully done to not spam the report every time a build fails or succeeds. 6) update the auto-tester grease monkey script to integrate the pull tester results with github's ui. I'll hopefully finish 1 and 2 tonight. I can do 3 manually until it's automated. I'm not sure about the ordering of 4-6. They're nice to haves rather than must haves. All these extra builds are going to cost a lot of time. There's about 100 open pull requests right now. The fastest runs are on the order of 10 minutes. That's 6 per hour or roughly 17 hours. The slowest are closer to an hour. So, obviously there's some growing pains to deal with. I'll probably add a way for github committers to prioritize pull requests so they build first. Luckily this stuff is trivial to throw hardware at.. it's super parallelizeable. Also the hardware I have for those long runs is super old. I think the freebsd/32 box is p4 era box. The win/32 is my asus eee atom based netbook. If anyone wants to volunteer build hardware, particularly for the non-linux platforms, please contact me via email (let's not clutter up the newsgroup with that chatter). Requirements: I need to be able to access it remotely. It needs to be have reliable connectivity (bandwidth is pretty much a non-issue.. it doesn't need much at all). It needs to be hardware you're willing to have hammered fairly hard at random times. I'll almost certainly write some code during the holiday to fire up and use ec2 nodes for windows and linux builds. With the application of just a little money, all of those runs could be done fully parallel.. which would just be sweet to see. Ok, I admit it.. I love working at amazon on ec2.. and I'm happy to finally have a project that could actually use it. Later, Brad

 With the application of just a little money

Out of curiosity, how much is a little? --f46d0408d57930a64604b43c4a7d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Thu, Dec 15, 2011 at 6:43 PM, Brad Roberts <span dir=3D"ltr">&lt;<a href= =3D"mailto:braddr puremagic.com">braddr puremagic.com</a>&gt;</span> wrote:= <br><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"m= argin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> On Thu, 15 Dec 2011, Andrew Wiley wrote:<br> <br> &gt; On Thu, Dec 15, 2011 at 6:04 PM, Robert Clipsham<br> &gt; &lt;<a href=3D"mailto:robert octarineparrot.com">robert octarineparrot= .com</a>&gt; wrote:<br> &gt; &gt; I really think github needs built in review tools (more advanced = than just<br> &gt; &gt; pull requests) to allow things like the auto-tester to be run, or= algorithms<br> &gt; &gt; like this to be used for manual review and so on.<br> &gt;<br> &gt; Well, Github does have an API to allow that sort of thing to happen.<b= r> &gt; In theory, a bot could examine a pull request, merge it and run tests,= <br> &gt; and post back the results.<br> <br> I&#39;m 90% done with adding pull testing to the existing auto-tester fleet= .<br> It&#39;s based on the work that Daniel did.<br> <br> The basic overview:<br> <br> server:<br> <br> =A0every 10 minutes, check github for changes to pull requests<br> =A0 =A0(if they had notifications for pull changes, I&#39;d use it instead= )<br> <br> =A0every time a trunk commit notification is received, check github for<br=

<br> <br> client:<br> <br> =A0forever {<br> =A0 =A0check for trunk build<br> =A0 =A0if yes { build trunk; continue }<br> =A0 =A0check for pull build<br> =A0 =A0if yes { build pull; continue }<br> =A0 =A0sleep 1m<br> =A0}<br> <br> <br> Left to do:<br> <br> =A01) deploy changes to the tester hosts (it&#39;s on 2 already)<br> <br> =A02) finish the ui<br> <br> =A03) trigger pull rebuilds when trunk is updated<br> <br> =A04) add back in support for related pull requests (ie, two pulls that<br=

<br> =A05) consider adding updating the pull request on github with tester<br> =A0 =A0 results. =A0This one needs to be very carefully done to not spam t= he<br> =A0 =A0 report every time a build fails or succeeds.<br> <br> =A06) update the auto-tester grease monkey script to integrate the pull<br=

<br> I&#39;ll hopefully finish 1 and 2 tonight. =A0I can do 3 manually until it&= #39;s<br> automated. =A0I&#39;m not sure about the ordering of 4-6. =A0They&#39;re ni= ce to haves<br> rather than must haves.<br> <br> All these extra builds are going to cost a lot of time. =A0There&#39;s abou= t 100<br> open pull requests right now. =A0The fastest runs are on the order of 10<br=

loser<br> to an hour. =A0So, obviously there&#39;s some growing pains to deal with. = =A0I&#39;ll<br> probably add a way for github committers to prioritize pull requests so<br> they build first.<br> <br> Luckily this stuff is trivial to throw hardware at.. it&#39;s super<br> parallelizeable. =A0Also the hardware I have for those long runs is super<b= r> old. =A0I think the freebsd/32 box is p4 era box. =A0The win/32 is my asus = eee<br> atom based netbook.<br> <br> If anyone wants to volunteer build hardware, particularly for the<br> non-linux platforms, please contact me via email (let&#39;s not clutter up = the<br> newsgroup with that chatter).<br> <br> Requirements:<br> =A0I need to be able to access it remotely.<br> <br> =A0It needs to be have reliable connectivity (bandwidth is pretty much a<b= r> =A0non-issue.. it doesn&#39;t need much at all).<br> <br> =A0It needs to be hardware you&#39;re willing to have hammered fairly hard= at<br> =A0random times.<br> <br> I&#39;ll almost certainly write some code during the holiday to fire up and= <br> use ec2 nodes for windows and linux builds. =A0With the application of just= <br> a little money, all of those runs could be done fully parallel.. which<br> would just be sweet to see. =A0Ok, I admit it.. I love working at amazon on= <br> ec2.. and I&#39;m happy to finally have a project that could actually use i= t.<br> <br> Later,<br> Brad<br> </blockquote></div><br><div>&gt;=A0<span style>With the application of just= =A0</span><span style>a little money</span></div><div><span style><br></spa= n></div><div><font color=3D"#222222" face=3D"arial, sans-serif">Out of curi= osity, how much is a little?</font></div> --f46d0408d57930a64604b43c4a7d--
Dec 16 2011
prev sibling next sibling parent reply Brad Roberts <braddr puremagic.com> writes:
On 12/16/2011 1:29 PM, Brad Anderson wrote:
 On Thu, Dec 15, 2011 at 6:43 PM, Brad Roberts <braddr puremagic.com
<mailto:braddr puremagic.com>> wrote:
 
 
     Left to do:
 
      1) deploy changes to the tester hosts (it's on 2 already)

done
      2) finish the ui

very ugly but minimally functional: http://d.puremagic.com/test-results/pulls.ghtml
      3) trigger pull rebuilds when trunk is updated

partly implemented, but not being done yet
      4) add back in support for related pull requests (ie, two pulls that
         separately fail but together succeed)
 
      5) consider adding updating the pull request on github with tester
         results.  This one needs to be very carefully done to not spam the
         report every time a build fails or succeeds.
 
      6) update the auto-tester grease monkey script to integrate the pull
         tester results with github's ui.
 
     I'll hopefully finish 1 and 2 tonight.  I can do 3 manually until it's
     automated.  I'm not sure about the ordering of 4-6.  They're nice to haves
     rather than must haves.
 
     All these extra builds are going to cost a lot of time.  There's about 100
     open pull requests right now.  The fastest runs are on the order of 10
     minutes.  That's 6 per hour or roughly 17 hours.  The slowest are closer
     to an hour.  So, obviously there's some growing pains to deal with.  I'll
     probably add a way for github committers to prioritize pull requests so
     they build first.
 
     Luckily this stuff is trivial to throw hardware at.. it's super
     parallelizeable.  Also the hardware I have for those long runs is super
     old.  I think the freebsd/32 box is p4 era box.  The win/32 is my asus eee
     atom based netbook.
 
     If anyone wants to volunteer build hardware, particularly for the
     non-linux platforms, please contact me via email (let's not clutter up the
     newsgroup with that chatter).
 
     Requirements:
      I need to be able to access it remotely.
 
      It needs to be have reliable connectivity (bandwidth is pretty much a
      non-issue.. it doesn't need much at all).
 
      It needs to be hardware you're willing to have hammered fairly hard at
      random times.
 
     I'll almost certainly write some code during the holiday to fire up and
     use ec2 nodes for windows and linux builds.  With the application of just
     a little money, all of those runs could be done fully parallel.. which
     would just be sweet to see.  Ok, I admit it.. I love working at amazon on
     ec2.. and I'm happy to finally have a project that could actually use it.
 
     Later,
     Brad
 
 
 With the application of just a little money

Out of curiosity, how much is a little?

I'll need to experiment. It's the kind of thing that the more money thrown at the problem the faster the builds could be churned through. The limit of that would be one box per platform per pull request. A silly (and not possible due to ec2 not supporting some platforms, such as osx) extreme though. One c1.medium running 24x7 at the current spot price is about $29/month. Spot is perfect because this is a totally interruptable / resumable process. I'll need to do a little work to make it resume an in-flight test run, but it's not hard at all to do. After watching the build's slowly churn through the first round of build requests, it's clear to me that one of the areas I'll need to invest a good bit more time is in scheduling of pulls. Right now it's pseudo-random (an artifact of how the data is stored in a hash table). Later, Brad
Dec 16 2011
parent Robert Clipsham <robert octarineparrot.com> writes:
On 17/12/2011 06:40, Brad Roberts wrote:
 On 12/16/2011 1:29 PM, Brad Anderson wrote:
 On Thu, Dec 15, 2011 at 6:43 PM, Brad
Roberts<braddr puremagic.com<mailto:braddr puremagic.com>>  wrote:


      Left to do:

       1) deploy changes to the tester hosts (it's on 2 already)

done
       2) finish the ui

very ugly but minimally functional: http://d.puremagic.com/test-results/pulls.ghtml
       3) trigger pull rebuilds when trunk is updated

partly implemented, but not being done yet

Idea: I noticed most pull requests were failing when I looked at it, due to the main build failing - that's a lot of wasted computing time. Perhaps it would be a good idea to refuse to test pulls if dmd HEAD isn't compiling? This would be problematic for the 1/100 pull requests designed to fix this, but would save a lot of testing. An alternative method could be to test all of them, but if the pull request previously passed, then dmd HEAD broke, then the pull broke, stop testing until dmd HEAD is fixed. -- Robert http://octarineparrot.com/
Dec 17 2011
prev sibling next sibling parent Brad Anderson <eco gnuk.net> writes:
--f46d042fde60931b8d04b444f266
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Dec 16, 2011 at 11:40 PM, Brad Roberts <braddr puremagic.com> wrote:

 On 12/16/2011 1:29 PM, Brad Anderson wrote:
 On Thu, Dec 15, 2011 at 6:43 PM, Brad Roberts <braddr puremagic.com<mailto:

     Left to do:

      1) deploy changes to the tester hosts (it's on 2 already)

done
      2) finish the ui

very ugly but minimally functional: http://d.puremagic.com/test-results/pulls.ghtml
      3) trigger pull rebuilds when trunk is updated

partly implemented, but not being done yet
      4) add back in support for related pull requests (ie, two pulls that
         separately fail but together succeed)

      5) consider adding updating the pull request on github with tester
         results.  This one needs to be very carefully done to not spam

         report every time a build fails or succeeds.

      6) update the auto-tester grease monkey script to integrate the pull
         tester results with github's ui.

     I'll hopefully finish 1 and 2 tonight.  I can do 3 manually until

     automated.  I'm not sure about the ordering of 4-6.  They're nice to

     rather than must haves.

     All these extra builds are going to cost a lot of time.  There's

     open pull requests right now.  The fastest runs are on the order of

     minutes.  That's 6 per hour or roughly 17 hours.  The slowest are

     to an hour.  So, obviously there's some growing pains to deal with.

     probably add a way for github committers to prioritize pull requests

     they build first.

     Luckily this stuff is trivial to throw hardware at.. it's super
     parallelizeable.  Also the hardware I have for those long runs is

     old.  I think the freebsd/32 box is p4 era box.  The win/32 is my

     atom based netbook.

     If anyone wants to volunteer build hardware, particularly for the
     non-linux platforms, please contact me via email (let's not clutter

     newsgroup with that chatter).

     Requirements:
      I need to be able to access it remotely.

      It needs to be have reliable connectivity (bandwidth is pretty much

      non-issue.. it doesn't need much at all).

      It needs to be hardware you're willing to have hammered fairly hard

      random times.

     I'll almost certainly write some code during the holiday to fire up

     use ec2 nodes for windows and linux builds.  With the application of

     a little money, all of those runs could be done fully parallel..

     would just be sweet to see.  Ok, I admit it.. I love working at

     ec2.. and I'm happy to finally have a project that could actually

     Later,
     Brad


 With the application of just a little money

Out of curiosity, how much is a little?

I'll need to experiment. It's the kind of thing that the more money thrown at the problem the faster the builds could be churned through. The limit of that would be one box per platform per pull request. A silly (and not possible due to ec2 not supporting some platforms, such as osx) extreme though. One c1.medium running 24x7 at the current spot price is about $29/month. Spot is perfect because this is a totally interruptable / resumable process. I'll need to do a little work to make it resume an in-flight test run, but it's not hard at all to do. After watching the build's slowly churn through the first round of build requests, it's clear to me that one of the areas I'll need to invest a good bit more time is in scheduling of pulls. Right now it's pseudo-random (an artifact of how the data is stored in a hash table). Later, Brad

That seems very reasonable. It sounds like this autotester will help immensely with processing the pull request backlog. More free time for Walter and everyone else who works on the project is a great thing. --f46d042fde60931b8d04b444f266 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Fri, Dec 16, 2011 at 11:40 PM, Brad Roberts <span dir=3D"ltr">&lt;<a hre= f=3D"mailto:braddr puremagic.com">braddr puremagic.com</a>&gt;</span> wrote= :<br><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"= margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> On 12/16/2011 1:29 PM, Brad Anderson wrote:<br> &gt; On Thu, Dec 15, 2011 at 6:43 PM, Brad Roberts &lt;<a href=3D"mailto:br= addr puremagic.com">braddr puremagic.com</a> &lt;mailto:<a href=3D"mailto:b= raddr puremagic.com">braddr puremagic.com</a>&gt;&gt; wrote:<br> &gt;<br> &gt;<br> &gt; =A0 =A0 Left to do:<br> &gt;<br> &gt; =A0 =A0 =A01) deploy changes to the tester hosts (it&#39;s on 2 alread= y)<br> <br> done<br> <br> &gt; =A0 =A0 =A02) finish the ui<br> <br> very ugly but minimally functional: <a href=3D"http://d.puremagic.com/test-= results/pulls.ghtml" target=3D"_blank">http://d.puremagic.com/test-results/= pulls.ghtml</a><br> <br> &gt; =A0 =A0 =A03) trigger pull rebuilds when trunk is updated<br> <br> partly implemented, but not being done yet<br> <br> &gt; =A0 =A0 =A04) add back in support for related pull requests (ie, two p= ulls that<br> &gt; =A0 =A0 =A0 =A0 separately fail but together succeed)<br> &gt;<br> &gt; =A0 =A0 =A05) consider adding updating the pull request on github with= tester<br> &gt; =A0 =A0 =A0 =A0 results. =A0This one needs to be very carefully done t= o not spam the<br> &gt; =A0 =A0 =A0 =A0 report every time a build fails or succeeds.<br> &gt;<br> &gt; =A0 =A0 =A06) update the auto-tester grease monkey script to integrate= the pull<br> &gt; =A0 =A0 =A0 =A0 tester results with github&#39;s ui.<br> &gt;<br> &gt; =A0 =A0 I&#39;ll hopefully finish 1 and 2 tonight. =A0I can do 3 manua= lly until it&#39;s<br> &gt; =A0 =A0 automated. =A0I&#39;m not sure about the ordering of 4-6. =A0T= hey&#39;re nice to haves<br> &gt; =A0 =A0 rather than must haves.<br> &gt;<br> &gt; =A0 =A0 All these extra builds are going to cost a lot of time. =A0The= re&#39;s about 100<br> &gt; =A0 =A0 open pull requests right now. =A0The fastest runs are on the o= rder of 10<br> &gt; =A0 =A0 minutes. =A0That&#39;s 6 per hour or roughly 17 hours. =A0The = slowest are closer<br> &gt; =A0 =A0 to an hour. =A0So, obviously there&#39;s some growing pains to= deal with. =A0I&#39;ll<br> &gt; =A0 =A0 probably add a way for github committers to prioritize pull re= quests so<br> &gt; =A0 =A0 they build first.<br> &gt;<br> &gt; =A0 =A0 Luckily this stuff is trivial to throw hardware at.. it&#39;s = super<br> &gt; =A0 =A0 parallelizeable. =A0Also the hardware I have for those long ru= ns is super<br> &gt; =A0 =A0 old. =A0I think the freebsd/32 box is p4 era box. =A0The win/3= 2 is my asus eee<br> &gt; =A0 =A0 atom based netbook.<br> &gt;<br> &gt; =A0 =A0 If anyone wants to volunteer build hardware, particularly for = the<br> &gt; =A0 =A0 non-linux platforms, please contact me via email (let&#39;s no= t clutter up the<br> &gt; =A0 =A0 newsgroup with that chatter).<br> &gt;<br> &gt; =A0 =A0 Requirements:<br> &gt; =A0 =A0 =A0I need to be able to access it remotely.<br> &gt;<br> &gt; =A0 =A0 =A0It needs to be have reliable connectivity (bandwidth is pre= tty much a<br> &gt; =A0 =A0 =A0non-issue.. it doesn&#39;t need much at all).<br> &gt;<br> &gt; =A0 =A0 =A0It needs to be hardware you&#39;re willing to have hammered= fairly hard at<br> &gt; =A0 =A0 =A0random times.<br> &gt;<br> &gt; =A0 =A0 I&#39;ll almost certainly write some code during the holiday t= o fire up and<br> &gt; =A0 =A0 use ec2 nodes for windows and linux builds. =A0With the applic= ation of just<br> &gt; =A0 =A0 a little money, all of those runs could be done fully parallel= .. which<br> &gt; =A0 =A0 would just be sweet to see. =A0Ok, I admit it.. I love working= at amazon on<br> &gt; =A0 =A0 ec2.. and I&#39;m happy to finally have a project that could a= ctually use it.<br> &gt;<br> &gt; =A0 =A0 Later,<br> &gt; =A0 =A0 Brad<br> &gt;<br> &gt;<br> &gt;&gt; With the application of just a little money<br> &gt;<br> &gt; Out of curiosity, how much is a little?<br> <br> I&#39;ll need to experiment. =A0It&#39;s the kind of thing that the more mo= ney thrown at the problem the faster the builds could<br> be churned through. =A0The limit of that would be one box per platform per = pull request. =A0A silly (and not possible due to<br> ec2 not supporting some platforms, such as osx) extreme though. =A0One c1.m= edium running 24x7 at the current spot price is<br> about $29/month. =A0Spot is perfect because this is a totally interruptable= / resumable process. =A0I&#39;ll need to do a little<br> work to make it resume an in-flight test run, but it&#39;s not hard at all = to do.<br> <br> After watching the build&#39;s slowly churn through the first round of buil= d requests, it&#39;s clear to me that one of the<br> areas I&#39;ll need to invest a good bit more time is in scheduling of pull= s. =A0Right now it&#39;s pseudo-random (an artifact of<br> how the data is stored in a hash table).<br> <br> Later,<br> Brad<br> </blockquote></div><br><div><br></div><div>That seems very reasonable. =A0I= t sounds like this autotester will help immensely with processing the pull = request backlog. =A0More free time for Walter and everyone else who works o= n the project is a great thing.</div> --f46d042fde60931b8d04b444f266--
Dec 16 2011
prev sibling next sibling parent Brad Roberts <braddr puremagic.com> writes:
On 12/17/2011 4:56 AM, Robert Clipsham wrote:
 On 17/12/2011 06:40, Brad Roberts wrote:
 On 12/16/2011 1:29 PM, Brad Anderson wrote:
 On Thu, Dec 15, 2011 at 6:43 PM, Brad
Roberts<braddr puremagic.com<mailto:braddr puremagic.com>>  wrote:


      Left to do:

       1) deploy changes to the tester hosts (it's on 2 already)

done
       2) finish the ui

very ugly but minimally functional: http://d.puremagic.com/test-results/pulls.ghtml
       3) trigger pull rebuilds when trunk is updated

partly implemented, but not being done yet

Idea: I noticed most pull requests were failing when I looked at it, due to the main build failing - that's a lot of wasted computing time. Perhaps it would be a good idea to refuse to test pulls if dmd HEAD isn't compiling? This would be problematic for the 1/100 pull requests designed to fix this, but would save a lot of testing. An alternative method could be to test all of them, but if the pull request previously passed, then dmd HEAD broke, then the pull broke, stop testing until dmd HEAD is fixed.

Yeah. I know I need to do something in that space and just haven't yet. This whole thing is only a few evenings old and is just now starting to really work. I'm still focused on making sure it's grossly functional enough to be useful. Optimizations and polish will wait a little longer. Thanks, Brad
Dec 17 2011
prev sibling next sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Sat, 17 Dec 2011 20:54:24 +0100, Brad Roberts <braddr puremagic.com>  
wrote:

 On 12/17/2011 4:56 AM, Robert Clipsham wrote:
 On 17/12/2011 06:40, Brad Roberts wrote:
 On 12/16/2011 1:29 PM, Brad Anderson wrote:
 On Thu, Dec 15, 2011 at 6:43 PM, Brad  
 Roberts<braddr puremagic.com<mailto:braddr puremagic.com>>  wrote:


      Left to do:

       1) deploy changes to the tester hosts (it's on 2 already)

done
       2) finish the ui

very ugly but minimally functional: http://d.puremagic.com/test-results/pulls.ghtml
       3) trigger pull rebuilds when trunk is updated

partly implemented, but not being done yet

Idea: I noticed most pull requests were failing when I looked at it, due to the main build failing - that's a lot of wasted computing time. Perhaps it would be a good idea to refuse to test pulls if dmd HEAD isn't compiling? This would be problematic for the 1/100 pull requests designed to fix this, but would save a lot of testing. An alternative method could be to test all of them, but if the pull request previously passed, then dmd HEAD broke, then the pull broke, stop testing until dmd HEAD is fixed.

Yeah. I know I need to do something in that space and just haven't yet. This whole thing is only a few evenings old and is just now starting to really work. I'm still focused on making sure it's grossly functional enough to be useful. Optimizations and polish will wait a little longer. Thanks, Brad

Another optimization idea. Put pull request that fail to merge on an inactive list, send a comment to github and wait until the submitter does something about them. martin
Dec 19 2011
prev sibling parent Brad Roberts <braddr puremagic.com> writes:
On 12/19/2011 4:05 AM, Martin Nowak wrote:
 On Sat, 17 Dec 2011 20:54:24 +0100, Brad Roberts <braddr puremagic.com> wrote:
 
 On 12/17/2011 4:56 AM, Robert Clipsham wrote:
 On 17/12/2011 06:40, Brad Roberts wrote:
 On 12/16/2011 1:29 PM, Brad Anderson wrote:
 On Thu, Dec 15, 2011 at 6:43 PM, Brad
Roberts<braddr puremagic.com<mailto:braddr puremagic.com>>  wrote:


      Left to do:

       1) deploy changes to the tester hosts (it's on 2 already)

done
       2) finish the ui

very ugly but minimally functional: http://d.puremagic.com/test-results/pulls.ghtml
       3) trigger pull rebuilds when trunk is updated

partly implemented, but not being done yet

Idea: I noticed most pull requests were failing when I looked at it, due to the main build failing - that's a lot of wasted computing time. Perhaps it would be a good idea to refuse to test pulls if dmd HEAD isn't compiling? This would be problematic for the 1/100 pull requests designed to fix this, but would save a lot of testing. An alternative method could be to test all of them, but if the pull request previously passed, then dmd HEAD broke, then the pull broke, stop testing until dmd HEAD is fixed.

Yeah. I know I need to do something in that space and just haven't yet. This whole thing is only a few evenings old and is just now starting to really work. I'm still focused on making sure it's grossly functional enough to be useful. Optimizations and polish will wait a little longer. Thanks, Brad

Another optimization idea. Put pull request that fail to merge on an inactive list, send a comment to github and wait until the submitter does something about them. martin

Way ahead of you, but low priority on it from a throughput standpoint. Those take almost no time to process. The benefit there is in getting the notification back to the pull submitter. But that's true for all failures. :) The biggest win I can think of right now is what I'll work on next: if one platform has failed the build, skip it unless there's nothing else to do. With that, only one build is wasted, and only on the platform that's making the fastest progress anyway. Later, Brad
Dec 19 2011