www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Volunteer for research project?

reply Brad Roberts <braddr puremagic.com> writes:
Would any of you be interested in helping out (read that as "doing") a research
/ data mining project for us?  I'd love
to take all of the regressions this year (or for the last year, or whatever
period of time can be reasonably
accomplished) and track them back to which commit introduced each of them
(already done for some of them).  From there,
I'd like to see what sort of correlations can be found.  Is there a particular
area of code that's responsible for them.
 Is there a particular feature (spread across a lot of files, maybe) that's
responsible.  Etc.

Maybe it's all over the map.  Maybe it will highlight one or a few areas to
take a harder look at.

Anyone interested?

Thanks,
Brad
Feb 20 2013
parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 21 February 2013 at 07:03:08 UTC, Brad Roberts wrote:
 Would any of you be interested in helping out (read that as 
 "doing") a research / data mining project for us?  I'd love
 to take all of the regressions this year (or for the last year, 
 or whatever period of time can be reasonably
 accomplished) and track them back to which commit introduced 
 each of them (already done for some of them).  From there,
 I'd like to see what sort of correlations can be found.  Is 
 there a particular area of code that's responsible for them.
  Is there a particular feature (spread across a lot of files, 
 maybe) that's responsible.  Etc.

 Maybe it's all over the map.  Maybe it will highlight one or a 
 few areas to take a harder look at.

 Anyone interested?

 Thanks,
 Brad

It sounds interesting, but what are you expecting to found? And how much are you sure you can found something? I would expect that often code which fixes some feature breaks the same feature in another aspect of functioning which is quite obvious. Sometimes one code relies implicitly on functioning of other code, so when you change the the latter, the former stops working correctly. You provide example with spreading across several files - how does knowing this helps in reducing regressions?
Feb 21 2013
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Feb 22, 2013 at 06:51:53AM +0100, Maxim Fomin wrote:
 On Thursday, 21 February 2013 at 07:03:08 UTC, Brad Roberts wrote:
Would any of you be interested in helping out (read that as "doing")
a research / data mining project for us?  I'd love to take all of the
regressions this year (or for the last year, or whatever period of
time can be reasonably accomplished) and track them back to which
commit introduced each of them (already done for some of them).  From
there, I'd like to see what sort of correlations can be found.  Is
there a particular area of code that's responsible for them.  Is
there a particular feature (spread across a lot of files, maybe)
that's responsible.  Etc.

Maybe it's all over the map.  Maybe it will highlight one or a few
areas to take a harder look at.

Anyone interested?

Thanks,
Brad

It sounds interesting, but what are you expecting to found? And how much are you sure you can found something? I would expect that often code which fixes some feature breaks the same feature in another aspect of functioning which is quite obvious. Sometimes one code relies implicitly on functioning of other code, so when you change the the latter, the former stops working correctly. You provide example with spreading across several files - how does knowing this helps in reducing regressions?

I would think he's referring to issues that are filed in the bugtracker. Obviously, we have no way of knowing if a code change broke something if nobody found any bug afterwards! So I'm thinking it's probably a matter of going through the regression bugs in the bugtracker, and making test cases to reproduce them, and then use git bisect to figure out which commit introduced the problem. T -- Public parking: euphemism for paid parking. -- Flora
Feb 21 2013
parent "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Friday, 22 February 2013 at 06:02:20 UTC, H. S. Teoh wrote:
 I would think he's referring to issues that are filed in the 
 bugtracker.
 Obviously, we have no way of knowing if a code change broke 
 something if
 nobody found any bug afterwards!

Yes, it is obvious that he refers to bugzilla issues.
 So I'm thinking it's probably a matter of going through the 
 regression
 bugs in the bugtracker, and making test cases to reproduce 
 them, and
 then use git bisect to figure out which commit introduced the 
 problem.


 T

This is also obvious. The question is what to do with such information next, how to analyze it and interpret the results. For example http://d.puremagic.com/issues/show_bug.cgi?id=9406 (there is commit which introduced regression). What can you infer from fixed regressions (http://d.puremagic.com/issues/buglist.cgi?query_format=advanced&bug_severity=regression&bug_status=RESOLV D&resolution=FIXED) which can be useful in fighting against non-closed ones? P.S. There is something wrong either with forum or with your answering. The discussion in mailbox is single piece, but in forum it is splitted into two threads. Posting message in one thread in answering to reply in another is strange. Do you use email for answering or forum?
Feb 21 2013
prev sibling parent Brad Roberts <braddr puremagic.com> writes:
On 2/21/2013 10:00 PM, H. S. Teoh wrote:
 On Fri, Feb 22, 2013 at 06:51:53AM +0100, Maxim Fomin wrote:
 On Thursday, 21 February 2013 at 07:03:08 UTC, Brad Roberts wrote:
 Would any of you be interested in helping out (read that as "doing")
 a research / data mining project for us?  I'd love to take all of the
 regressions this year (or for the last year, or whatever period of
 time can be reasonably accomplished) and track them back to which
 commit introduced each of them (already done for some of them).  From
 there, I'd like to see what sort of correlations can be found.  Is
 there a particular area of code that's responsible for them.  Is
 there a particular feature (spread across a lot of files, maybe)
 that's responsible.  Etc.

 Maybe it's all over the map.  Maybe it will highlight one or a few
 areas to take a harder look at.

 Anyone interested?

 Thanks,
 Brad

It sounds interesting, but what are you expecting to found? And how much are you sure you can found something? I would expect that often code which fixes some feature breaks the same feature in another aspect of functioning which is quite obvious. Sometimes one code relies implicitly on functioning of other code, so when you change the the latter, the former stops working correctly. You provide example with spreading across several files - how does knowing this helps in reducing regressions?

I would think he's referring to issues that are filed in the bugtracker. Obviously, we have no way of knowing if a code change broke something if nobody found any bug afterwards! So I'm thinking it's probably a matter of going through the regression bugs in the bugtracker, and making test cases to reproduce them, and then use git bisect to figure out which commit introduced the problem. T

Pretty much that. (Nearly) every bug comes with a test case already. The part that will be work is taking that test case and finding the exact commit that broke it. By definition, a regression once worked and something changed that broke it. My hope is that one or more people can spend some time going through each regression report in bugzilla and tracking down the exact commit for each. What will be uncovered by the effort? Who knows. It's better to not try to anticipate or predict since that can bias the analysis. The entire point of the exercise is to find out. If there is one or move obvious or detectible clusters, that gives us some interesting data. It might well point out a part of the code that's particularly sensitive to change. Or is very poorly covered by the test suite. Or is flawed in some other way. Regardless, if there are clusters, it's worth some study and pondering to consider what can be done to make it/them NOT hot beds of regressions. It's a research project. It might turn out to yield nothing useful. That's certainly a risk. I suspect it won't turn out to be fruitless. To seed the effort, here's all the regression bugs that have changed since the beginning of the year: http://d.puremagic.com/issues/buglist.cgi?chfieldto=Now&query_format=advanced&chfieldfrom=2013-01-01&bug_severity=regression&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED
Feb 21 2013