www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Bugzilla Reward System

reply Mike Parker <aldacron gmail.com> writes:
In my summary of last month's D Language Foundation meeting, I 
mentioned that we discussed a system intended to reward 
contributors who contribute pull requests that fix Bugzilla 
issues. This was Razvan Nitu's baby from conception to 
implementation, and we all think it is a great idea.

The system has been running in the background for a few weeks 
now, quietly gathering data and awarding points, proving that the 
programming side works as intended. Our first round of point 
scoring kicks off on September 20th. Razvan has put together a 
blog post explaining how the system works.

We'll revise and adapt the system as needed as time goes by. In 
the meantime, happy bug fixing!

The blog:
https://dlang.org/blog/2021/09/16/bugzilla-reward-system/

Reddit:
https://www.reddit.com/r/d_language/comments/ppbp7d/bugzilla_reward_system/
Sep 16 2021
next sibling parent ag0aep6g <anonymous example.com> writes:
On 16.09.21 13:56, Mike Parker wrote:

 The blog:
 https://dlang.org/blog/2021/09/16/bugzilla-reward-system/


 From there:



the patch.
 
 This is already an unwritten rule that applies to the DLang repositories, so
it should not surprise anyone.


Someone tell Walter about that unwritten rule. He regularly merges his 
own PRs.
Sep 16 2021
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Thursday, 16 September 2021 at 11:56:21 UTC, Mike Parker wrote:

 https://dlang.org/blog/2021/09/16/bugzilla-reward-system/


 From the post:


 The scoring is designed to reward contributors based on the 
 importance of the issues they fix, rather than the total number 
 fixed. As such, issues are awarded points based on severity:


In my experience, the only severity settings most people actually 
use when filing issues on Bugzilla are "enhancement", "normal", 
and "regression". And when people do use the other settings, 
there's no consistency to how they get applied. For example, the 
first two search results for priority "blocker", issues [22283][] 
and [22148][], have no indication of what (if anything) they 
block. Meanwhile, issues [14196][] and [13983][] are both 
enhancement requests but have their priority set to "major", and 
issue [22136][] is listed as "critical" even though it is 
actually a regression!

I don't blame anyone who files reports like these. The fact is, 
there is no official guidance anywhere about what distinguishes a 
"minor" issue from a "normal" one, or a "normal" issue from a 
"major" one, so people just guess. But treating the output of 
this guessing process as though it were meaningful data is 
probably a mistake.

[22283]: https://issues.dlang.org/show_bug.cgi?id=22283
[22148]: https://issues.dlang.org/show_bug.cgi?id=22148
[14196]: https://issues.dlang.org/show_bug.cgi?id=14196
[13983]: https://issues.dlang.org/show_bug.cgi?id=13983
[22136]: https://issues.dlang.org/show_bug.cgi?id=22136
Sep 16 2021
next sibling parent Mike Parker <aldacron gmail.com> writes:
On Thursday, 16 September 2021 at 14:35:08 UTC, Paul Backus wrote:


 In my experience, the only severity settings most people 
 actually use when filing issues on Bugzilla are "enhancement", 
 "normal", and "regression". And when people do use the other 
 settings, there's no consistency to how they get applied. For 
 example, the first two search results for priority "blocker", 
 issues [22283][] and [22148][], have no indication of what (if 
 anything) they block. Meanwhile, issues [14196][] and [13983][] 
 are both enhancement requests but have their priority set to 
 "major", and issue [22136][] is listed as "critical" even 
 though it is actually a regression!





 Severity levels are not always accurately set when issues are 
 first reported and may not have been updated since. The 
 reviewer of a pull request that closes a Bugzilla issue will 
 evaluate the issue’s severity level and may change it if he or 
 she determines it is inaccurate. I will moderate any 
 disagreements that may arise about severity levels.


Obviously, this isn't perfect, but it's the best we've got for 
the moment. I expect there will be quite a few kinks to work out 
as time goes by. The initial trial run will surely provide us 
with lessons we can apply going forward, and we will continue to 
refine the process as needed.


 I don't blame anyone who files reports like these. The fact is, 
 there is no official guidance anywhere about what distinguishes 
 a "minor" issue from a "normal" one, or a "normal" issue from a 
 "major" one, so people just guess. But treating the output of 
 this guessing process as though it were meaningful data is 
 probably a mistake.


They aren't meaningful because there has been no organized 
process to make them meaningful. Part of Razvan's job description 
is to oversee the Bugzilla issues, and this initiative is a 
product of that. But he's one man, and the issues are legion :-) 
I predict we'll see the publication of reviewer guidelines down 
the road, and will eventually ask for a handful of people to 
assist Razvan in making sure Bugzilla issues have roughly 
accurate severity levels before a PR is submitted. Those are the 
most obvious steps I see to improving accuracy.
Sep 16 2021
prev sibling parent reply RazvanN <razvan.nitu1305 gmail.com> writes:
On Thursday, 16 September 2021 at 14:35:08 UTC, Paul Backus wrote:

 On Thursday, 16 September 2021 at 11:56:21 UTC, Mike Parker 
 wrote:

 https://dlang.org/blog/2021/09/16/bugzilla-reward-system/


 From the post:


 The scoring is designed to reward contributors based on the 
 importance of the issues they fix, rather than the total 
 number fixed. As such, issues are awarded points based on 
 severity:


 In my experience, the only severity settings most people 
 actually use when filing issues on Bugzilla are "enhancement", 
 "normal", and "regression". And when people do use the other 
 settings, there's no consistency to how they get applied. For 
 example, the first two search results for priority "blocker", 
 issues [22283][] and [22148][], have no indication of what (if 
 anything) they block. Meanwhile, issues [14196][] and [13983][] 
 are both enhancement requests but have their priority set to 
 "major", and issue [22136][] is listed as "critical" even 
 though it is actually a regression!

 I don't blame anyone who files reports like these. The fact is, 
 there is no official guidance anywhere about what distinguishes 
 a "minor" issue from a "normal" one, or a "normal" issue from a 
 "major" one, so people just guess. But treating the output of 
 this guessing process as though it were meaningful data is 
 probably a mistake.

 [22283]: https://issues.dlang.org/show_bug.cgi?id=22283
 [22148]: https://issues.dlang.org/show_bug.cgi?id=22148
 [14196]: https://issues.dlang.org/show_bug.cgi?id=14196
 [13983]: https://issues.dlang.org/show_bug.cgi?id=13983
 [22136]: https://issues.dlang.org/show_bug.cgi?id=22136


Given that points are obtained depending on severity, my 
expectation is that reviewers will pay more attention to it when 
a PR is submitted. In addition, people that try to score as much 
points as possible will be interested in making sure that the 
competition does get the right amount of points. Therefore, I 
think that the rewarding system will improve the status quo with 
regards to labeling bugs.
Sep 17 2021
parent Paul Backus <snarwin gmail.com> writes:
On Friday, 17 September 2021 at 12:39:42 UTC, RazvanN wrote:

 Given that points are obtained depending on severity, my 
 expectation is that reviewers will pay more attention to it 
 when a PR is submitted. In addition, people that try to score 
 as much points as possible will be interested in making sure 
 that the competition does get the right amount of points. 
 Therefore, I think that the rewarding system will improve the 
 status quo with regards to labeling bugs.


My "null hypothesis" expectation is that, absent guidance, 
reviewers will continue to do what they have always done, which 
is to pay no attention to the severity at all (except for 
regressions).

I think that this system has the *potential* to improve the 
status quo, but it will take some effort to actually get 
reviewers to change their habits. At minimum, we will need to

1. Write down a set of guidelines for how to use the "severity" 
field on Bugzilla.
2. Copy+paste and/or link to those guidelines in all of our 
existing instructions for contributors, including:
     * The `CONTRIBUTING.md` files of the `dmd`, `druntime`, and 
`phobos` repositories.
     * Wiki articles like [Get involved][1], [Starting as a 
Contributor][2], and [Guidelines for maintainers][3].
     * The comment that [dlang-bot][4] adds automatically to all 
new PRs.

[1]: https://wiki.dlang.org/Get_involved
[2]: https://wiki.dlang.org/Starting_as_a_Contributor
[3]: https://wiki.dlang.org/Guidelines_for_maintainers
[4]: https://github.com/dlang/dlang-bot
Sep 17 2021
prev sibling next sibling parent M.M. <matus email.cz> writes:
On Thursday, 16 September 2021 at 11:56:21 UTC, Mike Parker wrote:

 ...
 We'll revise and adapt the system as needed as time goes by. In 
 the meantime, happy bug fixing!

 The blog:
 https://dlang.org/blog/2021/09/16/bugzilla-reward-system/
 ...


Nice idea to reward contributors. Happy to see that you just try 
and see how it works, and adjust if needed. I think the overall 
positive synergy of the community is important, and this 
initiative should not damage it. To achieve this, I would suggest 
to consider giving more than 3 prizes each evaluation period. 
Furthermore, I would suggest to think about rewarding "rookies" 
as well... But let's first see how this works.
Sep 16 2021
prev sibling next sibling parent Basile B. <b2.temp gmx.com> writes:
On Thursday, 16 September 2021 at 11:56:21 UTC, Mike Parker wrote:

 In my summary of last month's D Language Foundation meeting, I 
 mentioned that we discussed a system intended to reward 
 contributors who contribute pull requests that fix Bugzilla 
 issues. This was Razvan Nitu's baby from conception to 
 implementation, and we all think it is a great idea.

 The system has been running in the background for a few weeks 
 now, quietly gathering data and awarding points, proving that 
 the programming side works as intended. Our first round of 
 point scoring kicks off on September 20th. Razvan has put 
 together a blog post explaining how the system works.

 We'll revise and adapt the system as needed as time goes by. In 
 the meantime, happy bug fixing!

 The blog:
 https://dlang.org/blog/2021/09/16/bugzilla-reward-system/

 Reddit:
 https://www.reddit.com/r/d_language/comments/ppbp7d/bugzilla_reward_system/


This is a good move, I hope it will work so that D will keep 
contributors that are not already gone and gain new talented ones.

Another answer mentions that the lack of "issue triage" might 
cause problems.
I think to the opposite that this system could encourage

1. better triage
2. better reviews

But well, we'll see if this works next year. Let's not be 
negative ;)
Sep 16 2021
prev sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 9/16/21 4:56 AM, Mike Parker wrote:


 This was Razvan Nitu's baby from conception to implementation


Thank you, Razvan! Great job and a great article.

What I missed in the article is whether we are going to reward all 
contributors or whether certain people like Walter are excused? :)

Also, if a regression is best fixed by the person who caused it in the 
first place, regressions may become a good thing. ;)

Ali
Sep 16 2021
parent RazvanN <razvan.nitu1305 gmail.com> writes:
On Friday, 17 September 2021 at 01:07:09 UTC, Ali Çehreli wrote:

 On 9/16/21 4:56 AM, Mike Parker wrote:


 This was Razvan Nitu's baby from conception to implementation


 Thank you, Razvan! Great job and a great article.


Thank you, Ali!


 What I missed in the article is whether we are going to reward 
 all contributors or whether certain people like Walter are 
 excused? :)


Foundation people like Walter and myself are not going to be 
rewarded, however,
unaffiliated titans such as Iain, Vladimir and kinke have the 
opportunity of being rewarded. Of course, it is their decision if 
they do or don't want to participate in the race. Either way, the 
scoring system is going to track everyone's activity and then we 
can exclude foundation members and take into account potential 
prize yields.



 Also, if a regression is best fixed by the person who caused it 
 in the first place, regressions may become a good thing. ;)


If you are hinting at intentionally introduced regressions, I 
think that it is the burden of the reviewer to catch them at 
review time. If not, it really depends on who has time to fix it. 
I, personally, fixed tons of regressions and only a small part of 
those where introduced by me. I think that we can just assume 
good faith and acknowledge the fact that people make mistakes and 
it's fine to reward them if they fix them.

Cheers,
RazvanN


 Ali
Sep 17 2021