www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dsource and ranking vs. rating

reply Knud Soerensen <4tuu4k002 sneakemail.com> writes:
In the tread DSSS, Dsource and cpan it have been suggested that dsource should
implement a rating system like on the matlab site and the firefox extension
site. 

I would like to suggest a ranking system instead of the rating system.

In a ranking system every user have a list with rankings,
when ranking a new project the user add it into the ranking between 
two other projects.

Some of the problem i see with a rating system is:

  * Different users might use different bases.
  One user might examine code quality, user interface, documentation etc.
  and give 3 points on basis on this analyses.
  Another might just give 5 point to every project he likes.
  That a ranking system compare the project with each other help make a
  common basis for reference.  


  * It difficult for a user to change his base.
  Imagine a user discover a new brilliant project 
  that is much better than the project he already
  have rated with max points. 
  To rate the new project fairly he would have 
  to go back and re-rate all his rated projects. 
  With ranking he can just put it at the top.

  * Low differentiation between top projects.
  Given most users only rating the projects they like.
  As a result,an extremely good and a very good project will have almost the
same average score and 
  the total sum will tell more about the popularity than how good it is.   
   

A advantages of the ranking system is that it provide 
much more data for data mining.

Your can extract a value and make a simple average like with a ranking.
Your can run most voting system on the data, but you also have data which can 
tell you that n users think that project x is better than project y. etc.
Apr 17 2007
next sibling parent reply Brad Anderson <brad dsource.org> writes:
Knud Soerensen wrote:
 In the tread DSSS, Dsource and cpan it have been suggested that dsource should
 implement a rating system like on the matlab site and the firefox extension
site. 
 
 I would like to suggest a ranking system instead of the rating system.
 
 In a ranking system every user have a list with rankings,
 when ranking a new project the user add it into the ranking between 
 two other projects.
 
 Some of the problem i see with a rating system is:
 
   * Different users might use different bases.
   One user might examine code quality, user interface, documentation etc.
   and give 3 points on basis on this analyses.
   Another might just give 5 point to every project he likes.
   That a ranking system compare the project with each other help make a
   common basis for reference.  
 
 
   * It difficult for a user to change his base.
   Imagine a user discover a new brilliant project 
   that is much better than the project he already
   have rated with max points. 
   To rate the new project fairly he would have 
   to go back and re-rate all his rated projects. 
   With ranking he can just put it at the top.
 
   * Low differentiation between top projects.
   Given most users only rating the projects they like.
   As a result,an extremely good and a very good project will have almost the
same average score and 
   the total sum will tell more about the popularity than how good it is.   
    
 
 A advantages of the ranking system is that it provide 
 much more data for data mining.
 
 Your can extract a value and make a simple average like with a ranking.
 Your can run most voting system on the data, but you also have data which can 
 tell you that n users think that project x is better than project y. etc.

Interesting... How does this handle the differences in projects' goals? i.e. rating Tango higher than Derelict - because they're two different categories, or goals for end-user use? BA
Apr 17 2007
parent Knud Soerensen <4tuu4k002 sneakemail.com> writes:
On Tue, 17 Apr 2007 13:53:59 -0400, Brad Anderson wrote:
 Interesting... How does this handle the differences in projects' goals?   i.e.
 rating Tango higher than Derelict - because they're two different categories,
 or goals for end-user use?

What you can do is select the rankings base on the projects category and thereby calculate the projects ranking within the category.
Apr 17 2007
prev sibling next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Knud Soerensen wrote:
 Some of the problem i see with a rating system is:

Any ranking system is going to have misleading, unfair and inaccurate results. So will any ranking system using annointed moderators. Any attempt to make fine distinctions based on a ranking system is doomed to failure. But looking at it with a broad brush will be, I predict, fine. Look at sites with user or moderator ranking systems - slashdot, digg, imdb, amazon, download, youtube. By and large, they are successful at separating the ones worth a second look from the ones not.
Apr 17 2007
next sibling parent BCS <BCS pathlink.com> writes:
Walter Bright wrote:
 Knud Soerensen wrote:
 
 Some of the problem i see with a rating system is:

Any ranking system is going to have misleading, unfair and inaccurate results. So will any ranking system using annointed moderators. Any attempt to make fine distinctions based on a ranking system is doomed to failure. But looking at it with a broad brush will be, I predict, fine. Look at sites with user or moderator ranking systems - slashdot, digg, imdb, amazon, download, youtube. By and large, they are successful at separating the ones worth a second look from the ones not.

How about whatever system is used, when the user asks for data, they are never given actual numbers. In place of numbers the server converts the scores to percentile and then gives them all a random fuzz factor of about 5% and then orders them and spits out an order. That way if 2 things are within 5-10% they will get shuffled back and forth somewhat randomly.
Apr 17 2007
prev sibling next sibling parent reply Knud Soerensen <4tuu4k002 sneakemail.com> writes:
On Tue, 17 Apr 2007 11:30:35 -0700, Walter Bright wrote:

 Knud Soerensen wrote:
 Some of the problem i see with a rating system is:

Any ranking system is going to have misleading, unfair and inaccurate results. So will any ranking system using annointed moderators. Any attempt to make fine distinctions based on a ranking system is doomed to failure.

But from the users ranking in my way, you can generate many types of ranking lists and here you also have data of how the project compared to each other something which you don't have in ordinary rating. A thing I use in eigenpolls. http://all-technology.com/eigenpolls/
 But looking at it with a broad brush will be, I predict, fine. Look at 
 sites with user or moderator ranking systems - slashdot, digg, imdb, 
 amazon, download, youtube. By and large, they are successful at 
 separating the ones worth a second look from the ones not.

What I don't like with these type of moderation systems is that a expert option very often is down moderated by what I call school kids. (People which blindly believe what they learned in school.) What i like best is amazons "the users also bought this books" system but again I have discover better books with my eigenpolls.
Apr 17 2007
parent Walter Bright <newshound1 digitalmars.com> writes:
Knud Soerensen wrote:
 What I don't like with these type of moderation systems is
 that a expert option very often is down moderated by what I call 
 school kids.  (People which blindly believe what they learned in school.)

Sure, that's the big problem with democracy. On the other hand, every other system tried is worse.
Apr 17 2007
prev sibling parent Knud Soerensen <4tuu4k002 sneakemail.com> writes:
On Tue, 17 Apr 2007 13:58:23 -0700, Walter Bright wrote:

 Knud Soerensen wrote:
 What I don't like with these type of moderation systems is
 that a expert option very often is down moderated by what I call 
 school kids.  (People which blindly believe what they learned in school.)

Sure, that's the big problem with democracy. On the other hand, every other system tried is worse.

First I don't think that you should uncritical compare a comment moderation system with a system for electing government. When I comments on slashdot.org i like to read good comments from expert on the subject not the most comment option on the subject because that option I properly know already. The same way the ranking/rating on dsource I see as tool to discover good software not just popular software. When making decisions for a whole group the option of the majority is the best way to go. But to make a good decision about a problem it is important that the good solutions is discovered and analysed and it is this discovery part we are trying to solve. And last if we should compare the ranking system to democracy it is not a question of for or against but a question about what type of voting system. http://en.wikipedia.org/wiki/Voting_system And my suggestion would be a systems that allow one to capture the voters option about each candidate and then allow one to calculate the result of a plurality, runoff, approval or ranked vote.
Apr 17 2007
prev sibling next sibling parent reply Manfred Nowak <svv1999 hotmail.com> writes:
Knud Soerensen wrote

 when ranking a new project the user add it into the ranking
 between two other projects.

Not quite because this implies that the first two projects cannot be ranked without a third. Ranking ideally only compares two projects. But ranking has the same vulnerabilities as rating in respect to "bad guys". -manfred
Apr 17 2007
parent reply Knud Soerensen <4tuu4k002 sneakemail.com> writes:
On Tue, 17 Apr 2007 18:52:12 +0000, Manfred Nowak wrote:

 Knud Soerensen wrote
 
 when ranking a new project the user add it into the ranking
 between two other projects.

Not quite because this implies that the first two projects cannot be ranked without a third. Ranking ideally only compares two projects.

Sorry, of cause you can also put the project at the top or at the bottom of you list. You can thing of the list as a favorite list with ranking added.
 But ranking has the same vulnerabilities as rating in respect to "bad 
 guys".

I think that it might be best you compute different types of rating form the ranking lists. So, gaming might win on one type of rating but lose on another type.
 -manfred

Apr 17 2007
next sibling parent Dan <murpsoft hotmail.com> writes:
I tend to think that both have obvious deficiencies.

I would personally implement:

[] Did you use the project
[] Did you examine the source

[1 2 3 4 5] The maintainer is my best friend/family [ comments ]
[1 2 3 4 5] Project satisfied my needs [ comments ]
[1 2 3 4 5] Project was well designed [ comments ]
[1 2 3 4 5] Project was well documented [ comments ]
[1 2 3 4 5] Project performed well [ comments ]

This sort of system could then be fed into a user-defined weighted ranking
system.  If the user only cares about performance, they can ask for it.  If
they insist on documentation, they can ask for that.

That way we can search by our own priorities, rather than the priorities of the
people who voted.  We can also weed out people who vote without knowing shit,
or vote just because they like someone or hate someone else.

Developers can also get feedback from the comments and rankings so they know
what to work on rather than simply getting a "your project sucks" big fat zero.
Apr 17 2007
prev sibling parent Manfred Nowak <svv1999 hotmail.com> writes:
Knud Soerensen wrote

 Sorry, of cause you can also put the project at the top or at the
 bottom of you list.

That is still overly restrictive. In essence you require voters to impose a total ordering on the set of items they choose to vote on. But voters usually only have a partial knowledge of any such set and therefore only a partial ordering available. Every imposed total ordering then forces the voter to give out already distorted data. The evaluator of such a poll then goes off with this distorted data and some algorithm that is believed to produce a reliable result on undistorted data. The evaluator then preents some pseudo acribic ranking/rating like "item1 has score 0.393, item2 has score 0.374, ..." and requires the audience to conclude that item1 should be their choice. But Walter got it right when he wrote in this thread: | By and large, they are successful at separating the ones worth a | second look from the ones not. Everyone who is able to improve this outcome has found a philosopher's stone---and a future without qualms. -manfred
Apr 18 2007
prev sibling parent reply Carlos Santander <csantander619 gmail.com> writes:
Knud Soerensen escribió:
 In the tread DSSS, Dsource and cpan it have been suggested that dsource should
 implement a rating system like on the matlab site and the firefox extension
site. 
 
 I would like to suggest a ranking system instead of the rating system.
 
 In a ranking system every user have a list with rankings,
 when ranking a new project the user add it into the ranking between 
 two other projects.
 
 Some of the problem i see with a rating system is:
 
   * Different users might use different bases.
   One user might examine code quality, user interface, documentation etc.
   and give 3 points on basis on this analyses.
   Another might just give 5 point to every project he likes.
   That a ranking system compare the project with each other help make a
   common basis for reference.  
 
 
   * It difficult for a user to change his base.
   Imagine a user discover a new brilliant project 
   that is much better than the project he already
   have rated with max points. 
   To rate the new project fairly he would have 
   to go back and re-rate all his rated projects. 
   With ranking he can just put it at the top.
 
   * Low differentiation between top projects.
   Given most users only rating the projects they like.
   As a result,an extremely good and a very good project will have almost the
same average score and 
   the total sum will tell more about the popularity than how good it is.   
    
 
 A advantages of the ranking system is that it provide 
 much more data for data mining.
 
 Your can extract a value and make a simple average like with a ranking.
 Your can run most voting system on the data, but you also have data which can 
 tell you that n users think that project x is better than project y. etc.

What would be more important in such a system, the absolute position or the relative position? Say I have rated 5 projects (A-E), and someone else has rated 10 (F-O). Has E been ranked the same as J or as O? -- Carlos Santander Bernal
Apr 17 2007
parent Manfred Nowak <svv1999 hotmail.com> writes:
Carlos Santander wrote
 What would be more important in such a system, the absolute
 position or the relative position?

There exists a bunch of characteristica that might be of interest: http://en.wikipedia.org/wiki/Kemeny-Young_method#Characteristics Sorrily I could not find any evaluations of these characteristica for Knud's ranking method "eigenpoll". -manfred
Apr 17 2007