Posted on: January 30, 2008 10:06 am
Edited on: January 31, 2008 5:33 pm

Ratings Changes Effective Tomorrow

In this thread we announced some upcoming changes to the way posts are rated and how those ratings impact the reputation:


Now that the changes have been made I want to describe them here as the thread above may not have gotten all the details exactly right.

Why are we making this change?
There are two main reasons for us doing this:
  1. Members were using the ratings as currency to either game the system (content free ten-fest threads) or to annoy or get back at other posters for disagreements. Neither of those behaviors were good for the community as a whole.
  2. Members are not too interested in giving low ratings, and they just want to use the ratings to highlight good content. Perhaps some of the apprehension towards giving lower than the highest rating is due to fear of backslash.
So what is the change?
The changes we are making, in a nutshell is to simply remove the negative impact that any rating can have on a user's reputation. So even if someone gives you 1 star, that's still better than not getting a rating. The more ratings you get for each post the better your reputation will be.

Now for the details...
A lot of factors go into converting the number of stars someone gives you into the actual number that affects your reputation:
  1. The reputation of the user that submitted the rating. The higher someone's reputation is the more weight their rating has
  2. The scale the reviewer uses to rate posts. This is the dreaded 5 stars to 2.5 star adjustment. Users that tend to give everyone 5 stars will have their reviews be worth 2.5 stars only. While those that spread them around, will keep most of their value. This is to encourage users to use the entire spectrum of rating values and create more distinction between good and great posts. Those that do not want to do this are not hurt by this, and getting 2.5 stars is still better than getting none, so you may continue with this behavior without any consequence.
  3. The type of content being rated. Each type of content we allow has an internal weight relative to the amount of time a user spends writing or reading that type of content. You are expected to put more time into a blog entry than into a reply in a message board, so it's logical that those have a higher weight.
The total ratings a user gets across all their content is then added up and divided by the number of posts made by that user. Posting more messages will not help this part of the reputation calculation. (It will however help the participation part).
An Example
Let's walk thru a simple example (some of the math has been simplified here for the sake of clarity):

Let's say member X posts a message and it gets rated by:
  • Member A, who gives him 5 stars and has a reputation of 97. Member A only gives out 5 stars, so his rating is adjusted to 2.5. Multiplying the reputation (as a percentage) we get 2.5*.97 = 2.43.
  • Member B, who gives him 4 stars and has a reputation of 50. Member B spreads out his ratings so this one stays at 4, and factoring in the reputation becomes 4*.5 = 2.0.
  • Member C, who gives him 5 stars and has a reputation of 80. Member C gives mostly 5 stars, but a few others so his is adjusted to 3.5. Factoring in the reputation, we get 3.5*.8 = 2.80.
Then all these numbers are added: 2.43+2.0+2.8 = 7.23.

Now let's see what happens when Member D who is a troll with a reputation of 15 and gives him 1 star (which is not adjusted any lower). That will then result in a total of 1*.15 = 0.15. So this value will be added to the prior total resulting in 7.38. It didn't go up by much, but 7.38 is still better than 7.23. So the troll that gives a low rating still helps your value go up.
So to take it further, lets say that member X has posted 5 messages with totals of 7.38, 10.34, 5.10, 20.34 and 31.93. The first four are messages and but the fifth one is a blog entry. So the first 4 are multiplied by 5 each and the fifth one is multiplied by 9. The relative weight of a message is 5 while a blog entry is a 9. That sum is then divided by the sum of all the weights (4*5+9). The result is a 17.35 total value.

Final Thoughts
We don't know exactly what will happen with this new system. I'm sure some will find a way to abuse it or at least attempt to find a way around it. We will be closely monitoring things to see how it works out and undoubtedly will keep tweaking at it. Once concern is that those that hang out in the less trafficked areas will receive less ratings and thus will end up with a lower reputation. That's possible, but perhaps that's not so bad. If there are more people interested in football than in tennis, it should follow that those that write about football have a higher reputation. I may seem unfair to the tennis fan. However, I don't think that it will be that significant.

In then end, it's probably best to not worry too much about the mechanics of this. To that end, we have removed the Adjusted Rating column from the review lists. It is just causing too much unnecessary anxiety. There are plenty of users with very high reputations that don't care about it, they just got there by submitting good content and not playing any games.
Category: General
Posted on: January 22, 2008 11:46 am

Why we have ratings

I guess I'll start this thing in the most controversial possible way: discussing the reputation and rating system use by the message boards and now the blogs. Many of you wonder why we have ratings?

Having a system that rates users and their content has several benefits. The two main ones are:

1. Allows us to pick the best content and feature it on our community pages and soon in other areas of the site. When the community selects someone to be in the top 100 or 1000, we can be sure that this content is worth reading, and we can make it available to newcomers to the site. A lot of our users can't spend hours on the site finding the best content, so we use the community ratings as a way to help them do that. By introducing them to some of your best content, hopefully that will motivate them to consume more content and even join in on the discussion.

2. Deal with trolls and other undesirables. Most people will agree that trolls are a nuisance but they are hard to deal with, because attention is exactly what they want. The more attention they get from users and administrators, the better for them. So we want a system that makes it hard for trolls to get full access to the site and to annoy the vast majority of users that do not engage in that behavior.

That said, the system is not and will never be perfect. But we are constantly improving it. Or at least changing it, sometimes the changes backfire. The challenge in doing this is that we want a community that manages itself, but at the same time that it does not let itself be abused by some members. Any system can have loopholes that let some get an advantage. So we are always monitoring it and fixing things as necessary. But also we look at how the community evolves, and we adapt to reward things that the community values.
Category: General
The views expressed in this blog are solely those of the author and do not reflect the views of CBS Sports or CBSSports.com