Guinevere.ca

Is There a Perfect Rating Scale?

// August 19th, 2008 // content

Web sites that rate products or sites that allow users to rate content are common. A ranking system is simple tool to implement, which can provide a point of discussion or a way for users to have a say in what’s popular on your site. Yet these systems are varied from the most basic, which let users recommend a story to star ratings to percentage based systems. Which one of these work best? Is there an ideal rating scale for web content?

I recently heard an interesting insight from a community software developer: if you provide a way to vote up content, but don’t provide a way to vote down, abuse reports increase.

I’ve observed this in one of the communities I manage where discussion frequently gets heated. The software that powers that community provides a method for people to recommend a comment, but no way to vote a comment down. The result is that nearly every comment in the community is reported as abuse (which is intended for things like hate speech.) This flurry of activity in the abuse section makes that tool nearly useless for moderators.

The same developer says they’ve seen this issue disappear when communities include a method for voting down comments. It seems human nature to want to be able to express your dislike for something if you give them the opportunity to express in favour. The vote-up only system that I use on that one community seems inadequate and frustrating for my users.

Digg has a great solution for community ratings with one of the most simplistic scales: thumbs up or thumbs down. This basic system makes a lot of sense for all sorts of content. After all, often you either like something or you don’t.

Digg’s like-it-or-not voting scale is great if you expect a number of community votes. You can judge how the community feels about the content based on the number of diggs or buries. But what if you’re providing expert reviews from one person on a product? One vote up for a Canon SLR camera would not reveal as much about the reviewer’s opinion on the product as a 4/5 star rating or a 85% rating would.

Let’s take a look at video game sites which offer a constant stream of product rankings from experts, usually received by game fans in a flurry of controversy. As a gamer, I’ve noticed that the controversy is sometimes due to the ranking scale chosen.

Reviews on video game sites are often given by one, two or three experts who each rank the game on a pre-determined scale. The community can often agree or disagree with the rank in the comments, but the review itself is left up to the expert. What scale the experts use is by no means standard and vary site to site. A mix of percentage scales, rank out of 10 and rank out of 5 are all common.

Jeff Gerstmann’s 8.8 “great” ranking of Zelda: Twilight Princess was met with huge backlash from the community who collectively decided that the game deserved a 9.3. Take those same two scores and apply it to a 5/5 rating system. If you used full stars and not halves, both of those rankings would be a 4/5 and Jeff may have not had so many death threats.

A percentage scale out of 100 can sometimes make something seem a little worse than it is. If you used the full scale, you’d mark an average game at 50%. But if you used a 5 star system, you’d mark it at 2.5 (or maybe 3 if you didn’t use half-stars.) I would be more likely to rent or buy a game ranked 2.5/5 instead of 50%/100% even though they’re the same rating. It may not make sense, but the rank in the smaller scale looks average, whereas the larger scale makes the game sound bad. Then again, if your content is a quiz that’s longer than 5 or 10 questions, it makes sense to provide your user with the percentage or a grade.

Essentially we don’t give enough thought to the ranking scales that we use on our sites. Site owners need to let their content dictate what rank scale to use and deploy the simplest one that makes sense for that situation. If your site is looking for a wide range of community opinions, try thumbs up or thumbs down. For expert or in-depth product reviews, try a 5 star system. Higher scales get more complex and should be used when the others don’t make sense.

Careful consideration of your rank scale could mean a less frustrating, more interesting experience for your users and moderators.