Rating vs "Like"

11 Nov 2009 - 10:31am
5 years ago
13 replies
1336 reads
Rih Tand
2009

Hi Group,

Any thoughts on when it is best to use a rating system vs a simple
like/don't like voting feature? I currently need to decide which way to go
on a restaurant guide project in which the existing rating system is a 1-5
scale "star" (with no guide, higher stars = better).

Unless one has a known and universal explanation on how "stars" are valued,
it seems that the rating system is flawed. What are your thoughts on this?

Rih

Comments

12 Nov 2009 - 7:59am
Dru
2009

Rih,
It will be interesting to see what responses come back re: your post.
I am not aware of any "best practices" or universal approach for
rating, ranking, "favorite-ing", like/dislike, heart, etc. I have
worked in music on line, social networks, and product ecommerce and
finding that the sophistication and complexity of the algorithm
behind the "rate" system defines the quality of the end result.
When I was working for a music site, over time, we found that the
"average" for any song eventually reached 4.0-4.5. We also found
that participation by users to "like/dislike" (the action) was more
frequently done than the users clicking on a star to "rank".
Netflix tends to do the best job of explaining to the user (I think
it is in Netflix's help section) just what the ranking represents.
In your case, a menu item of food truly could be "awful, ok,
delicious" but even then it is so subjective because it truly
reflects the consumer's TASTE (or lack of) and opinion. Amazon's
reviews provides a bar graph representing rank to the number of
reviewers who share that same ranking opinion. That is also a fairly
compelling approach. My position is purely one of opinion but I think
providing a "I like/I dislike" or a thumbs up/thumbs down, is really
about as complicated a rating system needs to get unless you really do
plan to cross tab those ranking responses with other data about your
consumer base to arrive at some marketing profile or user persona.
So my knowledge there is no "guide". Good luck!
Druid

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=47342

12 Nov 2009 - 8:01am
Rih Tand
2009

I agree. So, you converted existing ratings and converted favorable ones to
"like"?

On Thu, Nov 12, 2009 at 7:49 PM, Nik Lazell <
nicholas.lazell at realadventure.co.uk> wrote:

> Hi Rih,
>
> We have recently changed from a star rating system to a simple 'like'
> functionality. Our findings showed users were more inclined to simply
> say 'yes we liked that' - requiring less effort than consider and
> justify a score out of 5.
>
> Nik
>
>
> -----Original Message-----
> From: discuss-bounces at lists.interactiondesigners.com
> [mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of R.
> Tan
> Sent: 11 November 2009 16:31
> To: discuss at ixda.org
> Subject: [IxDA Discuss] Rating vs "Like"
>
> Hi Group,
>
> Any thoughts on when it is best to use a rating system vs a simple
> like/don't like voting feature? I currently need to decide which way to
> go
> on a restaurant guide project in which the existing rating system is a
> 1-5
> scale "star" (with no guide, higher stars = better).
>
> Unless one has a known and universal explanation on how "stars" are
> valued,
> it seems that the rating system is flawed. What are your thoughts on
> this?
>
> Rih
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

12 Nov 2009 - 9:51am
Kai En Ong
2008

I've been thinking about this with regard to a vast range of
different kinds of media content and the stars Vs like/dislike
debate.

Does it depend on what you're rating?

Do users rate a book/song differently (and use the resulting info
differently) than a restaurant, which may have taste/value/service as
multiple dimensions to rate, and help with decision making?

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=47342

12 Nov 2009 - 9:57am
Dru
2009

Kai en ong,
Based upon my experience - it very definitely depends upon subject
mater that is being rated. The main thing to keep in mind is that
subjectivity. It's one person's opinion vs the next person's.
All subjective.
Dru

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=47342

12 Nov 2009 - 10:02am
Paul Sherman
2006

This post has hit upon an interesting area of design assessment. There
have been books - many in fact - written about scale-based assessment
and measurement. I couldn't possibly touch on all the relevant points
here, but there are some pointers I can provide:

- If your goal is (as dru mentioned) to crosstab responses or generate
measures of central tendency such as means, medians, or modes, then a
scale approach is more useful than a binary "like/dislike" measure.

- There are always positivity biases in your data, depending on what
you're measuring. But there are a whole raft of other biases -
extremity bias, negativity bias, centrality bias - that can come into
play depending on what your respondents are rating, and even what
culture they're from. So you go with your measurement, try to identify
and account for the biases you may encounter, and interpret your data
in light of the context.

- As some of the responses alluded to, it's sometimes difficult to
decide what a "3" or a "1" *means* to each user. One way to address
this is to behaviorally anchor your response options. That is, do
something like (and this is just a humorous example...) "5 (I would
step over my own grandmother to buy this)", "4 (I would buy this if I
were walking by the store, but wouldn't go out of my way to purchase
it)", etc.

- I strongly caution against using scales with more than 5 response
options. And I personally use 4 options quite often, which has the
effect of removing the "neutral" option. The reasons are pretty
straightforward:
Fewer response options: The more response options you use, the harder
it is for your respondents to discriminate between any two adjacent
options. That is, on a 9-point scale, what's the diff. between a "3"
and a "4", a "4" and a "5", etc? Better to use fewer response options.
They're more easily discriminable.
Removing the neutral option: Research into attitude measurement has
shown that people can't *really* hold a neutral attitude. When people
choose the "neither agree nor disagree" (or "neither like nor
dislike") option, it's actually more an indication that they don't
*care* about the item you're asking them to rate. That's different
than neither liking or disliking. And if you don't know what you've
measured, it's impossible to interpret. Forcing people to take a stand
actually gets them more engaged in the question you're asking.

Just M2c, and of course ymmv.

-Paul

- - - - - - -
Paul Sherman, Principal, ShermanUX
UX Design | Research | Strategy
paul at ShermanUX.com
www.ShermanUX.com
+1.512.917.1942
- - - - - - -

On Nov 12, 2009, at 8:01 AM, R. Tan wrote:

I agree. So, you converted existing ratings and converted favorable
ones to
"like"?

On Thu, Nov 12, 2009 at 7:49 PM, Nik Lazell <
nicholas.lazell at realadventure.co.uk> wrote:

> Hi Rih,
>
> We have recently changed from a star rating system to a simple 'like'
> functionality. Our findings showed users were more inclined to simply
> say 'yes we liked that' - requiring less effort than consider and
> justify a score out of 5.
>
> Nik
>
>
> -----Original Message-----
> From: discuss-bounces at lists.interactiondesigners.com
> [mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of
> R.
> Tan
> Sent: 11 November 2009 16:31
> To: discuss at ixda.org
> Subject: [IxDA Discuss] Rating vs "Like"
>
> Hi Group,
>
> Any thoughts on when it is best to use a rating system vs a simple
> like/don't like voting feature? I currently need to decide which way
> to
> go
> on a restaurant guide project in which the existing rating system is a
> 1-5
> scale "star" (with no guide, higher stars = better).
>
> Unless one has a known and universal explanation on how "stars" are
> valued,
> it seems that the rating system is flawed. What are your thoughts on
> this?
>
> Rih
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>
________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... discuss at ixda.org
Unsubscribe ................ http://www.ixda.org/unsubscribe
List Guidelines ............ http://www.ixda.org/guidelines
List Help .................. http://www.ixda.org/help

13 Nov 2009 - 2:12am
cfmdesigns
2004

On Nov 11, 2009, at 8:31 AM, R. Tan wrote:

> Any thoughts on when it is best to use a rating system vs a simple
> like/don't like voting feature? I currently need to decide which way
> to go
> on a restaurant guide project in which the existing rating system is
> a 1-5
> scale "star" (with no guide, higher stars = better).
>
> Unless one has a known and universal explanation on how "stars" are
> valued,
> it seems that the rating system is flawed. What are your thoughts on
> this?

In my experience, a star-based rating system has inherent flaws:

* Does 1 star mean passable or very bad?
* Does 0 stars mean unrated, neutral, or very bad?
* Does 3 stars out of 5 mean fairly good or neutral?

Everyone knows more or less what the extreme ends mean, but not the
middle. Is one star less than max only a little under tops or quite a
bit? Only the rater knows for sure.

I think what is generally needed for most cases is a four-option set:

* Hate it
* Neutral
* Love it
* Haven't rated it

-- Jim

13 Nov 2009 - 12:25pm
Dru
2009

Paul
You reference "books - many in fact"...on scale based, etc. Could
you list a couple of the titles of the books - ones that are possibly
not too statistical or "humanoide" in nature but that might speak to
the ranking/rating for those of us (myself) who might not be so
steeped in all the human stuff. Or direct me to a web site or source
that might have the reference materials you reference. :-).
Thank you.
Dru

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=47342

15 Nov 2009 - 5:59pm
Paul Sherman
2006

Just giving back a bit... Dru asked for a recommendation for a
readable resource about attitude measurement, scaling, etc.

I went back to my stats and survey research research but most of them
seemed either too dry, too statistics-heavy, or both.

I did find one book with a straightforward and concise description of
attitude measurement, questionnaire design and scaling. The relevant
section is Chapter 5.

Here's the reference:
Questionnaire Design: How to Plan, Structure and Write Survey Material
for Effective Market Research
http://www.amazon.com/gp/search?index=books&linkCode=qs&keywords=074944181X

-Paul

- - - - - - -
Paul Sherman, Principal, ShermanUX
UX Research | Design | Strategy
paul at ShermanUX.com
www.ShermanUX.com
+1.512.917.1942
- - - - - - -

16 Nov 2009 - 2:56am
Thomas Petersen
2008

Look at what steepster did
http://blog.steepster.com/post/226679106/better-rating-system

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=47342

16 Nov 2009 - 11:39am
Dru
2009

Great summary . Thanks Tom for sharing. Steepster seems to have
figured out a way to avoid the predictable "final resting place" of
something between 4.-4.5 (4.3). I am passing the site on to our UX
Designers and the business owners within our company. It's nice to
see something new/fresh.

Looks like this rate it topic could turn into a book for one of you
UX brainiacs :-). Given that it seems every social networking site
and e-com site seem to be moving in the direction of "rate it",
sure would be nice if we had some stats to back up it up. I wonder if
anyone has done a usability test to determine if users even like being
offered the chance to rate/rank.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=47342

16 Nov 2009 - 12:08pm
Thomas Petersen
2008

I would also recommend looking at what StackOverflow, their vote
up/down engine is pretty well thought out. You can test it out at
uxexchange.com where it's used.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=47342

16 Nov 2009 - 7:44pm
Holger Rothemund
2009

Very interesting discussion. However, I personally prefer uneven
scales, because I think that users do know what 3 on a 5-Point scale
means: not bad, but not good either. Why should I force them to
either a positive or a negative decision?

On the other hand you should offer a "don't know" option as well,
for those who just don't want or can't rate the item. We made and
still do make good experiences with those ratings and benchmarking
studies do show that respondents can easily cope with the "neutral"
option.

And if you think about it: sometimes there is just that odd item,
that you don't consider positive nor negative. If there would be an
even scale, you would be forced into a decision that you can't make.
Whatever you answer, it is basically wrong. On a scale with many
options this might be not so bad, but on a scale of 4, this can have
a huge impact on results (mean, top-boxes, etc.).

Some researchers are concerned that the respondents use the center of
the scale too often, because it is "easier", but from my experience
that isn't the case. Most respondents do have an opinion, and if it
is good or bad, they will tell you.

However, as a company you should know how you want to treat those
neutral responses. I personally recommend to mentally move them
towards the negative responses. Because even if the respondent
doesn't consider the item negative, the companies intention should
be to move all the responses towards the positive side of the scale
over time by actions to improve quality, etc.

@Paul: As my experience from research practice is quite opposite to
what you were writing I would like to get some more background
information about this. Do you have any scientific articles about
this? Would be interesting, because this can also depend very much on
the issues that should be rated.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=47342

17 Nov 2009 - 8:33am
Dru
2009

@Thomas - def will check out stackoverflow. thanks for suggestion -
send more of any examples that display "rank/rate/I heart/favorite"
other than the "typical" ways.
dru

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=47342

Syndicate content Get the feed