Research

James Swift

2 August 2021

Strategist’s Digest: Why online star ratings are useless 

Contagious digests the most interesting and relevant research from the world of advertising and beyond, because there’s just too much to read and too little time

Photo by Glen Carrie on Unsplash

Mass-scale emotionality reveals human behaviour and marketplace success. 

By Matthew Rocklage , Derek Rucker and Loran Nordgren.

Give it to me in one sentence.

Emotional comments in online reviews are better at predicting the success of products and services than star ratings.

Give me a little more detail.

A team of researchers analysed the comments in online reviews to see if emotionality was better at predicting the success of a product or service than star ratings.

Emotionality is the extent to which an attitude stems from a person’s feelings. It is distinct from positivity (or valance), which includes positive-but-sterile words like ‘impeccable’, and from arousal, which describes energy levels rather than emotions.

The researchers measured emotionality using a tool called Evaluative Lexicon, and they applied it to reviews for movies, books, adverts and restaurants.

For movies, the researchers analysed 13 years of online reviews from Metacritic.com, to see whether star-ratings or emotionality could predict a movie’s box office revenue.

For adverts, the researchers pitted the emotionality of tweets about Super Bowl commercials against the score each spot received from USA Today’s Ad Meter survey, to see which correlated with increases in Facebook followers for the brands involved.

In both instances – and in every other experiment the researchers ran – emotionality was a better predictor of the various measures of success than star ratings. In fact, only when they explored the link between restaurant reviews on Yelp and increases in table reservations did the researchers find average star ratings were a useful predictor.

Why is this interesting?

Star ratings skew so positively they are all but useless. The average Amazon star rating is 4.2 out of five, and nearly half of all Yelp reviews give maximum five-star scores. Of course, your average restaurant or movie-goer doesn’t have access to tools that can analyse emotionality, but we’re sure marketers can find a useful way to harness this information.

Any weaknesses?

The study doesn’t explain why emotionality is a good predictor or why so many star ratings are positive. But maybe we’re just asking for the moon on a stick.

Where can I find the whole report?

Here, and it’s free. Five stars.

This article was downloaded from the Contagious intelligence platform. If you are not yet a member and would like access to 11,000+ campaigns, trends and interviews, email [email protected] or visit contagious.com to learn more.