Reference bias: what does “good” look like?

A recent study by Heckman & Kautz ranked nations according to their self-reported conscientiousness. They then cross referenced the average number of hours a worker in that country actually works each year (see below).

National rank in conscientiousness vs. average number of hours worked annually

Reference bias_Heckman & KautzOverall there is little correlation between how conscientious people in each country think they are, and the reality of how much they work. As the authors state: “The results are surprising. South Korea ranks second to last in terms of conscientiousness but first in the number of hours worked.” France on the other hand put in 40% fewer hours each year than the South Koreans yet rate themselves as far more conscientious. Thanks to the Stirling Behavioural Science Blog sharing the paper.

It’s a reminder that measuring abstract concepts in surveys can be a challenge. If participants have to interpret what a concept like ‘conscientious’ ‘happy’ or ‘lazy’ means they will all reference different norms. This “reference bias” makes comparisons misleading.

What’s needed is a common frame of reference. Academics use “vignettes” (short written descriptions) to help anchor responses and give more standardisation across cultures. There is a great list of these here for anyone interested. “Ken” for example, is the anchor point of 5/5 happiness on a happiness scale – and sounds like great company:

“Ken loves life and is happy all the time. He never worries or gets upset about anything and deals with things as they come.”

Whilst commercial constraints are going to prevent the average market researcher developing vignettes for every survey, when you’re measuring abstract notions they reduce subjectivity. I’m resolved to giving them a go.


About Simon Shaw

I'm a Director at an insight consultancy. I'm interested in marketing, market research & consumer psychology. The views expressed are not necessarily those of my employer.
This entry was posted in Marketing Research, Methods and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s