The Whatcar? website has just released the 2012 JD Power Vehicle Owner Satisfaction Survey (VoSS) results.
Based on a sample of 18,000 car owners the survey is the most comprehensive car ownership survey in the UK. People who bought new cars from January 2009 and December 2010 were eligible to take part making the cars between one to three years old at the point of measurement.
This year’s top performing motor was the Kia Sportage, a mild-mannered Korean SUV. The worst, a full 117 places lower in the league table, wearing the dunce hat and sitting in the corner of the class was the Vauxhall Vectra.
Compare these owner reviews with the expert reviews on the Whatcar website. The Merc-murdering, Beamer-bashing Sportage, the best car in Britain today scored 3 out of 5. The Vectra, supposedly the worst car anyone can own? 4 out of 5. Anyone looking to buy a new car might be forgiven for being confused.
Why do experts and owners come to such different conclusions? Most importantly, if you’re trying to decide what to buy – which do you rely on?
Owners have the undoubted advantage of living with their car over a prolonged period compared to expert reviewers who at best might get a few days to make their assessments.
Experts and owners are measuring different things
Owners rate their motors on four categories: quality and reliability, appeal, service satisfaction and ownership costs. Each category is then weighted according to how important it is to them, which in practice means appeal (31%) and ownership costs (25%) have most impact on the overall score.
Expert reviewers score cars objectively on 9 criteria which encompass car ownership: performance, ride & handling, refinement, buying & owning, quality & reliability, safety & security, behind the wheel, space & practicality, and equipment.
Expert car reviewers are meant to be objective. A scan of past reviews might give you the impression that they are perhaps a little picky about upstart far eastern brands, perhaps.
With a third of the total JD Power rating based on the ‘appeal’ of a car I’d argue it is owners who are more subject to bias. Anyone shelling out a small fortune for new car is psychologically invested in it. This might affect results in a number of ways:
Bias – if you’ve bought it you want it to be good.
Expectation bias – if you’re not expecting much, it’s possible to be pleasantly surprised. But if you’ve spent 3 months waiting for your black sapphire Beamer with red leather interior – and its stitching comes loose – it’s going to grid your gears.
Treatment bias – reliability is as much about how a car is treated as how it is manufactured. The type of person who buys a Subaru Imprezza Turbo is wholly different in terms of demographics, attitudes and driving behaviour to the Honda Jazz owner. Whilst it’s entirely possible to drive a Jazz sideways I for one have never seen it.
It could be that JD Power control for these factors. The raw data, methodology and the questionnaire are not made publicly available.
Given that the JD Power survey is used to separate good from bad, there’s actually not that much separating the 118 cars – the world conquering Sportage scoring 83.8% (or about eight of out ten)– and the lowly Vectra 70.2% (or about seven out of ten). At least expert reviews are more directional.
Had the JD Power survey been conducted 30 years ago things might have been different. Back then there was much more variation in quality, and the survey would have separated the wheat from the chaff. Now we have cars which are more reliable, so could it be that what is really separating cars are the number of cupholders rather than the number of calls to the AA?
What to rely on?
I enjoy reading the survey every year, and will continue to do so. Owner information about reliability and running costs over time is for me the most useful element, but I’d take the results with a pinch of salt & cross-reference them with both expert reviews and some 3rd party reliability data from a warranty provider.