P r e d i c t i o ni nMa r k e t i n g R e s e a r c h( 2 ) :

T h e V a l i d i t y o f C u s t o m e r S a t i s f a c t i o n , P u r c h a s e I n t e n t i o n , a n d N e t P r o m o t e r S c o r e
b y J a nHo f me y r

Prediction in Marketing Research (2): The Validity of Customer Satisfaction, Purchase Intention, and Net Promoter Score
In this second note in the prediction series, I analyze the aggregate and respondent level associations of traditional loyalty measures with various market behaviours. I rely heavily on the work of Keiningham et al (2007), Morwitz et al (2007), and Hofmeyr et al (2008). I also draw attention to the indirect influence that the Fishbein-Ajzen equation (1975) seems to have had on the development of whats called the weighted additive model of preference measurement in our industry.

Headlines Heres what youll learn:

The aggregate correlation between customer satisfaction and various measures of real behaviour varies between R = 0.60 and R = 0.80+. Respondent level correlations average about R = 0.20. The aggregate correlation between purchase intention and what people go on to use is R = 0.52. Respondent level correlations average about R = 0.27. The aggregate correlation reported by Reichheld between NPS and past/ present firm growth rates is R = 0.83. He notes that respondent level correlations are less accurate than other loyalty measures. The correlations reported by Reichheld (2003) are based on a selection of 18 out of 50+ firms in 3 out of 12 industries. One wonders why he doesnt report the others. There is a big difference between the aggregate and respondent level validity of the above measures. Because of the importance of respondent level validity, classic loyalty measures like customer satisfaction, purchase intention, and Net Promoter should not be used as dependent variables for profiling or driver analysis if better measures are available.

Prediction in Marketing Research (2): The Validity of Customer Satisfaction, Purchase Intention, and Net Promoter Score 

An Important Old Equation

A good place to start is with an old equation that seems to underpin a lot of marketing research thinking (See, for example, Ryan and Bonrfield (1980) and Fishbein and Ajzen, 1975 dont be intimidated by the notation. Youll find you understand it, so read on):

The equation says that what a person does, i.e., behaviour, will be consistent with (), the strength of their intention to do it. This, in turn, will be a function of their beliefs about what will happen (Bi), their attitudes whether positive or negative to those outcomes (ai); and the overall importance of each of those outcomes (0) relative to the importance of social norms (1). Add the results to get a total. The second term builds social norms into the equation, but we dont need to be concerned with that here. The basic idea is simple: the stronger a persons behavioural intention, the more likely it should be that the behaviour will follow. In marketing research, what Bettman et al (1998) refer to as the weighted additive model of the consumer choice process has a similar structure (see below). The logic goes as follows: what a person uses or buys should be consistent with the strength of their intention to buy it. We can measure the strength of their intention simply by asking them, e.g., with a purchase intention question. Next, we try to understand why a person has that intention by measuring their beliefs about the products, services, or brands they might use/buy. This requires the following steps: Identify the attributes you believe may influence a persons intentions (indexed by i, i = 1, , m); Ask the person to rate the services/brands in terms of how they perform on each attribute (ai); Establish the importance of each attribute to the person (Bi); Multiply the ratings by the weights and add them up.

Prediction in Marketing Research (2): The Validity of Customer Satisfaction, Purchase Intention, and Net Promoter Score 

The result is a weighted additive model of purchase intention. In marketing research we often have a series of weighted additive measures on the right hand side of the equation, e.g., different touch point ratings or attribute associations. We use measured intention (the left hand side of the equation) as the dependent variable and then estimate the importance weights:

The model is non-comparative because it doesnt measure what a person thinks about competitors, and its computationally intensive because it implies a complex mental process, i.e. draw up a list of relevant characteristics; rate the product, service, or brand on each characteristic; give each characteristic an importance weight; and multiply the rating by the weight (in your head). Then add the numbers up in your head. Its worth noting that cognitive neuropsychologists dont think the brain works like this. In the classic model, marketing researchers know the strength of intention (the left hand side of the equation) and the attribute ratings. Importance weights are usually derived using purchase intention as the dependent variable.

How well does purchase intention correlate with real behaviour? i). The aggregate correlation of purchase intention with behaviour
A key aspect of the model is the left hand side of the equation which links the strength of a persons declared intention to their actual behaviour. In a wideranging literature search covering social psychology, economics, statistics, and marketing from 1940 to 2006, Morwitz et al (2007) found some 40 articles that linked a survey of purchase intention to real world outcomes. Twenty-five of the articles reported correlations. The average correlation in the academic literature at an aggregate level is R = 0.52. This is the correlation between penetration or actual sales on the one hand and a summary measure of purchase intention on the other. Various summaries were used including weighted scales, mean or median values, and top boxes. Scales varied from 5- to 21-points. A number of factors influenced the quality of the correlation. In general, purchase intention correlates better with penetration than actual sales. It also correlates better for existing rather than new products, and for durables rather than fastmoving consumer goods. The nature of the summary measure (i.e. mean, median, top box or weighted scale) made no difference. Nor did the number of scale points.
Prediction in Marketing Research (2): The Validity of Customer Satisfaction, Purchase Intention, and Net Promoter Score 

ii). Respondent level correlation of purchase intention with behaviour

On the Synovate website under Research on Research you will find Report 23. It studies the relationship between what people say they will do (i.e., buy or not buy a microwave in the next 18 months) and what they actually go on to do in the next 18 months. As I pointed out in the first note in this series, aggregate results were relatively good: 9% of the respondents said that they would buy a microwave in the next 18 months and 7% actually did. At the respondent level however, the results were not good: only 26.5% of the people who said that they would buy a microwave actually did. The fact that purchase intention may be wrong about who actually buys and why has long been known in market research. In a recent analysis of purchase intention scales, Wright and MacRae (2007, p. 617) write: Purchase intention scales suffer from severe theoretical and empirical problems. It has long been known that the majority of purchasers are nonintenders and that actual compliance with stated intention is low. An important question therefore is: what degree of respondent level validity can one expect from purchase intention questions? Morwitz et al dont report on respondent level correlations. But in a search of the marketing research literature from 1980 to 2005, I found 5 articles that reported respondent level correlations: Sewall (1981) measured womens intention to buy clothing in a mall intercept. The correlation between purchase intention and actual purchases was R = 0.27. LaBarbera and Mazursky (1983) tested the relationship of purchase intention to next brand bought in six product categories. The average correlation was R = 0.24. Chandon, Morwitz, and Reinhardt (2005) measured purchase intention in three product categories then observed behaviour for six months. The average correlation was R = 0.24. Seiders, Voss, Grewal, and Godfrey (2005) tested the relationship of purchase intention to high end retailer behaviour in terms of no of visits/share of wallet. The correlation was R = 0.10. Perkins-Munn, Aksoy, Keiningham, and Estrin (2005) tested the relationship of purchase intention to share of wallet for trucks and pharmaceuticals. The average correlation was R = 0.46. According to these results, the average correlation between purchase intention and various actual behaviours at respondent level in 13 product categories is about R = 0.27.

Prediction in Marketing Research (2): The Validity of Customer Satisfaction, Purchase Intention, and Net Promoter Score 

The American Customer Satisfaction Index (ACSI) and Net Promoter Score (NPS) i). The aggregate correlation of the ACSI and NPS with behaviour
In an award winning paper, Keiningham et al (2007) compare the aggregate predictive performance of numerous loyalty measures using two data sets. The measures they compare are: NPS, the ACSI, and various summary measures of satisfaction, repurchase intention, and recommendation. The two data sets are from the ACSI data-base (See Fornell et al, 1996) and its equivalent in Norway. The comparison standard is based on claims for NPS made by Reichheld (2003, 2006) and Satmetrix (2004). Keininghams analysis of the Norwegian data is difficult to summarize. So Ill not attempt a summary here (Ill just recommend that you read it). But its worth quoting his conclusion (Keiningham et al, p 42): There is no real indication that average levels of any of the satisfaction/ loyalty metrics are significantly correlated with the relative change in revenue within the respective industry. Reichhelds claim that NPS is the single most reliable indicator of a companys ability to grow isnt supported and the NPS itself doesnt correlate consistently with average growth rates. Keininghams comparison between the NPS and ACSI is easier to summarize than his analysis of the Norwegian data (see table 1). The results for both measures are good: R = 0.83 (NPS) and R = 0.82 (ACSI), but theyre aggregate correlations. What would happen if we were able to look at these results at respondent level?
Table 1: Correlations of two loyalty measures with firm growth rates

Prediction in Marketing Research (2): The Validity of Customer Satisfaction, Purchase Intention, and Net Promoter Score 

Neither Reichheld nor the ACSI report a respondent level analysis. But Keiningham et al (2007, p 41) quote Reichheld as follows: Net Promoter yields slightly less accurate predictions for the behaviour of individual customers than models consisting of data from multiple survey items The specifics behind this essentially negative evaluation of NPS metrics at respondent level arent made clear. It might have been raw scores, or perhaps it was the Net Promoter classification of people into segments: promoters, neutrals, detractors. Whatever the case, the NPS approach yields results that arent as good as customer satisfaction or purchase intention. Turning to customer satisfaction, as with purchase intention, there are few articles that test its validity at the respondent level. Here is a list I found in articles published from 1983 to 2005 (See Hofmeyr et al, 2008): LaBarbera and Mazursky (1983) tested the relationship between satisfaction and next brand bought in six product categories. The average correlation was R = 0.20. Gustaffson, Johnson, and Roos (2005) test the relationship between customer satisfaction and churn in the telecommunications market. The correlation of satisfaction churn was R = 0.13. Seiders, Voss, Grewal, and Godfrey (2005) test the relationship between customer satisfaction with high end retailers and no of visits/share of wallet. The correlation is R = 0.07. Perkins-Munn, Aksoy, Keiningham, and Estrin (2005) test the relationship between customer satisfaction and share of wallet for trucks and pharmaceuticals. The average correlation is R = 0.33. Hofmeyr (2010) tests the correlation between customer satisfaction and share of wallet for retailers in Italy in 2007 and again in 2008. The correlations are R = 0.27 (2007) and R = 0.22 (2008). The average correlation between customer satisfaction and various actual behaviours at respondent level is R = 0.20. This compares with correlations that vary between R = 0.60 to R = 0.80+ at aggregate level.

A further note about NPS validations

As Keiningham et al (2007) note, NPS has achieved widespread corporate acceptance. Its therefore worth pausing to consider the nature of the validations that Reichheld used as the basis for his original claim that NPS is the one number you need to grow.

Prediction in Marketing Research (2): The Validity of Customer Satisfaction, Purchase Intention, and Net Promoter Score 

Here are some facts. First, Reichhelds validations look at the relationship between average NPS from 2001/2 and average corporate growth rates from 1999/2002. In other words, the correlations are partly backward looking. He is not showing that a good NPS predicts future growth. Second, as Keiningham points out, Reichheld himself notes that NPS metrics dont correlate as well with individual behaviour as other classical loyalty metrics. In this note Ive shown that the correlation between classical loyalty metrics and individual behaviour of various kinds is quite poor averaging R = 0.27 for purchase intention, and about R = 0.20 for customer satisfaction. We can therefore conclude that NPS metrics are poor at respondent level. Third, Reichheld says that he has the data for some 50 companies in twelve industries. Yet his HBR article only reports the correlations for 18 companies in three industries. Satmetrix (2004) say that their correlations are significant in the majority of the industries they looked at. That means they have industries for which their results arent significant the reporting appears to be selective. Reicheld notes the common sense idea that high levels of recommendation should lead to adoption and therefore growth. As reported by Keiningham, he then notes that there are imperfections in the analytics: All we did was quantify this common sense in a way that made sense to business leaders... These practical leaders have little interest in advanced statistical methods. However, the reason Reichheld shows any correlations at all is, Im sure, because he realizes that business leaders do care about whether or not the measures they use correspond with reality. One has to ask about the industries for which there are no significant correlations, and about the results that are unreported.

Why respondent level validity matters

Our industry compensates in various ways for the limitations of classic loyalty metrics. For instance, we use norms and databases to help put the results into context. At an aggregate level, the resulting models are good and achieve their purpose. But weve become careless in failing to recognize the importance of the difference between aggregate and respondent level models.

Prediction in Marketing Research (2): The Validity of Customer Satisfaction, Purchase Intention, and Net Promoter Score 

I can further illustrate the problem by flipping around the Fishbein/Ajzen equation:

On the left were trying to establish a) What people believe about products, services, or brands; b) How they rate those products, services, or brands; and c) How important each of the potential characteristics/motivators is. In this hypothetical illustration of what we do, Ive supposed that we got an outstanding driver model from our survey R = 0.90. The problem is: it doesnt matter how good the driver model is - the link with the real world is broken at the respondent level. Lets use purchase intention to illustrate this. Weve seen that the aggregate correlation between purchase intention and real behaviour is R = 0.52. As in the Market Facts paper, the percent who use or buy a product/service/ brand correlates well with the percent who say they will use or buy it. But at the respondent level, the relationship breaks down: many people who say they will use or buy a product/service/brand, dont. A similar number who say they will not use or buy it, do. As a result, the respondent level correlation is only about R = 0.27. The problem is: youve arrived at conclusions about why people intend to use or buy it and who they are by looking at those who say they will use or buy it. But most of them dont. In the meantime, youve also arrived at conclusions about who will not use or buy and why but many of those do. You cannot have any confidence in your driver analysis or profiling. There is no problem with aggregate modeling as long as one keeps the model to the aggregate level, but as long as our measures identify the wrong people as loyal, we cant be confident about our answers to questions about who or why.

Summary
The relationship between what individual people say about how satisfied they are, or what they intend to do and what they actually do, is poor. We may ask satisfaction or purchase intention questions and get weights from a driver analysis. We may then work out how to increase peoples satisfaction or purchase intention. We may think that when we apply the resulting strategies, well increase customer loyalty. But we need to recognize the extent of the slippage between these models and actual individual behaviour. When we apply such strategies, we need to appreciate that we may be pulling marketing levers whose cogs are broken.
Prediction in Marketing Research (2): The Validity of Customer Satisfaction, Purchase Intention, and Net Promoter Score 

Bibliography
Anonymous Author, Date Unknown: Measuring buying intention: How valid is the estimate, Research on Research, 12, Market Facts Inc. Bettman, James R., Mary Frances Luce, John W. Payne: Constructive Consumer Choice Processes, Journal of Consumer Research, 25 (December), 187 - 217 Chandon, Pierre, Vicki G. Morwitz, and Werner Reinartz (2005): Do intentions really predict behaviour? Self-generated validity effects in survey research, Journal of Marketing, 69, 1 14 Fishbein, Marting and Icek Ajzen (1975): Belief, attitude, intention, and behaviour: An introduction to theory and research, Addison-Wesley, Reading Fornell, Claes, Eugene W. Anderson, Jaesung Cha, and Barbara E. Bryant (1996): The American Customer Satisfaction Index: Nature, purpose, and findings, Journal of Marketing, 60 (October), 7 - 18 Hofmeyr, Jan, Victoria Goodall, Martin Bongers, and Paul Holtzman (2008): A new measure of brand attitudinal equity based on the Zipf distribution, International Journal of Market Research, 50:2 Gustaffson, Anders, Michael D. Johnson, and Inger Roos (2005): The effects of customer satisfaction, relationship commitment dimensions, and triggers on customer retention, Journal of Marketing, 69:4 (October), 1 9. Keiningham, Timothy L., Bruce Cooil, Tor Wallin Andreassen, and Lerzan Aksoy (2007): A longitudinal examination of net promoter and firm revenue growth, Journal of Marketing, 71 (July) 39 - 51 LaBarbera, Priscilla A. and David Mazursky (1983): A longitudinal assessment of consumer satisfaction/dissatisfaction: The dynamic aspect of the cognitive process, Journal of Marketing Research, 20, 393 - 404 Morwitz, Vicki G., Joel H. Steckel, and Alok Gupta (2007): When do purchase intentions predict sales, International Journal of Forecasting, 23, 347 364 Perkins-Munn, Tiffany, Lerzan Aksoy, Timothy L. Keiningham, and Demitry Estrin (2005), Actual Purchase as a Proxy for Share of Wallet, Journal of Service Research, 7:3 (February), 245 256 Reichheld, Frederick F. (2003): The one number you need to grow, Harvard Business Review, 81 (December), 46 54 . (2006): The Ultimate Question: Driving Good Profits and True Growth, Boston: Harvard Business School Press Ryan, Michael J. and E. H. Bonfield (1980): Fishbeins Intention Model: A Test of External and Pragmatic Validity, Journal of Marketing, 44 (Spring), 80 - 95 Satmetrix (2004): The power behind a single number: Growing your business with Net Promoter, Satmetrix Systems white paper available from www.satmetrix.com. Seiders, Kathleen, Glenn B. Voss, Dhruv Grewal, and Andrea L. Godfrey (2005) Do satisfied customers buy more? Examining moderating influences in a retailing context, Journal of Marketing, 69, 26 - 43 Sewall, M. A. (1981): Relative information contributions of consumer purchase intentions and management judgments as explanators of sales, Journal of Marketing Research, 18. 249 253 Wright, Malcolm and Murray MacRae (2007): Bias and variability in purchase intention scales, Journal of the Academy of Marketing Science, 35, 617 624

Prediction in Marketing Research (2): The Validity of Customer Satisfaction, Purchase Intention, and Net Promoter Score