Measuring effectiveness
Merry Baskin
Warc Best Practice
July/August 2010
 
 

   Title: Measuring effectiveness

   Author(s): Merry Baskin
   Source: Warc Best Practice
   Issue: July/August 2010
 

Measuring effectiveness

This month, Merry Baskin advises on how to demonstrate the effectiveness of marketing budgets through ROI,
behavioural and intermediate measurements

The burden of proof for marketing funds to have been invested efficiently and effectively has always been Important. That
pressure has intensified with the increasing influence of client procurement departments and the frugality imposed by the
global recession.

In the 'Era of Accountability', we need to demonstrate not just the attitudinal or behaviour changes effected by our campaigns,
the perceptions of the brand we have altered, but also the financial return of those campaigns.

But only very rarely is there a direct link between the ad campaign and its impact on sales. However, there are other roles for
performance measurement: to monitor progress against objectives; to check the strategy is working as intended; to identify
ways in which communications can become more effective in future; to set and decide future budgets and media spends; and
to quantify the impact we have had (Return on Marketing Investment (ROI or ROMI). Even if your campaign bombs, a return on
learning (from your mistakes) is most valuable.

MEASUREMENT TYPES
There are three different kinds of measurement:

Intermediate measures: awareness, message communication, brand image and attitude shifts are all usually covered by
some sort of consumer tracking study.

Behavioural measures:

response rates, enquiries, or compliance with proposed changes (such as getting tax returns in on time or different usage
occasions).

Downloaded from warc.com

2 

 
 

Figure 1: The By, By, By Model – Police Recruitment

ROI measures and effects:

Sales correlations, monetised payback, efficiency, human and societal impact.

Some people dispute the value of intermediate measures versus hard commercial ones. Some claim that intermediate
measures are the most important, given they can't control the consumer experience beyond the advertising. However, the
reality is that the most effective campaigns tend to show a wide range of effects to build really compelling cases, so the more
measures that you can demonstrate to have shifted, the better.

A BRAVE NEW WORLD

Not only has accountability evolved from being a backroom role, moving the pointy headed econometricians and data wonks
into the limelight, but the received body of wisdom regarding effectiveness measurement has also shifted seismically.

As new media and marketing channels proliferate, so do the options for evaluation. It is important to be able to distinguish the
individual contribution of each component in the mix, not least to justify its continuing presence in the budget.

There are also exciting developments in how we understand communication works, such as neuroscience, which may rewrite
the rules. In particular, we have been forced to recognise the crucial role played by our emotions in how we not only make
decisions, but also how we respond to advertising and other brand stimuli. It has been powerfully demonstrated that emotions
are central to our decision-making and our rational consciousness is considerably less dominant.

These findings have huge implications for the research industry, those traditional providers of pre-testing and tracking
surveys, some of whom are still wrestling with the shift from face-to-face and telephone interviewing to online. For decades,
they have been promulgating research techniques (let alone fieldwork methods) that have barely moved on from the
Awareness, Interest, Desire, Action (AIDA) model of how communication works. Highly rational, high attention processing of
information, brand recall and recognition and persuasion scores, not forgetting the dreaded norms, are all trademark measures
of the wrong sort of measurement for today's marketers and their brands. Nowadays, measuring emotional response should be
central, not peripheral, to consumer communications research. It has been proved that the way people feel about an ad is a

Downloaded from warc.com

3 

 
 

better predictor of its potential effectiveness than rational, information-processing measures.

Let's not forget the typical multinational brand manager who measures their ad's effectiveness by putting a TV animatic
through some sort of quantitative pretest among 100 representative members of the UK population (who may or may not have
been the actual target audience). Back comes a key one number score generated by a black box technique that determines
whether or not the ad is going to get made. Once over that hurdle, and on air, a benchmark data point is established, and any
changes in brand and ad awareness, ad recall, brand image and purchase decision are tracked, and if positive, linked to the
timing of the campaign. I suspect that there are still many brand managers operating under such a system and I suspect their
days are numbered. It's time to wake up to the New World of Performance Measurement.

EVALUATING THE OPTIONS

The principles of evaluation are simple, and these have not changed; it is putting them into practice that can be tricky.

One key thing is to remember that, to know whether or not you have been successful is to know what you set out to achieve in
the first place, so start by revisiting your campaign objectives and see how they relate to the overarching business objectives.
They should be clear and well defined, giving a hint as to the quantitative results and timescale they are intended to perform
under.

The second thing is that it is not enough to know whether or not you succeeded in meeting your objectives; you also need to
understand how or why you did (or didn't). Only then does true learning start. So, you need to identify your communications
strategy, and figure out how that contributed to the outcome. If possible, you will also want to isolate the various effects of your
different channels and messages (rather than the fact you were running a BOGOF at the same time).

The third thing to understand is the context in which your campaign was operating – because it is never in a vacuum. Other
things will have been happening, which you may or may not have anticipated, and these may have also influenced the
outcome.

You need to develop a conceptual model of how your communication is going to work on the consumer. Some research
companies build this into their research development model and are often surprised at how hard some clients and agencies
seem to find it to state how they expect their campaign to work against the target to deliver the intended results. But all parties
agree how helpful it is to focus their thinking and unite the teams involved against a common goal.

It is known as the By, By, By technique (Figure 1) and it works by asking yourself: 'what are we going to achieve?' and 'how
are we going to achieve it?' at every stage. It was originated by Marilyn Baxter. Each layer, from Business Objectives to
Business Strategy, down to Advertising Objectives and Creative Strategy, answers that 'how?' in the form of 'we'll do that by
doing such and such' and 'we'll achieve that by doing something else'… and so on.

You can add as many layers as you like to this ladder, by setting out in more and more detail how your different strategies are
intended to work – sometimes right down to the executional detail of the creative or media strategy. The example in Figure 1
comes from M&C Saatchi's Police Recruitment case study, winner of APG Gold and IPA Silver awards.

Once the theoretical model of your strategy has been agreed, you can use it to identify the effects you expect to see if it works.
So, in this example, you would expect to achieve an increase in the number of policemen and women hired into the force (note
– not see more applications in your postbag). At level two, you would expect the quality of applications to improve. At levels

Downloaded from warc.com

4 

 
 

three and four, you would like to see the campaign generating these responses from the audience, and so on.

From here, you can figure out exactly how you are going to gather this data. You can use it to develop the methodology and
questionnaire design for any consumer research. You would want to know the number of new police recruits for starters,
compared with previous years. Also, an analysis of the educational calibre of the applicants, or possibly anecdotal feedback
from the recruitment interviews, compared with previous years. Then some qualitative research into the creative concepts to
gauge responses from the audience. Followed by image tracking on perceptions of how hard it is to be a member of the police
force, and how much respect you accord them (having established a benchmark first), analysed by those who had ad recall.
The important thing is not to overwhelm with too many measurements, figure out your KPIs, and include some business
numbers as well as behavioural and attitudinal measures. Keep an eye on the end game.

CONCLUSIONS
Communications ideas are powerful but expensive things. They can work in both the short-term and the long-term, and it is the
latter that tends to transform the brands and the businesses behind them.

These valuable successes do not happen overnight, however; they evolve from a virtuous circle of hypothesis, testing,
learning and feedback. Knowing how to predict and measure effectiveness will encourage clients to invest for the long-term
and value the contribution that communication makes.

It has never been easy to prove communication effectiveness, and it has just got a lot harder and more complicated. On the
plus side, it has become a lot more accurate. Reading around the subject, developing hypotheses, starting early, being well
prepared and investing appropriately, thinking both short and long-term, are all 'best behaviours'.

FURTHER READING ON WARC.COM

Peperami: The Consequences of Unleashing the Beast, Justin Kent, IPA Effectiveness Gold Winner, 1994

Home Office/Police: How Thinking Negatively Ended the Negative Thinking, Richard Storey, IPA Silver and Best
Insight award, 2002

Using an Emotional Model to Measure Ad Effectiveness, Orlando Wood, Admap Jan 2010

On the road to a New Effectiveness Model: Measuring Emotional Responses to Television Advertising, Anca
Cristina Micu & Joseph T Plummer, The Advertising Research Foundation White Paper, January 2007

How Effective is Creativity? Emotive Content in TV Advertising Does Not Increase Attention, R Heath, A Nairn, P
Bottomley, JAR Dec 2009

Payback Calculations – How to Make Sure You Get Your Sums Right, Les Binet, IPA

15 ways NOT to evaluate your communications, Les Binet, Admap, February 2006

OTHER READING
Marketing in The Era of Accountability, Les Binet and Peter Field, IPA/Warc 2007

Downloaded from warc.com

5 

 
 

How public service advertising works, Judie Lannon (ed), COI, IPA, Warc 2008

Best Practice Guide to Measuring Marketing Payback, www.ipa.co.uk

© Copyright Warc 2010

Warc Ltd.
85 Newman Street, London, United Kingdom, W1T 3EX
Tel: +44 (0)20 7467 8100, Fax: +(0)20 7467 8101

www.warc.com

All rights reserved including database rights. This electronic file is for the personal use of authorised users based at the subscribing company's office location. It may not be reproduced, posted on intranets, extranets
or the internet, e-mailed, archived or shared electronically either within the purchaser’s organisation or externally without express written permission from Warc.

Downloaded from warc.com

6