Introduction

A pilot experiment, also called a pilot study, is a small scale

preliminary study conducted before the main research, in order
to check the feasibility or to improve the design of the
research. A pilot, or feasibility study, is a small experiment
designed to test logistics and gather information prior to a
larger study, in order to improve the latter’s quality and
efficiency.

Pilot studies, therefore, may not be appropriate for case

studies. They are frequently carried out before large-scale
quantitative research, in an attempt to avoid time and money
being wasted on an inadequately designed project. A pilot
study is usually carried out on members of the relevant
population, but not on those who will form part of the final
sample. This is because it may influence the later behaviour of
research subjects if they have already been involved in the
research.

A pilot experiment is often used to test the design of the full-

scale experiment which then can be adjusted, a potentially
valuable insight. Should anything be missing in the pilot, it can
be added to the experiment to improve chances of a clear
outcome that will influence the full-scale (and more expensive)
experiment.

Often in engineering applications, pilot experiments are used to

sell a product and provide quantitative proof that the system
1
has potential to succeed on a full scale basis. Pilot experiments
are also used to reduce cost, as they are less expensive than
full experiments. If there is not enough reason to provide full
scale applications, pilots can generally provide this proof.

In sociology, pilot studies can be referred to as small-scale

studies that will help identify design issues before the main
research is done.

The term pilot study is used in two different ways in social

science research. It can refer to so-called feasibility studies
which are "small scale version[s], or trial run[s], done in
preparation for the major study"). However, a pilot study can
also be the pre-testing or 'trying out' of a particular research
instrument (Baker 1994: 182-3). One of the advantages of
conducting a pilot study is that it might give advance warning
about where the main research project could fail, where
research protocols may not be followed, or whether proposed
methods or instruments are inappropriate or too complicated.
These are important reasons for undertaking a pilot study, but
there are additional reasons, for example convincing funding
bodies that your research proposal for the main study is worth
funding. Thus pilot studies are conducted for a range of
different reasons.

Although pilot experiments have a well-established tradition in

public action, their usefulness as a strategy for change has
been questioned, at least in the domain of environmental
management. It is argued that extrapolation from a pilot study
2
to large scale environmental strategy cannot be assumed to be
possible, partly due to the exceptional resources and
favourable conditions that often accompanies a pilot study.

Reasons for Pilot Studies

A pilot study can reveal deficiencies in the design of a proposed

experiment or procedure and these can then be addressed
before time and resources are expended on large scale studies.
Animal experiments are not usually carried out in isolation, but
are part of a programme of research. A good research strategy
requires careful planning and a pilot study will often be a part
of this strategy.

A pilot study is normally small in comparison with the main

experiment and therefore can provide only limited information
on the sources and magnitude of variation of response
measures. It is unlikely, for example, that a pilot study alone
can provide adequate data on variability for a power analysis to
estimate the number of animals to include in a well designed
experiment. A systematic review of the literature or even a
single publication is a more appropriate source of information
on variability. The pilot study may, however, provide vital
information on the severity of proposed procedures or
treatments.

Other reasons include:

• Developing and testing adequacy of research instruments

• Assessing the feasibility of a (full-scale) study/survey
3
 • Designing a research protocol
• Assessing whether the research protocol is realistic and
workable
• Establishing whether the sampling frame and technique
are effective
• Assessing the likely success of proposed recruitment
approaches
• Identifying logistical problems which might occur using
proposed methods
• Estimating variability in outcomes to help determining
sample size
• Collecting preliminary data
• Determining what resources (finance, staff) are needed
for a planned study
• Assessing the proposed data analysis techniques to
uncover potential problems
• Developing a research question and research plan
• Training a researcher in as many elements of the research
process as possible
• Convincing funding bodies that the research team is
competent and knowledgeable
• Convincing funding bodies that the main study is feasible
and worth funding
• Convincing other stakeholders that the main study is
worth supporting

Logistical issues which may be revealed by a pilot study

4
A pilot study may address a number of logistical issues. As part
of the research strategy the following factors can be resolved
prior to the main study:

• Check that the instructions given to investigators (e.g.

randomisation procedures) are comprehensible;

• Check that investigators and technicians are sufficiently

skilled in the procedures;

• Check the correct operation of equipment;

• Check that the experimental animal can perform a task

(physical or cognitive)

• Check the reliability and validity of results

• Detect a floor or ceiling effect (e.g. if a task is too difficult or

too easy there will be skewed results)

• Assess whether the level of intervention is appropriate (e.g.

the dose of a drug);

• Identify adverse effects (pain, suffering, distress or lasting

harm) caused by the procedure, and the effectiveness of
actions to reduce them (e.g. analgesia dose rate and
schedule);

• Define early humane endpoints.

What to do with the Data / Information

The information obtained on logistical issues should be

incorporated into the main study design. As the purpose of a
5
pilot study is to assess the feasibility of an experiment it is very
rarely sensible to present more than summary statistics of the
data. In fact, the data might be irrelevant if problems with the
methods are discovered.

If a pilot study does not lead to modification of materials or

procedures then the data might be suitable for incorporation
into the main study. The sampling strategy used to select
subjects, and the possibility of changes over time should be
carefully considered before incorporating pilot data. Even if the
pilot data are not used in this way, and even if the final design
differs markedly from the pilot, it is useful to include
information on the pilot study in any publications or reports
arising from the main experiment as this can inform the design
of future experiments.

It may be necessary to carry out a second pilot study to assess

the revised main study or in some cases the main study may
have to be abandoned.

Before beginning such a full research project, researchers need

to know that their study is valid and the study’s design will be
able to capture the data they are looking for. They need to
know that the research they plan to do will be the most
accurate and reliable research possible. The best way to do this
is to perform a pilot study.

Think of a pilot study as a miniature study. Typically, it will call

for fewer literary resources, a shorter time frame, and fewer

6
research subjects. For example, if the full study is going to
reference 50 previous books or articles, the pilot study might
reference five or ten. If the full study will call for a survey of
100 people, the pilot study might rely on surveying a small
handful of individuals. If the full study is going to take many
months or years to complete, the pilot study can focus on two
or three lakes over the course of a few weeks.

Methods and Statistics Required For a Pilot Study

The researcher has many options. This research method could
be as simple as using interview guides, and the measurement
techniques used could be as simple as using averages, simple
percentages, and standard deviations. However, the
researchers may wish to employ more sophisticated methods
and analyses and may use Appendix A as a basic guide.

A small chemistry experiment in a college laboratory, for

example, costs very little, and mistakes or validity problems
easily rectified. At the other end of the scale, a medical
experiment taking samples from thousands of people from
across the world is expensive, often running into the millions of
dollars.

Finding out that there was a problem with the equipment or

with the statistics used is unacceptable, and there will be dire
consequences.

7
A field research project in the Amazon Basin costs a lot of time
and money, so finding out that the electronics used do not
function in the humid and warm conditions is too late.

To test the feasibility, equipment and methods, researchers will

often use a pilot study, a small-scale rehearsal of the larger
research design. Generally, the pilot study technique
specifically refers to a smaller scale version of the experiment,
although equipment tests are an increasingly important part of
this sub-group of experiments.

For example, the medical researchers may conduct a smaller

survey upon a hundred people, to check that the protocols are
fine.

The Amazon Researchers may perform an experiment, in

similar conditions, sending a small team either to the Amazon
to test the procedures, or by using something like the tropical
bio-dome at the Eden Project.

Pilot studies are also excellent for training inexperienced

researchers, allowing them to make mistakes without fear of
losing their job or failing the assignment.

Logistical and financial estimates can be extrapolated from the

pilot study, and the research question, and the project can be
streamlined to reduce wastage of resources and time.

Pilots can be an important part of attracting grants for research

as the results can be placed before the funding body.
8
Generally, most funding bodies see research as an investment,
so are not going to dole out money unless they are certain that
there is a chance of a financial return.

Unfortunately, and paper reporting the preliminary pilot study,

especially if problems were reported, is often stigmatized and
sidelined. This is unfair, and punishes researchers for being
methodical, so these attitudes are under a period of re-
evaluation.

Discouraging researchers from reporting methodological errors,

as found in pilot studies, means that later researchers may
make the same mistakes.

The other major problem is deciding whether the results from

the pilot study can be included in the final results and analysis,
a procedure that varies wildly between disciplines.

Pilots are rapidly becoming an essential pre-cursor to many

research projects, especially when universities are constantly
striving to reduce costs. Whilst there are weaknesses, they are
extremely useful for driving procedures in an age increasingly
dominated by technology, much of it untested under field
conditions.

Methods of Pilot Studies

Pilot studies can be based on quantitative and/or qualitative

methods and large-scale studies might employ a number of
9
pilot studies before the main survey is conducted. Thus
researchers may start with "qualitative data collection and
analysis on a relatively unexplored topic, using the results to
design a subsequent quantitative phase of the study"
(Tashakkori & Teddlie 1998: 47). The first phase of a pilot
might involve using in-depth interviews or focus groups to
establish the issues to be addressed in a large-scale
questionnaire survey. Next the questionnaire, e.g. the wording
and the order of the questions, or the range of answers on
multiple-choice questions, might be piloted. A final pilot could
be conducted to test the research process, e.g. the different
ways of distributing and collecting the questionnaires. For
example, a recent study exploring nurses' and midwives'
attitudes to research followed this pattern. In this study focus
groups were used to identify key issues from which a
questionnaire could be developed, and this was then piloted
prior to the study proper (Hundley et al. 2000). On a much
larger scale, the largest (decennial) survey in the UK, the
Census (of 29th April 2001), tested methodological and other
changes to the 1991 Census questionnaire on over 100,000
households in 1997. This 1997 Census Test "provided essential
information on public reaction to new questions and form style
as well as assessing the success of collection and processing
methods" (Office for National Statistics, General Register Office
for Scotland, Northern Ireland Statistical & Research Agency
1999: 15).

10
Pilot studies may also try to identify potential practical
problems in following the research procedure. For example, in
a recent Scottish study of maternity care the pilot phase
demonstrated that the proposed means of distributing the
questionnaires would not be adhered to (van Teijlingen et al.
2001). Without consulting the research team, the person
responsible for distributing the questionnaires from the hospital
records department decided that it was better to distribute
them through the community midwives. This was despite the
fact that the hospital itself had suggested the records
department as a means of distribution. Other problems such as
poor recording and response rates can also be identified and
precautionary procedures or safety nets can be devised.

The steps used to pilot a questionnaire on a small group of

volunteers, who are as similar as possible to the target
population, are listed in Table 2. Pilot studies can also uncover
local politics or problems that may affect the research process.
In the study described above, the managers of maternity
services had different perceptions of what the forthcoming
changes in the Data Protection Act (1998) allowed them to do
about the involvement of their clients in research. In the above
mentioned recent Scottish study of maternity care one head of
midwifery voiced ethical concerns about the use of follow-up or
reminder letters due to a previous local incident, where parents
of an ill baby had been sent a questionnaire which was felt to
be inappropriate, and as a result of changes to the UK Data
Protection Act. Consequently reminders were sent out via the
11
head of midwifery in case there were any problems with the
newborn baby.

Problems of Pilot Studies

It should be recognised pilot studies may also have a number

of limitations. These include the possibility of making
inaccurate predictions or assumptions on the basis of pilot
data; problems arising from contamination; and problems
related to funding. These issues are now discussed in turn.

Completing a pilot study successfully is not a guarantee of the

success of the full-scale survey. Although pilot study findings
may offer some indication of the likely size of the response rate
in the main survey, they cannot guarantee this because they
do not have a statistical foundation and are nearly always
based on small numbers. Furthermore, other problems or
headaches may not become obvious until the larger scale study
is conducted.

A further concern is that of contamination. This may arise in

two ways:

1. where data from the pilot study are included in the main
results;
2. where pilot participants are included in the main study,
but new data are collected from these people.

Social scientists engaged in predominantly quantitative

research are likely to argue that: "an essential feature of a pilot
12
study is that the data are not used to test a hypothesis or
included with data from the actual study when the results are
reported" (Peat et al. 2002: 57). The obvious concern is that if
there were problems with the research tool and modifications
had to be made in the light of the findings from the pilot study,
data could be flawed or inaccurate. However, where an
established and validated tool is being used and the pilot study
is determining other methodological aspects such as
recruitment rates, it could be argued that such data may be of
value.

A more common problem is deciding whether to include pilot

study participants or site(s) in the main study. Here the
concern is that they have already been exposed to an
intervention and, therefore, may respond differently from those
who have not previously experienced it. This may be positive,
for example the participants may become more adept at using
a new tool or procedure. However it may also be negative with
participants showing a decline in following a protocol because it
is no longer novel. Indeed both changes in behaviour have long
been recognised and a 'run in' period, where an intervention is
introduced prior to a study, is often used for these reasons.
The concern about including participants from the pilot study in
the main study arises because only those involved in the pilot,
and not the whole group, will have had the experience. In
some cases however it is simply not possible to exclude these
pilot-study participants because to do so would result in too
small a sample in the main study. This problem arises in
13
particular where the samples are clusters, for example schools,
prisons or hospitals. In such cases one can conduct a
sensitivity analysis (or sub-group analysis) to assess to what
extent the process of piloting influences the size of the
intervention effect.

Contamination is less of a concern in qualitative research,

where researchers often use some or all of their pilot data as
part of the main study. Qualitative data collection and analysis
is often progressive, in that a second or subsequent interview
in a series should be 'better' than the previous one as the
interviewer may have gained insights from previous interviews
which are used to improve interview schedules and specific
questions. Some have therefore argued that in qualitative
approaches separate pilot studies are not necessary (e.g.
Holloway 1997: 121). For example, a qualitative interviewer
conducting 15 focus group interviews will listen to the
recordings or read through the transcripts of the first three or
four in order to improve the questions, the way of introducing
the issues into the group interview or even to add new topics.
Thus, although there is no specific pilot study, analysis of the
earlier focus groups may help improve the later ones. However,
Frankland and Bloor (1999: 154) argue that piloting provides
the qualitative researcher with a "clear definition of the focus of
the study" which in turn helps the researcher to concentrate
data collection on a narrow spectrum of projected analytical
topics. Piloting of qualitative approaches can also be carried out

14
if "the researcher lacks confidence or is a novice, particularly
when using the interview technique" (Holloway 1997: 121).

Problems may also arise where a pilot study requires a

significant investment of resources, making it difficult for the
study team to call a halt to the research after an unsuccessful
pilot study. Researchers might be tempted to make
considerable changes in the main study, rather than deciding
that the proposed study is not possible with the available
resources, time, population, etc. In contrast, funding bodies
may be reluctant to fund a further study if the pilot has been
substantial as they may view the research as no longer
original, especially if results from the pilot study are published.

Why are Pilot Studies Not Reported?

Publication bias may occur because of a tendency for journals

to accept only papers that have statistically significant results
and not to report non-significant effects (Mahoney 1977;
Chann 1982; Dickersin, 1990). A recent study exploring
research on passive smoking found a difference of two years in
the median time to publication between findings from
significant and non-significant studies (Misakian & Bero 1998).
It follows that papers reporting methodological issues, such as
those identified during the pilot phase of a study, will also be
less attractive to publishers.

15
Selective publication of research results has been recognised as
a problem. It may lead to an overestimation of the
effectiveness of interventions, exposing patients to useless or
harmful treatments, while overestimation of adverse effects
may mean that patients are denied effective forms of care
(Oxman et al. 1994). In the past editors have recognised the
dangers of publication bias with respect to clinical trials and
have offered 'an amnesty for unpublished trials' in an attempt
to overcome these problems (Smith & Roberts, 1997).
However, it is equally important to ensure that lessons learned
with respect to the research method are shared, otherwise
patients may be subjected to poorly developed tools or money
may be wasted because methods of recruitment failed. A
consistent selection bias favouring reports of primary research
over papers on research methods, theoretical thinking, or
secondary analysis, can lead to many of researchers re-
inventing the wheel without having had the opportunity to
learn from other people's experience.

Conclusions

It has been said that pilot studies are likely to be

"underdiscussed, underused and underreported" (Prescott and
Soeken, 1989 p60). Full reports of pilot studies are rare in the
research literature (Lindquist, 1991; Muoio et al, 1995, van
Teijlingen et al. 2001). When reported, they often only justify
the research methods or particular research tool used. Too
often research papers only refer to one element of the pilot
16
study, for example, to the 'pre-testing' or 'pilot testing' of a
questionnaire (De Vaus, 1993). Such papers simply state: "the
questionnaire was tested for validity and reliability." When pilot
studies are mentioned in more detail in academic papers and
reports, researchers regularly comment that they "had learned
from the pilot study" and made the necessary changes, without
offering the reader details about what exactly was learnt. Some
of these processes and outcomes from both successful and
failed pilot studies might be very useful to others embarking on
projects using similar methods and instruments. This is
particularly important because pilot studies can be "time-
consuming, frustrating, and fraught with unanticipated
problems, but it is better to deal with them before investing a
great deal of time, money, and effort in the full study" (Mason
and Zuercher, 1995). It has also been argued that the current
research climate demands accountability from researchers,
which means that there is a need to ensure the best possible
use of research results (Crosswaite and Curtice 1994). We
would like to go one step further and argue that researchers
have an ethical obligation to make the best use of their
research experience by reporting issues arising from all parts of
a study, including the pilot phase. Well-designed and well-
conducted pilot studies can inform us about the best research
process and occasionally about likely outcomes. Therefore
investigators should be encouraged to report their pilot studies,
and in particular to report in more detail the actual

17
improvements made to the study design and the research
process.

18
References

Altman DG. (1991) Practical Statistics for Medical Research.

London: Chapman & Hall
Baker, T.L. (1994), Doing Social Research (2nd Edn.), New
York: McGraw-Hill Inc.
Bowling, A. (1997), Research Methods in Health: Investigating
Health & Health Services, Buckingham: Open
University Press. p 232.
Burns, N. and Grove, S.K. (1999), Understanding Nursing
Research (2nd edn.), Philadelphia: W.B. Saunders
Company. p 40.
Chann, S.S. (1982), The epidemiology of unpublished
randomised controlled trials. Clinical Research 30:
234A.
Cochran WG & Cox GM (1992) Experimental Designs (2nd
Edition). New York: John Wiley & Sons
Crombie, I.K. and Davies, H.T.O. (1997), Research in Health
Care: Design, Conduct and Interpretations of Health
Services Research, London: Wiley.
Crosswaite, C. and Curtice, L. (1994), Disseminating research
results-the challenge of bridging the gap between
health research and health action. Health Promotion
International 9: 289.
De Vaus, D.A. (1993), Surveys in Social Research (3rd edn.),
London: UCL Press.

19
Dickersin, K. (1990), The existence of publication bias and risk
factors for its occurrence. Journal of the American
Medical Association 263: 1385-1389.
Festing MFW, Overend P, Das RG, Borja MC & Berdoy M (2002)
The Design of Animal Experiments. London: Royal
Society of Medicine Press
Frankland, J. and Bloor, M. (1999), Some issues arising in the
systematic analysis of focus group material, In:
Barbour, R. and Kitzinger, J. (eds) Developing Focus
Group Research: Politics, Theory & Practice, London:
Sage
Haralambos, M.; M. Holborn (2000). Sociology: Themes and
Perspectives. Hammersmith, London: HarperCollins
Publishers.
Holloway, I. (1997). Basic Concepts for Qualitative Research,
Oxford: Blackwell Science.
Hundley, V, Milne, J, Leighton-Beck, L. et al. (2000). Raising
research awareness among midwives and nurses:
does it work? Journal of Advanced Nursing 31 (1): 78-
88.
Lancaster GA, Dodd S & Williamson PR (2004) Design and
analysis of pilot studies: recommendations for good
practice. Journal of Evaluation in Clinical Practice
10(2): 307-12
Lindquist, R. (1991). Don't forget the pilot work! Heart Lung
20: 91-92.

20
Mahoney, M.J. (1977), Publication prejudices: an experimental
study of confirmatory bias in the peer review system.
Cognitive Therapy Research 1: 161-175.
Mason, D.J. and Zuercher, S.L. (1995). Pilot studies in clinical
nursing research. Journal of the New York State
Nursing Association 26: 11.
Misakian, A.L. and Bero, L.A. (1998), Publication bias and
research on passive smoking. Journal of the American
Medical Association 280: 3: 250-253.
Muoio, R., Wolcott, L., and Seigel, H. (1995). A win-win
situation: The pilot program. Journal of Continuing
Education in Nursing. 26: 230-233.
Office for National Statistics, General Register Office for
Scotland, Northern Ireland Statistical & Research
Agency. (1999), 2001 Census Information Paper,
Government Statistical Services:
Ruxton GD & Colegrave N (2006) Experimental Design for the
Life Sciences (2nd edition). Oxford: Oxford University
Press
Teijlingen, E. R; V. Hundley (2001). The importance of pilot
studies. Social research UPDATE, (35).
http://sru.soc.surrey.ac.uk/SRU35.html.

21