The random sample survey is a commonly used method in market research and such surveys generally are
considered to provide more robust results than purposive or convenience (such as quota) samples. Simple random
samples (SRS), where every member of the population has an equal or predetermined chance of selection, and the
sampling is single stage, are the simplest to visualise and the level of precision of results derived from them is easy
to calculate. However, SRS are almost impossible to achieve in reality (due to imperfect sampling frames, nonresponse,
and so on) and, in any case, they are relatively inefficient ways of obtaining a particular number of
responses, in terms of fieldwork and travelling costs.
Much of the theory underlying this paper is not new, and many of the concepts covered relate to important work
published a number of years ago. However, the messages that this paper serves to portray are no less relevant
today than they were in the past, as cluster sampling continues to be a popularly used type of survey design. It is
less statistically efficient than simple random sampling, and this is as critical as ever today as researchers continue
their quest to obtain data in an ever-faster and less expensive way.
If one considers the situation of running a national survey of 10,000 people across the whole of Great Britain, it
would be impractical and expensive to send interviewers across the whole country to addresses which are
completely randomly scattered, because travelling time and interviewer costs are likely to be prohibitive. It would
be more economical and realistic to focus on specific localities (or clusters) across the country (e.g. constituencies
or wards) and sample from some localities (these particular localities being selected at random) but not from
others. There is, however, a price to pay for this saving in administrative time and costs, namely that one would
receive no information about the localities or the clusters from which one is not sampling. Correspondingly,
aggregate results would be based only on the clusters from which one is sampling. Also, there may be good
reasons why people's views within a small area are similar. This would lead to a reduction in precision and a
widening of confidence intervals as compared to the situation of running a survey on a totally SRS basis.
This paper begins by providing a brief background to the concept of the design effect, and how it can be used to
see how various aspects of survey design (such as clustering, but also stratification and weighting) can impact on
the level of statistical reliability of a survey. Focusing on clustering, a commonly used formula for calculating the
confidence interval (CI) for clustered samples is then introduced. Using this formula on a worked example covering
a specific national survey, it is shown how the level of confidence can vary with the number of clusters taken (which
is controllable) and with the differences between the clusters (which cannot be controlled). At this point, what
appears to be a surprising conclusion is verified by an alternative methodological approach. The paper then goes on
to present a theoretical and somewhat historical example which nevertheless is useful in illustrating that the design
effect is very different for different survey measures. The paper also touches on the issue of interviewer variability,
which is in itself a form and extension of clustering. Finally, the report is put together with appropriate costings to
look at the practical and financial implications of clustering-related losses of effective sample size.