Misrepresenting convenience
samples
A related and significant concern with nonprobability-based
sampling methods, both for
Internet-based and traditional surveys, is that
survey accuracy is characterized only in terms
of sampling error and without regard to
the potential biases that may be present in
the results. While this has always been a
concern with all types of survey, the ease
and spread of Internet-based surveys seems to
have exacerbated the practice. For example,
the results of an ‘E-Poll’ were explained as
follows:
THE OTHER HALF / E-Poll® Survey of 1,007
respondents was conducted January 16–20, 2003.
A representative group of adults 18+ were randomly
selected from the E-Poll online panel. At a
95% confidence level, a sample error of +/− 3%
is assumed for statistics based on the total sample of
1,007 respondents. Statistics based on sub-samples
of the respondents are more sensitive to sampling
error. (From a press release posted on the E-Poll
website.)
No mention was made in the press release
that the ‘E-Poll online panel’ consists of
individuals who had chosen to participate in
online polls, nor that they were unlikely to
be representative of the general population.
Rather, it leaves readers with an incorrect
impression that the results apply to the general
population when, in fact, the margin of error
for this particular survey is valid only for
adult members of that particular E-Poll online
panel.In response to the proliferation of these
types of misleading statement, the American
Association for Public Opinion Research
(AAPOR) has publicly stated that ‘The reporting
of a margin of sampling error associated
with an opt-in or self-identified sample (that
is, in a survey or poll where respondents are
self-selecting) is misleading.’ They go on to
say, ‘AAPOR considers it harmful to include
statements about the theoretical calculation of
sampling error in descriptions of such studies,
especially when those statements mislead the
reader into thinking that the survey is based
on a probability sample of the full target
population’ (AAPOR, 2007).