Psychology of Popular Media Culture
Testing the Reliability and Validity of Different Measures
of Violent Video Game Use in the United States,
Singapore, and Germany
Robert Busching, Douglas A. Gentile, Barbara Krahé, Ingrid Möller, Angeline Khoo, David A.
Walsh, and Craig A. Anderson
Online First Publication, August 26, 2013. doi: 10.1037/ppm0000004
CITATION
Busching, R., Gentile, D. A., Krahé, B., Möller, I., Khoo, A., Walsh, D. A., & Anderson, C. A.
(2013, August 26). Testing the Reliability and Validity of Different Measures of Violent Video
Game Use in the United States, Singapore, and Germany. Psychology of Popular Media
Culture. Advance online publication. doi: 10.1037/ppm0000004
Testing the Reliability and Validity of Different Measures
of Violent Video Game Use in the United States,
Singapore, and Germany
Robert Busching
University of Potsdam
Douglas A. Gentile
Iowa State University
Barbara Krahé and Ingrid Möller
University of Potsdam
Angeline Khoo
National Institute of Education
David A. Walsh
Mind Positive Parenting, Minneapolis, MN
Craig A. Anderson
Iowa State University
To examine the potential link between violent video game play and aggressive behavior,
reliable and valid measures of the level of violence in video games are required. A
range of different approaches have been used in the literature, and it is not clear to what
extent they converge or measure different constructs. To address this question, three
large longitudinal data sets covering at least 12 months were used from the United
States (N 1,232), Singapore (N 3,024), and Germany (N 1,715). Violent content
was measured through user ratings, expert ratings, and official agency ratings of
individual titles as well as through expert ratings of game genres. The different
measures were linked to aggressive behavior both cross-sectionally and longitudinally
in all three countries and to the normative acceptance of aggression in Germany and
Singapore. User ratings, expert ratings, and official agency ratings were found to be
reliable. They showed substantial correlations within each culture as well as between
the different cultures, indicating high convergent validity. Measures using nominations
of game titles and measures using genre lists showed similar relationships with
aggressive behavior and aggressive norms, both concurrently and prospectively over 12
months. Recommendations for a best practice approach to the assessment of violent
content in video games are derived from the findings.
Keywords: computer games, aggression, questionnaires, measurement, media violence
In recent years, many studies have examined
the effects of violent video games on aggressive
thoughts, feelings, and behaviors. These studies
have created a fair amount of controversy both
within and outside the scientific community.
Some of the controversy has focused on methRobert
Busching, Barbara Krahé, and Ingrid Möller,
Department of Psychology, University of Potsdam, Germany;
Douglas A. Gentile and Craig A. Anderson, Center
for the Study of Violence Iowa State University; Angeline
Khoo, Psychological Studies Academic Group, National
Institute of Education, Singapore; David A. Walsh, Mind
Positive Parenting, Minneapolis, MN.
The authors gratefully acknowledge the support of
Claudia Ahlert and Marianne Hannuschke. The U.S.
study was supported by grants from Medica Foundation,
Healthy and Active America Foundation, Cargill, and
Fairview Health Services (Walsh & Gentile, PIs). The
German study was supported by a grant from the German
Research Foundation (Krahé, PI). The Singapore
study was supported by a grant from the MOE and MDA
(Khoo, PI).
Correspondence concerning this article should be addressed
to Robert Busching, Department of Psychology,
University of Potsdam, Karl-Liebknecht-Str. 24-
25, D-14476 Potsdam, Germany. E-mail: robert.
busching@uni-potsdam.de
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Psychology of Popular Media Culture © 2013 American Psychological Association
2013, Vol. 2, No. 3, 000 2160-4134/13/$12.00 DOI: 10.1037/ppm0000004
1
odological issues, such as how to measure violence
in video games (Anderson et al., 2010;
Ferguson, 2010). Many different approaches
have been used to measure the violence levels in
people’s media diet, including user reports of
individual games (e.g., Anderson & Dill, 2000)
and ratings of game genres (e.g., Möller &
Krahé, 2009). Violent content has been assessed
in different ways, such as through ratings by
official agencies (e.g., Kutner & Olson, 2008),
experts (Wei, 2007), and users (Funk, Buchman,
Jenks, & Bechtoldt, 2003). This diversity
has positive and negative aspects. It is generally
beneficial to the field because it allows for conceptual
replication, testing the construct in a
stringent way by requiring that it be robust to
measurement differences (Roediger, 2012). It is
potentially harmful to the field if studies using
different methods find different results, which
can lead to difficulties in interpreting the overall
evidence. For example, it is becoming common
for researchers using one method to claim that
their findings fail to replicate others’, and to
interpret this as a lack of support for the link
between violent games and aggression. However,
as Roediger (2012) notes, the failure to
find evidence with one paradigm does not necessarily
have implications for the other paradigms.
The purpose of the present article is to
bring together longitudinal data sets from three
continents to compare directly different methods
of measuring violence in video games, testing
their reliability and convergent validity as
well as the predictive validity of each approach
in predicting outcomes a year later.
Ratings of violent video game contents are
often used to investigate the link between violent
video game use and aggression-related outcome
variables. Six meta-analyses, based on
over 100 studies of violent video games, have
been published (Anderson, 2004; Anderson &
Bushman, 2001; Anderson et al., 2010; Ferguson,
2007a, 2007b; Sherry, 2001). Although
they vary greatly in terms of how many studies
they include, they find almost identical effect
sizes for violent video games on aggressive
thoughts, feelings, and behaviors (approximately
r 0.15 to 0.25). The empirically de-
fined effect sizes are in the small to moderate
range, and scientists often differ in their interpretations
of the importance of a given effect
size. This is certainly the case in this field of
study, with some researchers arguing that the
small-to-moderate effect sizes indicate a lack of
importance and others arguing that they are
highly important. The purpose of this study is
not to address that debate, but to examine
whether different operationalizations of violent
game play converge and how they are linked to
aggression-related outcome measures.
Conceptually, it is possible to separate the
measurement of violent game play into three
independent parts. One part is to obtain information
about the media diet itself (i.e., what
games users play). A second is to assess the
frequency or amount of time played, and the
third part is to assess the games’ violent content.
The simplest approach toward measuring the
games users play is to ask them directly to name
their favorite or most frequently played games
(e.g., “Title of your ‘most played’ game,” Anderson
& Dill, 2000). This approach focuses on
the game as the smallest unit, but it is also
possible to shift the focus to a more general
level. Rather than asking players to nominate
specific games they play, they can be asked to
describe the game genres they typically play. In
this approach, participants are presented with a
list of common game genres and asked to indicate
the amount of time they usually spend
playing games belonging to each genre (e.g.,
“How often do you play first-person shooter
computer games?,” Richmond & Wilson,
2008).
The second part involves assessing how
much time a person spends playing games, either
in general, or for each specific game or
genre. The third part of assessing violent game
play involves a violent content judgment of
either the individual game titles or the broader
genres to which they may be assigned. Although
there is no single “gold standard” for
measuring violent content in video games, there
are three common approaches: (1) to ask the
participants to judge the amount of violence in
a game or genre, (2) to use official game ratings,
and (3) to use independent raters to assess the
level of violent content in the games or genres.
A straightforward and widely used approach
to measuring violent content is to ask the participants
to rate the level of violence of each of
their nominated games. They can either be
asked to rate the global amount of violence in
the game (e.g., “how violent is the content of
this game,” Anderson & Dill, 2000), or they can
be asked to rate specific behavioral aspects of
2 BUSCHING ET AL.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
violence in the game (e.g., “how often do you
kill players in this game,” Gentile, Choo et al.,
2011).
The second approach is to use the assessment
of expert raters who judge the violent content of
the games or genres listed by the participants.
At least two types of expert ratings can be
generated. First, people who know a lot about
video games, such as experts from the video
game industry or journalists working for gaming
magazines, can be asked to rate each genre
or game (e.g., Krahé & Möller, 2004). In the
measures included in the present study, experts
were first presented with a definition of violence
and then asked to provide an independent