1. Introduction
Network structures provide intuitive and useful representations for modeling semantic
knowledge and inference. Within the paradigm of semantic network models, we can ask at
least three distinct kinds of questions. The first type of question concerns structure and knowlCognitive
Science 29 (2005) 41–78
Copyright © 2005 Cognitive Science Society, Inc. All rights reserved.
Requests for reprints should be sent to Mark Steyvers, Department of Cognitive Sciences, 3151 Social Sciences
Plaza, University of California–Irvine, Irvine, CA 92697–5100; E-mail: msteyver@uci.edu or Joshua B. Tenenbaum,
Department of Brain and Cognitive Sciences, 77 Massachusetts Avenue, Cambridge, MA 02139; E-mail:
jbt@mit.edu
edge: To what extent can the organization of human semantic knowledge be explained in terms
of general structural principles that characterize the connectivity of semantic networks? The
second type of question concerns process and performance: To what extent can human performance
in semantic processing tasks be explained in terms of general processes operating on semantic
networks? A third type of question concerns the interactions of structure and process:
To what extent do the processes of semantic retrieval and search exploit the general structural
features of semantic networks, and to what extent do those structural features reflect general
processes of semantic acquisition or development?
The earliest work on semantic networks attempted to confront these questions in an integrated
fashion. Collins and Quillian (1969) suggested that concepts are represented as nodes in
a tree-structured hierarchy, with connections determined by class-inclusion relations (Fig. 1).
Additional nodes for characteristic attributes or predicates are linked to the most general level
of the hierarchy to which they apply. A tree-structured hierarchy provides a particularly economical
system for representing default knowledge about categories, but it places strong constraints
on the possible extensions of predicates—essentially, on the kinds of knowledge that
are possible (Keil, 1979; Sommers, 1971). Collins and Quillian proposed algorithms for efficiently
searching these inheritance hierarchies to retrieve or verify facts such as “Robins have
wings,” and they showed that reaction times of human subjects often seemed to match the qualitative
predictions of this model. However, notwithstanding the elegance of this picture, it has
severe limitations as a general model of semantic structure. Inheritance hierarchies are clearly
appropriate only for certain taxonomically organized concepts, such as classes of animals or
other natural kinds. Even in those ideal cases, a strict inheritance structure seems not to apply
except for the most typical members of the hierarchy (Carey, 1985; Collins & Quillian, 1969;
Rips, Shoben, & Smith, 1973; Sloman, 1998).
Subsequent work on semantic networks put aside the search for general structural principles
of knowledge organization and instead focused on elucidating the mechanisms of semantic
processing in arbitrarily structured networks. The network models of Collins and Loftus
(1975), for instance, are not characterized by any kind of large-scale structure such as a treelike
hierarchy. In terms of their large-scale patterns of connectivity, these models are essentially unstructured,
with each word or concept corresponding to a node and links between any two
42 M. Steyvers, J. B. Tenenbaum/Cognitive Science 29 (2005)
Fig. 1. Proposed large-scale structures for semantic networks: (a), a tree-structured hierarchy (e.g., Collins & Quillian,
1969); (b), an arbitrary, unstructured graph (e.g., Collins & Loftus, 1975); (c), a scale-free, small-world graph.
nodes that are directly associated in some way (Fig. 1B). Quantitative models of generic associative
networks, often equipped with some kind of spreading-activation process, have been
used to predict performance in a range of experimental memory retrieval tasks and to explain
various priming and interference phenomena (Anderson, 2000; Collins & Loftus, 1975; Deese,
1965; Nelson, McKinney, Gee, & Janczura, 1998).
As a result of research in this tradition, there is now a fair consensus about the general character
of at least some of the processes involved in the formation and search of semantic memory
(Anderson, 2000). By contrast, there is relatively less agreement about general principles
governing the large-scale structure of semantic memory, or how that structure interacts with
processes of memory search or knowledge acquisition. Typical textbook pictures of semantic
memory still depict essentially arbitrary networks, such as Fig. 1B, with no distinctive
large-scale structures. The implications for semantic network theories of meaning are not
good. Under the semantic net view, meaning is inseparable from structure: The meaning of a
concept is, at least in part, constituted by its connections to other concepts. Thus, without any
general structural principles, the semantic net paradigm offers little or no general insights into
the nature of semantics.
In this article, we argue that there are in fact compelling general principles governing the
structure of network representations for natural language semantics and that these structural
principles have potentially significant implications for the processes of semantic growth and
memory search. We stress from the outset that these principles are not meant to provide a genuine
theory of semantics, nor do we believe that networks of word–word relations necessarily
reflect all of the most important or deepest aspects of semantic structure. We do expect that semantic
networks will play some role in any mature account of word meaning. Our goal here is
to study some of the general structural properties of semantic networks that may ultimately
form part of the groundwork for any semantic theory.
The principles we propose are not based on any fixed structural motif such as the tree-structured
hierarchy of Collins and Quillian (1969). Rather, they are based on statistical regularities
that we have uncovered via graph-theoretic analyses of previously described semantic networks.
We look at the distributions of several statistics calculated over nodes, pairs of nodes, or
triples of nodes in a semantic network: The number of connections per word, the length of the
shortest path between two words, and the percentage of a node’s neighbors that are themselves
neighbors. We show that semantic networks, like many other natural networks (Watts &
Strogatz, 1998), possess a small-world structure characterized by the combination of highly
clustered neighborhoods and a short average path length. Moreover, this small-world structure
seems to arise from a scale-free organization, also found in many other systems (Barabási &
Albert, 1999; Strogatz, 2001), in which a relatively small number of well-connected nodes
serve as hubs, and the distribution of node connectivities follows a power function.
These statistical principles of semantic network structure are quite general in scope. They
appear to hold for semantic network representations constructed in very different ways,
whether from the word associations of naive subjects (Nelson, McEvoy, & Schreiber, 1999) or
the considered analyses of linguists (Miller, 1995; Roget, 1911). At the same time, these regularities
do not hold for many popular models of semantic structure, including both hierarchical
or arbitrarily (unstructured) connected networks (Figures 1A and 1B), as well as high-dimensional
vector space models such as Latent Semantic Analysis (LSA; Landauer & Dumais,
M. Steyvers, J. B. Tenenbaum/Cognitive Science 29 (2005) 43
1997). These principles may thus suggest directions for new modeling approaches, or for extending
or revising existing models. Ultimately, they may help to determine which classes of
models most faithfully capture the structure of natural language semantics.
As in studies of scale-free or small-world structures in other physical, biological, or social
networks (Albert, Jeong, & Barabási, 2000; Barabási & Albert, 1999; Watts & Strogatz, 1998),
we will emphasize the implications of these distinctive structures for some of the crucial processes
that operate on semantic networks. We suggest that these structures may be consequences
of the developmental mechanisms by which connections between words or concepts
are formed—either in language evolution, language acquisition, or both. In particular, we show
how simple models of network growth can produce close quantitative fits to the statistics of semantic
networks derived from word association or linguistic databases, based only on plausible
abstract principles with no free numerical parameters.
In our model, a network acquires new concepts over time and connects each new concept to
a subset of the concepts within an existing neighborhood, with the probability of choosing a
particular neighborhood proportional to its size. This growth process can be viewed as a kind
of semantic differentiation, in which new concepts correspond to more specific variations on
existing concepts, and highly complex concepts (those with many connections) are more likely
to be differentiated than simpler ones. It naturally yields scale-free small-world networks, such
as the one shown in Fig. 1C (see also Fig. 6).
Our models also make predictions about the time course of semantic acquisition, because
the order in which meanings are acquired is crucial in determining their connectivity. Concepts
that enter the network early are expected to show higher connectivity. We verify this relation
experimentally with age-of-acquisition norms (Gilhooly & Logie, 1980; Morrison, Chappell,
& Ellis, 1997) and explain how it could account for some puzzling behavioral effects of age of
acquisition in lexical-decision and naming tasks, under plausible assumptions about search
mechanisms in semantic memory.
Our growing network models are