either reducing the overlap of hidden unit representations, or
rehearsing old items as new items are learned (see Frean and
Robins [1997] for a brief review). Rehearsal based methods
are the most directly effective, with “broad but shallow”
rehearsal regimes (rehearsing many old items as each new
item is introduced but not necessarily training them to
criterion) producing particularly good results. Using such a
regime we can solve the catastrophic forgetting problem
shown in Fig. 1(b) and learn a function that fits both the old
population and the new item data points. The obvious
limitation of rehearsal based methods, however, is that they
require access to the population on which the network was
originally trained. This may be impractical [Sharkey and
Sharkey, 1995], particularly in the case of possible cognitive
mechanisms. In any case, retaining old items for rehearsal in
memory seems somewhat artificial, as it requires that they be
available on demand from some other source, which would
seem to make the memory itself redundant!