Figure 17.13 depicts a typical CC-NUMA organization. There are multiple independent nodes, each of which is, in effect, an SMP organization. Thus, each node contains multiple processors, each with its own L1 and L2 caches, plus main memory.
The node is the basic building block of the overall CC-NUMA organization. For
example, each Silicon Graphics Origin node includes two MIPS R10000 processors;
each Sequent NUMA-Q node includes four Pentium II processors. The nodes are
interconnected by means of some communications facility, which could be a switching mechanism, a ring, or some other networking facility.
Each node in the CC-NUMA system includes some main memory. From the
point of view of the processors, however, there is only a single addressable memory,
with each location having a unique system wide address. When a processor initiates
a memory access, if the requested memory location is not in that processor’s cache,
then the L2 cache initiates a fetch operation. If the desired line is in the local portion
of the main memory, the line is fetched across the local bus. If the desired line is in a
remote portion of the main memory, then an automatic request is sent out to fetch
that line across the interconnection network, deliver it to the local bus, and then
deliver it to the requesting cache on that bus. All of this activity is automatic and
transparent to the processor and its cache.
In this configuration, cache coherence is a central concern. Although implementations differ as to details, in general terms we can say that each node must
maintain some sort of directory that gives it an indication of the location of various
portions of memory and also cache status information. To see how this scheme
works, we give an example taken from [PFIS98]. Suppose that processor 3 on node
2 (P2-3) requests a memory location 798, which is in the memory of node 1. The following sequence occurs:
1. P2-3 issues a read request on the snoopy bus of node 2 for location 798.
2. The directory on node 2 sees the request and recognizes that the location is in
node 1.