and components and viewed the source signal as a superposition of sine waves. Our study of
channels will begin with this kind of analysis (often called Nyquist theory) to develop basic
results about sampling, intersymbol interference, and bandwidth.
Shannon’s view, however, was that if the recipient knows that a sine wave of a given frequency
is to be communicated, why not simply regenerate it at the output rather than send it over
a long distance? Or, if the recipient knows that a sine wave of unknown frequency is to be
communicated, why not simply send the frequency rather than the entire waveform?
The essence of Shannon’s viewpoint is that the set of possible source outputs, rather than any
particular output, is of primary interest. The reason is that the communication system must be
designed to communicate whichever one of these possible source outputs actually occurs. The
objective of the communication system then is to transform each possible source output into a
transmitted signal in such a way that these possible transmitted signals can be best distinguished
at the channel output. A probability measure is needed on this set of possible source outputs
to distinguish the typical from the atypical. This point of view drives the discussion of all
components of communication systems throughout this text.
1.2.1 Source coding
The source encoder in Figure 1.1 has the function of converting the input from its original
form into a sequence of bits. As discussed before, the major reasons for this almost universal
conversion to a bit sequence are as follows: inexpensive digital hardware, standardized interfaces,
layering, and the source/channel separation theorem.
The simplest source coding techniques apply to discrete sources and simply involve representing
each succesive source symbol by a sequence of binary digits. For example, letters from the 27
symbol English alphabet (including a space symbol) may be encoded into 5-bit blocks. Since
there are 32 distinct 5-bit blocks, each letter may be mapped into a distinct 5-bit block with
a few blocks left over for control or other symbols. Similarly, upper-case letters, lower-case
letters, and a great many special symbols may be converted into 8-bit blocks (“bytes”) using
the standard ASCII code.
Chapter 2 treats coding for discrete sources and generalizes the above techniques in many ways.
For example the input symbols might first be segmented into m-tuples, which are then mapped
into blocks of binary digits. More generally yet, the blocks of binary digits can be generalized
into variable-length sequences of binary digits. We shall find that any given discrete source,
characterized by its alphabet and probabilistic description, has a quantity called entropy associated
with it. Shannon showed that this source entropy is equal to the minimum number of
binary digits per source symbol required to map the source output into binary digits in such a
way that the source symbols may be retrieved from the encoded sequence.
Some discrete sources generate finite segments of symbols, such as email messages, that are
statistically unrelated to other finite segments that might be generated at other times. Other
discrete sources, such as the output from a digital sensor, generate a virtually unending sequence
of symbols with a given statistical characterization. The simpler models of Chapter 2 will
correspond to the latter type of source, but the discussion of universal source coding in Section
2.9 is sufficiently general to cover both types of sources, and virtually any other kind of source.
The most straightforward approach to analog source coding is called analog to digital (A/D)
conversion. The source waveform is first sampled at a sufficiently high rate