nucleation

LIT-764: A Mathematical Theory of Communication

2 min read

Properties5
StatusActive — read 2026-10-09 (NOTE-591); worth reading as the source of the measure and both coding theorems, in their first and plainest form: a source's entropy is the minimum rate it can be sent at (Theorem 9), a noisy channel's capacity C = max(H(x) − H_y(x)) is the maximum rate at which error can be made arbitrarily small, and errors do not force the rate to zero (Theorem 11, by averaging over random codes). Shannon says the axioms that single out −Σ p log p are not needed by the theory; the coding theorems are its justification. The capacity formula W log(1 + P/N) and the sampling representation are already here (Theorems 13, 17), as is the minimum rate for a source at a given fidelity (Theorem 21). Proofs are sketches at the typical-set level.
Tagsinformation-theory
SupportsCLAIM-050 — For report fidelity, fidelity is relative to the communicative decisions receivers make, and Blackwell's order on rendering procedures, over a designed distribution of situations, makes precise the ceiling a rendering sets on those decisions
Cited as “paper” byNOTE-591 — A Mathematical Theory of Communication
Cited as “source” byTHEORY-602 — An ergodic source's entropy is the least average rate at which it can be encoded, a noisy channel has a capacity below which any rate can be sent with arbitrarily small error and above which the equivocation cannot fall below the excess, and a source can be reproduced at a given fidelity over a channel exactly when its rate for that fidelity does not exceed the capacity; Shannon rests the justification of entropy on these theorems, not on the axioms that single out −Σ p log p

Graph View

C. E. Shannon (1948), Bell System Technical Journal 27(3):379–423 and 27(4):623–656 — DOI-10.1002/j.1538-7305.1948.tb01338.x

Key takeaways

  • Entropy as a rate (Part I). A discrete source is an ergodic Markov process; its entropy H = −Σ p log p per symbol is fixed by the fact that long sequences fall into about 2^{HN} typical ones carrying nearly all the probability (Theorems 3–4). A source can be sent over a noiseless channel of capacity C at C/H − ε symbols per second and not faster (Theorem 9). The three axioms that single out −Σ p log p (Theorem 2) are offered, Shannon says, for plausibility only; the coding theorems are the justification.
  • Capacity of a noisy channel (Part II). The rate of transmission is R = H(x) − H_y(x), where the equivocation H_y(x) is exactly what a correction channel would have to supply (Theorem 10); capacity is its maximum over input sources. Below C, a code with arbitrarily small error frequency exists, shown by averaging over random codes; above C the equivocation is at least H − C (Theorem 11). The 1% error example: 1000 bits per second carry about 919, not 990.
  • What it does not do. The proofs are sketches; no explicit code near capacity is given beyond Hamming’s seven-bit example, which Shannon connects to the difficulty of constructing near-random sequences. “Relative entropy” here means H over its maximum on the alphabet, not the Kullback–Leibler divergence.
  • Continuous parts (III–V, skimmed). Sampling at 1/(2W) (Theorem 13, proof referred to the 1949 paper, LIT-317), C = W log(1 + P/N) for white noise (Theorem 17), and a minimum rate for a source at a fidelity criterion (Theorems 21–22), the first form of rate-distortion theory, which the information bottleneck (LIT-338) inherits.

Standing in the record

Filed on 2026-10-09 at the owner’s request, as one of the works in the reference list of the owner’s working manuscript (October 2026) that the record did not yet hold. See the curation entry of that day.

It is the record’s primary source for entropy, equivocation and capacity. LIT-317 (Shannon 1949, Deferred) is credited in its summary with the sampling theorem and W log(1 + P/N); both are stated here first, and this paper defers only the sampling theorem’s proof to the 1949 one. The information bottleneck (LIT-338) starts from this paper’s exclusion of meaning and from Part V’s minimum rate at a fidelity.

Backlinks