nucleation

LIT-793: Semantic Rate-Distortion Theory with Applications

3 min read

Properties4
StatusActive — read 2026-10-09 (NOTE-638); worth reading as the formal version of "preserve the distribution over interpretations, not one meaning": the semantic distortion compares the posterior of the meaning S given the original observation with its posterior given the reconstruction, alongside an ordinary distortion on the symbols, and the paper proves the coding theorem for the resulting rate–distortion function and solves a binary case. Read it knowing three things the paper does not say. With Kullback–Leibler divergence its expected semantic distortion is exactly I(S;X) − I(S;Y), so its function is the information bottleneck; with total variation it bounds the extra Bayes risk of every bounded-loss decision about S. Its sequence distortion is defined as a maximum inside the expectation, while both proofs use the maximum of per-letter expectations, so the theorems hold for the latter. And its MNIST experiment trains with a classifier's cross-entropy against the true label, not with the posterior distortion the theory defines.
Tagsinformation-theory
Cited as “paper” byNOTE-638 — Semantic Rate-Distortion Theory with Applications
Cited as “source” byTHEORY-161 — In Zhao et al.'s (2025) and Chai et al.'s (2023) semantic rate–distortion frameworks, meaning is a latent variable with a known joint law with the observation, so their limits are indirect source-coding limits; Zhao et al.'s posterior-matching semantic distortion is, with KL divergence, exactly the information bottleneck, and with total variation a bound on the expected lost value of every bounded-loss decision

Graph View

Yi-Qun Zhao, Zhi-Ming Ma, Geoffrey Ye Li, Shuai Yuan, Tong Ye and Chuan Zhou (2025), arXiv preprint — ARXIV-2509.10061

Key takeaways

  • The model (Section 2). A message is a pair (s, x): s the intrinsic meaning, relative to a task known to both ends, x the observable string. Coding acts on x only; the decoder outputs a reconstructed observation y. Since one string can support several readings (“orange”, “several days”), what should survive is the conditional distribution p(S|x), not a single most probable meaning.
  • The distortion (Definitions 1–8). Semantic distortion d_p(p_S|x, p_S|y), for any divergence (total variation and KL are named), where p_S|y is the posterior of S given the reconstruction under the joint law the code induces (S–X–Y Markov); and an ordinary symbolic distortion d_o(x, y). R^I(D_p, D_o) = min I(X; Y) subject to both expected distortions.
  • The theorems (Section 3). The information function is achievable, by a Poisson functional representation code with common randomness (Theorem 1), and is the operational limit when it is lower semicontinuous (Theorem 2), which holds for finite alphabets and bounded or f-divergence distortions (Proposition 2).
  • Binary case (Theorem 3). For S, X, Y binary, X a symmetric channel of S with crossover q, d_p = TV, d_o = Hamming: R = 1 − h₂((1 − √(1 − 2D_p/C))/2) when the semantic constraint binds (D_p ≤ a(D_o)), and the classical 1 − h₂(min(D_o, ½)) otherwise, with C = |1 − 2q|. Whichever constraint is tighter sets the rate.
  • Illustration (Section 5). MNIST autoencoders trained on MSE plus γ times a classifier’s cross-entropy reach far higher digit accuracy at the same bit budget (at 4 bits, 95% with γ = 0.1 against 15% with γ = 0).

Standing in the record

Filed on 2026-10-09 from the manuscript bibliography of 2026-10-09 (work what-survives-translation) at the owner’s request. It is the “Zhao et al., 2025” the exchange named as close to its interest in “preserving distributions over interpretations rather than one purported intrinsic meaning” (CLAIM-088); the manuscript did not keep it. It is read here on its own merits. See the curation entry of that day.

The reading confirms the exchange’s description: the constraint is on posterior distributions over meanings, which is the preservation of interpretations the manuscript wanted. It also finds that the KL version of that constraint is the information bottleneck (LIT-338), so the idea is older than this paper, and that the total-variation version is a decision-relative fidelity in the sense of the record’s account of comparing experiments (THEORY-156). Both are stated, with their scope, in THEORY-161, sourced here. It cites Chai et al. (LIT-812) among the rate–distortion–perception works it builds on.

Backlinks