nucleation

LIT-1166: Causality, Feedback and Directed Information

2 min read

Properties4
StatusActive — read in full 2026-10-17 (NOTE-939); worth reading as the short source of directed information, I(X^N→Y^N) = Σ I(X^n; Y_n | Y^{n−1}), and of the point that the usual definition of a memoryless channel silently forbids feedback because it confuses statistical with causal dependence. Read it knowing that "causal" here is a factorization condition on a system of sources and channels, not an interventional notion; that directed information conditions on the current input as well as the past, so it is not transfer entropy; and that the copy read is the author's preprint, not the proceedings version. Filed as THEORY-711.
Tagsinformation-theory, causality
Cited as “paper” byNOTE-939 — Massey — Causality, feedback and directed information
Cited as “source” byTHEORY-711 — Directed information from an input sequence to an output sequence, the sum over time of the information the inputs up to now carry about the current output given the past outputs, is at most their mutual information and equal to it when the channel is used without feedback; in a causal system it bounds what the output reveals about the source, so feedback cannot raise a memoryless channel's capacity; and the usual definition of a memoryless channel holds only without feedback

Graph View

James L. Massey (1990), “Causality, Feedback and Directed Information”, Proceedings of the 1990 International Symposium on Information Theory and its Applications (ISITA-90), Waikiki, Hawaii — https://isiweb.ee.ethz.ch/papers/arch/mass-1990-1.pdf

Key takeaways

From the reading, NOTE-939.

  • The DMC without feedback (Eqs. 1–5). A discrete channel is memoryless if P(y_n|x^n y^{n−1}) = P(y_n|x_n). It is used without feedback if P(x_n|x^{n−1} y^{n−1}) = P(x_n|x^{n−1}). Only both together give the textbook P(y^N|x^N) = Π P(y_n|x_n), which therefore excludes feedback.
  • Directed information (Eq. 7). I(X^N→Y^N) = Σ_n I(X^n; Y_n | Y^{n−1}), a slight modification of Marko’s 1973 definition. In general I(X^N→Y^N) ≠ I(Y^N→X
  • Theorem 1. I(X^N→Y^N) ≤ I(X^N;Y^N), with equality if the channel is used without feedback.
  • Theorem 3. In a causal system, where a channel sees a source only through its own inputs and outputs (Eq. 8), I(U^K;Y^N) ≤ I(X^N→Y^N). With Theorem 2 this gives the capacity of a DMC unchanged by feedback.
  • The lesson. “Probabilistic dependence is quite distinct from causal dependence.” Mutual information is symmetric; directed information is not.

Standing in the record

Filed on 2026-10-17 and read the same day. Filed as THEORY-711. Directed information is the full-history, conditioned-on-the-present relative of transfer entropy (TERM-057). By the chain rule it splits into a sum of full-history transfer entropies and a sum of instantaneous terms, which NOTE-939 checks; that decomposition is the record’s, not Massey’s. causality is for its subject. Massey’s “causal” is a factorization condition, not intervention.

Backlinks