Contribution
Before this Letter, transfer entropy was routinely read as information flow, and its known weakness was the indirect path (X → Z → Y), which causation entropy fixes by conditioning on Z. After it, the reading fails for a different reason that conditioning cannot fix. Transfer entropy assigns to one source information that only the joint past carries. A measure built on conditional mutual information inherits the failure.
Key insight
Conditioning is not subtraction. I[X : Y | Z] can exceed I[X : Y]. In a partial information decomposition it equals X’s unique information plus the synergy of X and Z. So a transfer entropy is “how much better Y_t is predicted when X’s past is added to Y’s”, which is a fact about prediction. It is not “how much of Y_t came from X”.
Assumptions
- A definition of flow as localized information: “information that is currently in Y, the ‘cause’ of which can be solely attributed to X’s past” (p. 1). The critique is relative to this definition, which the authors align with Wiener and Granger’s intuitions.
- Stationary discrete processes, with transfer entropy over full pasts, T_{X→Y} = I[Y_t : X_{0:t} | Y_{0:t}] (Eq. 1).
- A PID exists with I[X₁ : Y] = R + U₁ and I[X₁ : Y | X₂] = U₁ + S (Discussion). No particular redundancy measure is needed for the examples, since every proposed one gives XOR pure synergy.
Key results
- Two series (Fig. 1). X_t, Y_0 i.i.d. fair bits, Y_t = X_{t−1} ⊕ Y_{t−1}. H[Y_t] = 1, I[Y_t : (X_{0:t}, Y_{0:t})] = 1, I[Y_t : Y_{0:t}] = 0, T_{X→Y} = 1, I[Y_t : X_{0:t}] = 0, I[Y_t : Y_{0:t} | X_{0:t}] = 1, co-information −1. Checked exactly: all six values; the I_min PID with target Y_t and sources X_{t−1}, Y_{t−1} gives R = U₁ = U₂ = 0, S = 1.
- Three series (Fig. 2). X, Y i.i.d. fair bits, Z_t = X_{t−1} ⊕ Y_{t−1}. T_{X→Z} = T_{Y→Z} = 0; I[Z_t : X_{0:t}] = I[Z_t : Y_{0:t}] = I[Z_t : Z_{0:t}] = 0; C_{X→Z|(Y,Z)} = C_{Y→Z|(X,Z)} = 1. Checked exactly: all seven.
- Diagnosis. T_{X→Y} = U(X past) + S(X past, Y past). “Only this unique information” is a candidate for flow, as Barrett (2015) and Williams and Beer (2011) proposed.
- Directed information (Massey) “is essentially a sum of transfer entropies and so inherits the same flaws”. Ay and Polani’s interventional flow and Granger causality are conjectured to fail for the same reason, conflating dyadic and polyadic relations.
- Networks. A transfer-entropy graph of the second example has no edges. A hypergraph is the suggested repair.
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | Transfer entropy can be positive with no information solely attributable to the source (overestimates flow) | strong | Example 1, exact; checked |
| C2 | Transfer entropy can be zero from each of two sources that jointly determine the target (underestimates influence) | strong | Example 2, exact; checked |
| C3 | Causation entropy fixes indirect paths but localizes joint information in the same way | strong | Example 2, exact; checked |
| C4 | Under PID, transfer entropy = unique + synergistic information | strong | from I[X₁:Y|X₂] = U₁ + S, which holds for any PID satisfying the consistency equations |
| C5 | Unique information, if it could be measured, may be a measure of flow | weak | proposal; “no broadly accepted quantification of unique information” [19] |
| C6 | Directed information inherits the flaw | moderate | ”essentially a sum of transfer entropies”; not shown here |
| C7 | Ay and Polani’s interventional flow and Granger causality fail for the same reason | conjecture | stated as a conjecture (p. 4) |
| C8 | Similar failures arise in Gaussian systems, so the issue is widespread | weak | cited to Barrett (2015); no Gaussian example given |
| C9 | Modelling complex systems as graphs with transfer-entropy edges is “intrinsically flawed” | interpretive | from C2 and the dyadic structure of graphs |
Concepts
- Information flow — information in Y_t solely attributable to X’s past, “localized” (p. 1).
- Influence — any dependence of Y_t on X’s past, localizable or not (implicit; the examples “make operational” the distinction).
- Causation entropy — I[Y_t : X_{0:t} | Y_{0:t}, Z_{0:t}], Sun and Bollt’s conditional transfer entropy (Eq. 2).
- Co-information — the central atom of a three-variable I-diagram, negative under synergy.
Connections
- Schreiber (2000, LIT-1177) introduced transfer entropy as a measure of information transfer. This is the critique of that reading, not of the quantity. The authors “do not object to the transfer entropy as a measure of the reduction in uncertainty about one time series given another”.
- Williams and Beer (2011, LIT-1176, NOTE-942). Example 1 is their d = 1 process, the controlled-NOT, with the roles renamed. They call its 1 bit “state-dependent transfer entropy”. This paper calls it synergy that is not flow. Both papers give the same decomposition; they disagree on what to call its synergistic part. The observation is mine.
- Barnett, Barrett and Seth (2009, LIT-1171) is cited for the identification of transfer entropy with Granger causality. With THEORY-641 that makes transfer entropy a predictive quantity, which this paper accepts. It denies the step from prediction to flow.
- CLAIM-333 concerns the step from Granger prediction to intervention, and this paper the step from prediction to localized flow. They are separate failures. In Example 1, intervening on X_{t−1} does change Y_t, so X has an interventional effect on Y. What fails is attributing the bit to X alone. My observation.
- THEORY-497 and NOTE-010 (ΦID). ΦID decomposes past→future information by source and target atoms, so it can separate what this paper says transfer entropy conflates. NOTE-799’s open question, a ΦID of transfer entropy, is the direction this paper points to. My connection.
- THEORY-568. The vulnerability numerator’s single-source terms I(X_j; Y | X_{−j}) are unique plus synergy in exactly this sense (NOTE-053’s connection makes T_{Y→X} one of them).
- THEORY-257 (dynamical independence) is defined as zero transfer entropy from micro to macro. By Example 2, zero transfer entropy from each micro variable does not mean the micro history carries nothing; Barnett and Seth condition on the whole micro history at once, which avoids that case. My observation, unchecked against NOTE-791’s details beyond its summary.
- THEORY-275 / NOTE-788 read transfer-entropy asymmetries in a Gaussian (Langevin) system as flow direction. This paper says such readings need the synergistic part separated; Barrett (2015) is cited for Gaussian cases. Not tested on that system.
Bearing on the record
- Filed as THEORY-707. It is the main ground for CLAIM-335.
- NOTE-788 and THEORY-275 speak of “information flow” and “backflow” between the colloids, the paper’s own words. By this Letter that names the direction of predictive improvement, not localized flow. Nothing there rests on the stronger reading, and THEORY-275 already disclaims coupling architecture. NOTE-122 relays Barrett et al. calling transfer entropy “only a proxy” for inter-regional transfer, which agrees.
- For the Anthology of the SOTA: no instruction for ML practice.
Limitations
- Both examples are deterministic XORs with uniform inputs, the case where every PID agrees. Away from it, how much of a transfer entropy is synergy depends on the redundancy measure, and the paper does not say which it would use.
- “Flow” is the authors’ definition. A reader who means predictive improvement by “transfer” is not refuted, only renamed.
- The Gaussian claim (C8) and the directed-information and Ay–Polani claims (C6, C7) are cited or conjectured.
- The ε-transducer cure is pointed to, with its informational atoms “in preparation”.
Open questions
- A measure of unique information that the field accepts, and whether it then behaves as flow should, for instance additively along chains.
- Whether interventional flow (Ay and Polani, LIT-1174) fails on Example 1. Intervening on X_{t−1} alone moves Y_t, so an interventional measure would credit X with the bit. Whether that is a failure depends on which definition of flow one keeps.