nucleation

LIT-1168: A Bivariate Measure of Redundant Information

2 min read

Properties9
StatusActive — read in full 2026-10-17 (NOTE-944); worth reading as the paper that shows I_min is not a measure of shared information, since it gives two independent bits copied into the target 1 bit of redundancy, and that replaces it, for two sources, with I_red, a redundancy built from reverse information projections that satisfies Williams's three axioms plus an identity axiom, I_red((X,Y); X, Y) = I(X; Y). Read it knowing that the measure is bivariate only, that "mechanistic redundancy" lets independent sources share information (the AND gate keeps I_min's 0.311 bit), that the Imin-upper-bound pattern is observed in random distributions with exceptions left unexplained, and that its transfer-entropy example shows the two measures can split the same transfer entropy oppositely. Filed as THEORY-708.
Version2
Tagsinformation-theory, complex-systems
ExtendsLIT-1172 — Nonnegative Decomposition of Multivariate Information
Compared againstLIT-1176 — Generalized Measures of Information Transfer, LIT-1162 — Quantifying unique information, LIT-1203 — Intersection Information Based on Common Randomness
CorrectsLIT-1172 — Nonnegative Decomposition of Multivariate Information
RivalsLIT-1201 — Measuring multivariate redundant information with pointwise common change in surprisal
Cited as “paper” byNOTE-944 — Harder, Salge and Polani — A bivariate measure of redundant information
Cited as “source” byTHEORY-708 — The I_min redundancy credits two independent sources with shared information whenever each reduces surprise about the same target outcomes, so it gives two independent bits copied into the target 1 bit of redundancy; a bivariate redundancy built from reverse information projections, each source's information less what is lost when its posteriors are replaced by mixtures of the other's, satisfies symmetry, self-redundancy, monotonicity and the identity property, gives a nonnegative decomposition, and agrees with I_min on XOR, AND and their compositions but can reverse the state-independent/state-dependent split of a transfer entropy

Graph View

Malte Harder, Christoph Salge and Daniel Polani (2013), “A Bivariate Measure of Redundant Information”, Physical Review E 87: 012130 — ARXIV-1207.2080

Key takeaways

From the reading, NOTE-944.

  • What is wrong with I_min (§II C). For Z = (X, Y) with X, Y independent uniform bits, I_min(Z; X, Y) = 1 bit, though the sources share nothing. I_min takes the minimum of amounts per outcome, ignoring that X and Y move p(z) in different directions.
  • The identity axiom (Eq. 5). I∩((X_A1, X_A2); A1, A2) = I(X_A1; X_A2): when the mechanism copies its inputs, redundancy is the inputs’ mutual information.
  • I_red (Eqs. 6–13). Project each posterior p(z|x) onto the convex hull of {p(z|y)} by reverse KL. The projected information is I(Z; X) minus the average residual divergence. I_red is the smaller of the two directions. All four axioms are proved, and so is nonnegative synergy (Lemma 7).
  • Examples (Table I). Copy: I_red = I(X; Y), I_min = 1 at every correlation. XOR 0, AND 0.311, RdnXor 1, XorAnd 0.5, matching I_min. RdnUnqXor 1, where I_min gives 2. The AND result is “mechanistic redundancy”: independent sources sharing information because of the mechanism.
  • Transfer entropy (§IV D). For Williams and Beer’s coupled pair both measures give the same split. With a bystander process X added and T_{Z→(X,Y)} decomposed, at d = 1 I_red gives all of it as state-independent and I_min all as state-dependent.

Standing in the record

Filed on 2026-10-17 as the critique of I_min that the record had named as missing, and read the same day. Filed as THEORY-708. It changes no number in the transfer-entropy filing: the worked examples behind CLAIM-335 and THEORY-712 come out the same under I_red, as recomputed for this reading. It does bear on THEORY-581 and NOTE-020, whose synergy bias is computed with I_min, and supplies a second redundancy function with the identity property for NOTE-010’s worked disagreement. It carries no instruction for machine-learning practice.

The Griffith view it argues against is now held: Griffith and Koch (LIT-1204), whose final version concedes the point, and Griffith et al.’s intersection information (LIT-1203). Rauh et al. (LIT-1202) show why its identity axiom cannot be carried to three sources with nonnegative atoms.

Backlinks