nucleation

LIT-963: Fundamental Limits of Perfect Concept Erasure

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-820) in full; worth reading for the clean information-theoretic cap on perfect concept erasure, I(Z;X) ≤ H(X|A) once Z is independent of A, and for the proof that a deterministic erasure reaches it only when the concept groups are permutations of one another. Read it knowing that it assumes finite supports and disjoint groups, so that the concept is a function of the representation, and that its claim that unequal groups cannot reach the cap is false for stochastic maps. Filed for QUESTION-010; the bound is held as THEORY-281.
Version2
Tagsinformation-theory, representation-learning, anthology-candidate
Cited as “paper” byNOTE-820 — Chowdhury et al. — Fundamental Limits of Perfect Concept Erasure
Cited as “source” byTHEORY-281 — Perfect concept erasure caps what a representation can keep at the information not about the concept: for X of finite support, if Z is independent of A then I(Z;X) is at most H(X|A), and a deterministic erasure reaches that cap only when the concept groups' distributions are permutations of one another

Graph View

Somnath Basu Roy Chowdhury, Avinava Dubey, Ahmad Beirami, Rahul Kidambi, Nicholas Monath, Amr Ahmed and Snigdha Chaturvedi (2025), arXiv:2503.20098 (AISTATS 2025).

Key takeaways

From the reading, NOTE-820.

  • The cap. If I(Z;A) = 0 then I(Z;X) = I(Z;X|A) ≤ H(X|A), with equality iff H(X|Z,A) = 0 (Lemma 3.3). It needs only A → X → Z.
  • When a deterministic map reaches it. Only when every concept group has the same distribution up to permutation; then a bijection of each group onto a common distribution does it (Definition 4.1, Method 4.2, Appendices A.3–A.4).
  • The assumptions are strong. Finite supports and disjoint groups make the concept a function of the representation. The cap is then H(X) − H(A).
  • One claim does not hold. For unequal groups the paper says the cap is unreachable and calls the optimal output distribution an open question. A stochastic map built on a common refinement of the groups’ distributions reaches the cap, so the impossibility holds only for deterministic maps.
  • The empirical part (PEF against INLP, LEACE, FaRM, KRaM and DCA) is ML practice and carries the anthology-candidate flag.

Standing in the record

Filed 2026-10-12 for QUESTION-010 and read 2026-10-14 in NOTE-820. Its bound is held as THEORY-281, Proposed, which carries a partial answers edge to QUESTION-010. That answer is one step: a chosen variable can be destroyed while everything about the representation not shared with it is kept, and no more. It stands beside THEORY-199 as the same shape of answer reached by design rather than by noise. It does not reach chains or relations. Its algorithm and benchmarks belong in the anthology. The bound is information theory and is held here.

Backlinks