anthology-of-the-sota

THEORY-114: A modified-backpropagation rule that keeps only non-negative relevance produces a matrix chain that converges to rank 1, so the later layers can only flip the map's sign

4 min read

Properties5
StatusActive
Version2
Tagsanalysis-and-evaluation, representation-and-encoding
SourceLIT-729 — When Explanations Lie: Why Many Modified BP Attributions Fail
ExplainsSOTA-430 — Before using an attribution map to debug a model or explain what it learned, check that the map changes when the weights are randomized and when the labels are permuted — do not validate it by how it looks

Graph View

Source

Sixt, Granz and Landgraf (2019), LIT-729 — ICML 2020, §2 and §4.

The account

The theorem, its geometric proof, the cosine-similarity-convergence measurement and both DeepLIFT arms are all LIT-729’s. It proved the rank-1 result for the z⁺ family and traced the collapse layer by layer on a small CIFAR-10 net, VGG-16 and ResNet-50.

A modified-backpropagation attribution sets a relevance vector at the output layer — for a classifier, the explained logit — and propagates it down with a custom rule instead of the gradient. For the z⁺-rule, used by Deep Taylor Decomposition, LRP-α1β0 and Excitation BP, each layer’s backward operator is a non-negative matrix, so the whole path is

C_k = ∏_{l}^{k} Z⁺_l

Theorem 1: a product of non-negative matrices converges to a rank-1 matrix. The proof is geometric and short: the first matrix’s columns all lie in the positive quadrant, each further multiplication takes non-negative linear combinations of columns already inside that cone, and the cone shrinks with every factor until it is a single direction. The paper’s simulations find the convergence exponential.

The consequence is the finding. Write the converged chain as C = c γᵀ. Then for any relevance vector v,

C v = c γᵀ v = λ c,   λ ∈ ℝ

The direction is c whatever v is. The output relevance survives only as the scalar λ — so it can rescale the map and, if λ < 0, invert it, and nothing else. Which class you asked about, and what the later layers’ weights are, both enter only through v. That is why these methods give the same map for “cat” and “dog”, and why their maps do not change when the last layers are randomized: those are one mechanism, not two coincidences.

Why this is Active rather than Proposed

Because it is a theorem about matrix products, and because the paper does the thing this record keeps asking for — it includes arms the account says must behave differently, in both directions.

  • A published exception. DeepLIFT is the one tested modified-BP method that does not converge, and the account says why: its linear-layer rule separates positive and negative contributions and intermixes them, so the chain is not non-negative.
  • An exception built to order. The authors then construct DeepLIFT Ablation, removing the intermixing so that W⁺ drives the positive rule and W⁻ the negative. The chains decouple, both become non-negative — a product of two non-positive matrices is non-negative — and “as predicted by the theory, it converges.” A prediction made in advance about a variant that did not previously exist.
  • A measurement of the predicted quantity, not of a proxy. Cosine similarity convergence traces the collapse layer by layer: every analysed method except LRP z and DeepLIFT reaches at least 0.99 on VGG-16 and ResNet-50.
  • And the behavioural consequence separates the same two groups twice. On random logits the converging methods give “SSIM very close to 1” against 0.4 to 0.8 for the rest; under cascading parameter randomization, the same clustering.

What it does not cover

  • Not Guided Backprop, Deconv or RectGrad. They apply a ReLU to the gradient, so the backward pass is not a linear map at all and Theorem 1 does not apply. The paper says so and hands those to Nie, Zhang and Patel (2018), 1805.07039, now held as LIT-730. Their failure is a third mechanism — partial image recovery, THEORY-115 — and it is as far from this one as this one is from the multiplier: guided backprop recovers the input in a random CNN, where a converging chain would need trained weights to have collapsed at all.
  • Not the gradient family. Integrated Gradients and SmoothGrad “rely on the gradient directly … which does not converge”. Their randomization behaviour is THEORY-113’s subject, the model-independent input multiplier, and the two accounts do not overlap or compete.
  • Not a rival to the switches story — a refutation of it. “Other than argued in (Gu et al. 2018), the class insensitivity is not caused by missing ReLU masks and Pooling switches.” Worth recording because that is the intuitive explanation and it is wrong.
  • Not a claim that non-convergence means faithful. LRP-α5β4 converges less on VGG-16 and “also produces rather noisy saliency maps”; ROAR scores Integrated Gradients and Guided Backprop “equally bad, worse than a random baseline”, so ROAR “does not separate converging from non-converging methods”. Escaping this mechanism is necessary, not sufficient — the same shape as SOTA-430 being a rejection rule rather than a certificate.
  • The threshold is reported, not derived. “Sufficiently converged” carries the practical weight and 0.99 is where the measurements land, not a bound.

Backlinks