SOTA-430: Before using an attribution map to debug a model or explain what it learned, check that the map changes when the weights are randomized and when the labels are permuted — do not validate it by how it looks

Source

Adebayo, Gilmer, Muelly, Goodfellow, Hardt and Kim (2018; NeurIPS 2018), LIT-713 — ARXIV-1810.03292, read in full (v3, November 2020) as NOTE-365. The instruction is the paper’s own: “our tests can be thought of as sanity checks to perform before deploying a method in practice.”

Do this

On your model, with the attribution method you intend to use:

1. Model parameter randomization. Re-initialize the weights from the output layer downward, one block at a time, recomputing the maps at each step. Also re-initialize one layer at a time with the rest left trained. If the maps stay similar while the weights they are supposed to explain are destroyed, the method cannot support debugging this model.

2. Data randomization. Train the same architecture on permuted labels to high training accuracy, and compare its maps with the real model’s on the same inputs. If they stay similar, the method cannot tell you what the model learned about the input–label relationship.

3. Compare with signed rank correlation as well as absolute or perceptual similarity. The two disagree. On Integrated Gradients and gradient⊙input the absolute-value rank correlation and SSIM stay high after randomization, while the signed rank correlation drops to about zero at once. A single metric can hand you either verdict.

   Three groups have now found this, on three model families. LIT-713 on Inception and MNIST; LIT-724 on Inception and BERT, where SSIM calls global Integrated Gradients insensitive to randomization and Spearman calls it sensitive; and LIT-725 on ResNet-50, where signed rank correlation runs 0.10–0.18 across five explanations and the absolute-value correlation on the same explanations runs 0.57–0.62. This step is the best-supported thing in the practice.

4. If a method fails, re-run it without the input multiplier before concluding anything about the method. Integrated Gradients, InputXGradient, DeepLift and Gradient SHAP all multiply a model-dependent quantity by (x − x₀), which does not depend on the model at all. LIT-724 traced the failure to that factor: drop it and the maps become parameter-sensitive, with SSIM for the local variant about half the global variant’s on Inception. Their recommendation is to “compare their explanations with and without this multiplier in order to understand the magnitude of these structural effects”, and THEORY-113 is the account. This does not rehabilitate the method — a pass is still not a certificate — but it tells you whether you have learned something about the attribution or about the picture.

And the negative half, which carries its own evidence: do not accept a method because its maps look like the object. An untrained edge detector produces maps “strikingly similar” to several methods’ — and THEORY-115 now says why, because partial image recovery is what an edge detector approximates.

This half has a number, and it is older than the practice. Grad-CAM’s own paper, LIT-732, put four visualizations of the same image in front of Mechanical Turk workers and asked which of the image’s two annotated categories was being shown — 90 image-category pairs, 9 ratings each:

visualization	humans identified the right class
Guided Backpropagation	44.44%
Deconvolution	53.33%
Deconvolution Grad-CAM	60.37%
Guided Grad-CAM	61.23%

Guided backprop is last. The method this practice’s source picked as its headline failure — the one whose maps “look as convincing as ever” after the weights are destroyed — is also the one humans read the class off least often, and its own authors’ eventual gloss says so: Deconvolution is “more class-discriminative than Guided Backpropagation … although Guided Backpropagation is more aesthetically pleasing”. Those four numbers are in the paper’s arXiv v1, 7 October 2016, two years before LIT-713. The sentence reading them is in v4, 3 December 2019, fourteen months after LIT-713 made the point another way. So a measurement that the best-looking map was the least class-discriminative one sat published and unglossed for three years, by the people who took it. It is evidence for this half of the practice; it is not evidence that anybody acted on it, and it is a human-legibility result rather than a faithfulness one.

Guided Backprop’s maps survive randomizing the weights above the lowest layers, and that verdict has three groups behind it and does not rest on anyone’s figures: LIT-713 on Inception and MNIST, LIT-724 on Inception and BERT, LIT-729 on VGG-16 and ResNet-50, each by a similarity curve rather than by eye. Worth knowing while reading the first of them: its published Figure 2 for Guided Backprop is wrong, LIT-729’s authors confirmed the implementation bug, and LIT-713’s own footnote 5 withdrew the stronger “entirely invariant” claim. The narrowed claim above is the one all three support, and it is the one this practice has always stated.

What the source found, which tells you what to expect

method	weights randomized	labels permuted
Gradient, SmoothGrad	changes	changes
GradCAM	changes once the last conv layer is randomized	changes (disconnected patches)
Guided Backprop, Guided GradCAM	invariant to higher layers; changes only with the lowest	visible change, still covers the object
Integrated Gradients, gradient⊙input	structure persists; sign decorrelates	sign changes; input structure “clearly prevalent”

The paper names gradients and GradCAM as passing and Guided Backprop and Guided GradCAM as failing. It gives Integrated Gradients (LIT-712) no verdict, and neither does this practice.

Rows two and three are one construction, and the table does not show it. GradCAM and Guided GradCAM are both defined in LIT-732, and Guided GradCAM is GradCAM’s map — 14×14 at the last conv layer of VGG-16, upsampled bilinearly — multiplied element-wise by a guided backprop map. So the passing row and the failing row differ by one factor, and every pixel of fine structure a reader sees in a Guided GradCAM image comes from the failing one. LIT-713 states the construction too, in a clause: “for pixel level granularity GradCAM, can be combined with Guided Backpropagation through an element-wise product.” Neither paper draws the consequence, so nothing is filed as a theory for it — but it is why the two rows agree, and worth knowing before treating them as two results.

That also bounds what GradCAM’s pass buys you. It is a function of one layer’s activations and the gradients into them, at that layer’s resolution, and its own authors report the maps get “progressively worse as we move to earlier convolutional layers”. Passing the randomization test and telling you something fine-grained are not the same property, and here they trade against each other.

And “passing is not a certificate” now has figures for the method that passes. GradCAM’s agreement with the references its own paper chose: rank correlation 0.254 with occlusion maps over 2510 images, 0.136 with human attention maps on VQA — which the paper reports as “statistically higher than chance or random attention maps (zero correlation)”, the comparison being against zero rather than against a useful floor — and 30.0% pointing accuracy for caption words against COCO segmentations. A method can clear both randomization tests and still agree with every independent proxy at a rank correlation under 0.3. That is the practice’s own sentence, with the method it names as passing supplying the numbers.

The failing row’s methods are now held, and its rule is one sentence. Guided backpropagation (LIT-727, §4.2) zeroes the gradient at each ReLU wherever either the top gradient or the bottom activation is negative — both masks at once, where plain backprop applies the second and the deconvnet (LIT-728) applies the first. Its authors’ stated reason is to “prevent backward flow of negative gradients”.

That is the rule, not the explanation, and the explanation is still not held. The cluster’s failures partition into three mechanisms:

family	mechanism	held as
Integrated Gradients, gradient⊙input, DeepLIFT	the model-independent input multiplier	THEORY-113
DTD, LRP-α1β0, Excitation BP, PatternAttribution	rank-1 convergence of a non-negative chain	THEORY-114
Guided Backprop, Deconv, Guided GradCAM	partial image recovery — the map is an approximate reconstruction of the input, unrelated to the decision	THEORY-115

The middle row covers no method in this table. The bottom row is this table’s headline failure and is now explained: LIT-730 proves s_k^GBP(x) ≈ x in a random three-layer CNN, regardless of the class, so the map’s invariance to the weights is not a defect in an attribution — it is what an approximate image reconstruction looks like. That also explains the edge-detector comparison in this practice’s negative half. The 1912.09818 reading is what settled which is which: it measures guided backprop and explicitly hands the explanation to Nie et al., because applying a ReLU to the gradient makes the backward pass non-linear so its own theorem does not reach. v4 of this practice said 1912.09818 was the paper that argues it. That was wrong, written from LIT-725’s citation rather than from the paper.

It also rules out the intuitive story by name: “Other than argued in (Gu et al. 2018), the class insensitivity is not caused by missing ReLU masks and Pooling switches.”

What the evidence does not cover

• A pass is not a certificate. The tests are necessary conditions. The plain gradient passes both, and the paper does not claim it is faithful.

• Image classifiers only. Inception v3 on ImageNet, and a small CNN and MLP on MNIST and Fashion-MNIST. The tests are defined for any method and any model. That they discriminate on a transformer or on token attributions is unmeasured.

• No threshold. “Similar” is read from curves and figures. There is no significance test, and the number of inputs behind the main figures is not stated.

• The paper corrected itself once. An earlier version said Guided Backprop was entirely invariant. v3 withdraws that after a bug report (footnote 5). The abstract’s “independent both of the model and of the data generating process” is the pre-correction strength. Cite the body.

• The verdict does not transfer across modality, which is now measured rather than suspected. LIT-724 runs the parameter-randomization test on a BERT classifier fine-tuned on SST-2 and finds that global Integrated Gradients — which fails on images — becomes parameter-sensitive there, because token scores are summed over embedding dimensions and the multiplier’s structure does not survive the sum. So “run these tests” means run them in your modality: a verdict imported from an image classifier can be the wrong sign.

• A failure can be a fact about the similarity metric. Step 3 above says the metrics disagree, on this source’s own data. A second group reproduces the split on a different model: SSIM reports global IG insensitive to cascading randomization while Spearman rank correlation reports both variants sensitive, in the same experiment. That is the strongest corroboration any part of this practice has, and it is corroboration of the caveat rather than of the headline.

• Part of what the test moves is numerical, not explanatory. Randomizing weights makes the network less smooth, which makes gradients noisier and an integral-based attribution’s approximation worse. Infidelity on Inception goes from 2.84 at the trained model to 1.27 × 10⁷ partway down the randomization cascade. If you score a randomization test with a faithfulness metric rather than a similarity metric, some of the movement is quadrature error — and infidelity is a squared error between Iᵀ Φ and the function difference (LIT-725, Definition 2.1), so a degraded approximation raises it by construction. SOTA-435 is the hygiene that goes with reporting either measure.

• Whether the model-randomization test measures what it claims is disputed, and the practice stays Proposed for it. The dispute is now held rather than gestured at: see the consensus_note and LIT-724’s standing section. What neither group has done — and what the promote_when asks for — is run the tests where the true dependence of output on input is known by construction.

Where this does not apply

If you only want a map that reflects the architecture’s prior, or an image-processing view of the input, a method that fails these tests can still serve. The paper says so (§5.1). But then it is not an explanation of what the model learned, and it should not be presented as one.

Known implementations

• None recorded in the record. captum links the source; that is a reference, not an implementation of the tests.