Kokhlikyan, Miglani, Alsallakh, Martin and Reblitz-Richardson (2021) — ARXIV-2106.07475
Key takeaways
-
The failure has a named cause, and it is not the attribution method. Global Integrated Gradients multiplies the path integral by
(x − x₀), a factor that does not depend on the model at all. Drop it — the “local” variant — and on Inception under cascading parameter randomization the maps do change: SSIM for the local variant is “almost half” the global variant’s. LIT-713 suspected the multiplier and says so, but “do not measure input multiplier’s quantitative impact”. This is that measurement. The same reasoning covers InputXGradient, DeepLift and Gradient SHAP, which carry the same multiplier; Guided Backprop and Guided GradCAM fail for a different reason, modifying gradients during back-propagation. -
The verdict does not survive the change of modality. On a BERT classifier fine-tuned on SST-2, both local and global IG are sensitive to cascading randomization — cosine-similarity medians near zero for both. The mechanism offered is specific: token-level scores are obtained by summing over the embedding dimensions, embedding dimensions carry no structural pattern comparable to an image’s contours, and the positive and negative contributions of the multiplier cancel in that sum. Global IG fails the test on images and passes it on text.
-
SSIM and Spearman disagree about the same experiment, in this paper’s own data. Appendix A: “Spearman correlation gets close to zero after the first randomization attempt of the last layer and it remains that way… We observe that behaviour both for local and global explanations.” So on Inception, SSIM says global IG is insensitive to randomization and Spearman says it is sensitive. Two metrics, one experiment, opposite answers.
-
The two metrics it scores with are now held, and one of them needs a caveat this note reported without. LIT-725 defines both. Max-sensitivity is minimised by a constant explanation, so “max-sensitivity remains almost unchanged” under randomization is a weaker signal than it reads as — a measure an input-ignoring attribution wins cannot carry trustworthiness alone. Infidelity is defined relative to a perturbation distribution, which this note does specify (
N(0, 0.03), andx − x₀for global IG); that is what makes its figures comparable to each other and not to anyone else’s. SOTA-435 is the practice. -
The test has a confound that is not about explanation at all. Randomizing weights makes the network function less smooth, which makes the gradients noisier and the IG integral approximation worse. Infidelity does not drift, it explodes: Inception, averaged over ten samples, global infidelity goes 2.84 at the original model to 1.23 × 10³ at Mixed_7 and 1.27 × 10⁷ at Mixed_6. The authors note this “depends on the chosen randomization technique” — theirs is Xavier — and propose two remedies, more integration steps or a smoother network.
-
Smoothing the network improves the explanation metric. Replacing ReLU with Softplus and MaxPool with LogSumExp drops infidelity “significantly”.
-
What the authors actually recommend, and it is one sentence: “we recommend that users compare their explanations with and without this multiplier in order to understand the magnitude of these structural effects.”
Standing in the anthology
This is the paper three documents pointed at and called unread.
SOTA-430 named it twice — in its promote_when (“2106.07475 is the
first to read”) and in its consensus_note (“The record holds none of it and
has not read it”) — NOTE-365 named it in a takeaway and again in an open
question, and LIT-713 named it as “the first thing to read against this one”.
Five sites, one bare arXiv id, no document.
It lands on two of NOTE-365’s open questions and answers one of them:
NOTE-365 asked | this paper |
|---|---|
| ”Do the verdicts hold for transformers and for token attributions?” | No. BERT on SST-2, token attributions: global IG becomes parameter-sensitive. |
| ”Can a threshold be set so that passes is a statistic rather than a judgement?” | Not answered — but it shows why the threshold has to name its metric, since SSIM and Spearman split. |
SOTA-430’s third step already told a reader to compare signed rank
correlation against absolute or perceptual similarity because “a single metric
can hand you either verdict”, on LIT-713’s own data. A second group, a
different model and a different question reproduces exactly that split,
which is the strongest corroboration any part of that practice has.
It also settles the practice’s consensus, which was unassessed because
nobody had looked. Somebody has. The trunk is agreed — both groups run the
tests and neither validates a map by eye — and the branch is not: LIT-713
reads a failure as evidence that gradient methods are “inadequate tools for
model explanation”, and this paper reads the same failure as an artefact of a
model-independent factor that you can simply remove.
The input-multiplier finding is filed as THEORY-113, and the practice gains a fourth step from it: if a method fails, re-run it without the multiplier before concluding anything about the method — which is the authors’ own recommendation, in their own sentence.
No practice is filed for the smoothing result. Replacing ReLU with Softplus and MaxPool with LSE to get better infidelity scores is a recommendation to change the model in order to explain it, evidenced by one figure and one metric on one architecture. Whether the explanation then describes the model you ship is exactly the question, and the paper does not ask it.
Limitations
- One architecture per modality: Inception for images, one BERT classifier on SST-2 for text. The headline curves are single-sample, with ten-sample averages in the appendices to show the shape is not an accident.
- No thresholds, the same gap
NOTE-365records forLIT-713. Verdicts are read from boxplots. - It is the captum maintainers’ own paper. captum implements Integrated Gradients, and the finding is that IG’s most-cited failure is attributable to a factor its API lets you switch off. That does not make the measurement wrong — the local-versus-global arm is a real control — but “captum links this critique” is not third-party assessment, and the record should not read it as such.
- The smoothness confound is diagnosed, not separated. Infidelity rising by seven orders of magnitude under randomization is attributed to integral approximation error; no arm holds approximation quality fixed while varying the weights.