anthology-of-the-sota

SOTA-435: Never rank attribution methods by a single faithfulness or stability score, and report the perturbation distribution when you report infidelity

4 min read

Properties7
StatusActive
Version2
Consensusunreplicated
Tagsanalysis-and-evaluation
SourceLIT-725 — On the (In)fidelity and Sensitivity for Explanations
Introduced byLIT-725 — On the (In)fidelity and Sensitivity for Explanations
ExtendsSOTA-430 — Before using an attribution map to debug a model or explain what it learned, check that the map changes when the weights are randomized and when the labels are permuted — do not validate it by how it looks

Graph View

Source

Yeh, Hsieh, Suggala, Inouye and Ravikumar (2019), LIT-725 — NeurIPS 2019, §2–3.

Do this

  1. Report infidelity and sensitivity together, never either alone. They have opposite degenerate optima. Max-sensitivity,

    SENS_MAX(Φ, f, x, r) = max_{‖δ‖ ≤ r} ‖Φ(f, x + δ) − Φ(f, x)‖

    is minimised by a constant explanation — the paper’s words: “the optimal explanation that minimizes the above max-sensitivity measure is simply a constant explanation that just outputs a (potentially nonsensical) constant value for all possible test inputs.” An attribution that ignores the input scores perfectly. Infidelity has no such degenerate winner, but it is the one that moves with the perturbation choice, below.

  2. Name the perturbation distribution whenever you report an infidelity number. Infidelity is

    INFD(Φ, f, x) = E_{I ~ μ_I}[ (Iᵀ Φ(f, x) − (f(x) − f(x − I)))² ]

    and μ_I is yours to pick: difference-to-baseline, a subset of it, a noisy baseline, random square patches. Propositions 2.2–2.5 show that several published explanations are each the infidelity optimum under some μ_I, and the paper demonstrates it by giving SHAP the lowest infidelity when the perturbation is SHAP’s own. An infidelity number without its perturbation is not comparable to anybody else’s.

  3. Do not read a low sensitivity as a virtue on its own. The paper’s second objection is the substantive one: “natural explanations might have a certain amount of sensitivity by their very nature, either because the model is sensitive, or because the explanations themselves are constructed by measuring the sensitivities of the predictor function”. A faithful explanation of a sensitive model should move. What the paper recommends is restrained lowering, and it shows kernel smoothing does that: Theorem 4.1 bounds the smoothed explanation’s max-sensitivity by the kernel-average of the unsmoothed one’s, and empirically “Smooth-Grad improves both sensitivity and infidelity for all base explanations across all datasets”.

    The parameters, and how weak they are. SmoothGrad is LIT-731: average the gradient over n copies of the input perturbed with N(0, σ²), at 10–20% noise (σ/(x_max − x_min)) with n ≈ 50, past which “there was little apparent change”. Those numbers were chosen by looking at pictures — “since quantitative evaluation of a map remains an unsolved problem, we again focus on qualitative evaluation” — and the paper adds that “the ideal noise level depends on the input”. So treat them as defaults to start from and re-select them on whichever measure you are reporting. This step’s case for smoothing rests on the theorem and the measurement above, not on those figures, which is why the recommendation survives its own source being qualitative.

Why this is Active on one paper

Both cautions originate in LIT-725, the paper that defined infidelity and max-sensitivity: it states the constant-explanation minimiser itself, proves in Propositions 2.2–2.5 that several published methods are each infidelity-optimal for some perturbation, and shows SHAP winning under SHAP’s own perturbation. Turning those properties into “report the pair, and name μ_I” is this practice’s reading of them rather than a sentence the paper writes.

Most of it is not the kind of claim replication tests. The constant-explanation minimiser falls out of max-sensitivity’s definition, and the perturbation dependence is proved rather than measured. What it costs to follow is a sentence naming μ_I and a second column in a table. What it costs to ignore is a comparison no one else can reproduce, or a score won by saying nothing.

How this sits against what the record already holds

SOTA-430 says to run the randomization checks and, in step 3, to compare signed rank correlation against absolute or perceptual similarity because “a single metric can hand you either verdict”. This practice is the same warning one level down, about the faithfulness metrics rather than the similarity metrics, and from a different group.

LIT-724 is the document that makes it load-bearing here: it reports infidelity and max-sensitivity as a pair of trustworthiness measures across its whole randomization study, including the 2.84 → 1.27 × 10⁷ infidelity climb SOTA-430 now carries. It specifies its perturbations — N(0, 0.03) and x − x₀ — which is step 2 done correctly, and it reports max-sensitivity “almost unchanged” without the caveat in step 1.

Conditions

  • These are properties of two specific measures, not of explanation evaluation in general. A different faithfulness measure needs its own degenerate-optimum check; the transferable habit is to run that check, and the finding is that two measures in wide use fail it in opposite directions.
  • Vision models, 2019. MNIST, CIFAR-10, ImageNet. The definitions are modality-agnostic and the experiments are not.
  • The infidelity-optimal explanations the paper introduces are optimal by construction, so the table where they win is not evidence that infidelity is the right objective. The human study is the evidence offered for that, and it is 16 users on one synthetic construction.

Backlinks