Springenberg, Dosovitskiy, Brox and Riedmiller (2014) — ARXIV-1412.6806
Key takeaways
-
Guided backpropagation, defined exactly, in §4.2. Three backward passes differ only in how they handle the ReLU:
method what the gradient is masked on plain backprop the sign of the bottom input (the forward activation) deconvnet (LIT-728) the sign of the top gradient, “ignoring the bottom input” guided backprop either being negative — both masks applied The paper’s own justification: “This prevents backward flow of negative gradients, corresponding to the neurons which decrease the activation of the higher layer unit we aim to visualize.” The name is for that: “it adds an additional guidance signal from the higher layers to usual backpropagation.”
-
The property that became its defect is reported here as its advantage. The deconvnet needs max-pooling “switches” recorded in a forward pass, and the paper’s architecture has no max-pooling, so it needed a version that works without them. It found one: “unlike the ‘deconvnet’, guided backpropagation works remarkably well without switches”, while “the ‘deconvnet’ approach fails completely in the absence of switches.” The explanation offered is the sentence to keep: “In a sense, the bottom-up signal in form of the pattern of bottom ReLU activations substitutes the switches.”
-
The architecture, which is the paper’s actual subject. Max-pooling can be replaced by a strided convolution “without loss in accuracy on several image recognition benchmarks”, and an all-convolutional net reaches 41.2% top-1 on ILSVRC-2012 centre-crop with under 10M parameters — comparable to the 40.7% of Krizhevsky et al. with six times fewer parameters, in about four days on one Titan. The paper is careful that this is not a full analysis: “a more thorough analysis is needed to precisely evaluate the effect of max-pooling on ImageNet-scale networks.”
Standing in the anthology
Filed with LIT-728, because guided backprop is defined here as a modification of that paper’s deconvnet and neither is readable alone.
This is the referent under SOTA-430’s headline example. That practice exists because LIT-713 found Guided Backprop’s maps on a network randomized above its lowest layers “visually and quantitatively similar” to the trained network’s, while looking as convincing as ever. Until now the record could state that verdict and not say what the method was.
Guided GradCAM is this method with an envelope on it. LIT-732 §3.2 defines Guided Grad-CAM as a Grad-CAM map — 14×14 at VGG-16’s last conv layer, upsampled bilinearly — multiplied element-wise by a guided backprop map. SOTA-430 puts guided backprop and Guided GradCAM in one row of its verdict table on the strength of LIT-713’s curves; the construction is why they belong there together, and every pixel of fine structure in a Guided Grad-CAM image is this method’s output. Replacing it with the deconvnet (LIT-728) “gives similar results”, with more artifacts.
And the same paper scores this method last on class discrimination. Four visualizations of the same image, 90 PASCAL VOC 2007 image-category pairs with two annotated categories each, 9 Mechanical Turk ratings apiece, asked which category is shown: Guided Backpropagation 44.44%, Deconvolution 53.33%, Deconvolution Grad-CAM 60.37%, Guided Grad-CAM 61.23%. The gloss the authors eventually added — in December 2019, three years after the measurement — is that Deconvolution is “more class-discriminative than Guided Backpropagation … although Guided Backpropagation is more aesthetically pleasing”. This paper’s contribution over the deconvnet was cleaner-looking maps, and the first study to ask whether humans could read a class off them put it last of four. Its own Table 3 separately finds this method’s ReLU rule roughly neutral for Grad-CAM’s localization (59.14 against 59.65) while costing class-discriminativeness — the same direction, in a different pipeline.
What the record can now state, and what it still cannot. It holds the masking rule (here), the measurement that the maps survive randomization (LIT-713), and LIT-725’s statement that guided backprop and guided GradCAM “do not pass randomization tests for different reasons” from the multiplier-carrying methods THEORY-113 covers. It does not hold the argument that joins them.
The temptation is to join them here, and the sentence above about the bottom ReLU pattern substituting for the switches makes it very tempting. It does not follow: bottom ReLU activations still depend on the weights, so “the structure comes from the input” is not what that sentence says. No theory is filed, and the switches story is not merely unproven — LIT-729 rules it out by name: “Other than argued in (Gu et al. 2018), the class insensitivity is not caused by missing ReLU masks and Pooling switches.”
The paper that supplies the account is Nie, Zhang and Patel (2018), A
Theoretical Explanation for Perplexing Behaviors of Backpropagation-based
Visualizations, held as LIT-730 with the account at
THEORY-115. It settles the question this note declined to answer,
in the direction the note declined to assert. Theorem 1 gives
s_k^GBP(x) ≈ x in a random three-layer CNN, so the objection raised above —
that bottom ReLU activations still depend on the weights — is removed by making
the weights i.i.d. Gaussian. The restraint was right about the argument;
Springenberg’s sentence still does not prove it. v1 of this note named Sixt, Granz and
Landgraf’s 1912.09818 instead, and that was wrong: it measures guided
backprop and explicitly hands the explanation on, because applying a ReLU to the
gradient makes the backward pass non-linear and its own theorem covers only
linear non-negative chains. See LIT-729’s standing section for how the
mistake was made.
The architecture half is out of scope and is not a gap. “Replace
max-pooling with a strided convolution” is a 2014 recommendation the field
adopted and then moved past; the record holds no practice it corrects, and
adoption is not evidence (ADR-017). It is recorded here because the
paper’s title is about it, and a reader who finds this note from SOTA-430
should not be surprised.
Limitations
- The visualization comparison is qualitative. Guided backprop against the deconvnet is judged on “a striking difference in image quality” across figures in the appendix, with no metric.
- The deconvnet is handicapped in that comparison and the paper says so. To get switches at all, max-pooling is inserted after training, replacing the stride — “one potential reason why the ‘deconvnet’ underperforms in this experiment is that max-pooling was only ‘artificially’ introduced after training.” A control on a model trained with max-pooling is offered, also qualitative.
- 2014, CIFAR and ImageNet classification, ReLU networks. The masking rule is specific to a rectifier; what it means in a network with smooth or signed activations is not addressed.