Zeiler and Fergus (2013) — ARXIV-1311.2901
Key takeaways
- The deconvnet, and the switches that make it work. Given a high-level feature map, leave one neuron non-zero and invert the data flow back to image space. Max-pooling is not invertible, so the method “requires first to perform a forward pass of the network to compute ‘switches’ — positions of maxima within each pooling region”. Those switches are what condition the reconstruction on the image, and the paper is explicit that because of them the deconvnet “is hence conditioned on an image and does not directly visualize learned features”.
- Occlusion sensitivity. Slide a grey square over the input and watch two things: the classifier’s probability for the correct class, and the summed activity of the strongest feature map in the top conv layer. Both drop when the occluder covers the region the visualization highlights.
- And the paper uses the second to validate the first, in its own words: “This shows that the visualization genuinely corresponds to the image structure that stimulates that feature map, hence validating the other visualizations.” A perturbation that moves the model is offered as evidence that a visualization is about the model. Three test examples, qualitative.
- The visualizations are used diagnostically, not decoratively. They motivate architecture changes that beat Krizhevsky et al. on ImageNet: 14.8% test error with multiple models, against 16.4%, described as “almost half” the best non-convnet result (26.2%).
- An early depth-against-parameters ablation. Removing the fully connected layers 6 and 7 “only gives a slight increase in error … surprising, given that they contain the majority of model parameters”; removing two middle convolutional layers is also small; removing both gives a four-layer model that is “dramatically worse”. Widening the middle conv layers helps; widening them and the FC layers together overfits.
Standing in the anthology
Filed with LIT-727 as a pair, because guided backpropagation is defined in that paper as a variant of this one’s deconvnet, and the record already holds the verdict on both without holding either.
Two families begin here and the record leans on each.
| what | where the record depends on it |
|---|---|
| deconvnet → guided backprop | SOTA-430’s verdict table; its headline failure case is guided backprop’s maps surviving weight randomization |
| occlusion sensitivity | LIT-725 names it as the origin of perturbation-based attribution, and its infidelity measure is a perturbation-based fidelity criterion |
The second is the one a dependence count would miss. LIT-725 says it
plainly — “perturbation-based attributions measure the prediction difference
after perturbing a set of features. Zeiler & Fergus 2014 use such perturbations
with grey patch occlusions on CNNs” — and its “Square” perturbation, uniform
over square patches, is a grey square with a distribution over it. The record
acquired a perturbation-based metric yesterday and did not hold the
perturbation-based method it descends from.
The occlusion test became a metric, and the record now holds the paper that made it one. LIT-732 §5.3 takes this paper’s grey-square occlusion as the reference explanation — “one obvious choice for such a visualization is image occlusion” — grids it, takes the L-1 norm of a candidate map’s intensity in each cell, and rank-correlates the two over 2510 PASCAL VOC 2007 val images. That supplies one of the three things the limitations below say this paper’s occlusion validation lacked — a metric — and not the other two: there is still no threshold, and still no baseline for what an unfaithful map would score under the same test. It arrives in 2016 rather than 2018 and in the accepting direction: it is how Grad-CAM argues it is more faithful than guided backprop (0.254 against 0.168). The scores are low in absolute terms, which is a fact about all of them and the reason the missing baseline matters.
And Grad-CAM measured what this paper’s backward-pass rule costs. Its Table 3 substitutes the deconvnet’s ReLU rule into Grad-CAM’s own backward pass and reports 83.95 top-1 localization error against 59.65 with actual gradients — a 24-point loss, read by those authors as evidence that “negative gradients also carry important information for class-discriminativeness”. The same table finds guided backprop’s rule (LIT-727) roughly neutral for localization at 59.14 while costing class-discriminativeness. The deconvnet’s rule is the one that loses badly, and that ordering is the reverse of the human study in the same paper, where Deconvolution beat Guided Backprop 53.33% to 44.44% as a standalone visualization. Both results are that paper’s; they are different questions and the record holds both.
No practice is filed for occlusion sensitivity. The positive-validation
move it makes — perturb what the attribution highlights and check the output
moves — is worth having, and the record already has it in quantitative form
as LIT-725’s infidelity, under SOTA-435. Filing the 2013 qualitative
version separately would be filing on antecedence rather than on a defect,
which is the call LIT-722 got two units ago. What this note adds is the
lineage: the modern metric’s family has a named origin and a three-example
qualitative start.
The ablation is out of scope and worth a line anyway. “Most of the parameters are in the layers that matter least, and depth is what matters” is a 2013 result in a record with a large scaling cluster, and nothing here cites it. It is not this unit’s subject and is not a defect in anything held.
Limitations
- 2013, AlexNet-scale, ImageNet classification. The deconvnet is defined against a max-pooling architecture and needs it.
- The occlusion validation is three examples and a figure. No metric, no threshold, no baseline for what an unfaithful visualization would have looked like under the same test — which is the gap LIT-713 closes five years later, in the rejecting direction.
- The visualizations are qualitative throughout, and the paper’s own argument for them is that they led to a better architecture. That is evidence the process worked, not that the maps are faithful.