LIT-641: Attention Approximates Sparse Distributed Memory

Bricken and Pehlevan (2021) — ARXIV-2111.05498

Key takeaways

• The correspondence, and the two conditions it needs. SDM’s read maps onto attention’s ξ^new = P_p · softmax(β P_aᵀ ξ) with keys as pattern addresses, values as pointers and the query as the read address — given that (i) attention L²-normalizes its vectors and (ii) β is chosen to approximate the near-exponential decay of SDM’s circle-intersection weighting. Neither is free; both are assumptions about the architecture, not consequences of it.

• The retrodiction is what makes it more than an analogy. “In requiring that Attention vectors be L² normalized and β fitted, SDM predicted Query-Key Norm.” Those are exactly the two things LIT-640 does — ℓ₂-normalize queries and keys, replace 1/√d with a learnable scalar — published a year earlier, motivated by softmax saturation in low-resource translation, with no reference to associative memory.

• The measurement. Trained Query-Key Norm heads learn β ∈ [10, 25] (obtained from the QK-norm authors in private correspondence). SDM has three different optimality criteria giving three different β — critical distance, signal-to-noise ratio, and memory capacity — and the learned range interpolates between them. GPT-2 is checked too, with “effective” β inferred from query-key dot product magnitudes, “more approximate but… largely in agreement”.

• The authors state the weakness themselves, which is why the theory filed from this is Proposed: “Attention learns useful pattern representations that are far from random so this SDM β that fits the optimal d*s are only a weak reference for what β values might be reasonable.” The band [10, 25] is wide, the three criteria it interpolates between span a range, and nothing here is a point prediction that could have failed.

• The mapping is used to read other components, which is the part that will age either well or badly: §4 interprets LayerNorm, the feed-forward block and the residual stream through the SDM frame. Filed here as recorded rather than endorsed — the record already has THEORY-081 reading feed-forward layers as key-value memories from direct measurement, which is a different and better-evidenced route to an adjacent claim.

Standing in the anthology

Unit 5 of #326, and the issue ranked it fifth and called it the weakest reversal — “a THEORY candidate, not a substrate defect, and an unadopted account is still an account nobody depends on.” That assessment holds for the theory and missed what reading it would surface.

The record has five theories about what self-attention does — clustering (THEORY-065), collapse (THEORY-066), entropy and spectral bounds (THEORY-061, THEORY-062), circuit formation (THEORY-028) — and none about attention as associative memory. THEORY-081 makes the memory claim for the feed-forward block. This is the attention-side counterpart, and it is the weaker of the two.

What it also did was send the reading to QK-norm, whose origin the record did not hold while naming the technique in thirty documents and recommending it in an Active practice sourced to an adopter. The weak paper was worth reading for what it cited, which is not a reason anyone can plan around.

compared_against: LIT-640 — the β comparison is a real measurement against that paper’s trained models, which is what ADR-011 asks of the relation.

The comparison itself: the β ∈ [10, 25] that LIT-640’s trained QKNorm heads learned, obtained from its authors, is set against the three SDM-optimal values and falls between them. LIT-640 never mentions associative memory, so the agreement is a retrodiction rather than a shared design.

What it builds on is Kanerva’s. From the book, LIT-666, it takes the memory itself — patterns written to and read from every neuron within a Hamming radius, so that a read weights each pattern by the intersection of two hyperspheres — and the critical-distance radius, reproduced in its Appendix B.5 as the book’s Figure 7.3. From Kanerva’s review, LIT-669 (which its §1 cites, as the 1993 chapter, for all three radii), it takes the other two optimal radii β is fitted against, the signal-to-noise optimum and the memory-capacity analysis, and the more accurate variance its Appendix B.4 uses in place of the book’s. What it adds is the mapping of that read onto softmax attention, with L²-normalized vectors and a fitted β as the two conditions.