nucleation

LIT-269: Attention Approximates Sparse Distributed Memory

1 min read

Properties6
StatusActive — read in full 2026-09-27 (NOTE-242); worth reading as the clearest statement of the SDM–attention correspondence. Its exact circle-intersection formula and its eight-algorithm convergence comparison are the parts to take away. Treat its analytic exponential derivation, its GPT-2 "confirmation" and its cerebellar implementation of attention as argument, not demonstration.
Version2
Tagsneuroscience, mathematics, information-retrieval, representation-learning
SupportsCLAIM-180 — Drift is retrieval failure outside a critical distance: a restatement close enough to an idea retrieves it, one that has drifted too far is taken for new, and a code bypasses retrieval by content altogether, CLAIM-298 — A frame can change a judgement by changing which of an interpreter's stored distinctions are reachable, without supplying any evidence about what is being judged
Cited as “paper” byNOTE-242 — Attention Approximates Sparse Distributed Memory
Cited as “source” byTHEORY-697 — Sparse distributed memory's read weights each stored pattern by the number of hard locations within the read radius of both query and pattern, a ball-intersection count that falls roughly exponentially in Hamming distance over a local window, so softmax attention over cosine similarity of unit-norm queries and keys, with a fitted inverse temperature, approximates the SDM read there; the approximation cannot hold globally, since the binary count vanishes beyond twice the radius, and its inverse temperature comes from a regression, not from the paper's analytic bound

Graph View

Bricken & Pehlevan (2021), NeurIPS 2021 — ARXIV-2111.05498

Standing in the record

Held in both records under ADR-013. The anthology holds it as ANTH-LIT-641, read for what it implies about attention in practice (the QK-norm retrodiction), with its account ANTH-THEORY-097.

It is filed here at the owner’s request of 2026-09-27, to be read for a question the anthology does not ask: the mathematics and neuroscience of associative memory. That means Kanerva’s content-addressable retrieval, its cerebellar mapping, and the SDM read as a kernel smoother. The reading sets it against the record’s retrieval-geometry and kernel threads (LIT-262, THEORY-004, THEORY-008).

It was filed Deferred, unread. NOTE-242 is the close reading of 2026-09-27, and it placed the work: Active — worth reading as the clearest statement of the SDM–attention correspondence. Its exact circle-intersection formula and its eight-algorithm convergence comparison are the parts to take away. Treat its analytic exponential derivation, its GPT-2 “confirmation” and its cerebellar implementation of attention as argument, not demonstration.

Backlinks