Google DeepMind (2025) — https://ai.google.dev/gemma/docs/gemma-3n
Key takeaways
- Per-Layer Embeddings. Besides the usual input embedding, each token id indexes a second table holding one small vector per layer. The public implementation uses 256 dimensions per layer, 35 layers and a 262,144-token vocabulary, which comes to roughly 2.35B parameters. At each layer the hidden state is projected down, passed through the activation, multiplied elementwise with that layer’s vector, projected back to model width, normalized and added to the residual. The lookup is keyed on the token alone, not on its context
- The point is where the parameters live. The table is addressed by token id, so which rows a sequence needs is known before any layer runs. That lets the table sit in CPU memory or fast storage and be fetched ahead. The documentation gives E2B as “over 5 billion parameters” with “an effective memory load of approximately 1.91 billion”. The developer guide says only about 2B (E2B) and 4B (E4B) “core transformer weights” need to be on the accelerator
- The quality claim is asserted, not measured. The developer guide says PLE “dramatically improves model quality without increasing the high-speed memory footprint”. Neither page gives a number, a baseline or an ablation
- Carried forward: the Gemma 4 technical report (ARXIV-2607.02770) says its E2B and E4B models “use per-layer embeddings as in Gemma 3n”, 2.3B and 4.5B effective out of 5B and 8B total
Standing in the anthology
Filed because a paper in the record cites it and the record could not follow the reference. LIT-152’s n-gram embedding section cites “Google DeepMind. Gemma 3n model overview, 2025” twice. First it names embedding-based memory as a way to add capacity. Second, and the more specific claim, it says deterministic addressing allows host-memory offloading and asynchronous prefetching.
It is not the source of the n-gram embedding. #163 called PLE the
source for Qwen3.8-Flash-Next’s n-gram embedding. The Qwen report is more
careful than that. It cites Gemma 3n among six sources for unigram lookup
memory, and a separate list for the n-gram generalization: N-Grammer,
Over-Tokenized Transformer, and Cheng et al.’s Conditional memory via
scalable lookup. The two designs also differ in shape. PLE reads a small
table at every layer, while Qwen reads one large n-gram table at one
layer. None of the n-gram papers is in the record yet.
Carries no practice, and could not. It has no measurements, so it is adoption and not evidence (DP-005). Its place in the record is on the consensus side of SOTA-326, as one of the groups that shipped embedding memory held off the accelerator.
It ships MatFormer (LIT-546) and adds per-layer embeddings beside it. The nesting is MatFormer’s: E4B contains a jointly trained E2B, the way a MatFormer’s larger FFN granularities contain its smaller ones, and the documentation cites the paper for it. What Gemma 3n adds is PLE and the off-accelerator placement of its table. As with PLE, Google published no ablation of the nesting at this scale, which is well above the 850M MatFormer itself tested.
Read — NOTE-293.