nucleation

LIT-976: Emergence of Quantised Representations Isolated to Anisotropic Functions

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-759) in full; worth reading as a clean ablation in which the activation function is the only part of an autoencoder that singles out a basis: elementwise tanh (B_n) and Leaky-ReLU (S_n) drive latent representations into narrow beams arranged by their group, even in layers no activation precedes, while radial O(n)-equivariant versions leave them spread. Read it knowing that the evidence is density plots from small CIFAR and MNIST autoencoders, five seeds a condition, with no statistic, and that its claims about grandmother neurons, superposition, neural collapse and better reconstruction are conjecture or preliminary. Held as THEORY-288.
Version2
Tagsrepresentation-learning, anthology-candidate
Cited as “paper” byNOTE-759 — Bird — Emergence of Quantised Representations Isolated to Anisotropic Functions
Cited as “source” byTHEORY-288 — In small CIFAR and MNIST autoencoders whose only basis-privileging primitive is the activation function, elementwise tanh and Leaky-ReLU concentrate latent representations into narrow beams arranged by the activation's discrete symmetry group, even in layers no activation precedes, while their orthogonally equivariant radial counterparts leave them spread, so axis-aligned discreteness there comes from the primitive, not the task

Graph View

George Bird (2025), arXiv:2507.12070.

Key takeaways

From the reading, NOTE-759.

  • The ablation. Autoencoders of width 12–32 with up to three hidden blocks are trained by momentum SGD on MSE. Everything but the activation commutes with rotations. Elementwise tanh, Σᵢ tanh(x·êᵢ)êᵢ, is set against tanh(‖x‖)x̂, and elementwise Leaky-ReLU against a radial one. Two signed-permutation Leaky-ReLUs are added.
  • The result. Elementwise activations give narrow beams.
    • Tanh’s lie along and against the axes.
    • Permutation-only Leaky-ReLU’s lie on the ±1 diagonal.
    • Signed-permutation Leaky-ReLU’s lie on the axes, like tanh’s.
    • About 2,000 samples pass the plotting threshold early, against about 200,000 late in training.
    • Radial activations leave the representations spread.
    • The effect is seen at the first latent layer, which no activation precedes.
  • The method. Representations are projected onto all planes spanned by pairs of basis vectors, within an angular threshold. App. A shows and partly controls its artefacts along the measured basis.
  • The mechanism offered. A first-order step through an elementwise activation rescales each gradient component by ψ′, orthant by orthant. Through a radial one it is a multiple of the gradient plus a radial term (App. C.2). This is description, not derivation.
  • What is conjecture. That this causes grandmother neurons, discrete superposition or neural collapse. That isotropic networks reconstruct better is preliminary (App. D) and mixed without input standardisation.

Standing in the record

Filed on 2026-10-12 from the owner’s papers-feed for QUESTION-058, and read on 2026-10-14 in NOTE-759. The ablation is held as THEORY-288, which answers QUESTION-058 in part: an activation function is one architectural source of a group’s structure in a representation, with no counterpart in the task. It complements THEORY-184, on what a permutation-equivariant update preserves. Its prescriptions for choosing activation functions are machine-learning practice, for the anthology.

Backlinks