nucleation

LIT-944: Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-771) in full; worth reading for its side-by-side comparison of networks that compute the same image in the same architecture: SGD reproduces three Picbreeder images without the evolved networks' reuse of symmetric parts or factoring of independent ones, and perturbing its weights breaks the regularities that perturbing theirs preserves. Read it knowing that it is a position paper, that the evidence is curated neuron maps and weight sweeps for three images with no measure of fracture, that the evolved networks were selected by users rather than trained on the image, and that its explanation by open-ended search and its extension to large models are conjecture. Held as THEORY-241.
Version2
Tagsrepresentation-learning, anthology-candidate
Cited as “paper” byNOTE-771 — Kumar, Clune, Lehman and Stanley — The Fractured Entangled Representation Hypothesis
Cited as “source” byTHEORY-241 — Fitting the same function in the same architecture does not fix the internal decomposition: SGD reproduces three Picbreeder images from random weights without the reuse and factoring of the evolved networks, and perturbing its weights breaks the regularities that perturbing the evolved networks' weights preserves

Graph View

Akarsh Kumar, Jeff Clune, Joel Lehman and Kenneth O. Stanley (2025), arXiv:2505.11581.

Key takeaways

From the reading, NOTE-771.

  • Same function, same architecture. A CPPN’s image is its whole input–output function on the grid. The evolved skull, butterfly and apple are rewritten exactly as dense MLPs, and SGD from random weights reproduces each image in that architecture. On the sparse evolved architecture itself SGD fails (Fig. 12).
  • Different internals. The evolved skull is symmetric from early layers, with 24 novel feature maps and separate neurons for its parts. The SGD skull is symmetric only at the output, with many high-frequency maps. The evolved apple separates background, body and stem; the SGD apple mixes them.
  • Different perturbations. Single-weight sweeps of the evolved networks open a mouth, wink an eye or swing a stem while keeping the image’s regularities. Those of the SGD networks distort it. The same holds in a PCA basis and along random weight directions (App. F), and for ReLU networks (App. E).
  • What is argued, not shown. That the order of discovery in open-ended search (symmetry, then a face, then a skull) is the cause. That large models have FER, on the strength of anecdotes about GPT-3, GPT-4o and cited interpretability work. That FER harms generalization, creativity and continual learning.

Standing in the record

Filed on 2026-10-12 from the owner’s papers-feed for QUESTION-060, and read on 2026-10-14 in NOTE-771. The comparison is held as THEORY-241, which answers QUESTION-060 in part: neither the objective nor the architecture fixes the internal variables when both are held fixed. The evolved networks were never trained on the objective, so this compares an objective-trained network with one found without a target. Its suggestions for training large models are machine-learning practice, for the anthology.

Backlinks