Henry W. Lin, Max Tegmark and David Rolnick (2016), Journal of Statistical Physics 168(6):1223–1247 (2017) — ARXIV-1608.08225, DOI-10.1007/s10955-017-1836-5
Key takeaways
- Cheap: a product costs four neurons. With any smooth σ having σ”(0) ≠ 0, [σ(u+v) + σ(−u−v) − σ(u−v) − σ(−u+v)]/4σ”(0) = uv(1 + O(u² + v²)); scaling inputs down by λ and the output up by λ² makes it exact as λ → ∞. So any polynomial is approximated to any accuracy by a network whose neuron count is fixed by the polynomial, though not its weights. For bits, a product of k bits is one sigmoid unit with a large gain.
- Why physics should make this enough (argued, not shown): physical Hamiltonians are polynomials of degree 2 to 4, local, and symmetric, and Gaussians and maximum-entropy distributions give low-order log probabilities. The authors concede that marginalizing or transforming such variables destroys these properties, and that weights of a trained network are no guide to whether the true Hamiltonian is simple.
- Deep: inference mirrors a Markov generative hierarchy. For a chain y₀ → y₁ → … → yₙ = x, the minimal sufficient statistic of P(yᵢ | x) is a function of that of P(yᵢ₊₁ | x), so P(y₀ | x) is a composition f₀ ∘ f₁ ∘ … ∘ fₙ(x) (Theorem 2 and Corollary 2). That each fᵢ is cheap, and that the composition cannot be cheaply flattened, is argued from examples, not proved for the composition.
- No-flattening. One hidden layer needs exactly 2^n neurons to multiply n inputs, where a deep binary tree of four-neuron gates needs about 4n (Appendix A; “approximate” means matching Taylor coefficients to degree n). For linear networks counted by nonzero weights: the FFT (O(n log n) against n²), fast matrix multiplication (a flattening cost of at least n^0.627), rank-k maps (n/2k) and products of random sparse matrices (up to 1/2p).
- Renormalization is feature extraction for a stated target. A coarse-graining keeps “relevant operators” only relative to what one wants to predict, usually long-wavelength quantities; RG “only makes sense if we specify what features we are interested in”. A Gaussian field toy (Eqs. 23–24) shows block averaging scaling the coefficient of the 2i-th derivative term by b^(2−2i). The journal text calls whether a procedure counts as renormalization “ultimately a matter of semantics”.
Standing in the record
Filed on 2026-10-09 at the owner’s request, as one of the works cited by the batch on hierarchy and hyperbolic geometry (nucleation#113 and #114, LIT-865 to LIT-877) that neither record held. The work in the batch that cites it is Mehta and Schwab’s mapping (LIT-882), whose reading (NOTE-683) named it as the third side of the triangle Koch-Janusz and Ringel (LIT-873) draw; LIT-873 cites it too. Read on its own merits (NOTE-688), with the dispute assessed there on all three texts and on Schwab and Mehta’s comment on v1 (LIT-891, arXiv only).
On the dispute, in short. The journal version does not engage LIT-882’s identity; it cites it once and speaks of uncited “misconceptions”. The arXiv v1 and v2 appendix did engage it, with a counterexample that refutes the “if and only if” of LIT-882’s Eq. 8, which Schwab and Mehta then called a typo, and with the sharper point that even their stricter trace condition is satisfied by hidden variables that never interact with the system. That point holds and produces THEORY-202; it extends THEORY-197, which says the correspondence has content only at the exact point, by showing the exact point does not select a coarse-graining either. The paper’s thesis that RG needs a specified target agrees with THEORY-194. Its conclusion that RG is supervised rather than unsupervised does not survive LIT-873, whose target is the far environment, taken from the data’s spatial structure rather than from labels. LIT-873’s and LIT-882’s readings describe it as denying any link between RG and deep learning; it denies a link to unsupervised distribution fitting and asserts one to supervised feature extraction and to depth.
On QUESTION-025. It bears only indirectly. Its hierarchy is a causal chain of generative steps, not attributes that imply one another, and there is no co-occurrence statistic or lattice. What it supplies is the general reason a top-level variable of a generative hierarchy is reached by composing level-wise statistics, not read off the data in one step, which fits Cagnetta and Wyart’s observation (LIT-883, THEORY-201) that a parent is not a linear feature of its input tuple until after a nonlinear layer; and its binary result that a conjunction of k bits is one threshold unit, which puts no bar on an attribute lattice’s meets being linear.
Its sequel by two of the authors, Rolnick and Tegmark’s The power of deeper networks for expressing natural functions (arXiv:1705.05502), is not held. Neither is Schwab and Mehta’s comment (arXiv:1609.03541), read for this filing.