Source
Boncoraglio, Erba, Troiani, Xu, Krzakala & Zdeborová (2025), LIT-857, Appendix A, read in NOTE-661. The paper proves the identity. It is re-derived here as a check.
What was actually shown
- Tied weights. For W ∈ ℝ^(d×p), ‖W‖²_F = Tr(WWᵀ). WWᵀ is symmetric positive semi-definite, so its singular values are its eigenvalues and ‖WWᵀ‖* = Tr(WWᵀ). So λ‖W‖²_F = λ‖WWᵀ‖* exactly.
- Untied factors, the bound. For any U, V with UVᵀ = M, Hölder’s inequality for Schatten norms gives ‖M‖* = ‖UVᵀ‖* ≤ ‖U‖_F ‖V‖_F, and AM–GM gives ‖U‖_F ‖V‖_F ≤ (‖U‖²_F + ‖V‖²_F)/2.
- Untied factors, attained. Let M = PΣQᵀ be a singular value decomposition and take U = PΣ^(1/2), V = QΣ^(1/2). Then UVᵀ = M and ‖U‖²_F = ‖V‖²_F = Tr Σ = ‖M‖*. So min over UVᵀ = M of (‖U‖²_F + ‖V‖²_F)/2 is ‖M‖*.
- The consequence for training. If the loss depends on the weights only through the product, minimizing loss + λ(‖U‖²_F + ‖V‖²_F)/2 over (U, V) is minimizing loss + λ‖M‖_* over M. With tied weights the same holds for the PSD map S = WWᵀ. In both cases the product’s rank is at most the inner dimension p, so the nuclear-norm problem is over matrices of rank at most p.
What could have failed: a loss that depends on the factors other than through their product, such as a nonlinearity between them. Then the reduction does not apply.
What this does not say
- What the trained product looks like. The law η·ReLU(S0 + δZ − εI) for the learned map is THEORY-180’s. It is derived by replica and AMP arguments under a replicon condition checked only numerically, and it stays Proposed.
- Anything for losses that see the factors separately. Then weight decay is not a nuclear-norm penalty on anything.
- That gradient methods reach the minimizer. The identity concerns the objective, not the dynamics.