SOTA-411: To let a model spend more compute at inference without more parameters, train a depth-recurrent core on a randomly sampled iteration count, re-injecting the input at every iteration

Source

Geiping et al. (2025), LIT-683.

What to do

• Shape: a prelude of a few layers, a core block iterated r times, and a coda. The source uses (2, 4, 2).
• Input re-injection: concatenate the prelude’s output with the state at every iteration and project it back to width.
• Random initial state: s₀ drawn from noise.
• Random depth in training: sample r per step from a log-normal Poisson with a heavy tail (mean 32 in the source). Backpropagate through only the last k iterations (k = 8). The prelude still gets gradient every step, through the re-injection.
• At inference, choose r per task, or exit per token when successive states stop changing.

Why

The model is trained to improve its state for as long as it is iterated. So extra test-time iterations are a direction it has learned to use, not an extrapolation. Training at a single depth does not give this: the same architecture run once through the core, or the recurrent model evaluated at r = 1, is far below the recurrent model at r = 32.

The recipe as recommended here — prelude, re-injected core and coda, noise initial state, heavy-tailed random r with truncated backpropagation — is Geiping et al.’s, LIT-683, and so is the evidence. In their one controlled comparison (180B tokens, same data) the single pass through the core scores 73.20 on SciQ and 37.34 on HellaSwag against 80.60 and 48.80 at r = 32, and zero-shot accuracy rises with r until a task-dependent saturation: HellaSwag is near its peak by 8 iterations, MMLU climbs from 23.39 at 4 to 31.38 at 32.

Conditions

• Per parameter, not per FLOP. The source shows accuracy per parameter. It has no standard transformer trained at matched compute. The same authors found, at small scale and equal wall-clock, that shared recurrent layers gave no gain over unshared ones (LIT-681). Use this where parameters or memory bind. Where FLOPs bind, the case is not made.
• Training was fragile. Two runs failed: one by representation collapse, one by the model learning to ignore its state. The one that worked used sandwich normalization and a much smaller learning rate. Expect to need similar measures, and do not assume the listed recipe elements are each necessary. None was ablated at scale.
• One 3.5B model on code- and math-heavy data, with a constant learning rate that was never decayed.

Known implementations

• Huginn-0125, the source’s released model.