ADR-031: A scheme for explanations, complementing the practice registry

Context

This record has two schemes because a paper’s standing and a recommendation’s standing are different facts that were being stored in one field, and ADR-002 is the decision that separated them. A third fact has been leaking in the same way, and it is the one a reader of a practice registry most often wants: why does this work?

The leak is visible from both ends.

From the practice registry. Eight of 208 practices state a behaviour rather than an instruction, which is what the scaffold asks for and what the template’s first line says a practice is:

SOTA-010	skip connections promote training stability by smoothing out the loss landscape
SOTA-012	sharpness in the loss landscape correlates with test error
SOTA-037	LM in-context learning emerges at scale
SOTA-038	ICL permits few-shot task adaptability
SOTA-040	larger models are more sample efficient
SOTA-064	Warmup needed scales sub-linearly with model size
SOTA-092	smaller batch sizes are more sample efficient
SOTA-097	Optimal batch size scales approximately with compute budget

None of these is a mistake and none should be deleted. What they have in common is that their bodies end up arguing for their own presence: SOTA-037’s last paragraph reaches a negative consequence and then says outright that it “is the reason this belongs in a registry of practice.” A document that has to justify its filing is usually filed wrong.

From the reading list. 48 of 225 notes are cited by no practice — and in analysis-and-evaluation, the topic whose own blurb says “theory, interpretability and debugging belong here too”, it is half of them, 6 of 12. Two say so in their own words. LIT-019 and LIT-039 each carry a section headed “Carries no practice, deliberately”, and LIT-019’s explains why: “the procedure it describes costs a full dense training run before it produces anything, so as an instruction it reads train the model, then train a smaller one — which is a finding about neural networks rather than advice about how to train one.”

The sharpest case is LIT-223. It was filed weeks ago to correct a conflation, its own body says “the practices that turn on the mechanism — why BN permits what it permits — belong here”, and nothing in the record cites it. Nothing can. What it says is that batch normalization does not work for the reason it was published with, and that is not an instruction; it does not change what anyone should do.

That last case is also the one this scheme is shaped around, because it needs three statuses at once and there were only two places to put them. The technique is standard. The paper that introduced it is Active. The explanation in that paper’s title is wrong. Under two schemes, the third fact had to be either dropped or written into a practice body, where it reads as a practice arguing against itself.

Decision

A third scheme, THEORY, holding one document per claim about why something happens. A practice is an instruction; a theory is a finding.

Its relationship to LIT is exactly the one SOTA has, for exactly the reason ADR-002 gives: a note is a paper’s standing, a claim is a claim, and a claim outlives the paper that made it. Internal covariate shift was believed by far more papers than the one that proposed it, and was refuted by one that proposed nothing.

• source: is required and is a list, primary first, on the same rule and for the same reason as a practice’s: an explanation with no paper behind it is a hunch. published: derives from source[0].
• explains: names the practices the account underwrites, and luria link --fix writes explained_by: back onto each of them — a crossing converse, which luria has permitted since LU-ADR-097. It is optional, and the seed set is why: the two lottery-ticket documents explain nothing, because the record holds no pruning practice for them to explain, and requiring the field would have left them exactly where they are now.
• extends: and corrects: run between theories, with the distinction ADR-017 draws elsewhere: a refinement of an account that holds, versus a document that exists because an account breaks. Both edges in the seed set are corrections.
• Its own status vocabulary, theory-statuses, in which Rejected means disbelieved — “and the thing it explained may still work perfectly well.”
• The same thirteen topics, by ADR-026’s rule and with ADR-026’s mechanism: the words are one table in vocabularies.topics and only the filing rule is repeated per scheme. Copying the glosses a third time is how ten of thirteen drifted the last time.
• No introduced_by:. ADR-030 required it on practices because a recommendation and the evidence for it are separate acts, so the origin can differ from source[0] and an absent field was ambiguous. A theory document’s primary source is the work that states the account; there is no second act for the field to name. If a case turns up — an account proposed in one paper and evidenced in another — that is when to add it.

Four documents are filed with this decision: the batch-normalization pair (THEORY-003 correcting THEORY-001), and the lottery-ticket pair (THEORY-004 correcting THEORY-002). They are not a migration. They are the four claims the record was already carrying in prose with nowhere to put them.

Alternatives considered

• A status or a tag on SOTA. The cheapest move, and the one that breaks first. Rejected on a practice means do not do this; on an explanation it means this reason is wrong, and batch normalization needs both words on the same day pointing at different documents. A vocabulary whose words mean two things depending on a tag is the failure ADR-002 already fixed once.
• Leave explanations in the LIT note’s takeaways. This is where they are today, and it is why nothing cites them: a note is a paper’s standing, so an explanation stored there can only be referenced by referencing the paper, and a claim held by thirty papers has thirty homes and no address. It also cannot be given a status of its own without giving the paper one it does not deserve — LIT-002 is worth reading, and its title is wrong.
• Leave explanations in the practice body. Works until the explanation is wrong, at which point the choice is to rewrite the body — losing what the record stopped believing, against DP-003 — or to leave a practice whose prose argues with its own recommendation. And it offers nothing at all when no practice exists, which is the lottery-ticket case and 48 notes’ worth of the general one.
• Require explains:. It would make the scheme’s purpose enforceable, and it would have made both lottery-ticket documents unfilable. A finding that underwrites nothing yet is still a finding; requiring the link recreates, at one remove, the gap this decision is closing.
• Re-file the eight practices above as part of this decision. Tempting, and deliberately not done — see below.

Consequences

The eight finding-shaped practices stay where they are, for now. Moving one is not a status change, and “retire by changing status, never by deleting” does not cover it: a practice re-filed as a theory leaves a SOTA code that is neither Rejected nor Superseded but simply somewhere else. That needs its own decision about what the old code says and what happens to links into it, and it should be taken with a curation pass rather than

bundled into the scheme that makes it possible. SOTA-010 and SOTA-012 are the clearest candidates; both are sourced to LIT-014 and both are readings of a figure.

The scheme starts with five documents and five statuses, which is a bet. The honest version of DP-002’s warning applies to schemes as well as fields: if a year from now THEORY holds a handful of documents filed in this contribution and nothing since, then explanations did not in fact need a home of their own and this was a folder.

The fifth is the first one that was not reconstructed. THEORY-005 — dense feed-forward layers already being mixtures of experts — was filed because someone asked whether the lottery-ticket documents had anything to do with the record’s mixture-of-experts practices. They did, and the answer was a claim about why those practices work: a THEORY, and not a document this record could previously hold. That is weak evidence and it is the right kind, because the scheme was reached for by a question rather than by its author.

This decision is therefore filed Proposed, with a condition for promotion:

A case arrives in which a practice and the explanation under it move in OPPOSITE directions — an account rejected while the practice it justified stays Active, or a practice retired while the finding beneath it stands — and is filed without either document being rewritten to accommodate the other.

The batch-normalization pair is that case reconstructed, which is why it does not satisfy the condition: the four seed documents were written by someone who already knew what the scheme was for. The condition asks for an event rather than a count, because a count of documents is satisfiable by filing documents.

docs/theory/ joins the generated views, and the practice pages gain an explained_by: line where one exists — four practices today. On SOTA-006 and SOTA-020, both batch-normalization practices, what the link adds is a correction: the reason in the practice’s own source is not the reason the record believes. On SOTA-149 and SOTA-150 it adds the thing a practice registry is usually missing — an argument for why the recommendation works, as distinct from the argument for why you would want it to.

A fourth answer to “what does this status mean” now exists, after practices, papers and readings. That is not obviously a good thing, and it is the strongest argument the rejected alternatives had. The defence is that the four questions really are four — is this worth reading, should you do it, how carefully was it read, is the reason right — and that the record has been answering the fourth one badly rather than not at all.