ADR-102: The TOML crossing carries comments, because a comment is not a value

Context

luria upgrade yaml is the only way across the boundary ADR-098 drew: the new version does not read TOML at all, so a record that has not crossed cannot be linted, indexed or repaired. Every project on luria runs it exactly once, and what it does is what that project’s config becomes.

It parsed with tomllib and wrote with a YAML emitter, for a good reason its own docstring gives: a regex in a uid does not survive being moved as bytes, because the two formats escape differently. Every value has to be re-encoded by a writer that knows its own rules.

Comments are not values, and nothing carried them. tomllib returns a dict; the comments are gone before any writer is chosen. The crossing printed what it folded and what nothing read any more, and said nothing at all about the prose it had just dropped.

The amounts are not incidental. This record’s own config lost 85 comment lines when it crossed. anthology-of-the-sota, which had not crossed yet, carried 273 — why each reference field is declared rather than merely required, what promote_when is for, why contested_by is held to the same standard as source:, what each of the two chains is. That is the reasoning a config needs most, because a config’s values say what and almost never why.

Nor is the loss confined to the TOML. Each scheme’s vocabulary file was read with yaml.safe_load before being inlined into vocabularies:, so the prose above the thirteen topics went the same way.

Decision

The crossing carries comments.

A comment block is a run of comment lines plus the path of the first key or table that follows it — that is all toml_comments.blocks recovers, and all it needs to. The values still go through tomllib; the prose is attached afterwards, to the key it was written above, through the ruamel API yaml_edit already wraps.

Three details are load-bearing:

The document is round-tripped once before the comments go on. The values come back from tomllib as plain dicts, and a plain dict has nowhere to hold a comment. Dumping and reloading is what turns them into structures ruamel can annotate.

A dotted key is a path. uris.title = "..." nests exactly as [remotes.ARXIV.uris] plus title would. Read as a single key named uris.title, its comment lands nowhere — which is how the real config’s one stranded block was found, and it was the only one.

Prose that cannot be placed is printed in full, not counted. tags, statuses and tag_groups do not exist on the far side: each splits into a vocabulary named centrally and a field that names it. Comments on those follow the field where the mapping is unambiguous; anything left is printed whole, because a count tells you something was lost without telling you what, which is the failure this whole change exists to stop (DP-1).

Alternatives considered

• Leave them in git history. The TOML is not deleted by the upgrade, and git show can recover any of it. But a config is read in the editor that has it open, and reasoning that requires knowing to look for a deleted file is reasoning nobody reads. The comments were written next to the keys because that is where they work.
• Move the prose into blurb: fields. This is what luria’s own config did after it crossed, and for vocabulary entries it is the right answer — a blurb is structured, and it renders into the docs. It does not generalize: there is no field on references.contested_by whose value is “why this is held to the same standard as source:”, and inventing one for every key that might want a paragraph is a schema for prose.
• Copy the comment bytes along with the value bytes. Rejected for values for a reason that does not apply to comments — escaping — but the shape is tempting because it is one pass. It would mean a TOML-shaped emitter writing YAML, which is how the escaping bug gets in through the back door.
• Status quo. Every project on luria crosses this boundary exactly once, and each one silently loses everything it wrote about its own config. Two had already paid it when this was written.

Consequences

The anthology’s crossing now carries all 42 of its blocks; before the dotted-key fix it carried 41 and reported the 42nd. Its luria.yaml has 375 comment lines against the TOML’s 273 — the difference is the vocabulary-file prose, which now survives too.

A paragraph break inside a block needed one more thing: ruamel writes an empty comment line as an empty line, which is not a comment, so the prose below it detached from the key it documented. toml_comments.rejoin puts the bare # back. Cosmetic in a ten-line config and not in a 490-line one.

convert_config returns a fourth value, which is a signature change to a function two callers use, both in this repository.

What this obliges: the carry is best-effort by construction, and it says so. A record whose TOML puts comments somewhere this does not look — inside a multi-line array, say, where a comment documents an element rather than a place — still loses them. That is a narrower hole than “all of them”, and the report is what makes it visible rather than silent.