ADR-096: A rate limit is an answer, not a reason to ask again

Context

_fetch treated every non-answer alike: ATTEMPTS = 3, sleeping COURTESY then double, then returning whatever the last attempt said. That policy was written to fix a real bug — the version before it retried everything, including 404s, and reported every failure the same way, so a throttled batch of fifteen wrote nothing to the lockfile and read afterwards as fifteen documents in agreement. Separating “upstream has no such identifier” from “upstream would not talk to me” was correct and is not what this revisits.

What it got wrong is the schedule. A rate limit is a property of the window, not of the request. Asking arXiv again three seconds after it declined is asking the same host the same question inside the same window, and it answers the same way — so the retries cost 9 seconds per identifier and changed the outcome in none of the cases that motivated them.

Two things make that expensive rather than merely wasteful:

• luria lint asks. Under [luria.lint] network = "auto", the default, the lint reaches out for identifiers the lockfile cannot answer — normally the one citation a contribution just added, which is exactly when a citation is most likely to be wrong. Under a sustained throttle, every unknown citation pays the full 9 seconds.
• --resolve walks everything. It asks about every identifier, settled or not, with a 3-second courtesy pause between each. On a 225-identifier record that is an 11-minute floor before any throttling, it wrote the lockfile only at the very end, and it walked in the record’s own order — so a run interrupted partway left the same tail unresolved, run after run, and kept nothing it had learned.

The observed failure is a project where one new citation made luria lint unpredictably slow, and where the command that would have fixed it was the one nobody could afford to finish.

Decision

Fail fast, honour what the host actually asked for, and stop asking a remote that is refusing.

• _fetch makes one request. A throttled answer is returned as it stands.
• Retry-After becomes a value on Fetched rather than text inside detail, and a wait at or under RETRY_AFTER_CAP (10s) is honoured with exactly one retry. A host that says when to come back has named the request that will succeed; a longer wait is reported instead of slept through, because a build that blocks for minutes on a metadata courtesy API is worse than one that says what it could not check.
• ask() holds a per-run circuit breaker keyed by remote. The first throttle ends that remote for the run and its remaining identifiers are answered from the breaker without a socket. It lives in ask() rather than in resolve() so every caller inherits it — the lint’s per-identifier check most of all.
• resolve() asks about unsettled identifiers first, shuffled within each group, and checkpoints the lockfile as it goes.

The ordering is the part that makes an interrupted run cumulative. An identifier the lockfile has no entry for is one no run has ever settled, so starting there means a throttle costs the re-verification pass and never the new work. The shuffle — deliberately unseeded — is so that one identifier that always errors cannot sit at the head of the queue every run and spend the whole window on itself.

Alternatives considered

• Keep the retries, lengthen the backoff. The obvious one, and it is backwards: the fault is not that we waited too briefly but that we waited at all inside a window the host had already closed. A longer backoff makes the lint slower in exactly the case it is already too slow.
• A token-bucket rate limiter across remotes. The right answer for a client that must saturate an API. This one asks a metadata endpoint a few hundred times a month; a limiter is a lot of machinery, and it still would not have told an interrupted run where to resume — which is the half of the problem that actually hurt.
• Persist the refusal into the lockfile. Tempting: the next run could skip the remote outright. Rejected, and it is the one alternative worth being precise about. The lockfile is committed, so a refusal in it would travel to CI and to other contributors as though it were a fact about the world, when it is a fact about one machine’s last five minutes. The next run is a new window. Nothing about a rate limit should outlive the process that met it — and nothing needs to, because the unsettled queue already carries the only durable part: which identifiers still have no answer, which the lockfile records by their absence and has recorded all along.
• Have --resolve skip settled identifiers entirely. It would be fast, and it would quietly change what the command means: re-verifying a title is how an upstream revision gets noticed. Ordering is enough to get the benefit without taking the guarantee away.
• Status quo. Costs the lint its predictability under throttle, and costs --resolve the ability to make progress at all.

Consequences

luria lint under a throttle now reports could not be checked after one round trip per remote instead of 9 seconds per identifier, and --resolve makes durable progress every run even when it is cut off. The reports are unchanged in shape: a throttled identifier is still unchecked, still acknowledgeable, and still never written to the lockfile as agreement.

What it costs: one retry fewer against a host having a genuinely transient moment. That case is now handled by the next run rather than by this one, which is the trade — and Retry-After covers the hosts that say so.

Filed Proposed. The breaker and the ordering are provisioned, and provisioned is not working (DP-006): the firing that matters is a real throttle on a real record, and the devlog entry beside this decision is where it will be recorded when it happens.