Source
Bai et al. (2022), LIT-082 — ARXIV-2212.08073.
The method
Two stages. First supervised: sample responses to harmful prompts, have the model critique each one against a principle drawn from a written list, revise it, and fine-tune on the revisions. Then reinforcement: have the model choose between response pairs using those same principles, train a preference model on the resulting comparisons, and optimize against it. Human labels remain in the loop for helpfulness; the harmlessness half is model-generated.
This is the method of LIT-082 (Constitutional AI), where the practice originates. Its reported result is a model both more harmless and less evasive than the RLHF baseline: it engages with a harmful request and explains its objection rather than refusing flatly.
Why this is a training-data practice, not an alignment philosophy
The expensive, slow, unpleasant input to preference training is the human harm-comparison label, and this replaces it with generated data at the cost of writing down the criteria. The criteria being written down is the part that transfers — the supervision is now an artifact you can read, version and argue with, instead of a distribution implicit in a labeling contract.
Why emerging
The general move — AI feedback in place of human preference labels — is now widespread and appears in most open post-training pipelines. The specific apparatus of a principle list consulted per-critique is much less so; several groups get the same effect from a reward model or a rubric without the constitution framing. What is agreed is that the labels can be synthetic. What is not agreed is what the model should be asked to consult.
The condition worth carrying
This shifts the failure mode rather than removing it. A gap or an ambiguity in the written principles propagates silently into every generated comparison, where a human labeler would have produced visible disagreement instead.
Known implementations
- Claude, and the RLAIF stage in most open post-training recipes