anthology-of-the-sota

LIT-082: Constitutional AI: Harmlessness from AI Feedback

2 min read

Properties5
StatusActive
Version3
Tagsadaptation-and-tuning, data-pipeline
Cited as “introduced by” bySOTA-183 — Generate the harmlessness preference labels with the model itself, against a written set of principles
Cited as “source” bySOTA-183 — Generate the harmlessness preference labels with the model itself, against a written set of principles

Graph View

Bai et al. (2022) — ARXIV-2212.08073

Key takeaways

  • The supervised stage: sample a response to a harmful prompt, ask the model to critique it against a principle drawn from an explicit written list, ask it to revise accordingly, and fine-tune on the revisions. The model is its own labeler, and the label is a rewrite rather than a score.
  • The reinforcement stage: have the model choose between pairs of responses using the same principles, train a preference model on those comparisons, and optimize against it. This is RLHF with the H replaced — the paper’s own name for it is RLAIF.
  • Human labels do not disappear. Helpfulness comparisons stay human; it is the harmlessness half, the expensive and unpleasant half, that becomes synthetic.
  • The reported result is a model that is both more harmless and less evasive than the RLHF baseline — it engages with a harmful request and explains its objection rather than refusing flatly, which the paper treats as the point rather than a side effect.
  • The transferable contribution is not the specific principles. It is that the supervision becomes a document: readable, versionable, and arguable, where a labeling contract’s implicit distribution is none of those.

Standing in the anthology

The source of SOTA-183.

Unfiled for a year because of where it sat in the tag vocabulary. This note is tagged adaptation-and-tuning, which was not one of the seven topics a practice could carry, and that read at a glance like a scope boundary. It was not one: the record files preference-training practices under training-optimization and synthetic-data practices under data-pipeline, and has several of both.

Both halves of that have since been fixed at the source. ADR-021 added adaptation-and-tuning to the practice vocabulary, and ADR-026 made the two vocabularies identical — no LIT tag can read as a scope boundary now, because every one of them is a word a practice can carry. The LIT and SOTA topic vocabularies overlap rather than nest, so a note’s topic decides nothing about whether a practice can be drawn from it.

Backlinks