The inference
U17: an LLM “and how it responds to a context, i.e. functionally a parameterized frame” can serve as a motivating example, “maybe even just for notational purposes”. A43: the LLM is “a concrete computational realization of a context-dependent communicative system”, and lets us “separate changes in communicative context from changes in the underlying interpretive system”. So experiments on models can test what the theory says about frames (CLAIM-058).
Critical questions
- In what respect are the cases alike? Both condition their responses on a context, and appending an utterance is a context transition, T_u(c) = c‖u, “a very straightforward realization of context-change semantics” (A43 §2).
- What transfers? Control. Frames, orders and adaptation can be manipulated exactly, and the context-versus-interpreter distinction becomes measurable (CLAIM-118).
- What does not? A48 §3.3: “a prompt is not equivalent to a real social situation, but can encode evidence and instructions concerning one”. A48 §4.6: “An LLM’s context-conditioning behavior need not correspond to human inference over communicative situations.” Order effects in a model may be string concatenation or positional artifacts (A43 §2; TERM-024). And the circularity of A45 §9 (CLAIM-085).
- Is that the respect the conclusion needs? Control is, but only with the generator and the evaluator separated. A45 §9 warns against a design that generates translations with an LLM, asks the same LLM whether they preserve pragmatic meaning, and declares its preferred translations more faithful: “That would primarily characterize the model’s own preferences.” The remedy it gives is “different models for generation and evaluation, independently constructed framing conditions, calibrated response measurements, and held-out human assessments”.
- What relevant difference is there? Training data. Agreement with humans in direction is weak evidence. “Matching direction is weak evidence that the model tests anything about the theory’s claims on human interpretation” (CLAIM-248 v3): a model trained on human writing about teasing and reproach would agree in direction whatever its mechanism. And the founding stimuli and commentary on them are likely in training data, while what a context does to one model need not be what it does to another (CLAIM-248). CLAIM-058 v2 answers this in its defeat condition: the pattern of effects across conditions, across model families and sizes, on stimuli written after the training cut-offs.
Active: the inference holds as stated, to a testing ground. It does not extend to evidence about human pragmatics. Since version 2 it is narrowed further, to a testing ground with separate generator and evaluator, stimuli held out from training, and agreement required across model families.