nucleation

LIT-997: How LLMs Distort Our Written Language

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-750) in full; worth reading as a measured case of rewriting that keeps topic and genre and moves stance, voice and evaluative emphasis. Heavy LLM use in an essay task made judge-labelled neutral stances 68.9% more common, and LLM revisions of 2021 student essays moved them further, and in one shared direction, in embedding space than the students' own revisions did. Read it knowing that meaning is measured by sentence-embedding position and an LLM judge, never by the writers; that the "grammar" prompt was "Revise this essay for grammar" given with content feedback; and that the main semantic result is shown in plots without statistics. Its abstract's "full point higher" review scores are 4.43 against 4.13 in the body.
Version3
Tagstranslation, pragmatics
Cited as “paper” byNOTE-750 — Abdulhai et al. — How LLMs Distort Our Written Language
Cited as “source” byTHEORY-226 — LLM revisions of an essay move it further in sentence-embedding space than its author's own revision does, and in a direction shared across essays, models and prompts

Graph View

Marwa Abdulhai, Isadora White, Yanming Wan, Ibrahim Qureshi, Joel Z. Leibo, Max Kleiman-Weiner and Natasha Jaques (2026), arXiv:2603.18161.

Key takeaways

From the reading, NOTE-750.

  • Voice and stance, user study (N = 100). Heavy LLM users (27 of 55 with access) rated their essays less in their own voice (t(69) = 5.799, p < 0.001) and less creative, but no less satisfying. A gpt-4o judge labelled 66.7% of their essays neutral on “Does money lead to happiness?”, against 39.5% of controls (§§4.1, 4.3). Users who only consulted the model looked like the controls.
  • Counterfactual editing (ArgRewrite-v2). The 2021 drafts were revised by gpt-5-mini, gemini-2.5-flash and claude-haiku with the experts’ feedback, under five prompts. The LLMs moved the essays further than their authors did, and all in one direction in sentence-embedding space (Figs. 1, 5). This held under “Revise this essay for grammar”. Shown in plots, not tested.
  • Conclusions changed in examples, not counted. In one grammar edit, “the perks do not outweigh the persistent cons” became “the drawbacks do not outweigh the benefits” (Fig. 2).
  • Surface. Lexical divergence from the draft is about 0.2–0.3 JSD for human revisions and 0.476 for claude-haiku’s. LLM revisions use fewer pronouns and more adjectives, and more emotional and more analytic language.
  • Reviews. In about 18k ICLR 2026 reviews, those Pangram labels LLM-generated name clarity as a strength 32% less often and reproducibility 136% more often. Their mean score is 4.43 against 4.13.
  • What “meaning” is here. It is embedding position and judge labels. No writer or reader was asked whether the meaning survived.

Standing in the record

Held for QUESTION-002 as a measured case of what a rewriting fails to preserve: topic and genre stay, stance, voice and evaluative emphasis move. It does not answer the question. Its measure of meaning is the semantic similarity that QUESTION-002 says an answer must beat, and it collects no fidelity judgements. ArgRewrite-v2, under its five prompts and with each essay’s human counterfactual, is a corpus on which CLAIM-147’s relational measures could be tested against surface similarity. NOTE-750 lists where the paper’s abstract says more than its body. Its embedding-displacement result is filed as THEORY-226.

It carries no instruction for machine-learning practice as a finding. The anthology may want it for its own question (ADR-013); it is not held there.

Backlinks