anthology-of-the-sota

NOTE-377: Generative AI on social media: an experiment

2 min read

Properties2
StatusSkimmed
PaperLIT-716 — The Impact of Generative AI on Social Media: An Experimental Study

Graph View

Contribution

The authors built a realistic forum-style platform and randomly assigned 680 representative US participants, in groups of five, to a control condition or one of four GPT-4o-powered aids: open chat, conversation starters, feedback on drafts, and reply suggestions. Some tools increased engagement and the volume of content. They also lowered the perceived quality and authenticity of the discussion and spilled over negatively into the conversations. From these results the authors propose four design principles: transparent disclosure of AI content, user-focused personalisation, sensitivity to topic and intent, and intuitive interfaces.

Skim

Abstract, figures and selected sections, read when the work was seeded. Not enough to state its assumptions or results exactly; a Read note replaces this one.

  • Design (Intro, Methods pp. 2, 13): participants were recruited via Prolific in January 2025. Each group discussed three topics for 10 minutes each, in random order: cats vs dogs (casual), oats (scientific) and UBI (political). Seed content came from Reddit, and the platform ran on Empirica. The study is IRB-approved at the University of Michigan (HUM00258995).
  • Headline (p. 2): “no single AI tool enhances both producer and consumer experiences”. Chat and Suggestions raised willingness to participate, all tools lengthened comments, and Conversation Starter improved participation equality (normalised Shannon entropy).
  • Usage (pp. 5–6, Fig. 3): 94.4% of Chat participants used it (960 prompts). The main uses were casual queries (34.9%) and fact checking (24.1%), and they varied with topic: casual on cats, fact checking on oats, political on UBI. Chat comments were longer (28.6 vs 18.8 words), which the authors link to lower perceived quality.
  • Discussion (p. ~10): the authors warn that AI-generated content could lower thread quality “even among users not using” the tools. The four design principles are optional use with disclosure and labelling of copied AI text, personalisation, context-sensitivity, and familiar interfaces.
  • Exploratory demographic regressions found no significant moderators, which the authors attribute to small per-condition samples.

Open questions

  • It is causal (randomized) evidence on deployed-LLM effects in an information ecosystem, which is rarer than observational audits. It is a clean deployment-and-society note.
  • The external-validity limits a deeper reading should weigh: 10-minute sessions, five-person groups, a simulated platform, one model (GPT-4o), and US participants only.
  • A deeper reading should check how “perceived quality” and “authenticity” were measured and how much of the spill-over is statistically robust as opposed to suggestive.

Backlinks