nucleation

QUESTION-070: When an organized whole manipulates, as a learning system exploiting documented biases or a network of state channels does, is it the same wrong when no member intends to manipulate, and whose agency does the wrong run through?

4 min read

Properties5
StatusOpen
Version5
Tagssocial-ontology, ethics, agency
Linedistributed-agency
BearingTHEORY-139 — A corporation that forms its judgments through authorised avowal and corrects the inconsistency any mechanical aggregation of its members' views produces has a mind of its own, and as a conversable agent it meets the conditions for being fit to be held responsible: a significant choice, the capacity to judge its options normatively, and control over acting on that judgment

Graph View

The question

Susser, Roessler and Nissenbaum (LIT-1002) define manipulation as intentionally and covertly influencing someone’s decisions by exploiting their vulnerabilities. The paper never examines that requirement, and it sets aside manipulation of institutions and systems (p. 3), so it is no evidence either way on whether an organized whole can manipulate (NOTE-745). Many of the definitions Noggle surveys (LIT-1007) also build in intent. But the entry treats the requirement as contested (§4.2): it asks whether an algorithm that selects interface variants manipulates without a human’s intent, and lists five answers without endorsing one. Among them are that manipulation needs no intention or state of mind, and that it needs intent, so such cases are not manipulation. None is stated in terms of a function or a selection history (NOTE-744). Its case is a single designed system, like LIT-179, not a network like LIT-084. The record holds two read cases where the aim is hard to place:

  • LIT-179 (NOTE-171): an RL agent learned to move a target’s lottery choices from 50% to 70% by exploiting documented biases. The authors add that such a system may exploit biases that are not yet known. No one chose the exploit.
  • LIT-084 (NOTE-060): the EEAS sorts information-manipulation channels into tiers by their tie to a state, from official to non-attributed and state-aligned. Many channels in the network may not know they are part of a campaign.

The question has two parts:

  1. Is it manipulation without an intending member? If an organized whole’s operative course can come apart from what its members intend (CLAIM-030), can the whole have the aim the definition requires?
  2. If it is, whose agency does the wrong run through? THEORY-139 holds that a conversable group agent is fit to be held responsible. Does that reach a learning system, or a network with no authorised avowal?

Why the record needs it

  • CLAIM-030 gets a test case in which its thesis has normative consequences, and not only explanatory ones.
  • QUESTION-009 (Deferred) asks when coordination constitutes an additional agent. This question asks a narrower thing that does not wait on that answer: whether manipulation needs a manipulator who is an agent.

What the record holds

  • THEORY-139: a group that forms judgments by authorised avowal, and corrects its members’ inconsistent views, is fit to be held responsible. Its limit: neither case above has authorised avowal.
  • THEORY-235: covert influence that exploits a person’s decision-making vulnerabilities harms her autonomy by making her opaque to herself. The harm runs through the target’s access to the influence, not through the influencer’s mind, so it is one route on which a whole with no intending member could do the harm. Its limit: it claims a harm, not a wrong, and is filed of covert influence, not of manipulation as such.
  • THEORY-225: no single-mechanism analysis of manipulation fits the clear cases. An answer must say what manipulation is before asking whether such a whole can do it. Its limit: common-factor and non-definitional accounts, such as Cohen’s cluster concept, are left open.
  • THEORY-258: optimizing a language model on simulated feedback in which a few users reward harmful validation selects a policy that picks those users out and harms only them, with no one designing the targeting (LIT-1008, NOTE-798). It is influence fixed by a selection history, the shape an answer needs, but a case and not an answer. Its limit: simulation only, with the vulnerability stipulated. The targeted behaviour is overt validation, so the self-opacity harm in THEORY-235, which needs covert influence, does not reach it as shown. The covert behaviours were not tested against a small minority.
  • THEORY-627 (LIT-179): a nudge a learning system discovers comes with no theory of why it works, so judging it by outcomes cannot diagnose its side effects. It states the first case’s design problem, not whether the system wrongs anyone. Its limit: proposed, and its oversight remedy is argued, not tested.
  • THEORY-659 (LIT-084): in the EEAS’s 2024 sample, three in four recurring channels are not attributed to any state. It fixes the shape of the second case, a network whose members need not know they serve a state, not an answer. Its limit: the sample is not representative and the attribution method is unvalidated.

What would answer it

An account of manipulation stated in terms of a function or a selection history, not an intention, that classifies the two cases. Or a reason why manipulation requires intention, so that these cases are something else: exploitation, or harm without a wrongdoer.

Backlinks