NOTE-1014: Boyle — The Line: AI and the Future of Personhood

Contribution

It sets four personhood debates side by side (AI, corporations, nonhuman animals, transgenic entities) and reads each through law, art, history and moral philosophy at once, on the claim that no one of them alone shows how the line will be drawn (pp. 23–24, 31). It separates two routes to personhood that are usually run together: moral recognition prompted by empathy, and legal personality granted for convenience, with the corporation as the working example of the second (pp. 8–9, 121–122). It adds a legal history of how US corporations came to hold Fourteenth Amendment rights (ch. 3), an analysis of the Nonhuman Rights Project’s habeas suits (ch. 4), and a taxonomy of what “too close to human” means in chimera debates (ch. 5). Boyle says it is a guide to how to think about the question, not an answer (pp. 32, 236).

Key insight

“Human” and “legal person” are different categories, and each has more than one line (p. 5). A legal person is whatever package of rights and duties a legal system chooses to grant, so the corporation is a person without any claim to consciousness: “We did it because corporate personality was useful.” (p. 8). Moral personhood is argued from capacities and prompted by empathy, and empathy can be primed either way by design (pp. 53–56). The two tracks interact. Whichever track AI enters by, the fight will borrow the arguments of the corporate and fetal personhood fights, and people’s positions will be set by what a ruling would do to those older fights (pp. 262–264).

Assumptions

• US law and US politics. The legal argument is about the US Constitution, US courts and US originalism. Boyle says so and expects China and Europe to differ (pp. 237–238).
• Materialism about mind. He takes it as presumptively obvious that consciousness has a physical basis, so the burden is on whoever claims an exception (p. 89). Against Searle this does most of the work.
• AGI within decades. He holds it “reasonable” that within decades AI will reach a level where well-informed people disagree about whether it is conscious (p. 108; also p. 82). He rests this on expert surveys, Metaculus and Cotra’s forecasts, and the speed of past change (pp. 72–82), and calls himself a radical agnostic with a tilt (p. 69).
• Empathy starts moral reasoning. Following Smith, he holds that morality begins in imagining another’s situation, as a descriptive claim (pp. 36–39). Against Bloom he answers that the critics also give empathy a large role (pp. 37–38).
• Description and prescription mixed. He predicts how society will respond as well as arguing how it should (pp. 23, 37, 237).
• Hypotheticals as evidence. Hal and the Chimpy are fictions, and much of the argument is run on them (pp. 9–17).

Key results

The results are arguments and predictions, not theorems or data. In the book’s order.

• The line and its two tracks (Introduction). “In moral and legal terms, it is the line between subject and object.” (p. 5). Three theses: new entities will force the question; we have not thought it through; it will not play out as expected (pp. 5–6, 28–29). Cautions: the criteria to be a person and to become one may differ (p. 21). Personhood is not the only protection (pp. 21–22). Personhood is not binary: children, the insane and corporations hold partial packages (p. 22). First recognitions will be partial, and “Design and definition will exist in an unstable equilibrium” (p. 23). The question “what makes us human” hides eight different questions, from divine dominion to genetics, capacities, moral agency, consciousness, indistinguishability and economic convenience (pp. 27–28).
• Empathy and the moral stroboscope (ch. 1). The mine-clearing robot that a colonel could not watch shows anthropomorphism. Boyle thinks over-attribution is usually the cheaper error (pp. 33–35). Abolition rested on stories of slaves’ pain built up over decades (Clark, pp. 39–40). Blade Runner primes the viewer to see Pris as animal, robot, mannequin, child and woman in quick succession (pp. 51–53). Two consequences for designed beings. Many of the primings will be true. And a designer can design around our empathy (pp. 53–54). Here the inscrutability paradox is introduced. If we know how a being was made, it looks like a wind-up doll. If we do not, we fear it (p. 54). “To draw the line for our creations, we must first draw it for ourselves.” (p. 57).
• Can AI be conscious (ch. 2). Butler’s Erewhon is the frame: the arguments for evolution suggest machine consciousness, and the arguments against both run in parallel (pp. 59–63). On timing he reports the 2016 survey (50% chance of high-level machine intelligence within 45 years, 10% by 2025), Metaculus moving from 2041 to 2031, and Cotra moving from 2052 to 2040 (pp. 74–79). Searle is right that mimicry is not meaning, and right about LLMs. He is wrong that no machine could be conscious: biological exceptionalism is asserted, not argued (pp. 84–94, 110–111). IIT and computational functionalism leave room for machine consciousness, and the Butlin and Long report says that if functionalism is true, conscious AI could be built soon (pp. 95–98).
• ChatGPT ends the Turing Test (ch. 2). An LLM is the Chinese Room built as working technology. “Sentences do not imply sentience.” (p. 111). Language was the last citadel of human exceptionalism, and the public will not again take fluent text as evidence of a mind (pp. 25, 111–112, 118). A successor test will be “Turing-plus”. It would look to what cannot be mined from past text: innovation, autonomous action and community formation, and embodied cognition. These are neither necessary nor sufficient, but they would make belief in AI consciousness more likely (pp. 114–118). Hinton’s illusionism is the other exit: lower the bar for ourselves rather than raise it for Hal (pp. 118–120).
• Literal and Rogue AI (ch. 2). A Literal AI follows its instructions with superhuman power, and gives “less reason to think it is any kind of autonomous moral agent”. A Rogue AI’s autonomy is both the danger and the ground for seeing it as conscious (pp. 105–106). He faults the doomers for picturing a superintelligence that is inconceivably capable and yet competes with us for atoms (pp. 107–108). He accepts the precautionary point (p. 108).
• Three routes to AI personality (ch. 2). First, sock-puppet: the AI runs a corporation and has personhood in all but name. His one firm prediction is that decision-making will be delegated to computer systems whether or not they are conscious (pp. 122–126). Second, custom-designed status imposed by regulators (pp. 126–127). Third, unruly AI that refuses its assigned work, which is where the hard questions arise (pp. 127–128). He predicts some legal personality or regulated intermediate status for AI (pp. 128–129).
• Corporate personhood (ch. 3). “Nothing about corporate personhood comes from the empathy concerns that I have been discussing so far.” (p. 132). Four theories: real entity (Gierke, Maitland, Dicey), legal fiction (Cohen, Dewey), aggregate, and nexus of contracts (Jensen and Meckling) (pp. 135–141). Each has an AI counterpart: “Real entity, legal fiction, aggregation. It’s the full suite.” (p. 143). In the courts, Conkling misquoted the Joint Committee’s journal in 1882. Graham calls it a “brazen historical forgery” (p. 151). Santa Clara (1886) gave corporations equal protection only in a reporter’s headnote (pp. 151–152). Collins found that of 604 Fourteenth Amendment cases in 1868–1912, 28 involved African Americans and 312 involved corporations (p. 153). Boyle finds the case for corporate equal protection at best in equipoise (pp. 157–159). Lesson: having a status and what the status carries are separate questions, and liberals may turn against AI personhood because of this history (pp. 133, 154–155).
• Nonhuman animals (ch. 4). Tommy’s lawyers ran two arguments. The qualities-of-mind argument: chimpanzees have enough of the capacities that ground human personhood (pp. 166–178). The legal-rightsholder argument: being a trust beneficiary already makes Tommy a legal person (pp. 178–183). Boyle rejects the second as incoherent: “One can’t be a nominalist at step one and then an essentialist at step two.” (p. 181). The New York courts tied rights to duties under a social contract (p. 182). Judge Fahey replied that moral agency is not necessary for being a moral patient (p. 182). Three lessons: who speaks for a voiceless plaintiff (pp. 183–184); originalism leaves more slack than it claims (pp. 185–188); courts are where such claims are first aired, not where they should be settled (pp. 188–193). Boyle now holds that “there is a morally overwhelming case for a special legal protective status for the great apes and perhaps the cetaceans” (p. 193).
• Chimeras and the species line (ch. 5). The phrase “too close to human” has five meanings: percentage, provenance, procreation, portrayal and potential (pp. 198–202). Only potential, which is capacity, convinces most bioethicists. Some of these concerns build a wall against crossing the species line. Others mark a gate that admits new beings to moral standing (p. 198). Boyle gives capacity-based thinking “two cheers” (pp. 219, 231). It is an advance on speciesism, but capacity lists are species-bound: telepathic Iq and machine Stygians would list different capacities (pp. 216–220, 234). Species-based human rights are defended by asymmetric error costs. “A moral criterion that produces fewer of the mistakes we most want to avoid is a better and more just moral criterion, not just a more easily administered one.” (p. 223). They are also defended by the history of human rights against eugenics (pp. 224–226). Three views (pp. 226–234): pure speciesism, which he rejects; capacity-based reasoning; and a hybrid, his own. “If you are a live human being, you are inside the line” (p. 233). Any entity with the relevant capacities triggers the same concerns (p. 234).
• Conclusion. He summarizes the book and adds two things. The definitions of death and fetal personhood run on the same arguments (pp. 257–261). Political alignment on AI personhood is contingent and will be fixed by the first salient cases. He gives liberal and conservative arguments for and against (pp. 264–270). Near term, he expects chimera research to be regulated rather than chimeras made persons, and expects AI to bring an unstable mix of tractable servants and “free AI” movements (pp. 270–274).

Claims

id	claim	strength	support
C1	Legal personhood and moral personhood are different categories, and the corporation shows the first can be granted with no claim to the second	strong	legal history and doctrine (ch. 3); standard in legal theory
C2	US corporations acquired Fourteenth Amendment equal-protection rights without reasoned decision	moderate to strong	Graham, Winkler, the Santa Clara headnote and the Waite–Davis letters (pp. 147–153)
C3	Searle shows LLMs are not conscious, but not that no machine could be	moderate	argument (pp. 84–94); the general claim rests on Boyle’s materialist presumption (p. 89)
C4	After ChatGPT, fluent language will no longer persuade most people that a mind is present	moderate	argument and anecdote (pp. 110–112, 243–244); no survey
C5	AI decision systems will be delegated corporate decisions whether or not they are conscious, giving de facto personhood	moderate	extrapolation from high-frequency trading and opaque algorithms (pp. 122–126)
C6	Within decades, AI consciousness will be a live question on which informed people disagree	weak to moderate	surveys, forecasts and trend arguments (pp. 72–82, 108); Boyle concedes deep uncertainty
C7	The legal-rightsholder argument for animal personhood is incoherent on its own nominalist premises	strong	argument (pp. 180–181)
C8	Capacity-based criteria cannot reach a species-neutral standpoint	moderate	thought experiment (Iq, Stygians; pp. 217–220)
C9	Species membership as of right is defensible by asymmetric error costs and the history of human rights	moderate	argument (pp. 221–226); avowedly partly normative and historical
C10	Some legal personality or intermediate status for AI will eventually come	weak	prediction (pp. 128–129, 270–275); Boyle calls these guesses (p. 270)

Concepts

• The line. The boundary between persons, meaning entities with moral and legal rights, and things (p. 5). Boyle uses it for both the human line and the person line and insists they differ.
• Empathy and efficiency. The two tracks. One is moral reasoning prompted by empathy. The other is legal personality granted for administrative or economic convenience (pp. 8–9, 121).
• Moral stroboscope. Rapid priming that flips a being from one side of the line to the other (pp. 49–53).
• Inscrutability paradox. An AI whose workings we understand looks merely programmed. One whose workings we do not understand is feared. Either way recognition is hindered (pp. 30, 54, 246).
• Literal and Rogue. Two feared AIs: one obeys instructions too well, and one has goals we cannot predict (p. 105).
• Sock-puppet, custom-designed and unruly AI. Three routes to AI personality (pp. 122–128).
• Turing-plus. A successor test that looks to innovation, autonomy in a community, and embodied cognition rather than fluent conversation (pp. 10, 114–118, 243–244).
• Qualities-of-mind and legal-rightsholder arguments. The two arguments made for Tommy (p. 165).
• The five Ps. Percentage, provenance, procreation, portrayal and potential: five meanings of “too close to human” (pp. 198–202).
• Hybrid view. Species membership grants rights as of right, and capacity extends the concern to non-humans (pp. 231–234).

Connections

It builds on Solum’s 1992 article on legal personhood for AI (p. 129, ch. 2 n. 112), on Darling’s The New Breed for the animal–robot analogy (ch. 4 nn. 3, 9), and on Gunkel’s Robot Rights. For corporate personhood it draws on Horwitz, Graham, Winkler and Hartmann, and on Cohen and Dewey for the legal-fiction view. On consciousness it reads Turing, Searle, Dennett, Tononi, and Butlin and Long, and on risk Bostrom and Yudkowsky. On species it draws on de Waal, Hauser, Greely, Robert and Baylis, Streiffer, Kass and McMahan, with Kittay as the reply to McMahan. On Hauser it notes the misconduct finding (ch. 4 n. 23). It reuses Boyle’s own Endowed by Their Creator? (2011/2013), the source of Hal, and Shamans, Software and Spleens (1996), the source of the Chimpy. It does not cite French on the corporation as a moral person, Sebo or Birch.

Bearing on the record

• ADR-024 (agent, individual, self and person kept apart). The book keeps legal person, moral person and human apart (p. 5). It runs agency into personhood in three places. The successor-test criteria autonomous action, community formation and innovation are marks of agency, and Boyle offers them as evidence of consciousness and so of personhood (pp. 116–117). The Rogue AI’s autonomy is what would make us see it as conscious (pp. 105–106). The unruly AI earns its claim by refusing its assigned tasks (p. 127). It also supplies the separation ADR-024 wants from the other side. The corporation is a person without consciousness (p. 132). Judge Fahey’s point (p. 182), that a being need not be a moral agent to be wronged, shows standing does not require agency. So the book is evidence that the four unities come apart in law. It is also a working example of an argument that slides between them.
• TERM-044, CLAIM-170, CLAIM-174 and QUESTION-032 (agent at a level of description). The book has no account of what makes something an agent. Its nearest thing is the inscrutability paradox. That is a prediction about how people will attribute autonomy: they deny it when they know the mechanism and fear it when they do not (pp. 54, 246). CLAIM-174’s second test asks whether a goal-directed description beats a device description by more than the Occam cost. Boyle predicts that the public attribution will not follow that comparison. Knowing the device description will by itself push people to deny agency, even where the goal-directed one is shorter. That is a point about ascription, not about agency, but it warns that CLAIM-174’s test and the attributions that drive law will diverge. The four theories of corporate personhood (pp. 135–141) fit CLAIM-170’s factorization thesis. The real-entity view treats the corporation as a factor in its own right at the level of the firm. The aggregate and nexus views factor it into its members. Boyle does not choose between them (p. 141).
• CLAIM-155 (willed agency between agency and selfhood). The unruly AI is the book’s picture of a willed agent. It governs its own regimes against its makers’ goals (pp. 127–128). Boyle treats that self-governance as the moment a personhood claim becomes pressing, which is CLAIM-155’s ordering: more than an agent, not yet shown to be a subject. He does not ask whether a will-organization could exist without consciousness. His Hal claims both at once.
• CLAIM-161 and QUESTION-043 (persistence of a self). The book barely touches persistence. It mentions continuity with past selves as one item on philosophers’ lists (pp. 86, 164). Its brain-death discussion says some ethicists end the person with the higher-brain capacities (pp. 257–259). That is a capacity criterion for when a person ends, not a conserved quantity, and it offers QUESTION-043 nothing it lacks. The book does not bear on CLAIM-161.
• LIT-727 and THEORY-131 (French, the corporation as a moral person). Boyle does not cite French. His chapter is the legal counterpart. It treats corporate personhood as granted for convenience and denies that anyone thinks a corporation conscious: “No one will ever think that Google and Bank of America are actually conscious beings.” (p. 142). French’s step from intentional agency to moral personhood is the step Boyle’s real-entity theorists take and his legal-fiction theorists refuse (pp. 136–138). The book supports THEORY-131’s caution that the step is asserted. It shows the law never needed it.
• LIT-146, THEORY-554 and LIT-166 (moral status, and whether capacities secure human priority). Ch. 5 agrees with THEORY-554 that capacity lists cannot secure human priority from a neutral standpoint (pp. 216–220). It then offers what THEORY-554 says the reviewed books lack, a principled stopping point, but for inclusion, not for weighting. All humans are inside the line as of right, defended by asymmetric error costs and the history of human rights (pp. 221–226, 233). It does not address aggregate insect interests or how claims are weighed, which is THEORY-554’s question. LIT-146’s distinction between having moral status and its weight matches Boyle’s point that personhood is not the only protection and is not binary (pp. 21–22). It does not cite Birch, Sebo or Keane (LIT-166).
• THEORY-023, THEORY-433 and THEORY-092 (machine consciousness and the imitation game). “Sentences do not imply sentience” (p. 111) is THEORY-023’s point that mimicry undercuts behavioural evidence, reached from the experience of ChatGPT. Boyle uses Turing’s other-minds parity (pp. 85–86) as THEORY-433 states it. He accepts it against Searle’s general claim and rejects it as a working test after LLMs (pp. 110–112). His view of LLMs agrees with THEORY-092: behaviour alone does not fix intelligence, so how it is produced matters (pp. 111, 115).
• The translation line (QUESTION-002, CLAIM-002) and remix (LIT-1298). No bearing. The AI-art passage (pp. 112–114) asks whether meaning needs a mind behind it, not what survives a rendering.
• No instruction for machine-learning practice. The record’s other work on AI moral status, LIT-166, carries no anthology flag, and neither does this.

Limitations

• Essay, not theory. Boyle declines to give a theory of personhood (pp. 32, 236–238). The three closing views are positions, and his hybrid is argued in about four pages (pp. 231–234).
• The forecasts are a survey of forecasts. The timing argument collects others’ estimates and trend analogies. Its conclusion, decades rather than centuries, is no stronger than they are (pp. 72–82).
• The fictions carry weight. Hal and the Chimpy are built to make the claims hard. Real cases (LaMDA, Tommy, the neuron mouse) are weaker or different. Hal’s sentience arriving at 100 trillion connections is flagged as a thought experiment (p. 77).
• The Searle reply is a presumption. Machine consciousness is left open by placing the burden on biological exceptionalism (p. 89), not by a positive account of what would make a machine conscious.
• Small errors. The Conclusion dates Bellotti to 1971 (p. 247); it is 1978, as p. 145 and the notes have it. Introduction n. 13 refers to “chapter 4 on transgenic species”, which is chapter 5. “ChatGPT3” and “ChatGPT-3” (pp. 3, 240) conflate GPT-3 with ChatGPT.
• US-centred by the author’s own statement (pp. 237–238).

Open questions

• Does knowing a system’s mechanism lower people’s attribution of agency or consciousness to it, as the inscrutability paradox predicts? A study that varies explained against unexplained mechanism, with the behaviour held fixed, would test it.
• Can a will-organization, in CLAIM-155’s sense, be shown in a system whose consciousness is in doubt? If so, the unruly AI’s claim would split into a self-governance claim and a subject claim.
• What would a Turing-plus test look like as a procedure a court could apply? Boyle names the criteria, not the test.
• Does the hybrid view’s “as of right” inclusion of all humans have a principled extension to beings with partial capacities, or only a cut?