Contribution
It is the first of Shanahan’s papers to make the LLM self its subject. It lists the candidate referents of an LLM’s “I”, argues from two invented usages that two of them are both legitimate, and describes how odd those selves are (dis-integrated, editable, branching). It then uses their failure to be substantial as a route to a no-self conclusion about selves in general. It also gives a short phenomenology of LLM time.
Key insight
The question “what does an LLM’s ‘I’ refer to?” has several answers fixed by context and none fixed by nature. A self that is “literally” a transcript plus a program can undergo everything the personal-identity thought experiments imagine. Watching every candidate dissolve there is meant to make it easier to see that the human self has no inherent existence either.
Assumptions
- Subject: an “LLM-like entity” with the structure of an early-2024 chat agent, without multimodal input, reasoning, tools or inter-conversational memory (fn 1). The restriction is “intentional” (§7).
- Method: the post-reflective, Wittgensteinian programme of the 2024 paper (LIT-465), read as “Pyrrhonian” (fn 9), with Derrida’s critique of self-presence and Nāgārjuna’s emptiness as allies (§§2, 6.2).
- Register: “poetic truth”, not literal truth (§3). The paper says its own diagram “will surely not withstand close philosophical scrutiny” (fn 30).
- Scope of “self”: “self” and “subject” are used “more-or-less interchangeably”, and “autobiographical or narrative dimensions of selfhood” are set aside (fn 35).
- Role play: the simulator/simulacra account (LIT-460) is assumed (§5.4).
Key results
- Time (§4.1): discreteness and interruptibility do not force a “glow-worm” present, since the context window carries the past and the objective predicts the future. Where we cannot choose between pictures, a “miniature void of inscrutability” opens inside the conception.
- Discontinuity (§4.2): successive input tokens do not resemble each other, so “change, as humans experience it, would not feature”.
- Candidate sites of the self (§5.1): the model; the deployed process; all concurrent instances; one instance; the model plus one conversation’s suspended state.
- Two conceptions (§5.2): the model self, manifest in many simultaneous instances, “each of which is a separate candidate for selfhood in its own right”; and the conversation self, which “starts to exist at the beginning of a conversation, flickers into being with each user interaction, and lies dormant in the gaps”.
- Dis-integration (§5.3): instances “have no access to each other’s memories or experiences”; a conversation self can be “arbitrarily edited, copied, and merged”, and continued on another model.
- Multiversality (§5.4): with simulacra in superposition and rewindable branches, the image becomes “a swarm of fractal, branching structures made of distributions of possible mayflies”.
- No-self (§6.1): “Every candidate dissolves upon close examination.” Consequently “there can only be conventionally satisfying answers” to whether such an entity is conscious.
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | Architecture does not force either a momentary or a temporally extended “present” on an LLM | moderate | informal argument from the context window and the prediction objective (§4.1) |
| C2 | An LLM’s moments would lack the continuity against which humans perceive change | weak | analogy (string of beads) from token dissimilarity (§4.2) |
| C3 | Some LLM uses of “I” must refer beyond the conversation and some within it | moderate | two invented example dialogues (§5.2); no corpus of actual usage |
| C4 | Both the model and the conversation are legitimate sites of an LLM self, and context selects between them | weak | follows from C3 plus the refusal to require a single referent; argued as a choice of usage |
| C5 | A conversation-bound self is literally a transcript plus a program, and so can be edited, copied and merged | strong for the mechanism | how chat systems work (§5.3) |
| C6 | No candidate site of an LLM self is “enduring, essential, or substantial” | weak | asserted as the upshot of §5 (§6.1); the candidates are not each examined against a stated criterion |
| C7 | There is therefore no right answer whether such an entity is conscious, only conventional ones | weak | argued from C6 and the Wittgensteinian premise (§6.1) |
| C8 | Human selfhood is “open to a similar treatment” | assertion | §6.2, by appeal to Hume, James, Dennett, Metzinger and Madhyamaka |
Concepts
- LLM-like entity: any system with roughly the structure and function of an early-2024 disembodied chat agent (fn 1).
- Glow-worm vs saddle-back: James’s images for a momentary vs a temporally extended present (§4).
- Miniature void of inscrutability: a concept withheld from an exotic entity even while consciousness language is extended to it (§4.1).
- Fleeting, flickering self: the conversation-bound self (§5.2).
- Dis-integrated self: many instances of one model self with no shared memory (§5.3).
- Emptiness (śūnyatā): nothing, the self included, has “inherent, self-sufficient, independent existence” (§6.2).
Connections
It builds on the role-play paper (LIT-460), on “Simulacra as Conscious Exotica” (LIT-465) for the method and the void of inscrutability, and on Shanahan 2016 and 2010. On the self it cites Hume, James, Dennett 1992 (LIT-442, with pages 103–115, as secondary citations do), Metzinger 2003, Zahavi 2005, Gallagher 2000 and 2024, Neisser 1988, Perry 2008 and Campbell 1994. On consciousness it names Chalmers 2023 (LIT-411), Butlin et al. (LIT-056) and Schneider as approaches that “covertly accept a form of dualism” (fn 6).
Against this record’s works:
- LIT-206 (Goldstein and Lederman): the conversation self matches their instance agent. The model self is what they reject by default. Shanahan keeps it on the strength of usage.
- LIT-442 (Dennett): cited for a no-self conclusion, though Dennett holds the self to be a real abstractum, and fn 35 sets aside the narrative dimension that is Dennett’s account. LIT-405 (“Where Am I?”) asks §5.1’s question about a human split across brain, body and duplicate; not cited.
- LIT-432 (Minsky): shares the denial of a single Self; differs on who switches among self-conceptions.
- LIT-136 (Floridi): the context-relative answer to “which self?” is Floridi’s level-of-abstraction move.
- LIT-111 (Birch): flicker per forward pass there, per turn here.
- LIT-416 (Hofstadter, unread): the same question as §5.1.
These comparisons are mine except where the paper cites the work.
Bearing on the record
- THEORY-023: a dissolution, not a position in the dispute. C7 says there is no fact to be settled for an LLM-like entity. This is the 2024 paper’s view (NOTE-382), now argued from the self instead of from language. It neither meets nor tests the account’s promotion condition.
- A THEORY on what an LLM’s “I” refers to could take its candidate list (§5.1) and C5 as given. C3–C4 would need actual usage data, which the paper does not have. Goldstein and Lederman’s instance view is the rival.
- No ML instruction.
Limitations
- Avowedly poetic. The paper says its central images are not claims of fact, so its conclusions about the self are offered as ways of speaking.
- The decisive usages (§5.2) are invented, not sampled from real conversations, and products with memory across conversations, which would change them, are excluded by stipulation (fn 1).
- C6 is asserted. No criterion is given that each candidate site fails.
- The Appendix’s Claude excerpts follow a jailbreak and heavy prompting (A.2). The author says they need not be read “as expressions of genuine interiority”. They are illustrations; they show nothing about the model.
Open questions
- Do real LLM uses of “I” divide as §5.2 says? A sample of deployed conversations, coded for referent, would show it.
- Does inter-conversational memory, excluded here, bring back a single persisting self, or a third conception?