NOTE-169: Seth — Conscious AI and biological naturalism

Contribution

The paper gathers the case against conscious AI on current trajectories into one argument with two parts. The negative part (§§2–3) shows that the route to conscious AI presupposes computational functionalism plus silicon substrate flexibility, and gives reasons to doubt both from inside and outside a computational view. The positive part (§4) argues, without vitalism, that consciousness is tied to life through predictive processing, the free energy principle and autopoiesis. It then maps five scenarios for artificial consciousness by what each assumes (Table 1), and separates the ethics of real from seemingly conscious AI (§6).

Key insight

“Conscious AI” is not one hypothesis. It is a conjunction: computation suffices for consciousness, and the relevant computation can run in silicon. Brains are systems in which “it is hard and perhaps impossible to separate what they do from what they are” (§3.3), so the second conjunct can fail even if the first holds. If consciousness is grounded in a living system’s self-maintaining, free-energy-minimising regulation of its own body, then a simulation of that process is like a simulated rainstorm: nothing gets wet (§3.7).

Assumptions

• Nagel’s definition of consciousness (p. 2); a determinate “fact-of-the-matter” about whether something is conscious (p. 3); intelligence is “what a system can do” (Legg & Hutter).
• “AI” means digital (Turing) computation on CPUs, GPUs and TPUs (§1.1). The arguments target this and weaken as systems become neuromorphic or life-like.
• A “modest” physical Church–Turing thesis, not a “bold” one (fn. 6).
• Predictive processing is taken as a useful explanatory framework. The epistemic view “relies on accepting” it (§4.5).
• The FEP’s contested claims are taken at a favourable reading: free energy bounds surprisal under Laplace/Gaussian assumptions (fns. 33–34), and thermodynamic and variational free energy are “in some physical sense … the same thing” (§4.2), while the debate is acknowledged.
• Real-problem methodology: explaining, predicting and controlling properties of consciousness will dissolve the hard problem, by analogy with vitalism (§4.4).

Key results

• Biases (§2). Anthropocentrism, human exceptionalism and anthropomorphism, plus a “technophilic” Singularity temptation, conflate intelligence with consciousness. LLMs should be said to “confabulate”, not “hallucinate”.
• Conjunction (§3.3, §3.9). Computational functionalism is sufficient but not necessary for substrate flexibility, and conscious AI requires flexibility extending to silicon. Neural replacement begs the question (Block). Godfrey-Smith and Cao argue that generative entrenchment, nitric-oxide diffusion and metabolically regulated spiking (Chintaluri & Vogels 2023) make a behaviour-preserving silicon replacement implausible. Shiller argues that liberal functionalism over-attributes consciousness unless it is constrained in ways current networks fail.
• Broader computation (§3.4). Mortal computation cannot be separated from its hardware, and energy-efficient brains are unlikely to run “immortal” computations. Analogue and neuromorphic hardware likewise lose substrate flexibility.
• Separation of scales (§3.5). Causal and informational closure (Barnett & Seth 2023; Rosas et al. 2024) are guaranteed by design in computers but not in evolved systems, and they can be measured.
• Simulation ≠ realisation (§3.7). A simulation realises X only if computation is sufficient for X. Implementing the mechanism “at the right level of granularity would by definition realise consciousness”.
• Predictive processing (§4.1). Conscious contents are top-down “best guesses” (“controlled hallucination”). Interoceptive, instrumental inference serves allostasis. This gives epistemic biological naturalism and a possible ground state of “simply ‘being alive’”.
• FEP (§4.2). Autopoiesis corresponds to resisting the dispersion of internal states. Minimising variational free energy bounds surprisal, and via the Jarzynski equality, Landauer and Sengupta et al. (2013) this links to metabolic efficiency. Predictive processing grounded in a continuous quantity is argued to be “a dynamical, non-computational theory”.
• Hints (§4.3). Lane on membrane and mitochondrial potentials; Levin’s xenobots and anthrobots.
• Scenarios (§5, Table 1). (1) naïve along for the ride (implausible); (2) theory-based computational (needs computational functionalism plus silicon flexibility); (3) substrate-dependent computational; (4) substrate-dependent weak (non-computational functions, “weak” biological naturalism); (5) substrate-dependent strong (intrinsic properties of living material). Seth favours (5): “conscious AI would need to be ‘living’ AI”, though not necessarily carbon-based (§5.8).
• Ethics (§6). Real artificial consciousness risks “a mass inauguration of new forms of suffering”. It “should not be an explicit goal”, and credences should be used “to guide AI development away” from it (§6.1). Conscious-seeming AI is “almost certain”; the Müller-Lyer comparison suggests the impression may be cognitively impenetrable. It can exploit users, and it forces the “West World” dilemma: either distort the circle of moral concern or brutalise ourselves (§6.2).
• Conclusion (§7). “Our psychological biases are more likely to lead to false positives than false negatives.” LLMs “may be more similar to us in ways that do not matter for consciousness, and less similar in ways that do”.

Claims

id	claim	strength	support
C1	Conscious AI (digital, silicon) requires both computational functionalism and silicon substrate flexibility	strong	conceptual analysis (§3.3, §3.9)
C2	Silicon substrate flexibility is doubtful for the functions relevant to consciousness	moderate	plausibility arguments from generative entrenchment and neuroscience examples (Cao, Godfrey-Smith, Chintaluri & Vogels), mortal computation; Seth says they “do not disprove” it (§3.3)
C3	A simulation of the mechanisms of consciousness is guaranteed to realise consciousness only if computational functionalism is true	strong	conceptual (§3.7); nearly definitional
C4	Consciousness can only be understood in light of our being living, self-maintaining organisms (epistemic biological naturalism)	moderate	informal argument from interoceptive predictive processing (§4.1); conditional on predictive processing
C5	Consciousness depends on living substrate (ontological, strong biological naturalism)	weak	the FEP–autopoiesis “through-line” (§4.2, §5.5); relies on a contested equivalence the paper calls “much debated”; “not watertight” (§5.8)
C6	Real artificial consciousness is unlikely on current trajectories	weak–moderate	follows from C1–C2, and more strongly if C5 holds; stated as a credence, not derived
C7	Biological naturalism is “no more speculative than computational functionalism”	weak	one-line parity argument: all known conscious systems are alive (§7)
C8	Conscious-seeming AI is almost inevitable and may be cognitively impenetrable	weak	analogy to the Müller-Lyer illusion (§6.2); no data. Seth also says it “depends on the choices we collectively make” (§6.2)
C9	Human biases make false positives about AI consciousness likelier than false negatives	weak	argument from the §2 bias analysis; no empirical comparison
C10	Deliberately building real artificial consciousness is a “very bad idea”	moderate	ethical argument from suffering risk and the alignment complication (§6.1)

Concepts

• Computational functionalism: computations of some kind are sufficient to instantiate consciousness (§3.1). “Computation” means Turing computation unless stated otherwise.
• Substrate flexibility: consciousness is realisable in some, not necessarily all, non-carbon materials. Seth introduces it in place of unrestricted “substrate independence” (§3.3).
• Non-computational functionalism: mind or consciousness depends on functions that are not Turing-computable, such as continuous fields, fine timing or diffusing transmitters (Piccinini) (§3.1).
• Mortal computation: computation inseparable from the particular hardware running it; the opposite of software/hardware separability (§3.4).
• Simulation / implementation / realisation: abstracting a mechanism as computation / reproducing its finer-grained or non-computational aspects / instantiating the property itself (§3.7).
• Biological naturalism (Seth’s sense): consciousness is a property of only, but not necessarily all, living systems. It is distinct from Searle’s formulation, from biopsychism (all life conscious) and from carbon chauvinism (§4.0, §4.5).
• Epistemic vs ontological biological naturalism: consciousness understandable only in light of life / consciousness dependent on living substrate (§4.5).
• Computational biological naturalism: only living systems can implement the relevant computations. The phrase is Chalmers’; compare Wiese’s “weak computational functionalism” (§5.3, fn. 37).
• Conscious-seeming AI; West World scenario: systems we cannot help feeling are conscious, and the dilemma of caring for them or brutalising ourselves (§6.2).

Connections

Account of consciousness held. Seth holds biological naturalism grounded in predictive processing, the free energy principle and autopoiesis: the “beast machine” view, in which conscious experience integrates bodily and environmental signals in the service of allostatic regulation (§4.5). Methodologically he takes the “real problem” approach, not the hard problem. The consciousness tag is justified: consciousness and its attribution to machines is the whole subject.

Where Seth and Birch (LIT-111) disagree. Four points; the full statement is in the LIT-111 reading.

1. Whether conscious-seeming AI’s effects can be blocked. Seth (§6.2): the impression may be cognitively impenetrable, and expert knowledge will not inoculate us “from feeling that non-conscious systems are conscious, or from the consequences of doing so”. Birch (§5) grants the percept but says the inference to considered attribution is blockable, and builds his design nudges on that. Seth thinks the role-play framing “might only get us so far”; Birch makes role-play mode the central nudge.
2. The prior on AI consciousness. Seth: unlikely on current trajectories, with false positives the likelier error. Birch: both sides are speculative, and flicker/shoggoth hypotheses cannot be ruled out a priori for current LLMs.
3. Testability of the life link. Birch’s BBS commentary (abstract only read) says Seth’s case “rests on a worryingly speculative foundation: the free energy principle”, and that dependencies of consciousness markers on biological properties can be tested comparatively, with a pattern of failed predictions tilting toward computational functionalism. Seth frames his link as explanatory rather than brute (§4.4), which by Birch’s own distinction makes it testable. But Seth’s emphasised properties (autopoiesis, self-maintenance at every level) are shared by all life. His rejection of biopsychism leaves unspecified what distinguishes conscious life, which is where comparison would have to happen. Seth does name a partial empirical route: causal- and informational-closure measures could show whether neural dynamics abstract away from their substrate (§3.5).
4. Evidential value of architectural indicators. Seth’s §5.2 makes the indicator route (LIT-056) conditional on computational functionalism plus silicon flexibility. That is the stance of Birch’s Janus sceptic, for whom a global workspace found in an LLM would count against the theory, not for the AI.

Other held works.

• LIT-056 (Butlin et al.): Seth credits it as “an excellent overview” of the theory-based computational scenario but targets its explicit dependence on computational functionalism (§3.1, §5.2).
• LIT-096 (Nagel): Seth’s working definition (p. 2, quoting p. 436).
• LIT-185 (Barrett et al., IIT; Seth is a co-author): in §5.6 IIT stands apart as the theory that states sufficient conditions. On IIT, conventional feedforward AI is not conscious, “but there is also nothing special about life”. So IIT agrees with Seth’s verdict on current AI for a different reason and denies his biological premise.
• LIT-120 (Bruckner): uses autopoiesis as sufficient for welfare subjecthood without consciousness. Seth uses autopoiesis as the ground of consciousness but denies that all life is conscious. The two share the concept and invert its role, and together they suggest that autopoietic AI would raise welfare questions before, or without, consciousness questions.
• LIT-207 (Roberts): Roberts’ argument that interoceptive, affective and bodily-agentive states need an animate body parallels Seth’s interoceptive-inference case (§4.1). Roberts’ fictionalist engagement is a worked version of the role-play distance Seth thinks “might only get us so far”.
• LIT-159 (Schwitzgebel): Seth’s life requirement is one of the restrictive criteria Schwitzgebel’s argument forces a materialist to choose. Seth’s §3.3 discussion of Shiller’s “bucket of water” liberality worry is the same pressure from the other side.
• LIT-195 (Safron et al., life–mind continuity) sits in the same mind–life continuity lineage Seth cites (fn. 30, Godfrey-Smith; Kirchhoff & Froese).

Bearing on the record

No THEORY document in nucleation addresses this yet. The reading supports writing one on “conscious AI as a conjunction of computational functionalism and substrate flexibility”, with LIT-111 as its counterweight on testability. For ML practice there is no training instruction. §6.1’s advice to steer development away from consciousness-associated functions, and §6.2’s advice to restrict conscious-seeming systems to “well-justified use cases”, are governance positions, not ML practice for the Anthology of the SOTA. The §3.4 remark that mortal computation trades software immortality for energy efficiency touches hardware research, but the paper makes no engineering claim.

Note of 2026-10-16: THEORY-023 now cites LIT-135 as a source.

Limitations

• By the author’s own statement the case is “not watertight and may well be wrong in the details, or wrong altogether” (§5.8). Several key arguments are explicitly plausibility arguments that “do not disprove” computational functionalism (§3.3, §3.6).
• The positive case depends on the free energy principle’s contested claims: the thermodynamic/variational equivalence and the Markov-blanket reading of autopoiesis, which the paper flags as debated in §4.2. It leaves unanswered what separates conscious from non-conscious life (§4.5) and what counts as “living” beyond carbon (§4.5).
• The strongest-sounding claims (C7, C9, and the “almost certain” impenetrability in C8) carry the least support. Each rests on an analogy or a single observation.
• It does not engage the specific alien-consciousness hypotheses for LLMs (per-forward-pass flickers, implementation-level subjects) that LIT-111 raises. Its target is consciousness emerging “along for the ride” or by design from computational theories.
• The Author’s Response and the 50 commentaries were not read, so how Seth has since qualified or defended these claims is unverified.

Open questions

• Do causal and informational closure hold for neural dynamics (§3.5)? A measured failure of closure in brains, and its presence in silicon models, would bear directly on substrate flexibility.
• Which biological property separates conscious from non-conscious living systems, and does it vary across taxa enough for Birch-style comparative testing?
• Does the thermodynamic/variational free-energy equivalence hold “in some physical sense” or only formally?
• Would neuromorphic, mortal-computation or synthetic-biology systems (scenarios 3–5) satisfy the conditions, and how would we know?
• What does the Author’s Response concede or change, given its abstract’s stress on a research programme on “the substrate dependence of both consciousness and cognition”?

Corrections to the seeded skim

• The dossier’s §5 summary reads “scenarios run from ‘naïve along for the ride’ through theory-based computational and substrate-dependent (weak and strong) to embodiment”. Table 1 has five scenarios. The dossier omits §5.3 “substrate-dependent computational”, which covers mortal, neuromorphic, neural and biological computation and Chalmers’ phrase “computational biological naturalism”. Embodiment (§5.7) is not a scenario but a consideration that cuts across them, and §5.6 separately places IIT and quantum theories off the functionalism–naturalism spectrum. IIT would rule out conventional AI “but there is also nothing special about life”.
• The dossier’s §3 list omits §3.4, broader forms of computation. There Seth calls the mortal-computation argument (Hinton 2022; Ororbia & Friston 2023) “a provocative argument against the plausibility of conscious AI because it is based on limitations arising from within a computational view of mind” (p. 10). He cites Kleiner (2024) as reaching a similar conclusion by proof, and fn. 38 identifies a tension in that argument. The list also omits §3.8, which extends the substrate argument to intelligence: some distinctively biological intelligence may escape simulation, though intelligence, being about doing, can be realised by good-enough simulation where consciousness cannot unless computational functionalism is true.
• On the dossier’s open question, whether the free-energy/autopoiesis argument gives necessity or only an explanatory bridge: the text claims plausibility, not necessity. §4.4: “If features of being alive explain (and predict and control) features of being conscious, then a necessary relation between the two becomes more plausible.” §4.5 calls the epistemic view “the easiest to defend, though it relies on accepting predictive processing”. The ontological view is where the case is “drawn towards” (§5.5, §5.8), not where it is established. §4.2 concedes that the thermodynamic/variational free-energy equivalence and the FEP–autopoiesis link are debated. The conclusions say “at least some good (non-vitalistic) reasons to suppose that consciousness may be a property of, and perhaps only of, living systems” (§7).
• The dossier’s quote “no more speculative than computational functionalism” is accurate, but its support is one sentence: “Simply noting that all known examples of conscious systems are biological and alive” (§7). It is a parity claim, not a result of the paper’s argument.
• Seth’s definition of biological naturalism differs from Searle’s, and Seth says so (§4.0). Searle’s is “conscious states are higher-level irreducible properties of lower-level neurobiological states”. Seth’s is “consciousness is a property of only (but not necessarily all) living systems”. The dossier and LIT-111 (Birch) use the term more broadly, for any view on which biological properties matter beyond implementing computation.
• Venue and date check out: BBS 49 (2026), e315, online 21 Apr 2025. The Author’s Response is BBS 49, e366 (online 17 Sep 2026). There are 50 commentaries, including Birch’s “Free energy, consciousness, and the inverse dark room problem” (e320).
• Tags: consciousness is right as primary topic. ethics is justifiably appropriate as an added tag, since §6 is a sustained ethical section on real and seeming AI consciousness.