Contribution
It fixes the concept of existential risk, places it on a grid of scope and severity, and classifies it. It argues that on many theories of value reducing it is the dominant priority. It adds the normative and motivational analysis that the 2003 paper lacked. It also reframes sustainability as a trajectory, not a state.
Key insight
What makes an existential catastrophe bad is not the deaths but that it would “destroy the future” (p. 17). Parfit’s comparison of peace, a war that kills 99% and one that kills 100% puts the point (pp. 17–18, quoting Parfit 1984, pp. 453–454). The gap between the last two is the gap between the history so far and all the history to come.
Assumptions
- Aggregative, time-neutral value (p. 16). Lives usually have significant positive value, two similar lives are worth twice one, and value does not depend “on when it occurs or on whether it already exists or is yet to be brought into existence”. Bostrom calls this “the simplest case” and does not defend it.
- Epistemic probability: the credence an ideally reasonable observer should assign on current evidence (p. 16). What cannot be known safe is risky in the sense relevant to decision.
- Humanity means Earth-originating intelligent life, not Homo sapiens (pp. 19–20).
Key results
- The size of the risk (pp. 15–16). Estimates of 10–20% total existential risk this century are “fairly typical” among those who have examined it, and rest on subjective judgement. One informal expert poll gave a median 19% for extinction before 2100 (p. 28, n. 1). Natural risks are small; the bulk is anthropogenic and technological.
- Meta-level uncertainty (p. 16). If an analysis puts a catastrophe’s probability very low, the chance the analysis is flawed may be higher, and most of the risk may sit there (Ord, Hillerbrand and Sandberg 2010).
- Risk categories (p. 17, Figure 2). Scope (personal to pan-generational) by severity (imperceptible to crushing). Existential risks are at least crushing and pan-generational.
- The expected loss (pp. 18–19).
- Earth alone, a billion people for a billion years: at least 10^16 lives. So a millionth of one percentage point of risk reduction is worth at least a hundred times a million lives.
- Biological lives in the accessible universe: at least 10^34 life-years. Emulations: at least 10^54 subjective life-years, or 10^52 ordinary lives.
- Maxipok (p. 19). “Maximise the probability of an ‘OK outcome’, where an OK outcome is any outcome that avoids existential catastrophe.” It holds “whenever we act out of an impersonal concern for humankind as a whole”. It differs from maximin, which would have us “start partying as if there were no tomorrow”.
- Four classes (pp. 19–22, Table 1). Human extinction (premature, that is, before technological maturity); permanent stagnation (unrecovered collapse, plateauing, recurrent collapse); flawed realisation (unconsummated, ephemeral); subsequent ruination. The last two are defined normatively (p. 24).
- Capability and value (pp. 22–23). Long-run efficiency matters more than early output; even a delay of several million years is “brief”. “Even from a so-called ‘person-affecting’ moral perspective”, one should judge flawed realisation by the long integral of value, since digital or repaired agents could live indefinitely (p. 23).
- Other ethical perspectives (pp. 23–24).
- Adams (1989, pp. 472–473): a commitment to the future of humanity as “a vast project, or network of overlapping projects, that is generally shared by the human race”. “We owe, I think, some loyalty to this project of the human future. We also owe it a respect that we would owe it even if we were not of the human race ourselves.”
- Preference satisfaction; democratic deliberation, “even if hypothetical future people are not included as stakeholders”.
- Custodial duties: “We do not want to be the failing link in the chain of generations” (p. 23).
- Theological views and intuitions about genocide scaling up to omnicide.
- It is “on no account a conceptual truth” that existential catastrophe is bad (p. 24).
- Normative uncertainty and sustainability (pp. 24–26). If unsure whether X is bad but sure it is not good, X is bad in the decision sense. Option value favours reaching a state of more wisdom, coordination and capability with uncorrupted values. Like a rocket in mid-air, humanity should seek a sustainable trajectory, not a sustainable state.
- Barriers (pp. 26–27).
- Academic neglect: fewer Scopus papers on “human extinction” than on “dung beetle”, “star trek” or “zinc oxalate” (Figure 6, August 2012).
- “Mitigation of existential risk is hampered by a lack of understanding, but also by a deficit of motivation” (p. 26). It is a pan-generational public good: even a world state captures only the benefits to people now alive. “The quadrillions of happy people who may come to exist … would be willing to pay the present generation astronomical sums”, but the trade is impossible.
- “Moral motivations, too, may fail to measure up”: scope insensitivity, quoting Yudkowsky on extinction entering “a ‘separate magisterium’” (pp. 26–27).
- Grounds for optimism (pp. 27–28). The ideas are new; political integration has widened over history; movements for peace, the environment and human rights might take the concern on board. Bostrom doubts that the prospect of extinction motivates those movements now: “Perhaps the amount of concern would be roughly the same even in the face of an iron-clad guarantee that any catastrophe would stop short of human extinction” (p. 29, n. 37).
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | Existential risk this century is significant | weak: subjective estimates and one informal poll, which Bostrom flags as such | pp. 15–16, n. 1 |
| C2 | On aggregative, time-neutral views the expected loss from existential catastrophe exceeds the value of 10^16 lives | strong conditional on the axiology; the number is a lower bound on Earth alone | pp. 18–19 |
| C3 | Maxipok is a useful rule of thumb for impersonal moral action | moderate: follows from C2 for that kind of action; Bostrom limits it himself | p. 19 |
| C4 | Many ethical views, not only utilitarian ones, support existential risk reduction | weak to moderate: a survey of perspectives without derivation; the priority varies | pp. 23–24 |
| C5 | Existential risk reduction is undersupplied for lack of understanding and of motivation | moderate as theory (public goods, scope insensitivity); the motivational claim is not tested | pp. 26–27 |
Concepts
- existential risk: “one that threatens the premature extinction of Earth-originating intelligent life or the permanent and drastic destruction of its potential for desirable future development” (p. 15).
- maxipok: maximise the probability of an OK outcome, one avoiding existential catastrophe (p. 19).
- technological maturity: capabilities giving productivity and control over nature close to the feasible maximum (p. 19).
- permanent stagnation, flawed realisation, subsequent ruination: the three non-extinction classes (Table 1).
- sustainable trajectory: a path that minimises existential risk, against a “sustainable state” one could stay in (pp. 25–26).
- pan-generational public good: one whose benefits fall mostly on future generations, who cannot pay for it (p. 26).
Connections
- Bostrom 2003 (LIT-1739). The arithmetic is restated with new figures, and the 2003 claim that a delay of millions of years is “brief” is repeated (p. 22).
- Greaves and MacAskill (LIT-1684) cite p. 18 for the value of tiny reductions in extinction risk.
- Parfit (LIT-469) is quoted for the 99%/100% comparison (pp. 17–18).
- Scheffler (LIT-1712, LIT-1754). Adams’ shared project and the “chain of generations” are a ground in what the living value, the kind QUESTION-299 assigns to Scheffler.
- Schubert et al. (LIT-1748). Bostrom predicts scope-insensitive lay judgement of extinction; that study tests it.
- Gardiner (LIT-1681). The pan-generational public good is the same structure as Gardiner’s intergenerational storm.
Bearing on the record
- The owner’s thesis (CLAIM-491). This is the longtermist text
that says most about motivation, and what it says fits the thesis.
These are inferences; Bostrom does not discuss the self.
- The impersonal ground is named as such. Maxipok is for “an impersonal concern for humankind as a whole” (p. 19). Bostrom grants that “Unrestricted altruism is not so common” (p. 19). On his account the obligation outruns the motive.
- His diagnosis of the gap is the thesis’s prediction. Each producer of safety, even a world state, captures only the share that accrues to people now alive (p. 26). So present people would supply safety in proportion to their own stake, and the future’s beneficiaries cannot pay. Bostrom frames this as a market failure. Read through the thesis, it says concern reaches as far as the self reaches, and no further.
- Adams splits the ground in two. “Loyalty” to humanity’s project is owed by its members, as a commitment continuous with “the projects of our own personal futures”. “Respect” is owed “even if we were not of the human race ourselves”. The first is a temporally extended collective self of the thesis’s kind. The second is the impersonal ground and its hard case.
- Note 37 is a test the thesis needs. If concern for catastrophe would be the same with a guarantee against extinction, then those movements’ concern is for people now alive, not for the far future.
- The triviality objection (CLAIM-492). The “failing link in the chain of generations” (p. 23) suggests a criterion of extent apart from caring: felt membership in a lineage. It is a phrase, not a measure.
- Proposed dated note on CLAIM-491: “Bostrom (2013; NOTE of 2026-10-29) grounds existential-risk priority in ‘an impersonal concern for humankind as a whole’ and grants that ‘unrestricted altruism is not so common’. He explains its undersupply by the fact that people now alive capture only their own share of the benefit and moral sentiment is insensitive to scope. That is what the thesis predicts: concern reaches as far as the self does. Through Adams he also offers a ground that fits the thesis, loyalty to humanity’s shared project continuous with one’s own, beside one that does not, respect owed even by outsiders.”
- No THEORY filed.
- No ML instruction. Machine superintelligence appears as a risk area, not as practice.
Limitations
- The risk estimates are subjective, as Bostrom says (p. 15, n. 1).
- The axiology is assumed, not argued (p. 16), and the other perspectives are surveyed in a page.
- The motivational claims are untested. Scope insensitivity is cited to Yudkowsky’s review; note 37 is a conjecture.
- An understated figure. On p. 19, a 1% chance of 10^52 lives and a risk reduction of a billionth of a billionth of a percentage point give 10^30 lives in expectation; the text says “a hundred billion times as much as a billion human lives”, 10^20. The understatement does not hurt the argument.
- Typos and slips.
- “the potential exist for” (p. 18).
- “a civilisation starting could reach” (p. 22).
- Figure 2’s note lacks a closing parenthesis after “numerous generations” (p. 17).
- Note 10 promises “section 5”, but the paper has four sections (p. 28).
- In the references, “Accessedviewed” and “Cambrige” (p. 30); Schröder and Smith (2008) is given as volume 368, which appears to be 386 (p. 30).
Open questions
- Does concern about catastrophe track the loss of the far future or the deaths of people now alive? Note 37’s guarantee is an experiment: Schubert et al. (LIT-1748) may run a version of it.
- Is Adams’ “loyalty” to the human project measurable as identification with humanity over time, and does it predict support for risk reduction apart from impersonal concern?