Contribution
The author’s own update of the risk estimates in The Precipice. It takes four of the largest risks (climate change, nuclear war, pandemics, AI) and asks what changed between the book’s text being fixed and early 2024. It is a talk, not a paper: no new estimates, no sources given, and no moral argument. Its use to the record is as a reliable restatement of the book’s headline numbers and some of its framing, by the book’s author.
Key insight
Risk estimates change in two ways, and they should be kept apart. “Sometimes we have made the risks lower (or higher), sometimes we’ve discovered that they always were” (Climate Change). Lower emissions are a causal change, brought about by human response. A narrower climate sensitivity is an evidential one: the risk was always lower than feared.
Assumptions
- Existential risk is human extinction “and other ways that humanity’s entire potential could be permanently lost” (opening). The talk sets aside other harms from the same sources.
- Much of an existential risk sits in the tail. For climate, “much of the risk comes from the worst case possibilities”, so narrowing a range lowers existential risk even when the middle stays put.
- The book’s numbers are rounded, “usually to the nearest factor of 10” (Conclusions), so moderate changes do not move them.
Key results
- Climate change: down.
- Emissions: “we are not at all on RCP 8.5”; the path is “headed somewhere between the lower two paths”. The figure shows the four RCPs (2.6, 4.5, 6.0, 8.5) to 2100.
- Climate sensitivity: when the book was written, the IPCC’s likely range was 1.5–4.5°C, a two-thirds range, so about a 1 in 6 chance of exceeding 4.5°C, “where a lot of the existential risk was coming from”. The range was first proposed the year Ord was born and did not change for 40 years. AR6 narrows it to 2.5–4°C and gives a 95% chance the sensitivity is below 5°C.
- Nuclear: up.
- The Russian invasion of Ukraine brought nuclear powers “substantially closer to the brink”: calls for a no-fly zone, threats of retaliation, raised readiness. Not Cuban-missile-crisis level.
- New START was renewed at the last minute and cannot be renewed again; it expires two years from the talk. Ord thinks rising stockpiles “the most likely possibility”, after a decline since 1986 (figure: totals peak above 60,000 in the mid-1980s and fall to about 10,000 by 2020).
- In the book, one reason nuclear risk was not rated higher was that with current arsenals there is no known scientific pathway from an all-out war to extinction “(nor a solid story about how it could permanently end civilisation)”. Larger arsenals weaken that reason.
- The MacArthur Foundation, about 45% of funding for nuclear peace and arms reduction, withdrew; funding for the field halved.
- Pandemics: mixed.
- Covid was “not an existential risk” but had four implications. It exposed weaknesses: unity against a common threat was “quite a temporary phase — you get a few months of that”, then the response polarised “along existing national and political lines”; the WHO and CDC fell short. It was a warning shot that “punctured” our “air of invincibility”, yet produced less response than expected: no panic stage, and weak biodefence spending in the UK, worse in the US. Vaccines arrived within a year, where the fastest before had taken 10 years (figure: measles, 1953–1963). And gain-of-function research continues, though credible inquiries into Covid’s origin are “about evenly split” on a lab escape that may have killed “more than 25 million people”.
- Two protective technologies: metagenomic sequencing, which could flag a novel pathogen from a patient or wastewater, and clean air (filtration, germicidal and far-UVC light). They cover novel and engineered pathogens, need little behaviour change, help against everyday disease, and are not dual use.
- AI in biology: AlphaFold took known protein structures from about 50,000 to 200 million. Language models extend the democratisation of biotechnology, so unless the abilities are removed they “will indeed increase the risk from bioterrorism”.
- AI: mixed.
- From reinforcement-learning agents (AlphaGo, AlphaZero, AlphaStar, OpenAI Five) to generative language models. These are not inherently agents, so “the classical threat scenario of Yudkowsky & Bostrom (the one I focused on in The Precipice) doesn’t directly apply”. They have read about a trillion words, so “the challenge is no longer getting them enough information about ethics, but about making them care”. Trained on human text, their abilities may plateau near the human range instead of passing through it quickly (three schematic figures). Their compute needs tie labs to corporate partners.
- Racing: from 2023 a race between “the largest companies on Earth”. Microsoft’s search chatbot threatened users; Google merged Brain and DeepMind.
- Governance: US chip export controls (October 2022); ChatGPT; Hinton and Bengio’s warnings and the extinction-risk statement (May 2023); The Elders; the US executive order; the Bletchley Declaration (28 countries and the EU); US and UK AI safety institutes.
- Net: language models mixed, racing bad, governance good.
- Conclusions. The book’s per-century estimates were 1 in 1,000 for climate and for nuclear, 1 in 30 for pandemics, 1 in 10 for unaligned AI. None has “moved that far”. Climate down, nuclear up, pandemics and AI “no clear direction”. The book’s “existential security” is “a hypothetical future state of the world where existential risk is low and kept low”. The current work is putting out fires; the longer work is “establishing the moral seriousness of existential risk”, then norms, treaties and governance, over decades. The UN and The Elders now treat existential risk as a priority; a British prime minister quoted The Precipice at the UN.
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | Existential risk from climate change is lower than when the book was written, because emissions avoid the worst pathway and the IPCC has narrowed climate sensitivity | moderate: reports IPCC ranges; the step to existential risk is Ord’s judgement | Climate Change |
| C2 | Nuclear existential risk is up, through war between nuclear powers by proxy, a likely new arms race and halved civil-society funding | moderate: events are public; the size of the change is unquantified | Nuclear |
| C3 | Unity against a visible common threat lasts months, then divides along existing national and political lines | weak: one case, Ord’s observation, no data | Pandemics |
| C4 | Language models weaken the book’s main AI threat model because they are not inherently agents and have abundant information about human values | weak to moderate: an argument, with Ord’s own caveat that agents can be built from them | AI |
| C5 | Racing among the largest companies raises AI risk and improved governance lowers it, with no clear net direction | weak: judgement | AI |
| C6 | The book’s rounded estimates stand | moderate as a report of Ord’s view | Conclusions |
Concepts
- existential risk: extinction or another permanent loss of humanity’s entire potential.
- causal versus evidential change in risk: risk made lower (or higher) by action, versus risk discovered to have been lower (or higher).
- existential security: a future state in which existential risk is low and kept low, with “the mechanisms to ensure fire no longer poses a substantive threat”.
- panic–neglect cycle: the expected pattern after a crisis, too much caution then too little. Covid, Ord says, skipped the panic.
Connections
- The Precipice (LIT-1678), unread. See section 3. The talk quotes one passage of the book (on great-power war “this century”), and reports the four risk estimates, the focus on the Yudkowsky–Bostrom agent scenario, the reasoning behind the nuclear estimate, and the notion of existential security.
- Thorstad (NOTE-1308) reports other numbers from the book: one in six overall this century and one in 10,000 from natural risks. The talk does not restate them. It gives no new total, so whether the one in six still stands is left to inference from “none of them have moved that far”.
- Ord’s chapter (NOTE-1317-ordtraj), the companion reading: the framework in which reducing near-term existential risk multiplies the whole future’s value.
- Bostrom 2013 (NOTE-1321) found a “deficit of motivation” for existential safety. The talk’s Covid observations (no panic stage, biodefence spending weak, indifference to the lab-leak question) are anecdotal evidence of the same deficit after a warning shot.
Bearing on the record
- QUESTION-299. Little directly. The talk is about how large the risks are, not what is owed or why. It confirms the empirical premise of the impersonal ground as Ord holds it: risk this century is high, and he sees no reason to lower it. Its programme for existential security runs through norms, treaties and institutions, “establishing the moral seriousness of existential risk”, not through anyone’s concern for future people as such.
- The owner’s thesis (CLAIM-491). These are inferences; Ord says
nothing about selves or motivation.
- Ord’s Covid observation bears on a humanity-wide self. A visible common threat produced humanity-level unity “at first”, then the response split along national and political lines within months. Self-categorization theory (THEORY-1114) would predict that an outside threat makes the most inclusive category salient, and that salience is fixed by context and does not last when the context changes. If so, “humanity” is available as a self only while something makes it salient. A standing concern for far-future people could not rest on it.
- The talk confirms that the book’s case is not motivational. Its remedy for a deficit of concern is institutions that hold when concern fades.
- Ord’s line that the AI problem is “no longer getting them enough information about ethics, but about making them care” is about machines. It is not evidence about people, and is noted only because it is the same distinction CLAIM-492 turns on: knowing what is good is not caring about it.
- The triviality objection (CLAIM-492). Not addressed.
- No dated note on CLAIM-491. The one point (unity against a common threat is brief) is one case, observed and not measured. It belongs as a line in the dated note on QUESTION-299 (section 4).
- No THEORY filed.
- No ML instruction. The AI section describes the field; it gives no instruction for machine-learning practice.
Limitations
- A talk, lightly edited. No references; figures are unsourced on the page (the RCP, stockpile and vaccine charts are recognisable standard charts, but the post does not cite them).
- No new numbers. Directions only. The overall one in six is not restated or revised.
- Dated to the talk. “Now” is February 2024, though posted in July.
- Not the book. It quotes the book once and reports its estimates; the book’s moral argument is absent.
- Typos. “beleivable”; “issomething”; “and and Yoshua Bengio”; “Le Sedol” for Lee Sedol; “MRNA” for mRNA; “the typical time from vaccine development” for “for”; “it may be possible for first labs” (missing “the”), and that sentence lacks its full stop.
Open questions
- Does the one-in-six total still stand on Ord’s view? The talk implies so without saying it.
- Was the brevity of humanity-wide unity in Covid measured? A study of superordinate identification (“humanity”, “all people”) across the pandemic would test whether a threat-made human self fades as self-categorization theory predicts.