1 hour ago
37
The latest catalyst: Jacob Coxon, a safety researcher at Anthropic, resigned on September 9, warning that leading AI labs are barreling toward self-improving superintelligence without adequate safeguards.
The existential alarm bells keep getting louder
Coxon’s resignation didn’t happen in a vacuum. Anthropic CEO Dario Amodei published a 3,800-word essay calling for a global slowdown of AI development. The piece cited a buffet of nightmare scenarios: loss of control over advanced systems, mass economic disruption, and the potential for bioterrorism enabled by increasingly capable models.
Paul Christiano, a respected voice in AI alignment research, went further. He stated the industry is on course for near-term catastrophic loss of control, with some researchers estimating a greater than 10% probability of human extinction within a decade.
These warnings build on a foundation laid more than a decade ago by figures like Stephen Hawking and Elon Musk, and they’ve been reinforced by surveys of AI researchers that consistently peg the median estimate for extremely negative outcomes at around 5%.
The case for looking at what’s already broken
The existential framing has drawn sharp criticism from researchers who study AI’s real-world impacts today. Timnit Gebru, who has long advocated for accountability in AI development, argues that fixating on hypothetical extinction scenarios does a disservice to people already being harmed by these systems.
Milton Mueller and Tristan Harris have echoed this critique, pointing out that extinction-focused narratives can actually serve the interests of the very companies stoking them.
The focus on speculative catastrophe, critics argue, creates a convenient distraction from the governance failures that are measurable, documentable, and fixable right now.
There’s an uncomfortable irony here. The labs warning loudest about existential risk are also the ones racing hardest to build the technology they claim to fear. Anthropic markets itself as the “safety-focused” AI company, yet one of its own safety researchers just walked out the door saying the approach isn’t working. OpenAI dissolved its superalignment team in 2024, and the competitive pressure between these firms has only intensified since.
Regulation remains conspicuously absent
The competitive environment among AI labs continues to escalate without significant intervention from Washington. No binding federal legislation governs the development of frontier AI systems. No agency has clear jurisdiction over the kind of self-improving architectures that Coxon warned about. The EU’s AI Act remains the most comprehensive framework in existence, but its enforcement mechanisms are still catching up to the pace of development.
Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy.
English (US) ·