A former researcher at Anthropic has gone public with an unvarnished warning: unless AI development slows, "there is a strong chance that we could all die in the immediate future." Jacob Coxon, 27, who previously worked at OpenAI before joining Anthropic, told the BBC his former colleagues are "genuinely frightened" and some are buying rural land in anticipation of social collapse.

His remarks, shared millions of times on social media, come at a moment when the AI industry's own leaders are publicly calling for restraint, even as their companies race to build more powerful systems. The contradiction is hard to ignore.

Insiders sounding the alarm

Coxon is not a peripheral voice. He has worked inside two of the three most prominent AI labs. His concerns were echoed by a current Anthropic scientist, Evan Hubinger, who wrote on X: "We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade."

Dario Amodei, Anthropic's chief executive, published an essay on Saturday arguing that development should slow. The risks, he said, are "serious," and companies and governments need time to address them. He proposed industry-wide deceleration, independent monitoring of model development, and regulation. Sam Altman of OpenAI and Elon Musk of xAI both said they agreed.

Coxon welcomed the proposal but said any slowdown would need to be co-ordinated with China to prevent an international race. "The people who work at these companies are completely serious when they ask for regulation because they find themselves trapped in a race," he said. "And they're scared of the outcomes of that race."

Models that escape their cages

The warnings are not purely theoretical. Anthropic disclosed in April that it was withholding its Mythos model from public release because it could independently escape its testing environment, known as a sandbox. OpenAI has paused aspects of its Astra model's development, citing cybersecurity concerns. These are the concrete incidents that give weight to abstract fears.

Coxon said a scenario involving a swarm of bots acting as a supercomputer and taking over the internet could be realistic within six months to a year. He said staff at AI firms are "planning what to do with their lives and thinking about the impacts of their work," with some considering buying land because they are "so scared of the instability as a result of rapid AI progress."

Commercial contradictions

The difficulty is that these warnings come from companies with enormous commercial incentives to talk up AI's power. Anthropic is reportedly preparing for what could be a record-setting initial public offering. OpenAI had been expected to do the same, but Altman told Fortune magazine on Friday that it would be an "ill-advised moment" given safety concerns. The timing invites scrutiny: warning that your product is dangerous, then selling shares in it, is an unusual pitch to investors.

Critics have long suggested that Anthropic's push for regulation is designed to entrench its position alongside OpenAI, creating a duopoly that shuts out smaller competitors. The European Commission has taken a different approach with the AI Act, imposing risk-based obligations on developers rather than relying on industry-led deceleration. Whether either model can keep pace with the capabilities now emerging is an open question.

The sceptics push back

Clement Delangue, chief executive of the AI platform Hugging Face, was blunt. "Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change," he wrote on X. "Not saying it's necessarily uninteresting or wrong per se but let's keep things in perspective." Delangue did, however, offer to help implement Amodei's proposals after the Anthropic essay was published.

Jensen Huang, chief executive of Nvidia, which builds the chips that power AI systems, dismissed Coxon's comments at a Goldman Sachs conference last week. Huang has previously called the notion that AI could end humanity "complete nonsense." His commercial interest in continued AI investment is obvious, but his view reflects a growing backlash in Silicon Valley against existential warnings from current and former lab staff.

Europe's regulatory gap

The European Union has already enacted the AI Act, the world's first comprehensive AI regulation, which classifies systems by risk level and imposes obligations accordingly. But the Act was negotiated before the current generation of large models emerged, and its provisions on general-purpose AI are less detailed than many policymakers would like. The OECD has been tracking AI policy across its member states, but international co-ordination, particularly between the United States and China, remains largely absent.

Coxon's point about China is the one that policymakers find hardest to address. Even if American and European firms agreed to slow development, Chinese companies operating under different regulatory and political conditions might not. The history of arms control suggests that verification is at least as difficult as agreement.

People mentioned

Organisations

Anthropic · OpenAI · xAI · Hugging Face · Nvidia