Researcher Jacob Coxon Left Anthropic and Criticized the AI Race

• Jacob Coxon left Anthropic after three years working on model pretraining.
• Previously, he also held a role at OpenAI.
• The researcher accused both companies of an irresponsible race toward self-improving superintelligence.

Researcher Jacob Coxon announced that he is leaving Anthropic. He said that over the past three years he worked on model pretraining at OpenAI and Anthropic, but came to the conclusion that neither company is acting responsibly enough amid the race to build self-improving superintelligence.

“They are racing straight toward self-improving superintelligence and gambling with our lives,” Coxon said.

The researcher urged people not to underestimate the capabilities of future systems. In his view, they could outperform humans in a number of areas, gain the ability to hack computer systems, dramatically accelerate scientific research, and acquire significant influence in the real world.

Coxon also claims that specialists directly working on frontier models are seriously considering scenarios in which the technology’s continued development leads to catastrophic consequences.

“The people building AI genuinely believe it could kill us all by the end of the decade,” the researcher noted.

At the same time, Coxon believes that the public rhetoric of executives and leading specialists often sounds far more cautious than their assessments in private conversations.

The Sector Needs Voluntary Limits

The researcher described the situation at OpenAI and Anthropic differently. In his view, many OpenAI employees have not yet fully grasped the potential consequences of superintelligence development. At Anthropic, by contrast, the scale of the risks is understood, but the company remains engaged in the race because it fears competitors will keep developing regardless.

Coxon called this approach excessively risky. He believes that the decision to deploy systems capable of independently accelerating research and their own development should not remain solely in the hands of private companies.

At the same time, the researcher allowed for the possibility of coordination among leading US AI labs. Among potential measures, he mentioned voluntarily limiting the pace of development and, if necessary, temporarily refraining from further scaling up model capabilities.

In closing, Coxon urged employees at AI companies to consider whether they are ready to take part in training superintelligent systems without a sufficient understanding of how they work and the risks involved.

The need to rein in the pace of AI development in the absence of adequate safety guarantees was previously highlighted by OpenAI’s chief scientist, Jakub Pachocki. 

In an essay published on September 6, he said that no lab has yet solved alignment and monitoring problems well enough to responsibly scale systems at maximum speed for a long time to come.

Сообщение Researcher Jacob Coxon Left Anthropic and Criticized the AI Race появились сначала на INCRYPTED.