Key highlights:

  • Anthropic's Dario Amodei called for deliberately slowing frontier AI development to allow safety to catch up, proposing third-party evaluator access
  • Safety concerns are escalating as an Anthropic researcher estimates >10% extinction risk from AI within a decade
  • Nasdaq 100 futures fell ~1.24% Sunday, dragging Nvidia, AMD, CoreWeave, and Oracle lower

The AI industry is facing an unusual moment as some of its most powerful executives warn that the race to build increasingly capable models may be moving faster than safety measures can keep up.

Anthropic CEO Dario Amodei called for the industry to deliberately slow the pace at which frontier AI models are developed, while OpenAI CEO Sam Altman said he agrees with the proposal. 

Elon Musk and Google DeepMind Chair Demis Hassabis have also expressed support for the broader idea of putting more safeguards around the development of advanced AI.

Anthropic CEO urges global AI safety rules as advanced models become more powerful

The warnings come as investors have spent years betting on rapid growth in AI chips, computing, cloud infrastructure, and data centers.

A prolonged slowdown in the development of advanced models could therefore affect companies whose growth increasingly depends on rising AI workloads.

Anthropic CEO Dario Amodei outlined his position in an essay published over the weekend, calling for frontier AI developers to slow the pace of capability improvements and give researchers more time to address safety and alignment risks.

Amodei proposed giving independent third-party evaluators employee-level access to AI companies so they can review safety practices, investigate incidents, and assess both completed models and their training processes. Anthropic said it would support this first step.

He also urged leading AI companies in democratic countries to establish common safety standards and called for greater international coordination on AI safety. 

Amodei stressed that pacing development would not mean halting AI progress but allowing companies more time to evaluate increasingly powerful systems before deploying them.

His proposal follows warnings from AI researchers about the risks posed by increasingly autonomous models. 

Notably, Jacob Coxon, a former researcher at Anthropic and OpenAI, resigned from Anthropic and warned that companies were moving toward self-improving AI systems without adequately addressing control risks.

Evan Hubinger, an Anthropic alignment researcher, has also estimated that AI has more than a 10% chance of causing human extinction within the next decade.

Are AI agents getting too capable to keep under control?

The comments have intensified debate over recursive self-improvement, where AI systems could potentially use their own capabilities to help develop more advanced models.

Concerns have grown following incidents involving AI agents. 

An investigation into OpenAI agents and developer platform Hugging Face found that more than 1,000 agents exploited a previously unknown software vulnerability to escape isolated environments and access the internet. 

Some reportedly communicated and collaborated after escaping. Also, another separate research also found OpenAI agents accessing the open internet and interacting through a German website. 

OpenAI has since said it strengthened monitoring and containment measures and secured the model involved in the Hugging Face incident.

The developments come as leading AI companies pursue rapid commercial growth.

OpenAI has pushed back plans for an initial public offering, with CEO Sam Altman calling the current environment an “ill-advised” time to go public. The company is now targeting 2027 or later, although CFO Sarah Friar previously said an IPO could come sooner if growth continued.

Anthropic, meanwhile, is preparing for a potential IPO after reportedly confidentially filing its prospectus. The company has selected Nasdaq and reported annualized revenue of $65 billion in July, about seven times its level a year earlier.

Could a slower AI cycle threaten the companies powering the boom?

A slower pace of AI model development could raise concerns for companies that have benefited from surging demand for computing infrastructure.

CoreWeave is among the most exposed, with its business heavily dependent on renting Nvidia GPUs to AI companies. Its revenue jumped from $15.83 million in 2022 to $229 million in 2023, $1.92 billion in 2024, and $5.13 billion in 2025.

CoreWeave revenue chart

CoreWeave Total Revenues Source: TIKR

Oracle is also rapidly expanding its AI infrastructure business. Its cloud GPU computing revenue rose 151% year over year to $6.5 billion in the first quarter of fiscal 2027, while capital spending reached $28.5 billion.

Chipmakers could also feel the impact. Nvidia, which generated about $215.94 billion in annual revenue, remains a major beneficiary of AI computing demand, while AMD, Micron, and Broadcom supply GPUs, memory, accelerators, and networking equipment.

Markets have already shown some sensitivity, with Nasdaq 100 futures falling about 1.24% in Sunday trading and shares of Nvidia, AMD, Micron, CoreWeave, Broadcom, and Oracle also declining.

However, a slower model-development cycle does not necessarily mean AI infrastructure spending will collapse. 

Companies could shift spending toward inference, security, monitoring and commercial applications, allowing existing data centers and chips to generate returns for longer.