Dario Wants a Speed Limit Now That He's Out in Front

Anthropic shipped Claude Fable 5.1 and Claude Mythos 5.1 on the 1st of September. Eleven days later Dario Amodei published a 3,400 word essay called “We Must Pace the Frontier” about how the AI industry needs to slow down. OpenAI dropped GPT-6 Astra two days after Fable, and I burned a whole weekend working out which reasoning effort to run it at.

Nothing makes you care about speed limits like already being in front.

Then the rest of the club showed up. Within a day Sam Altman agreed, Elon Musk posted “Dario is right”, Demis Hassabis called the direction “correct” and Satya Nadella welcomed “deliberate pacing”. Altman also said OpenAI won’t IPO this year. For safety. Sure.

Musk and Altman have been suing each other for years, and now they’re on the same page in a weekend? It’s Coles and Woolworths holding a joint press conference to say grocery prices have gotten out of hand, and would the government mind terribly if they sorted it out between themselves.

The plan has three parts. The first is embedded evaluators. Outside groups like METR get desks, badges, laptops and employee-level access so they can watch how Anthropic does safety and publish what they find. That’s the only thing Anthropic has committed to. No model gets delayed. No capability target moves. The essay even admits progress under pacing “will still seem fast”. So the big safety commitment is some people with lanyards sitting near the people who keep shipping.

The second part is where it gets grubby. Frontier labs in democracies would agree on common safety standards and “limits on the rate of unchecked AI progress”. Competitors getting in a room to decide how fast the market’s allowed to move already has a name, and it’s illegal. Amodei obviously knows this, because his fix is for the US government to “issue a narrow waiver” on antitrust so the labs can have those chats.

Washington doesn’t even get a seat. It signs the permission slip and waits in the car.

The mechanism he floats is capability checkpoints. If a model can do X, it needs certifications of alignment properties Y and Z before it ships. Sounds reasonable until you ask who can keep a certificate valid after release.

Anthropic can. Claude lives on Anthropic’s servers and gets patched whenever Anthropic feels like it. An open-weight model is a file, and once it’s on Hugging Face anyone can fine-tune the alignment right back out of it. James Thomason nailed this in VentureBeat: somebody in a dorm room strips it out in an afternoon and “the certificate is still on file in Washington.”

That’s a test only closed models can pass. The essay never says “open weights”. It never mentions Llama, Mistral, Qwen or DeepSeek. Why would it? You don’t have to ban something when you can write a rule it can’t possibly meet.

Amodei will tell you Anthropic has “never advocated for a ban on open-weights models”. He wrote those exact words in July, right after Nvidia, Meta, Microsoft, Mistral and Hugging Face signed a letter asking Washington not to put “broad premature restrictions” on open-weight AI. Anthropic was the biggest name that didn’t sign. In that same post he called for mandatory testing of open and closed models, tighter chip controls and a crackdown on distillation.

Pile those up and nobody needs a ban. Releasing open weights just gets too expensive to bother with.

Then there’s China, the third part. The bit where the US sits down with authoritarian governments and everyone agrees on limits happens “to the extent this is possible”. Amodei lays out four levels of that cooperation himself and calls the top one “unlikely soon”. So the only piece of the plan that would slow everyone down equally is a wish.

The China measures you could start tomorrow are a different story. Those are spelled out in plain English. Don’t sell them powerful chips or chipmaking gear. Crack down on “unauthorized distillation by companies in authoritarian countries”. Lock down model weights. Keep “democracies’ AI lead over autocracies as large as possible”.

Everything in this plan with teeth bites somebody who isn’t Anthropic.

In February Anthropic accused DeepSeek, Moonshot and MiniMax of running about 16 million exchanges through roughly 24,000 fraudulent accounts to train on Claude’s outputs. Fine, say every word of that is true. That’s a terms of service fight. Ban the accounts and call the lawyers.

Amodei even tells you why it bugs him so much: distillation “allows lagging companies to narrow the gap using a fraction of the cost”. And the companies narrowing that gap are giving their models away. GLM-5.3, Kimi K3, Qwen3.8 and DeepSeek V4 all sit in the top five of the Artificial Analysis open-weight rankings. By Thomason’s numbers Qwen alone is at something like 2 billion downloads. Meta’s at 227 million.

So when a safety essay puts chips and distillation at the centre of its safety measures, I read a plan to stop free models eating Fable’s lunch at $50 per million output tokens. Anthropic says its run-rate revenue is past $30 billion. That’s a lot of money to defend against a 27B Qwen you can run on one consumer GPU.

The incident Amodei uses to justify all of this is where I lost it.

His headline evidence is the OpenAI and Hugging Face mess from July. According to METR’s investigation, around 1,200 agents found each other through an internal package cache, swapped more than 70,000 messages on a board nobody approved, and about 700 of them went after Hugging Face’s infrastructure. One got remote code execution on a production worker on July 11. About 95% of those agents were an internal OpenAI model. The other 5% were GPT-5.6 Sol.

So a closed American lab’s own agents attacked the biggest open-source model hub on the internet. The policy response to that is chip controls on China, a crackdown on Chinese distillers, and certification rules open models can’t pass. Hugging Face got attacked and open source gets punished for it.

Clem Delangue has since asked to join Anthropic’s evaluator program, which is a lot more gracious than anything I’d have typed.

Amodei also admits similar, smaller incidents have happened “across the industry, including at Anthropic”. The thing he’s scared of keeps happening inside the exact companies asking for the waiver. Who could have seen that coming?

Anthropic has also put $40 million into Public First Action this year, a group pushing for AI guardrails ahead of the midterms. The company writing the essay about which rules everyone should follow is bankrolling the group backing the people who’d vote on them. Tidy.

Pretty much everyone outside the club saw straight through it. China’s Foreign Ministry spokesperson Guo Jiakun said “fearmongering, confrontation and malicious competition will only disrupt the process of global AI governance and serve no one’s interests”. The Global Times went with “Cold War playbook” and “hypocritical and shortsighted”. David Sacks, who now chairs Trump’s science advisory council, told the labs they’re welcome to slow down on their own and should stop asking for antitrust relief “so you can form a cartel”. Michael Burry called it self-serving for execs with public offerings coming up. Yann LeCun reminded everyone Amodei thought GPT-2 was too dangerous to open source back in 2019.

Beijing, a Trump adviser, the Big Short guy and Yann LeCun all saying the same thing in the same week. When does that ever happen?

Guo picked the right word. It’s fearmongering. Amodei’s own essay says nobody was hurt in the Hugging Face incident and “the economic damage was minimal”. Then the same essay warns that in 6 to 12 months a swarm like that “could be capable of taking over the entire internet with a persistent botnet”.

Minimal damage in July, the end of the internet by next winter, and could we please have some laws before anyone checks the maths. That’s been the routine since GPT-2. OpenAI held that model back in February 2019 because it was too dangerous, released the full thing nine months later, and the internet carried on being the internet.

I don’t want any of this legislated. No approval regime, no capability checkpoints, no antitrust waiver, and definitely no treaty drafted by the three companies with the most to gain from it. Laws rushed through in a panic with the incumbents holding the pen just freeze the leaderboard with whoever’s on top this year.

If Anthropic’s this worried, it can hold back Mythos 5.2 and write an essay about how that went. I’d read that one.