Dario Amodei - Firethering

For years, the AI race has been built around one basic assumption: whoever can build the most capable systems first wins. Now some of the people leading that race are saying it has started moving too fast. Anthropic CEO Dario Amodei says the frontier needs to be paced. OpenAI CEO Sam Altman agrees and says OpenAI will give independent evaluators employee-like access to its systems. Anthropic alignment researcher Evan Hubinger says pacing is necessary if humanity is going to survive. Elon Musk says Dario is right. Bill Gates says the world needs a global plan. That is a remarkable amount of agreement from people who normally have every reason to keep pushing. But there is a problem. Just days before Amodei's proposal, Anthropic researcher Jacob Coxon resigned after arguing that the companies building these systems are locked in a race because they don't believe their competitors will act responsibly. So what happens when everyone agrees the race is becoming dangerous, but nobody wants to be the one who slows down first? That is the question behind the new push to pace frontier AI. And answering it means looking beyond what the CEOs are promising to the incentives that could make those promises difficult to keep. Table of ContentsThe Problem Coxon Saw ComingThe Accelerator Inside the RaceThe First Test: Can Anyone Verify What Labs Are Doing?The Second Test: Can Competitors Actually Agree?The Third Test: What Happens When Countries Don't Agree?So, Will the AI Race Actually Slow Down? The Problem Coxon Saw Coming Jacob Coxon had already identified the problem a few days before the new calls for pacing. When he resigned from Anthropic, his criticism wasn't simply that AI was becoming dangerous. He argued that Anthropic understood the stakes but was still caught in a race because it believed other companies might keep pushing ahead. That creates a difficult incentive. A lab can believe that slowing down would make AI development safer while also believing that slowing down alone could leave it behind. If a competitor keeps improving its models, the lab that pauses could lose talent, investment, capabilities and influence. In that situation, telling everyone to be more careful isn't enough. Someone has to believe that the other players will be careful too. That is what makes the recent statements from Amodei, Altman and others worth looking at beyond the headlines. They are not only arguing that AI safety matters but also starting to talk about mechanisms that could make slowing down possible without asking one company to take the risk alone. The question is whether those mechanisms can work once the incentives of the AI race are taken seriously. The Accelerator Inside the Race There is another reason this race is becoming harder to manage, the technology being raced over can actually help with the race itself. AI systems are already useful for writing code, running experiments, analyzing results and helping researchers explore new approaches. As those systems become more capable, more of the work involved in developing the next generation can potentially be handed to AI. That creates a feedback loop. A better model can help build a better model, which can then help build the one after that. The question is what happens to the amount of time humans have to understand each new jump. Imagine two labs improving their systems at roughly the same pace. If one develops an AI that becomes substantially better at AI research itself, its advantage isn't limited to the capabilities of that model. It may also have a faster way of producing the next one. Now the competitive pressure changes. Falling behind doesn't necessarily mean having a slightly weaker model. It could mean falling behind in the ability to improve models at all. That helps explain why the coordination problem Coxon described is so difficult. If every lab worries that another lab might discover a faster development loop, waiting can look increasingly expensive even when everyone agrees that more caution would be safer. And this is where the argument for pacing becomes different from simply asking researchers to be careful. The goal would be to make sure the systems helping accelerate AI development do not move faster than the methods humans use to understand, evaluate and control them. That is the problem Dario Amodei is trying to address. The harder question is whether a set of agreements can actually create that breathing room when every participant has an incentive to keep its lead. The First Test: Can Anyone Verify What Labs Are Doing? Amodei's first proposal is also the easiest one to understand. Anthropic would give independent evaluators ongoing, employee-like access to its AI systems and development process. They would be able to examine safety practices, investigate incidents and assess how well the models are behaving as they are trained. OpenAI says it will do the same. That is important because one of the big problems with AI safety commitments is verification. A company can say it is taking precautions. An outside evaluator can ask whether those precautions are actually being followed. Amodei is proposing something more intrusive than a traditional audit. The evaluators would have access comparable to employees, and they would be able to publish key findings without Anthropic controlling the conclusions. That creates a useful mechanism for accountability. But it doesn't answer the hardest question yet. An evaluator can tell you whether a lab is following its safety procedures. It cannot, by itself, make competing labs agree on how quickly they should develop more capable systems. In other words, this can make the race more visible without necessarily making the race slower and that distinction becomes important with the next two parts of Amodei's plan. The Second Test: Can Competitors Actually Agree? The second part of Amodei's plan is much harder. He wants frontier AI companies in democratic countries to coordinate on common safety standards and limits on how quickly unchecked capabilities can advance. That sounds reasonable. If every major lab follows the same rules, no company has to worry that slowing down will simply hand its advantage to a competitor. But agreeing on the principle is easier than agreeing on the rules. What exactly counts as moving too fast? Should limits be tied to the size of a training run, the amount of compute used, the capabilities a model demonstrates or how much AI is being used to develop the next generation of AI? And who gets to decide when a model has crossed that line? These questions become even harder because the companies involved are competing for the same customers, researchers and investment. A system that slows everyone equally may reduce the competitive problem, but only if everyone trusts the others to follow the agreement. There is also a legal problem. Amodei acknowledges that some forms of coordination between competing companies could raise antitrust concerns. His proposal therefore involves government support, potentially giving companies a legal framework in which they can coordinate on safety without effectively agreeing to restrict competition. That changes the nature of the proposal. This isn't something Anthropic and OpenAI can simply settle between themselves. It would require companies, regulators and governments to agree on what should be limited, how those limits should be measured and how violations would be handled. The independent evaluators address one problem, whether companies are doing what they said they would do. This part has to solve another, whether everyone is agreeing to the same rules in the first place. And even if the companies manage that, there is still a much bigger player that cannot be brought into the room by a corporate agreement alone. The Third Test: What Happens When Countries Don't Agree? Amodei's third proposal is about getting governments to coordinate globally, including with China. And he is unusually clear about the problem. Pacing only works within the limits of the lead that the United States and its allies already have. If the U.S. slows down by more than that margin while China keeps advancing, Amodei argues that CCP-associated projects could pull ahead. That creates a contradiction at the heart of the proposal. The reason to slow down is to give researchers more time to make increasingly capable AI safer. But the same slowdown could reduce the technological lead that Amodei believes is important for U.S. national security. His answer isn't to ignore that problem. He argues that the U.S. should use export controls, stronger protection against model-weight theft and restrictions on unauthorized distillation to preserve or even widen its lead while creating room to pace development. He believes those measures could widen America's lead over China over the next three to five years, giving democracies more breathing room to improve AI safety. But that still leaves the larger question: what happens when the goal is to slow the frontier everywhere? Amodei proposes several levels of global cooperation with China. The easiest would involve narrow agreements around obviously dangerous uses, such as AI-assisted biological weapons. Another possibility would be requiring models to be tested for serious cyber, biological and alignment risks before release. He sees a harder possibility further up the ladder: an agreement limiting the rate of recursive self-improvement. A full global pause sits at the very end of that ladder and Amodei is skeptical that it could happen anytime soon. The problem is verification. If one side could secretly defect from an agreement and gain a major strategic advantage, the incentive to do so would be enormous. That brings the race problem back in a different form. A company might worry that another company will move faster. A government might worry that another country will do the same. In both cases, slowing down becomes much easier if everyone can trust that the others are slowing down too. And the more powerful AI becomes, the more costly that trust becomes to get wrong. So, Will the AI Race Actually Slow Down? Maybe the most revealing part of this whole debate is that the people asking for a slower race are still building the systems they want to slow down. Dario Amodei isn't calling for AI development to stop. Sam Altman isn't either. They are arguing that the frontier needs enough breathing room for safety work to catch up with capability. Whether that is possible will depend on something much harder than getting CEOs to agree in public. They have to make slowing down feel less dangerous than continuing to race. And nobody has really solved that yet. For now, the industry has something it rarely has: broad agreement that the current pace deserves a second look. The next question is whether that agreement survives when slowing down starts to carry a real cost.

Tag: Dario Amodei

Most Read

Everyone Agrees the AI Race Needs to Slow Down. But Can Anyone Do It?

LLM Wiki: An AI Knowledge Base That Builds Itself

Anthropic Researchers Fear the AI Race Could Cause Human Extinction

Tired of Being a Tenant in Your Own PC? These 7 Open Source Tools Give You Back Control.