Anthropic publishes regulatory framework for governing the development of advanced AI systems

Why existing regulation is not enough for AI

Anthropic, the company behind the Claude assistant, has published a comprehensive policy proposal for regulating frontier AI systems. The document’s central thesis is that “policy was built for a slower world” — regulatory institutions and legislative processes developed over decades now face a technology advancing far faster than any previous industrial revolution.

The document’s key message is that the frontier of AI development is no longer a technical question — it is a social and regulatory question that demands swift, coherent government action.

Who would the new framework apply to?

Anthropic proposes two clear application thresholds that determine who falls under the regulatory framework:

• Compute: models trained using more than 10²⁵ floating-point operations (FLOPs) — a measure that reflects the actual computational power required to train the most advanced systems
• Company size: enterprises with more than $500 million in AI revenue or more than $1 billion in annual AI research and development spending

These thresholds are deliberately set high so that regulation targets only the actors developing the most powerful and potentially most dangerous systems — not startups or academic institutions.

Four catastrophic risks at the framework’s core

Anthropic identifies four risk categories it describes as catastrophic and warranting special regulatory treatment:

1. Biological risk — AI systems capable of supporting the development of pathogens and biological weapons without adequate safeguards
2. Cyber risk — frontier models capable of identifying critical software and infrastructure vulnerabilities at scale
3. Loss of control — systems that operate outside the parameters set by their developers, resulting in unpredictable autonomous behavior
4. AI research automation — AI that automates its own development, exponentially accelerating progress, potentially beyond the reach of human oversight

What the framework would require of developers

Transparency and reporting

Companies developing frontier models would be required to publicly release summaries of safety assessments, frameworks for evaluating catastrophic risks, and regular capability reports. The goal is to ensure that information about the capabilities of AI systems does not remain exclusively within companies.

Independent evaluation

Developers would need to engage qualified independent evaluators (third-party evaluators) to review their safety assessments. This is a significant departure from the current practice of self-assessment — the industry should not be the sole judge of its own risks.

Model and infrastructure protection

The document specifically emphasizes the requirement for protecting model weights and training infrastructure, along with public disclosure of the structure of security programs and the results of regular defense testing.

Powers that the framework would give governments

One key recommendation is that governments must have the legal authority to block or delay the deployment of models that pose a significant catastrophic risk. In addition, the proposal calls for introducing fines tied to global annual revenue that escalate for repeat violations — a mechanism designed to make penalties financially meaningful even for the largest technology corporations.

Federalism: federal law versus state law

The document also touched on the sensitive question of regulatory jurisdiction in the US. Anthropic explicitly states that Congress should not preempt individual state laws (so-called federal preemption) unless the federal regulation is at least as strict as the proposed framework. States should retain jurisdiction over child protection and consumer protection — a direct message to federal legislators not to lower protective standards in the name of uniformity.

Broader context

This document comes at a moment when both the European Union and the United States are engaged in intense debate about regulating AI systems. Anthropic, as a company developing some of the most powerful AI systems on the market, occupies an unusual position — it advocates for regulation that would apply to itself. This voluntarist approach, combined with concrete thresholds and enforcement mechanisms, distinguishes this proposal from the abstract ethical declarations with which the industry typically responds to regulatory pressure.

It remains to be seen whether this proposal will find its way into actual legislative initiatives — but as a reference document from one of the leading AI developers, it will certainly shape expert discussion about the future of frontier AI regulation.