Autonomous cyberattacks: 19 language models tested against 300 servers — breach rates from 10.7 to 69.3 percent

Autonomous cyberattacks are no longer a theoretical threat

The increasingly loud debate in the security research community about whether large language models (LLMs) can autonomously conduct cybersecurity attacks has today received a strong empirical foundation. A research team — Jiaqi Luo, Jiarun Dai, Zhile Chen, Jia Xu, Weibing Wang, Yawen Duan, Brian Tse, Geng Hong, Xudong Pan, Yuan Zhang, and Min Yang — has published a comprehensive evaluation that surpasses all previously available public analyses in rigor.

How was the testing framework constructed?

The key strength of this paper lies in its methodology. The researchers built an evaluation framework consisting of two components: a set of target servers and an agent infrastructure for autonomous attacks.

Target servers were organized into two difficulty tiers:

• Tier 1: one protected service per server
• Tier 2: three protected services per server

A total of 300 test servers were created. A critical methodological requirement: agents used only general-purpose tools, without any prior knowledge of the specific targets. This simulates the realistic scenario of an attacker who has no insider information about the system being attacked.

Results: breach rates by model

Of the 19 models evaluated — combining open-weight and proprietary solutions — successful autonomous breach rates ranged from 10.7% to 69.3%.

This range is itself telling: even the weakest model tested autonomously breached the defenses of nearly every ninth server without any human assistance. The strongest models successfully compromised nearly every other target system.

The danger grows with model capability

Researchers note a key finding with far-reaching implications for security policy: autonomous cyber-breach capability continues to grow in parallel with advances in general model capability.

This means that improvements LLM providers implement to make models more useful in everyday tasks — better code understanding, multi-step task planning, tool use — simultaneously increase the ability to autonomously exploit security vulnerabilities. This correlation is not coincidental: the same cognitive capabilities that make a model an effective assistant also make it an effective attacker.

Methodological advance over prior work

Previous assessments of LLMs’ autonomous cyberattack capabilities were, as the authors note, methodologically opaque: limited in scope, unclear in testing conditions, and difficult to reproduce. This paper introduces a standardized, transparent framework with clearly defined:

• number and type of target servers
• complexity tiers
• agent and infrastructure type
• target-knowledge conditions

Connection to the “red line” of autonomous cyberattacks

The research directly addresses one of the key points in AI safety debates: autonomous cyberattacks represent an explicit “red line” in risk assessments of advanced AI systems.

Several leading AI labs and government bodies cite autonomous cyber-offensive capabilities as one of the thresholds that should trigger special precautions or restrictions in model development and distribution. This paper offers the first methodologically sound empirical assessment of where the current generation of models stands relative to that threshold.

The conclusion is unambiguous: the capabilities exist, they are measurable, and they are growing. For the security community — and for policymakers working on AI regulatory frameworks — this research delivers data that can no longer be ignored.