AgentGym2: New Benchmark Tests LLM Agents Under De-Idealized Conditions

Do current benchmarks for LLM agents measure true capability — or merely the ability to pass through laboratory obstacles? A research group led by Zhiheng Xi and 24 co-authors has released AgentGym2, an evaluation framework that asks exactly that question and offers a concrete answer: existing benchmarks use overly simplified, idealized setups, which causes them to dramatically underestimate how hard the real work is for agents in production environments. The paper was accepted as a main conference contribution at ACL 2026.

Every standard benchmark gives agents pre-prepared tools, clean and complete input data, and unambiguously defined tasks. These are conditions that in real systems almost never exist. In production deployment, an agent regularly encounters unknown tools it must discover on its own, noisy inputs that do not provide all the information it needs, and multi-step tasks where a mistake in one phase cascades into failures in everything that follows. AgentGym2 was created precisely to bring that reality into evaluation.

Four Dimensions That Separate the Lab From the Real World

AgentGym2 introduces a structured framework built on four evaluation dimensions, each targeting an aspect that previous tests have systematically avoided:

End-to-end procedure execution measures the agent’s ability to complete an entire workflow without relying on external interventions. An error in an early phase cascades through all subsequent steps — mirroring the fragility of real automated pipelines.

Tool discovery through environment exploration requires the agent to independently identify which options are available, rather than receiving a ready-made list with instructions. This simulates conditions where an agent enters an unknown environment without a prior manual.

Tool composition for unseen tasks tests the ability to combine multiple tools in ways not encountered during training. Generalization, not memorization of ready-made solutions, is what is truly being measured.

Robustness to noise and incomplete information deliberately introduces poorly formatted inputs, contradictions, and incomplete context. In real environments, users and systems rarely write perfectly structured queries, and agents must remain functional despite this.

What Do the Results of Testing 15 Models Reveal?

The authors evaluated 15 models on the AgentGym2 benchmark, including the leading proprietary systems — GPT-5 and Gemini — as well as a range of open-source alternatives. The conclusion is unambiguous: all of them struggle significantly. Not one of the tested models reached a performance level acceptable for autonomous production deployment without intensive supervision and verification.

What draws particular attention is the scale of the difference compared with results the same models achieve on idealized benchmarks. The performance drop is neither marginal nor merely unexpected — the authors describe the gap as dramatic, as clear evidence that previous leaderboards were measuring something fundamentally different from what agents are supposed to do in real systems. A high score on standard tests no longer implies readiness for production.

AgentGym2 Sets a New Reference Standard for the Research Community

Acceptance at ACL 2026 — one of the most prestigious annual conferences for natural language processing — confirms that the research community recognizes the seriousness of the problem. Existing benchmarks persist because they are practical: easy to automate, fast to evaluate, and straightforward to compare across models. But if they measure ease rather than reality, they lead research in the wrong direction over the long term.

AgentGym2 offers an alternative grounded in de-idealized scenarios that more closely match the conditions agents encounter in enterprises, automated workflows, and user-facing applications. The four evaluation dimensions together cover aspects that have been regularly omitted until now: discovery, composition, robustness, and full end-to-end task execution.

For development teams building production agent systems, the message is clear: a high score on the old tests is no guarantee that an agent will function where the real work actually demands it.