arXiv:2607.14642: MCPEvol-Bench Shows Both GPT-5.4 and Claude Lose Accuracy When MCP Tools Change

What is MCPEvol-Bench, and why does it matter?

MCPEvol-Bench is a new benchmark introduced by Huanxi Liu and colleagues to test AI agents’ adaptability to changing tool interfaces. The Model Context Protocol (MCP) is a standard that gives AI agents access to external tools such as databases, APIs, and file systems. Existing benchmarks assume tool interfaces stay fixed, but in production, MCP servers are constantly updated and change their interfaces.

How does the benchmark simulate a tool change?

The team defined 11 mutation operators — simulated tool-interface changes, such as renaming parameters, changing data types, or restructuring responses — applied across 123 MCP servers. Agents must perform the same task before and after mutation, and the difference in success rate measures how resistant a model is to tool evolution.

How much do frontier models lose on evolved servers?

Twelve frontier LLMs were tested, and the results show that even the most advanced models struggle to keep up with changes. GPT-5.4 records a performance drop of 13.7% on evolved servers compared to static ones, while Claude-Sonnet-4-6 drops even further — 14.4%. The comparison shows Anthropic’s model is marginally more sensitive to tool-interface changes than OpenAI’s, though the difference is small, only 0.7 percentage points.

What this means for the future of agents

Alongside the drop in accuracy, planning and reasoning errors also increase — agents misinterpret changed parameters or try to call an outdated version of a tool that no longer exists on the server. The researchers’ conclusion is unambiguous: the dynamic tool landscape remains a weak point even for the most advanced models, regardless of whether they come from the OpenAI or Anthropic family.

The authors conclude that MCPEvol-Bench opens up room for new adaptability-training strategies, since static benchmarks overestimate agents’ actual readiness for a production environment where tools change almost daily.