arXiv: CODA-BENCH reveals that leading agents achieve only 61.1% on tasks combining code and data

A team of researchers from Renmin University of China — Yuxin Zhang, Ju Fan, Meihao Fan, Shaolei Zhang, and Xiaoyong Du — has introduced CODA-BENCH, the first benchmark that simultaneously evaluates the ability of AI agents in both code generation and data retrieval within a data environment. The paper has been accepted at ICML 2026 and addresses a clear gap in existing evaluation frameworks for agents that aim to be autonomous engineers.

The gap between code-centric and data-centric tests

Previous benchmarks for AI agents have mostly chosen one of two dimensions: either testing the ability to generate code (the agent writes a program that solves a task), or testing the ability to work with data (the agent searches and analyzes information within a dataset). Real development environments do not afford that luxury of separation. An engineer solving an analytical problem must first locate the relevant data within a complex file structure, understand the schema, and only then write and iteratively improve the code that uses that data. CODA-BENCH is designed to test precisely this combined set of capabilities — and the results reveal that this is substantially harder than either isolated benchmark has suggested.

Why is integrating these capabilities particularly demanding?

The problem is non-trivial for several mutually reinforcing reasons. First, the data space is large and unstructured: the agent must actively explore file hierarchies and deduce which data is relevant to the task, without explicit instructions about the location. It must then infer the data schema — column types, formats, encodings — which in real-world datasets is not always obvious or consistent. Finally, the code the agent writes based on incorrect schema assumptions will likely fail to execute or produce wrong results, and the agent must iteratively debug in an environment with limited feedback. Each of these three steps introduces independent sources of error that multiply — rather than simply add — in the combined task.

Realistic sandbox: Kaggle, 31 communities, 980 files

The authors construct a Linux sandbox based on the Kaggle ecosystem, containing hundreds of datasets. Each environment contains on average 980 files, simulating the real scale and information noise an agent encounters in actual development environments. The benchmark covers 1,009 tasks across 31 task communities, spanning diverse domains of data analysis. Agents must actively explore the file structure to locate relevant data, then write and execute analytical code that addresses the specified task. This combination — large data scale, uncertain location of relevant resources, and the need to generate code that actually works — makes CODA-BENCH systematically more demanding than previous benchmarks that tested these dimensions separately.

Leading agents reach only 61.1% success

Evaluation of advanced agents on CODA-BENCH reveals a clear limitation: even top-performing systems achieve a success rate of only 61.1%. This is significantly below the performance the same systems achieve on isolated benchmarks for coding or data work, confirming that integration of the two capabilities — not their sum — is the fundamental challenge. For tasks that require both data discovery and code generation, even state-of-the-art agents fail in nearly 4 out of 10 cases. The authors emphasize that this result is not a problem specific to a particular model or architecture, but reflects a systemic gap in the way current agents integrate retrieval and coding.

Three failure classes and future research

The authors identify three primary failure classes that dominate the error cases. The first is file discovery: the agent fails to find relevant data within a complex hierarchy, especially when relevant resources are in unexpected locations or have unconventional names. The second is schema inference: the agent incorrectly reasons about data structure — column types, formats, or encodings — leading to code that fails to execute or produces incorrect results. The third is iterative debugging under data constraints: the agent is unable to effectively test and correct code when feedback from the environment arrives slowly, partially, or only after a lengthy execution.

Acceptance at ICML 2026, one of the most influential machine learning conferences, confirms that the research community recognizes this type of benchmark gap as an important contribution. CODA-BENCH, together with the identified failure classes, sets concrete goals for future systems: agents that aspire to be useful in real development environments must learn to explore file structures, reason about data schemas, and debug their own code under conditions of uncertainty — all simultaneously.