arXiv:2606.04261: Can agents curate training data on their own?

The quality of large models heavily depends on the quality of the data they are trained on. But selecting the right data — curation — is a laborious, iterative task. A team of authors around Feiyang Kang and Ruoxi Jia asks whether an AI agent can take it over.

What does Curation-Bench measure?

The authors introduce Curation-Bench, a benchmark designed to test whether coding agents can independently run the entire data curation flow. Curation involves selecting, filtering, and preparing examples for training so that the final dataset is as informative as possible. The benchmark puts the agent in the role of a researcher who must devise and improve a data-selection policy through successive attempts.

How successful are agents on their own?

The results are twofold. Out-of-the-box agents reach strong published baselines within just ten iterations, showing they can carry out the basic curation loop. The problem lies in exploration: the agents mostly refined existing approaches instead of trying fundamentally different strategies. In other words, they know how to optimize, but not how to explore boldly.

What does structured guidance change?

The key shift comes from scaffolding — structured guidance that instructs the agent to cite, instantiate, and adapt a prior method in each iteration. With such instruction, the agent autonomously assembles a selection policy that surpasses strong published baselines, and does so on a tenth of their data budget. The authors’ conclusion: reliable data exploration requires guided adaptation of methods, not just open-ended prompting.