arXiv:2605.30434: LongDS-Bench reveals AI agents' decline on long analyses

A new paper titled “LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis” presents a benchmark that evaluates AI agents on multi-turn data analysis tasks. The authors are Kewei Xu, Xiaoben Lu, Shuofei Qiao, Zihan Ding, Haoming Xu, Lei Liang and Ningyu Zhang.

What is LongDS-Bench?

LongDS-Bench is an evaluation suite composed of 68 tasks and 2,225 turns derived from real Kaggle notebooks. The tasks cover six domains, including geoscience, business and education. The goal is to test whether agents can maintain analytical context across long, multi-turn interactions.

How poorly do models perform over long horizons?

The results are damning: the best model achieves only 48.45% average accuracy. In testing five leading models, performance drops by almost 47 points from early to late turns. Long-horizon errors make up between 52% and 69% of all failures.

What is the real cause of failure?

The key finding is that the main bottleneck is maintaining the correct analytical state, not increasing the interaction budget. In other words, additional computation steps in themselves do not solve the fundamental challenge of preserving state across long tasks. The researchers released the benchmark to encourage the development of more reliable agentic systems for data analysis.