ArXiv: HORIZON — Where and Why AI Agents Fail on Long-Horizon Tasks

A research team has presented HORIZON, a new benchmark that systematically diagnoses where and why LLM agents fail on long-horizon tasks — those requiring tens or hundreds of consecutive steps.

Key Findings

Instead of testing only the final result, HORIZON analyzes every potential failure point throughout the agent chain. The results show:

• Cumulative degradation — each step carries a small probability of error, but across 20+ steps this becomes a near-certain failure
• Context loss — agents gradually “forget” the original goal as their context window fills up
• Faulty recovery — when an agent makes a mistake, recovery attempts often make the situation worse

Why It Matters

Most existing benchmarks test agents on short tasks (5-10 steps). In the real world — autonomous coding, research, planning — tasks involve tens to hundreds of steps. HORIZON shows that impressive results on short benchmarks do not translate to reliability on long-horizon tasks.

Practical Implications

The results suggest that current approaches to agentic AI need fundamental changes in context management and error recovery, rather than just larger models or longer context windows.