arXiv:2607.10891: SETA — 4,500+ RL environments boost terminal agent results

What are terminal agents and why they need specialized environments

A terminal agent is an AI agent that solves tasks exclusively through commands in a terminal or shell — without a graphical interface. It can execute scripts, manage files, configure systems, and debug code directly on the command line. Such agents have practical applications in DevOps automation, system administration, and engineering tasks, but their training requires specialized environments that simulate real terminal tasks — and this is exactly where the problem lies: existing reinforcement learning (RL) datasets are small and limited in diversity.

SETA: scalable generation of RL environments

Shen and colleagues from Oxford and KAUST, together with 17 collaborating institutions, propose SETA (Scaling Environments for Terminal Agents): a system that automatically generates verifiable RL environments at scale. SETA combines two datasets: SETA-Synth (synthetically generated tasks) and SETA-Evol (evolutionarily modified tasks with difficulty control). Together they produce more than 4,500 verifiable environments — significantly more than existing benchmark datasets.

The comparison with manually curated datasets is direct: manually creating a single verifiable terminal environment requires substantial human effort, while SETA automates this process while preserving verifiable outputs.

Results on Terminal-Bench 2.0

Evaluation on Terminal-Bench 2.0 shows measurable progress. DeepSeek-V4-Flash trained with SETA environments improves pass@1 from 40% to 43% and pass@5 from 54% to 58%. Pass@1 measures success on the first attempt; pass@5 measures success if the agent has five attempts — both measurements are relevant for production use.

Implications for agent system development

SETA addresses the fundamental bottleneck in terminal agent development: the lack of diverse and verifiable RL training environments. By increasing the number and diversity of environments — with difficulty control through SETA-Evol — the system enables more robust and generalizable training. For researchers and engineers building DevOps agents or infrastructure automation tools, SETA offers a practical platform for scaling training without a proportional increase in the cost of manual data creation.