WindowsWorld benchmark: leading computer-use agents fall below 21% success rate on tasks spanning multiple desktop applications

A research team from the Harbin Institute of Technology (Jinchao Li, Yunxin Li, Chenrui Zhao, Zhenran Xu, Baotian Hu, and Min Zhang) published a new benchmark for autonomous GUI agents on ArXiv on April 30, 2026. WindowsWorld shifts the focus from isolated tasks within a single application toward realistic professional work that crosses the boundaries of multiple programs, and reveals that leading computer-use agents achieve less than 21% success on such tasks.

What does WindowsWorld measure differently from OSWorld?

OSWorld and related benchmarks predominantly measure agents within a single application: open a browser, double-click, fill out a form. WindowsWorld explicitly covers multi-app workflows — for example, extract data from a table in Excel, prepare an email proposal in a mail client, and create a presentation with the results in a presentation tool, all within a single task. 78% of the total 181 tasks are inherently multi-application, and the average task has 5.0 sub-goals across 17 different desktop applications. Tasks were generated through a multi-agent framework guided by 16 professional roles (occupations), then refined through human review and executed in a simulated environment.

Why do agents fail when a task spans three applications?

The authors’ main experimental finding is inconsistent across benchmarks. While leading agents perform well on single-app tasks, they fall below 21% success on multi-app workflows. The specific problems are conditional reasoning across three or more applications — agents stall at early sub-goals or repeat the same steps. Another issue is low execution efficiency: agents even exceed the human ceiling in number of steps, yet still fail to complete the task. In other words, the problem is not solely about step count but about the ability to maintain state across contextual transitions between programs.

Implications for deploying agents in office work

Computer-use agents are among the fastest-growing AI products — Anthropic, OpenAI, and Google DeepMind are actively promoting agents as replacements for routine office work. WindowsWorld signals that the current generation of agents is far from reliably executing multi-application tasks characteristic of real professional daily work. The benchmark should become a realistic direction of progress for the next generation of agents, similar to how SWE-bench defined the direction for coding agents.