arXiv:2605.25707: AgentHijack Benchmark Reveals Critical Fragility of AI Agents in Computer Use

What Is AgentHijack and Why Is It Important for AI Agents?

Researchers Jingwei Sun, Jianing Zhu, Yuanyi Li, Tongliang Liu, Xia Hu, and Bo Han introduced AgentHijack — a new benchmark accepted at ICML 2026 that systematically measures the robustness of multimodal LLM agents against realistic disturbances while performing tasks on a computer. Computer use agents are LLM systems that visually monitor the screen and autonomously control the mouse and keyboard.

While earlier research primarily examined intentional adversarial attacks, AgentHijack focuses on practical disturbances that users encounter daily: pop-up notifications, display changes, and similar disruptions that do not arise from intentional malicious action.

What Types of Disturbances Does AgentHijack Test?

The AgentHijack benchmark includes 9 configurable categories of environmental disturbances that replicate real-world desktop usage conditions:

• Pop-up windows (notification pop-ups, dialogs, advertisements) that interrupt the task flow
• Display changes (resolution, zoom, scaling of UI elements)
• Other disturbances that affect the agent’s visual perception and control precision

The paper’s key finding is unambiguous: “Even minor instances of corruption can result in significant performance drops,” indicating a systemic fragility of current multimodal LLM agents such as Claude Computer Use, GPT-4o, and related systems.

How Does AgentHijack-Agent Address the Robustness Problem?

The authors not only identified the problem but also proposed AgentHijack-Agent — a two-component framework for improving robustness:

1. Action generator with enhanced grounding precision that better maps visual elements to semantic actions
2. Onlooker module that continuously monitors agent behavior and verifies the state of the environment, detecting anomalies caused by disturbances

Experimental testing confirmed the effectiveness of the proposed framework. Code, environments, baseline models, and datasets are publicly available through the project page, making reproducibility and further development easier for researchers.

AgentHijack arrives at a time of intense development of computer use agents by Anthropic, OpenAI, and Google DeepMind — robustness to real-world disturbances is a critical requirement for reliable deployment in production environments.