arXiv: UCS-Bench tests the spatial memory of AI agents in egocentric video

The vision of always-on AI assistants — embedded in AR glasses or wearable cameras that record everything a user sees — brings one key engineering challenge: how can an agent know where an object is if it has not seen it in the past hour, or even several hours? A team of researchers led by Yun Wang, Junbin Xiao, Han Lyu, Yifan Wang, Jing Zuo, Zhanjie Zhang, Hong Huang, Dapeng Wu, and Angela Yao has published a paper accepted at ICML 2026 that directly addresses this problem by introducing a new benchmark and accompanying framework for spatial memory.

Benchmark for continual spatial intelligence

The authors introduce UCS-Bench (User Centric Continual Spatial Intelligence Benchmark), a dataset encompassing over 170 hours of egocentric visual observations. Within this corpus they have formulated more than 8,100 temporally annotated questions designed to diagnose a model’s ability in spatial reasoning and long-term memory aligned with the user’s movement.

The benchmark is designed for what the authors call user-centric continual spatial intelligence. The agent must not only know where objects are — it must express that location relative to the user’s current position and orientation, as the user continuously moves and changes viewing angle throughout the video. This dynamism distinguishes the benchmark from static scene sets or short video clips that have previously dominated spatial understanding research.

Can AI remember where you left your keys?

This question — in a literally everyday sense — motivates DirectMe, the framework the authors propose as a solution. DirectMe incrementally builds and maintains structured spatial memory from streaming egocentric observations, continuously tracking object locations relative to the user’s movement.

Unlike approaches that process video as a static segment or short clip, DirectMe is designed for the always-streaming scenario: the agent is never switched off but continuously receives and integrates visual information. Memory is not a simple list of locations — the system must resolve ambiguities that arise when the same object is seen from different angles at different moments, and when the spatial reference frame shifts along with the user’s movement.

Two classes of problem that the paper highlights are particularly demanding: long-horizon queries that require recalling the user’s prior interactions with objects, and resolving viewpoint ambiguities across multiple observations of the same object recorded from different positions. Both problems directly correspond to real situations that wearable camera users encounter every day.

Performance on leading multimodal models

Experimental results show that DirectMe improves the spatial reasoning capability of leading multimodal large language models. On tasks requiring spatial awareness and understanding of long video streams, the system outperforms comparable models that had previously achieved the best results on this type of task.

The authors note that code and data are publicly available via a GitHub repository, allowing the research community to independently reproduce results and build further on the same infrastructure. This openness is especially important because UCS-Bench thereby becomes a shared reference point — a measure of success against which future systems will be compared.

Implications for agent development and AR technology

The paper arrives at a moment of rapid growth in the wearable AI device industry — from AR glasses to body-mounted cameras that record users’ daily lives. For these always-on camera scenarios, previous benchmarks were inadequate: they tested short clips and static scenes, not continuous streams of hours of content with a dynamic user at the center.

UCS-Bench and DirectMe together represent a concrete step toward agents that could deliver practical utility in everyday life — from supporting people with memory impairments to boosting the productivity of professionals who need to quickly locate a tool or document in a physical workspace. Acceptance at ICML 2026, one of the most influential machine learning conferences, signals that the research community recognizes this class of problem as a legitimate and time-sensitive research frontier.

The question that remains open is scalability: how does a system like DirectMe behave when streaming continues for days or weeks, and how should the memory base be managed when it could grow to impractical proportions under such conditions. The paper does not address these challenges directly, but by establishing the benchmark and a reference solution it opens space for future iterations to tackle them systematically.