arXiv:2607.12385: PM-Bench measures 'prospective memory' in agents — best GPT-5.4 scores only 65.1% F1

What is prospective memory and why do agents fail at it?

Prospective memory is the ability to remember that an intended action must be executed at the right future moment — for example, that an agent should send an email when a certain condition is met, or initiate a reservation a week in advance. Unlike retrospective memory (remembering past events), prospective memory requires tracking open intentions over longer time periods and triggers. Authors Genglin Liu and Saadia Gabriel from the University of Washington identified precisely this ability as a critical gap in the evaluation of modern LLM agents.

PM-Bench: design and methodology

PM-Bench is a benchmark accepted at the COLM 2026 conference, and its design directly draws on the cognitive-psychological paradigm of Virtual Week. The platform simulates a 7-day period of everyday agentic tasks in which the agent must track future triggers and execute deferred actions. The evaluation covers 8 state-of-the-art LLMs tested in 8 different configurations, examining the impact of prompting, context length, and memory strategies on overall performance.

The benchmark distinguishes several types of prospective tasks: time-conditional actions, event-conditional actions, and combinations of both. Each task is tracked through an F1 metric that combines precision and recall in correctly triggering the intended action.

Results: even GPT-5.4 doesn’t exceed 65.1% F1

The results are unambiguous — all tested models show a significant gap in prospective memory. GPT-5.4 achieves 65.1% F1 as the absolute maximum, meaning that in more than a third of cases the agent either forgets the task or triggers it at the wrong moment. For comparison, the average person in analogous psychological tests achieves considerably higher success rates.

No tested strategy — neither extended context, nor explicit reminders, nor structured memory modules — consistently outperformed the others across all scenarios. This suggests that the problem of prospective memory is not a matter of prompting architecture, but a fundamental limitation in how modern LLMs internally manage intentions across longer sessions.

Implications for agent development

PM-Bench results establish a concrete baseline for measuring progress in this area. Unlike previous benchmarks focused on knowledge or short-term reasoning, PM-Bench quantifies a capability that is critical for autonomous agents in the real world — planning assistants, workflows, and multi-day tasks. The authors emphasize that without solving prospective memory, claims about “autonomous agents” remain partially unsubstantiated.