ArXiv: Hodoscope — Monitoring AI Agents Without Predefined Error Categories

How do you monitor AI agents when you don’t know in advance what kinds of errors to look for? Hodoscope solves precisely this problem — it detects suspicious AI agent behavior without the need to define categories of possible errors.

An unsupervised approach

Unlike traditional tools that look for specific known issues, Hodoscope uses an unsupervised approach — it compares behavioral distributions across groups of agents and automatically identifies statistically significant deviations. If one group behaves substantially differently from the others, the system flags it for review.

Concrete results

Hodoscope achieved impressive results during testing:

• Discovered a previously unknown vulnerability in the Commit0 benchmark — unsorted git history allowed agents to access information they should not have had
• Successfully reproduced known exploits on the ImpossibleBench and SWE-bench benchmarks
• Reduced the required human review by 6-23x compared to random sampling
• Discovered patterns also improve the accuracy of LLM-based judges for automated evaluation

Why does this matter?

AI security currently relies on manually defining possible threats — but a new type of agent may exhibit behaviors that were never anticipated. Hodoscope represents a shift toward proactive monitoring that can discover unknown problems, not just those we have prepared for in advance. This is especially important as AI agents take on increasingly complex tasks with access to real systems.