A Probe Cascade Predicts AI Agent Failure from Hidden Activations at the First Step, Saving Up to 47% of Tokens

AI agents solving multi-step tasks — web search, code writing, business process automation — often continue toward failure long after the outcome is effectively determined. Every unnecessary step consumes tokens, memory, and money. Researchers Kai Ruan, Zihe Huang, Ziqi Zhou, Qianshan Wei, Xuan Wang, and Hao Sun developed a system that recognizes such “doomed” episodes earlier than ever before — ideally at the very first interaction with the environment, not when the error materializes as a visible agent action.

Hidden States as a Failure Predictor

The paper’s core insight is that the failure signal is not hidden in what the agent says or does, but in what is happening inside the model itself. Lightweight probes on hidden activations of the neural network — tensors produced between transformer layers that encode the model’s internal state at every moment — outperform all behavioral approaches that analyze externally visible actions.

This is an important conceptual distinction. Behavioral scorers wait until an error becomes visible in the agent’s output or action, which can require several steps or rounds. Internal probes capture the signal while the model is still processing the first interaction with the environment. Hidden representations encode information about the episode’s future trajectory far earlier than that information becomes available through externally visible signals — and empirical comparison with behavioral baselines confirms this: behavioral scorers typically achieve roughly half the savings.

How Does the Calibrated Abort-Gate Cascade Work?

The system is implemented as a cascade of early-stopping gates, with one gate per round of the agent episode. Each gate uses a probe calibrated with a distribution-free method that guarantees a per-round recall budget — a precisely defined percentage of successful episodes that may be incorrectly stopped in that specific round.

The critical difference from a naive multi-gate approach is that optimization is performed at the episode level. The user specifies an episode-level recall target — for example 90% — and the system guarantees that at least that proportion of episodes that would have completed successfully pass all gates and are completed. Individual gates are calibrated jointly to satisfy that condition, not each independently.

The cascade architecture combines probes with independent per-round signals, achieving 1.6–1.7× better efficiency than any single-gate policy. Each gate eliminates a portion of doomed episodes that earlier gates failed to recognize, and the cascade’s aggregate precision strictly exceeds the capability of any individual gate placed at the optimal point in an episode.

Empirical Results on Qwen and Llama Models

The authors evaluated the cascade on two open-source models at the 90% recall target, measuring the percentage savings in total tokens processed:

For Qwen-2.5-7B: the cascade achieves 47.1% ± 10.3% savings. Almost half of all tokens consumed were going to episodes the system correctly identifies as doomed and aborts early — without successful episodes being incorrectly stopped at a rate exceeding 10%.

For Llama-3.2-3B: savings amount to 37.2% ± 8.8% at the same recall target. The difference between models suggests that the internal representations of the larger model are more informative for outcome prediction — consistent with the hypothesis that larger models encode richer internal representations of task state.

Both results substantially outperform behavioral scorers, which achieve approximately half the savings under the same conditions, empirically confirming that hidden states carry significantly more prognostic information than externally visible actions.

Implications for Production Environments

Early termination of doomed episodes has direct implications for the operating costs of agentic systems. Compute is consumed proportional to episode length: every token spent on a trajectory that ends in failure anyway is a cost with no return. On systems with high task volumes, even a conservative 37% token saving translates into a significant reduction in infrastructure costs.

The recall parameter enables the cascade to be adapted to different business priorities without retraining. For critical tasks — where incorrectly stopping a successful episode is more expensive than computational inefficiency — recall can be moved to 95–97%, allowing a smaller share of doomed episodes to pass through for greater safety. For high-volume, less critical environments, more aggressive token savings may be the priority. This flexibility makes the cascade applicable to a wide range of production scenarios without compromise to the underlying architecture.