arXiv: ASSAY framework sets new record for LLM agents through selective skill masking

A team of researchers from the University of North Carolina at Chapel Hill — Yixuan Wang, Yiyang Zhou, Yiming Liang, Congyu Zhang, Fuxiao Liu, Jiawei Zhou, and Huaxiu Yao — has published a paper identifying a fundamental problem in LLM agents that learn from experience: agents have no principled way of distinguishing helpful from harmful skills in their library. The problem is not merely theoretical — an agent that learns from experience but cannot select knowledge may over time accumulate a library in which inferior resources cancel out the good ones. The paper introduces ASSAY (Attributed Skill curation for Agent Systems), a framework that solves this problem through empirical causal attribution, without any modification of model weights.

Causal heterogeneity: the same skill, opposite effects

Modern LLM agents can accumulate skills from experience in the form of natural-language descriptions and use them to better solve future tasks. The problem the authors identify is called causal heterogeneity: individual skills routinely help on some task types while simultaneously harming others. When these opposing effects are aggregated at the full library level they cancel each other out — which is why global correction methods that look at a skill’s average effect cannot detect the problem. The agent continues to use all skills indiscriminately, unaware that some of them are not neutral in a given context but actively harmful.

Why a global library correction approach does not work

Previous approaches to skill library correction relied mainly on the LLM’s own judgment — the model assessed which skills were useful. The authors argue this is conceptually wrong for the correction task: skill generation from experience is a creative act at which LLMs excel, but deciding whether that skill is truly useful requires empirical evidence across a range of tasks. A model cannot assess its own blind spots. Combining both tasks in the same LLM judgment leaves the system without an objective metric and without a mechanism for detecting contextual harm.

How does ASSAY separate the creative from the empirical?

ASSAY divides correction into two separate phases. In the first phase, skill generation remains a creative task performed by the agent in the standard way — from experience and interactions. In the second phase, ASSAY measures the causal contribution of each skill through randomized masking on a small development set: for each task, a skill is randomly withheld and the effect on task outcome is measured. From these measurements the system builds a per-skill attribution picture — for which task type the skill helps, and for which it hurts. Based on this picture, ASSAY dynamically suppresses skills with a predicted negative effect for each specific test task at inference time. No weight modifications, no global skill deletion — exclusively contextual filtering at runtime.

New record on AppWorld and tau-bench

Experiments cover 7 base models from 4 providers and 2 benchmarks — AppWorld and tau-bench. On the hardest portion of the AppWorld benchmark, DeepSeek-V3 with ASSAY achieves 69.3% task goal completion — a relative improvement of 47.4% over the baseline and a new state-of-the-art including methods that use weight tuning. On the tau-bench retail split, GPT-4.1 with ASSAY records a relative improvement of 8.7% and outperforms o4-mini, o1, and GPT-4.5 on the public leaderboard — without any weight tuning. Results are consistent across all tested models and both benchmarks, suggesting the finding is not an artifact of a specific model.

Per-task masking: the key factor behind improvement

Ablations precisely locate the source of the improvement. Global removal of skills with a negative average effect yields a smaller gain than per-task masking. The critical difference lies in the granularity of context: the same skill that is bad for the average task may be useful for a specific task type, and vice versa. Per-task masking captures this distinction; a global approach cannot. The authors conclude that the bottleneck for agents that learn from experience is aligning skills with task type at inference time — not global library cleanup. This distinction has practical implications: it suggests that a system that knows which task it is solving can dynamically select the right subset of knowledge, rather than carrying the entire accumulated library without filtering. Code is publicly available on GitHub.