arXiv:2606.17819: First Systematic Benchmark of 500 Agentic Skills Across 19 Model Configurations

A new preprint presents the first systematic agentic skills benchmark, an area that has been poorly measured despite the rapid deployment of agents in production.

What does the benchmark measure?

The framework evaluates 500 real-world skills and generates 1,000 tasks with rubrics that separately score instruction following and goal completion. A skill here is a package of instructions and tools that enables an agent to perform a specific task. Testing was conducted across 19 configurations of proprietary and open models, yielding a broad comparative picture.

What is the key finding?

Models show significant differences in gain depending on how precise the instructions for each skill are. In other words, the same skill produces very different results across different models, and instruction quality critically influences the outcome. This suggests that agent success is not just a matter of model choice, but also careful skill design.

Why does this matter?

The authors publicly released the evaluation set, enabling repeatable measurements and further research. For teams deploying agents, the finding is practical: model choice and skill definition precision must be measured together, because the wrong combination can significantly reduce reliability in production.