arXiv: OrchRM — reward model for multi-agent AI orchestration without human annotations

The orchestration problem in multi-agent AI systems

How do you coordinate multiple AI agents to jointly solve complex tasks — and how do you learn which coordination moves actually lead to good results? That is the core problem addressed by the research presented in “Reward Modeling for Multi-Agent Orchestration” (OrchRM). Orchestration — deciding which agent receives which task, in what order, and with what context — has until now required either expensive human supervision or re-running entire agent subsystem cycles.

OrchRM offers an alternative: a self-supervised framework that learns from data generated by the task execution itself, without any additional annotation costs.

How does OrchRM work?

The key innovation lies in what it learns from. Rather than waiting for a final outcome or requesting human judgment, OrchRM builds win/lose pairs from intermediate execution artifacts — temporary outputs, intermediate results, and traces that agents generate while working.

From those pairs, a Bradley-Terry reward model is trained — a statistical framework that assesses the relative quality of two outcomes. Crucially, training occurs directly at the orchestration level, not at the level of individual agent subsystems. This avoids the most costly step: re-running entire agent chains just to collect training data.

Performance results

Results are consistent across three different application domains — mathematical reasoning, web-based question answering (web QA), and multi-hop tasks requiring the combination of information from multiple sources.

Across all three domains, OrchRM achieves:

• Up to 10x better efficiency in token usage during training
• Up to 8% higher accuracy when scaling multi-agent systems at inference time (test-time scaling)

Token efficiency is not a trivial metric — it directly affects the cost of developing and operating systems. Achieving comparable or better accuracy with ten times fewer tokens means OrchRM can be used even in resource-constrained environments.

The importance of the orchestration level

Previous approaches to improving multi-agent systems focused mainly on improving individual agents — better models, more precise instructions, more sophisticated tools. OrchRM shifts attention to another dimension: how agents coordinate is just as important as how capable each agent is on its own.

A reward model at the orchestration level can recognize patterns that lead to good coordination outcomes — which order of task delegation works, when it is useful to seek verification from another agent, when it is optimal to consolidate results — all without ever having to explicitly encode those rules.

A scalable path for multi-agent development

The authors position OrchRM as a scalable direction for robust multi-agent orchestration. The self-supervised nature of the method means the system can continuously improve as more execution data is collected, without growing annotation costs.

Code is available on GitHub (github.com/Wang-ML-Lab/OrchRM), allowing researchers and practitioners to apply and extend the method directly. For the community building multi-agent AI systems, OrchRM represents a concrete tool for improving coordination intelligence without proportional increases in development cost.