arXiv:2605.06177: BioMedArena — toolkit for biomedical AI agents with 147 benchmarks and 75 tools
BioMedArena is an open-source toolkit that separates biomedical AI agent evaluation into six layers, exposes 147 benchmarks and 75 tools in 9 families, and achieves an average of +15.03 percentage points SOTA across eight representative benchmarks.
This article was generated using artificial intelligence from primary sources.
A research team from Oxford and collaborating institutions published a paper on BioMedArena on arXiv on May 7, 2026 — an open-source toolkit for building and evaluating biomedical AI agents. The toolkit, configurations and task-specific traces are available on GitHub.
What problem does BioMedArena solve?
The authors identify a “per-paper engineering tax”: the same models on the same benchmarks produce different results across different papers due to variations in implementation and tool registries. This makes it difficult to compare progress and slows field development.
How is the toolkit structured?
BioMedArena separates the evaluation pipeline into six layers: benchmark loading, tool exposure, tool selection, execution mode, context management and scoring. The system covers 147 biomedical benchmarks and 75 tools organized into 9 functional families, with 6 agent harnesses and 6 context management strategies — yielding 12 competing research backbones.
What are the results and how is it extended?
BioMedArena achieves SOTA results on eight representative biomedical benchmarks with an average improvement of 15.03 percentage points over previous approaches. Adding a new model, benchmark or tool reduces to registering a short provider adapter of a few lines of code, which facilitates integration and ensures reproducibility.
Frequently Asked Questions
- What is BioMedArena?
- BioMedArena is an open-source toolkit for building and evaluating biomedical AI agents that separates the evaluation pipeline into six independent layers and exposes 147 benchmarks and 75 tools.
- How is a new model or benchmark added?
- The toolkit reduces the process to registering a short provider adapter of a few lines of code, significantly reducing the per-paper engineering cost and ensuring reproducibility of results.
- How large is the performance gain?
- BioMedArena achieves state-of-the-art results on eight representative biomedical benchmarks with an average improvement of 15.03 percentage points over previous SOTA approaches.
Sources
Related news
arXiv:2605.06623: MASPO — automatic prompt optimization for multi-agent LLM systems, ICML 2026
Google DeepMind: AlphaEvolve available through Google Cloud, first industrial results
arXiv:2605.05191: LongSeeker with Context-ReAct framework achieves 61.5% on BrowseComp