Allen Institute for AI Released olmo-eval: Evaluation Platform Designed for the Active LLM Development Cycle
AI2 has released olmo-eval, an open evaluation platform that extends the OLMES standard from one-shot benchmarking toward a full development loop. The key innovation is per-question pair comparison across checkpoints, with built-in statistical measures for distinguishing genuine improvements from data noise.