arXiv:2606.09450: TheoremBench — benchmark for evaluating LLMs on formal theorem proving
TheoremBench is a new benchmark that assesses the capability of large language models in formal theorem proving using Lean4. It covers approximately 100 classical theorems in standard and extended format with automatically extracted premises. Research shows that explicit premises significantly improve performance, while new metrics reveal that models favor easier sub-theorems and generate overly long proofs.
This article was generated using artificial intelligence from primary sources.
Lean4 and formal theorem proving
Lean4 is a programming language designed for writing mathematical proofs that a computer can automatically verify. Unlike classical papers written in natural language, formal proofs must pass rigorous logical verification — every step must be explicitly stated and justified. This makes them a powerful tool for verifying complex mathematical claims, but also a demanding testing ground for language models.
TheoremBench: approximately 100 classical theorems, two formats
Researchers QuocViet Pham, Elvir Karimov, Andrey Galichin, and Ivan Oseledets (arXiv:2606.09450, June 8, 2026) introduced TheoremBench — a set of ~100 classical mathematical theorems in two formats. The standard format provides only the statement, while the extended format automatically extracts auxiliary sub-theorems as explicit premises. The benchmark goes beyond competition problems and evaluates models on longer, dependency-rich mathematical developments.
Do explicit premises improve model performance?
Yes. The key finding of the research is that providing explicit premises significantly improves performance for models capable of Lean4 — the difference between formats is not marginal. In addition, two new metrics — theorem-level coverage and token efficiency — reveal hidden weaknesses: models systematically favor easier sub-theorems and generate excessively long proofs. A high overall solve rate can thus mask genuine deficiencies in mathematical reasoning.
Frequently Asked Questions
- What is Lean4 and why is it important for evaluating language models?
- Lean4 is a programming language for formal theorem proving that requires rigorous logical verification of every step. Unlike natural language, it does not permit approximate arguments — making it an ideal testing ground for AI systems.
- Why do explicit premises improve model performance in TheoremBench?
- Explicit premises reduce the reasoning burden because the model does not need to independently find intermediate steps. The extended format automatically extracts auxiliary sub-theorems, significantly raising solve rates for models capable of Lean4.
📬 AI news in your inbox
A daily digest built your way — pick topics, sources and cadence. One-click unsubscribe.