arXiv:2607.14541: Atrex-Bench Reveals LLM-Generated GPU Kernels Reach Only 10% of Target Hardware Speed

What is Atrex-Bench, and what does it measure?

Atrex-Bench is a benchmark introduced by Lingyun Yang and colleagues to test the quality of GPU kernels — low-level code that performs a specific operation, such as matrix multiplication, directly on the graphics processor. The benchmark uses 30 operators and 440 shapes collected from real production inference traces, unlike synthetic tests that don’t reflect actual workloads.

How fast is LLM-generated code compared to the hand-tuned optimum?

Six frontier coding agents were evaluated, and the results reveal a problem: even the best “vanilla” model reaches only about 10% of the hardware roofline — the theoretical maximum of hardware performance — on production operators. Comparison with hand-optimized production baselines reveals a huge gap: the high pass-rate figures from earlier evaluations are partly misleading, as they often reflect PyTorch fallback code rather than an actually generated LLM kernel.

The solution: Atrex-Kernel-Agent and a knowledge base

Alongside the benchmark, the team presents Atrex-Kernel-Agent (AKA), an agent with an “optimization dropout” technique and a knowledge base of 298 reference kernel files and 244 optimization documents. AKA turns non-functional fallbacks into kernels comparable to hand-optimized baselines. The benchmark and agent are released as open source.