arXiv:2607.14573: Alipay-PIBench Measures Coding Agents on Payments — Skill Boosts RPR by 10.31 Percentage Points

What is Alipay-PIBench?

Alipay-PIBench is a benchmark released by Alipay for testing coding agents — AI systems that autonomously write and integrate code — on realistic payment system integration. The benchmark covers 9 projects specific to real products and a total of 18 task instances.

Basic and Advanced tasks

Tasks are split into two categories: “Basic,” which checks functional completeness of the integration, and “Advanced,” which tests risk-aware hardening — the agent’s ability to recognize and fix security flaws before code goes into production.

Results across six coding-agent models

Researchers tested six different coding-agent models on both task types. With access to the dedicated “alipay-payment-integration” skill, the average rubric pass rate (RPR) — a metric measuring how well a solution satisfies a predefined scoring rubric — ranges from 68.58% to 91.37%, depending on the model.

How much does the skill actually help?

The skill alone raises RPR by an average of 10.31 percentage points compared to working without it, though the difference varies by model, product, and scenario. The comparison shows that specialized knowledge of payment integration measurably reduces the number of errors, without eliminating the capability gaps between models.