ArXiv SPPO: Sequence-level PPO solves the credit assignment problem in long reasoning chains

The problem with token-level PPO

Standard PPO (Proximal Policy Optimization) is the central algorithm for aligning LLMs on reasoning tasks with verifiable rewards. However, token-level PPO struggles with two problems:

1. Credit assignment instability — across long Chain-of-Thought (CoT) chains, assigning credit to individual tokens becomes unstable
2. Prohibitive memory costs — the value model requires significant resources

Critic-free alternatives like GRPO mitigate these issues, but require multiple sampling to estimate the baseline, which drastically limits training throughput.

SPPO — the best of both worlds

The team (Wang, Li, Li, Chen, Huang et al.) introduces Sequence-Level PPO (SPPO) which reformulates the reasoning process as a Sequence-Level Contextual Bandit problem.

Key innovation: a separate scalar value function that provides low-variance signals without the need for multi-sampling.

Results

On mathematical benchmarks, SPPO:

• Significantly outperforms standard token-level PPO
• Achieves the performance of computationally expensive group methods (GRPO)
• Dramatically more efficient — no multi-sampling overhead

For researchers training reasoning models, SPPO offers a practical alternative: GRPO performance at costs closer to standard PPO.