SWARR: Reinforcement learning makes sliding-window attention competitive in mathematical reasoning

The standard self-attention mechanism driving modern large language models has a fundamental limitation: it scales quadratically with context length. Every token must be compared to every other token, which becomes computationally prohibitive for long inputs. An alternative exists — sliding-window attention (SWA), which limits each token to a local window of neighboring tokens and offers linear complexity — but historically at the cost of a significant performance drop on demanding tasks.

Eight researchers — Kai Liu, Peijie Dong, Xinchen Xie, Jianfei Gao, Qipeng Guo, Xiaowen Chu, Shaoting Zhang, and Kai Chen — published on arXiv a paper presenting SWARR (Sliding-Window Attention via Reinforcement Learning): a two-stage approach that significantly reduces that performance drop.

The problem of quadratic scaling and long context

Full self-attention gives every token access to all other tokens in the context — extremely powerful for tasks requiring long-range dependency tracking. Mathematical reasoning is precisely such a task: a model must follow multi-step deduction chains, maintain intermediate results, and refer back to earlier steps.

Sliding-window attention physically restricts this: a token sees only neighboring tokens within a fixed-size window. Previous attempts to convert models from full attention to SWA resulted in significant accuracy drops on exactly these kinds of tasks, limiting the practical adoption of this otherwise attractive approach.

How SWARR solves the problem in two stages

SWARR is a two-stage approach:

Stage 1 — Supervised Fine-Tuning (SFT): A pre-trained model with full self-attention is converted to a sliding-window attention model via standard SFT. This stage is a necessary starting foundation, but the authors show it is insufficient on its own for demanding reasoning tasks.

Stage 2 — Reinforcement Learning (RL) policy adaptation: The model undergoes an RL phase in which reasoning trajectories are optimized specifically tailored to the architectural constraints of the sliding window. This is the key innovation of SWARR.

Key insight: data-architecture mismatch

The authors identify the fundamental problem that SFT cannot solve: data-architecture mismatch. Training datasets are typically generated by models with full self-attention, which freely exploit long-range dependencies. When such a model is converted to a sliding window, a fundamental disconnect arises — learned reasoning strategies assume access to information the sliding window physically cannot reach.

SFT optimizes the model on ready-made data that was generated without SWA constraints. Reinforcement learning, by contrast, generates its own reasoning trajectories — an on-policy approach that is inherently aligned with the architecture. The model learns to reason in a way that corresponds to what it can actually see. The authors position this insight as the paper’s central contribution.

Results on mathematical reasoning

Experiments on mathematical reasoning tasks show that SWARR significantly narrows the gap between sliding-window attention models and full-attention models — while simultaneously retaining the efficiency gains of linear complexity at long contexts.

The authors conclude that RL “fundamentally changes the conclusions about the applicability of sliding-window attention” to mathematical tasks — transforming it from a merely “acceptable” substitute into a genuinely competitive alternative.

Implications for efficient language models

SWARR opens the way to models that can process long contexts without quadratic computational cost, while maintaining high performance on demanding reasoning tasks. This is relevant for a broad spectrum of applications: from analysis of long legal documents and source code, to solving complex mathematical problems.

More broadly, the paper shows that carefully designed RL can be the key tool for adapting LLM architectures originally trained with one attention mechanism to architecturally different, more efficient alternatives — without the need to retrain from scratch.