arXiv:2605.15514: RoPE mathematically cannot distinguish positions or tokens in long contexts — theoretical proof of a fundamental limitation

What is RoPE and why does it matter for all modern LLMs?

Large language models (LLMs) are based on transformer architecture, which cannot inherently know where each token is located in a sequence. Positional encoding solves this problem: it assigns each token information about its position in the context. Without it, a model would not distinguish “dog bites man” from “man bites dog.”

Rotary Positional Embeddings, better known as RoPE, are today’s dominant standard for that task. Introduced in a 2021 paper, they have since become an integral part of nearly all relevant architectures: Meta Llama across all generations, Mistral, Qwen, GPT-NeoX and numerous derivatives. RoPE encodes relative positions between tokens via rotations in vector space — an elegant mathematical solution that works well in short and medium-length contexts.

What RoPE mathematically cannot do in long contexts

A new arXiv paper (2605.15514) “RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably” by Yufeng Du, Phillip Harris, Minyang Tian, Eliu A. Huerta, Srikanth Ronanki, Subendhu Rongali, Aram Galstyan and Hao Peng presents a formal theoretical proof of two fundamental limitations.

Loss of local position bias. In normal operation, the attention mechanism should favor nearby tokens — semantic context usually comes from neighboring sentences, not from distant paragraphs. The authors prove that as context length grows, RoPE ceases to exhibit this bias: the model becomes equally likely to direct attention to a token at position 1 as to a token at position 10,000. The error rate in distinguishing near from far positions approaches 50%.

Loss of token consistency. An even more serious problem is that the same token can receive diametrically opposite attention score values at different positions in the context. A key vector that receives high attention at one position may receive low attention at another — without any semantic justification. Moreover, the attention score can remain unchanged even when a token is moved or replaced with a different token.

Both degradation effects in the theoretical analysis converge toward an error rate of 50% — which is practically equivalent to random guessing.

What are the implications for long-context LLMs?

The practical consequences are significant. Industry has been intensively working in recent years to extend LLM context windows — from 4,000 tokens to 128,000, 1 million and beyond. Models are marketed precisely by their ability to process long documents, knowledge bases and complex queries. This paper mathematically calls into question the foundations of that capability for all architectures using RoPE.

The authors specifically examined whether the problem is solvable within the existing RoPE framework. Tuning the base parameter (RoPE base), a technique already used for extending the context window, shows an inverse relationship: increasing the base improves token distinction but inevitably sacrifices position distinction. This is a fundamental trade-off, not a technical detail that can be patched. Neither deeper networks nor multi-head attention architectures can bridge this theoretical limitation.

What comes next — new positional mechanisms?

The authors conclude that the deep integration of RoPE into all leading architectures does not mean the problem was known or accepted, but rather that it has only now been formally proven. Their recommendation is clear: fundamentally new mechanisms for encoding positions and token order in transformer models are needed.

The paper spans 35 pages and 11 figures, and represents one of the rare works that — using theoretical tools rather than purely empirical benchmark tests — addresses a fundamental architectural weakness of an entire generation of LLMs. Whether this will prompt research labs like Meta AI, Mistral AI or Alibaba (Qwen) to redesign positional encoding in the next generation of models remains an open question.