arXiv:2605.15514: RoPE mathematically cannot distinguish positions or tokens in long contexts — theoretical proof of a fundamental limitation
arXiv paper 2605.15514 provides a mathematical proof that Rotary Positional Embeddings (RoPE), the positional mechanism used by nearly all modern large language models including Llama, Mistral, Qwen and GPT-NeoX, loses the ability to distinguish positions and tokens in long contexts. The authors conclude that fundamentally new architectural mechanisms are needed.