arXiv:2605.29157: Parallax local linear attention accelerates decode phase 12.9× compared to FlashAttention
Parallax is a new attention mechanism for large language models that replaces standard softmax attention with local linear estimation, achieving a 12.9× speedup of the decode kernel compared to FlashAttention. Researchers from Northwestern University and collaborators demonstrated consistent perplexity improvements during pretraining of 0.6B and 1.7B parameter models, claiming the first empirical demonstration of strong architecture-optimizer co-design for attention mechanisms.