Allen Institute: which tokens does a hybrid model (OLMo 3) predict better?

What are hybrid architectures and why do they matter?

A hybrid architecture combines an SSM (state-space model — a model that processes text sequentially with linear complexity) with classic Transformer layers. While Transformers use an attention mechanism that looks at all tokens simultaneously, an SSM processes the sequence step by step, similar to recurrent networks. The Allen Institute (AI2) investigated how this combination affects which tokens a model predicts more accurately.

Where does the hybrid win — and where does it lose?

The analysis of OLMo 3 and OLMo Hybrid models reveals a clear division. Hybrid architectures better predict semantic, context-dependent tokens — those requiring an understanding of the broader meaning of a sentence or paragraph. Pure Transformers, however, retain an advantage when the task is verbatim text copying, where the model must reproduce an exact sequence without interpretation.

Connection to the open OLMo line

Both analyzed models are part of the open OLMo 3 line that AI2 develops as a transparent alternative to closed LLMs. Token-level research helps the team optimize the ratio of SSM to Transformer layers in future versions — instead of random mixing, design becomes empirically driven.