Allen Institute: EMO — MoE language model with natural semantic modularity from data

The Allen Institute for AI (Ai2) published EMO on May 8, 2026 — a sparse Mixture-of-Experts (MoE) language model that develops natural semantic modularity among its experts without manual labels. The model has 1 billion active and 14 billion total parameters, with 128 experts of which 8 are active per token, and was trained on 1 trillion tokens.

How does EMO achieve emergent modularity?

EMO uses document boundaries as a weak supervision signal: all tokens within the same document must choose active experts from a shared pool. This simple restriction is sufficient for experts to self-organize into coherent groups during training that can be selectively used and combined. MoE (Mixture of Experts) is an architecture in which only a subset of all available expert networks is activated per token, enabling large capacity at lower computational cost.

What do the pruning experiment results show?

When only 25% of experts are activated (32 out of 128), EMO loses just ~1% absolute performance, while at 12.5% of experts (16 out of 128) the drop is around 3%. Standard MoE models degrade dramatically under the same conditions, suggesting that EMO has functionally separate expert subsets covering distinct thematic domains.

Into which domains do experts cluster?

Visualization of expert activation shows clusters corresponding to semantic domains: “Health, Medical & Wellness”, “News Reporting”, “US Politics & Elections”, “Film & Music”. Standard MoE instead groups tokens by surface syntax — prepositions, definite articles and punctuation are scattered across clusters.

What is publicly available?

Ai2 released the full EMO model and a comparable standard MoE baseline on Hugging Face, training code on GitHub, and an interactive visualizer (emovisualization.netlify.app) that allows real-time exploration of expert activation by domain.