arXiv:2606.04557: Cartridges at Scale for KV cache

The Cartridges at Scale paper scales KV cache compression (the attention memory of language models) to document collections larger than one million tokens. The method beats monolithic cartridges by 10 to 31 points at the same token budget and, with retrieval, achieves accuracy equivalent to RAG while using 3 to 4 times fewer prompt tokens.

What does the Cartridges at Scale paper deliver?

Cartridges at Scale is a research paper published on 3 June 2026 on arXiv under the identifier arXiv:2606.04557 (version v1) that scales compression of the KV cache of language models to document collections larger than one million tokens. The KV cache (the key-value memory a model builds while reading the input text) traditionally grows linearly with context length, which makes large collections expensive and slow to process. The paper shows that the “cartridge” approach, until now limited to smaller amounts of text, can be extended to million-token contexts.

How large are the accuracy gains?

At the same token budget, the method improves results by 10 to 31 points compared to monolithic cartridges, that is, cartridges that condense an entire collection into a single block. Instead of one large summary, Cartridges at Scale builds several smaller, specialized cartridges and manages them together. The difference in results grows with the size of the collection, confirming that the monolithic approach loses information at large contexts.

How close is the method to full in-context learning?

The accuracy of the so-called oracle cartridge stays within 2 to 6 points of full in-context learning (learning in which the model has the entire content directly in the prompt), even at high compression. This is an important result because it shows that aggressive memory compression need not significantly sacrifice answer quality. The oracle cartridge in the paper serves as an upper bound that practical variants aim to reach.

How does Cartridges at Scale compare to RAG?

Combined with retrieval (fetching relevant parts of the collection), the method achieves accuracy equivalent to RAG (Retrieval-Augmented Generation), but with 3 to 4 times fewer prompt tokens. RAG is the standard approach that inserts retrieved passages of text into the prompt, but in doing so it consumes many tokens. Cartridges at Scale achieves the same result with a substantially smaller prompt, which directly reduces the cost per query and speeds up answer generation.

Which techniques enable the scaling?

The paper introduces two key components. The first is dynamic distractor mixing, a training technique that injects confusing distractor examples into the process so that the cartridge learns to better distinguish relevant from irrelevant content. The second is a memory-efficient budget manager that allocates resources across hundreds of cartridges per document without exceeding the available memory. Together, these components allow the approach to remain practical even at collections larger than one million tokens, where naive memory management would quickly become untenable.

Why does this matter in practice?

For organizations working with large document bases, such as legal archives, technical documentation or scientific literature, Cartridges at Scale offers a concrete path to lower costs without loss of quality. Reducing prompt tokens by 3 to 4 times while retaining RAG-equivalent accuracy means cheaper and faster production, which is crucial for systems with a high volume of queries. The paper thereby shows that attention memory compression can become a standard tool for working with million-token contexts.

Frequently Asked Questions

What is a cartridge in the context of language models?

A cartridge is a compressed form of the KV cache (the key-value memory a model builds while reading text) for some collection of documents. Instead of loading the entire content into the prompt every time, a cartridge condenses that memory into a smaller, reusable block. The goal is to reduce the number of tokens needed per query while preserving accuracy.

How many tokens does Cartridges at Scale save compared to RAG?

Combined with retrieval, the method achieves accuracy equivalent to RAG (Retrieval-Augmented Generation) but with 3 to 4 times fewer prompt tokens. This means lower costs and faster responses when working with large document collections.

What are dynamic distractor mixing and the budget manager?

Dynamic distractor mixing is a training technique that deliberately mixes in confusing (distractor) examples so that a cartridge better distinguishes relevant content. A memory-efficient budget manager allocates resources across hundreds of cartridges per document without exceeding memory.

arXiv:2606.04557: Cartridges at Scale compresses KV cache to millions of tokens