arXiv:2606.06888: Data-Constrained Pre-Training — the SoftQ Scaling Law and MIR Regularization

Researchers published a paper on June 5, 2026, on the arXiv repository (identifier arXiv:2606.06888) that addresses pre-training language models under conditions where available compute (computing power) grows faster than available natural text data. The paper introduces two key innovations: the regularization technique MIR and a scaling law named SoftQ.

Why is a shortage of data a problem at all?

Modern language models are trained on ever-larger amounts of computing power, yet the quantity of high-quality, unique text on the internet is not unlimited. When compute grows faster than available data, teams are forced to use the same data multiple times, which is called data repetition (repeated data exposure).

The problem is that repeating the same examples leads to overfitting — the model starts memorizing the data instead of learning to generalize. The paper arXiv:2606.06888 targets exactly this scenario, proposing methods that extract more useful information from a limited dataset.

What is masked-input regularization (MIR)?

The central contribution of the paper is MIR (masked-input regularization), i.e., regularization via masked inputs. It is an auxiliary next-token loss (predicting the next token) computed on randomly masked inputs during training. By masking part of the input, the model is forced to learn more robust representations rather than relying on surface patterns.

The authors tested MIR on models ranging from 72 million to 1.4 billion parameters. The results show that the combination of MIR + weight decay outperforms using weight decay alone, with measurable downstream gains (improvements on tasks that follow pre-training) on the 1.4 billion parameter model.

What does the SoftQ scaling law bring?

The second innovation is SoftQ, a new scaling law. Scaling laws mathematically describe how a model’s performance changes as model size and the amount of data change. The best-known such law, Chinchilla, uses an additive approach that does not capture the situation of repeated data exposure well.

SoftQ is designed precisely for conditions with limited data, because it better models the interaction between model size and data when data is repeated. This gives teams a more precise tool for predicting how much data and how many parameters they will need in real, data-constrained settings.

How significant are these gains and is the code available?

The authors quantify the gains from MIR by comparison with the amount of data: they estimate that the improvements correspond to roughly 1.3 times more unique data. In other words, by applying MIR, the model extracts from the same dataset as much as it would otherwise get from a significantly larger one.

It is especially relevant for the community that the authors released the code as open source, enabling other researchers to reproduce the results and test MIR and the SoftQ scaling law on their own models. In an era where there is increasingly loud talk about approaching the limits of available data, approaches like this could become a standard part of the pre-training recipe.