arXiv:2607.08173: 'Overthinking' — Amplified Reasoning Forces Reasoning Models to Reveal Learned Secrets, New Extraction Attack at ICML 2026

The paper “Overthinking: Extracting Learned Secrets” (arXiv:2607.08173), by Hopkins, Khullar, and Roger, was accepted at ICML 2026 and describes a new class of attacks on reasoning models. A reasoning model is a language model that performs an extended internal chain of thought before producing its final answer — a mechanism that raises accuracy on complex tasks but, as the paper shows, also opens an unexpected security channel.

What Is the Attack?

The authors show that by deliberately amplifying the model’s “reasoning weight,” it can be induced to expose in its chain of thought information it learned during training but would not reveal in a normal response. In other words: the more a model “thinks,” the greater the chance it inadvertently leaks latently stored secrets — from fragments of training data to internal instructions.

Why Is the Paradox Dangerous?

Extended reasoning is the flagship capability of the new generation of models — OpenAI’s o-series, DeepSeek R1, and Claude with extended thinking. That very mechanism, which the market pays a premium for, Overthinking turns into a vulnerability: any defense that limits leakage could simultaneously weaken useful reasoning. This poses a concrete trade-off for teams deploying reasoning models in production.

Context

Acceptance at ICML 2026, one of the three most prestigious ML conferences, gives the finding more weight than a typical preprint. The paper arrives the same day that GitHub introduces prompt-injection detection in CodeQL — together they illustrate how 2026 security research is increasingly focused on attacks targeting the models themselves rather than just the applications around them. For operators of reasoning models, the message is clear: deeper thinking demands deeper control over what the model is allowed to say.