Google: New statistical framework for auditing data forgetting in AI models

Mónica Ribero, a researcher at Google Research, published on June 10, 2026, a new statistical framework for auditing machine unlearning — the technical mechanism by which AI models should “forget” specific data that was part of their training. The paper was accepted for presentation at AISTATS 2026.

Why is forgetting in AI systems a problem?

Imagine a company that used personal user data to train a recommendation model, and a user subsequently requests deletion of all their data under the GDPR Right to be Forgotten. Fully retraining the model from scratch — without that data — is computationally extremely expensive and rarely feasible in practice. Instead, machine unlearning algorithms are developed to modify an already-trained model so that it “forgets” targeted data.

But how do you verify that forgetting actually succeeded? This is precisely the regulatory and technical problem this paper addresses.

The fundamental flaw of existing methods

The standard approach to unlearning auditing used two-sample tests: the distribution of the unlearned model is compared to a baseline model and the statistical difference is measured. The problem is fundamental — models can never perfectly forget data through parameter adjustment alone. A difference always remains, which leads standard tests to incorrectly flag correctly trained, safe models as non-compliant.

Ribero identified this weakness and designed a framework that structurally avoids it.

Regularized f-Divergence Kernel Tests: multiple sensors for distributions

The central innovation is called “Regularized f-Divergence Kernel Tests” — an adaptive framework that simultaneously uses multiple different divergence measures, analogous to having several sensors in a single instrument, each detecting a different type of deviation:

• Chi-squared divergence — emphasizes smooth, localized differences in the distribution
• Kullback-Leibler (KL) divergence — the standard measure for global deviations in probability distributions
• Hockey-stick divergence — specifically designed for privacy and unlearning definitions, with a controlled threshold that corresponds to formal privacy guarantees

A key advantage: the framework automatically selects the optimal divergence and hyperparameters for each scenario, without manual tuning, which was previously a source of errors and non-reproducibility.

Three-sample relative-distance test

The most important methodological innovation is the three-sample relative-distance test. Instead of the binary question “does the unlearned model differ from the baseline?”, the test asks a three-way question:

Is the distribution of the unlearned model closer to the safe, retrained version or to the compromised original version?

This shift in perspective eliminates false positives: a difference between the unlearned model and the retrained model always exists, but that difference does not necessarily mean that unlearning failed — it only means the model is not identical to a freshly retrained one. The relevant signal is which model is distributionally closer.

Experimental results of the framework

Evaluation of simplified unlearning algorithms produced clear and somewhat concerning results:

• Random label — passed evaluation; the three-sample test correctly identifies this variant as safe
• Finetuning — ineffective; targeted data remains present
• Pruning — ineffective; the network retains the requested information
• Selective Synaptic Dampening — ineffective; dampening synaptic connections is insufficient for true forgetting

Additionally, standard two-sample tests incorrectly flagged safe, retrained models as non-compliant — precisely the opposite of the desired outcome — while the new framework correctly identifies them.

Hockey-stick divergence outperforms DP-Auditorium

In the domain of differential privacy auditing — closely related to machine unlearning — the hockey-stick divergence variant of the framework outperforms the reference tool DP-Auditorium:

• It detected privacy violations in the sparse vector technique using only thousands of samples — compared to the millions required by the DP-Auditorium approach
• It significantly reduces the computational resources needed to audit large models

Practical implications for regulators and auditors

The framework addresses one of the most concrete regulatory challenges in applying AI legislation: mathematical proof of forgetting that auditors can submit to regulatory bodies without access to the internal parameters of the model or the original training data.

For GDPR compliance this is highly relevant: until now, organizations could only claim that unlearning had been performed. This tool enables statistical verification — the difference between a claim and evidence.

The paper is publicly available through the Google Research blog, with the full academic text expected in the AISTATS 2026 proceedings.