arXiv:2607.13562: AI Advice Suppresses People's Willingness to Say 'I Don't Know' — Even When Wrong

AI Advice Nearly Eliminates the ‘I Don’t Know’ Response

New research by a trio of authors — Chiara Marcoccia, Walter Quattrociocchi, and Valerio Capraro from Italian universities — shows that AI suggestions dramatically change how people assess their own uncertainty. In five experiments with a total of 3,132 participants (four pre-registered and one replication), respondents answered questions presented alongside deliberately wrong AI suggestions, while accuracy was financially incentivized. The results are troubling: willingness to say “I don’t know” nearly vanished, accuracy fell to one-third, and confidence almost doubled.

What Is Overreliance and Why Is It Dangerous?

Overreliance — or excessive dependence on AI — means uncritically accepting AI answers without independently evaluating their correctness. The research quantifies this clearly: where participants without AI help were accurate in, say, 60% of cases and chose “I don’t know” when uncertain, with AI suggestions (even wrong ones) they far more frequently gave an answer — and were wrong. More precisely, accuracy dropped to approximately one-third, meaning participants with AI support made errors two to three times more often than without it.

Is It Enough for the User to Simply Not Ask for Advice?

No. One of the study’s key findings is that even passive display of AI suggestions — without the user consciously requesting them — has an identical negative effect. This distinguishes the phenomenon from classic “automation bias,” where the user chooses to rely on the system. Here, the mere display of information changes epistemic behavior: people stop treating their own uncertainty as a legitimate option. Comparison with a control group (without AI) confirms this: the difference in the rate of “I don’t know” responses is statistically significant across all experiments.

Why Are These Findings Important for AI System Deployment?

The results directly affect the design of AI interfaces in education, medicine, and legal systems — anywhere high-stakes decisions are grounded in the user’s knowledge. The authors warn that the effect cannot be reduced to bad intentions or ignorance on participants’ part: incentivizing accuracy did not eliminate overreliance. This suggests that the architecture of how AI suggestions are displayed encourages cognitive surrender of responsibility to the machine, even when the machine is wrong. The study calls for regulatory and design interventions that would restore visibility of uncertainty to the user.