arXiv:2605.04012: SymptomAI in the Fitbit app with 13,917 patients outperforms independent clinicians in differential diagnosis

A team including researchers from Google and Fitbit has published a preprint on SymptomAI, a conversational agent integrated into the Fitbit app for everyday symptom assessment. The study deployed five different AI agents to approximately 13,917 participants to measure their diagnostic utility in real-world conditions.

What did the study actually measure?

The conversational agent guides the user through a structured conversation about symptoms and offers a ranked differential diagnosis — a list of possible diseases by probability, which clinicians use before confirming the final diagnosis with tests.

In the clinical evaluation, 1,228 participants had confirmed diagnoses, while 517 underwent blinded assessment through clinician panels with over 250 hours of annotation. Results were validated on an additional 1,500+ participants from general US panels.

How reliable is the result?

The diagnostic recommendations of the SymptomAI agent achieved statistically significantly better results than independent clinicians who evaluated the same conversations, with an odds ratio of 2.47 (p < 0.001). Agents that used a dedicated symptom interview and gathered additional information before diagnosis significantly outperformed user-guided variants.

An important caveat: the paper is a preprint without peer review, the focus is on evaluating conversational transcripts, and clinicians in the control group had no access to the patient in person, physical examination, or additional tests. The authors themselves note the limitation of self-reported ground truth when analyzing data from wearable devices across nearly 400 conditions. This work demonstrates the potential of home AI symptom assistants, but does not change clinical practice until it undergoes independent replication and regulatory evaluation.