SymptomAI: Towards a conversational AI agent for everyday symptom assessment

Researchers conducted a large-scale study testing whether conversational AI agents could assess symptoms and generate diagnostic suggestions as accurately as human clinicians. The study involved thousands of research participants who described their symptoms to AI agents, which then provided lists of possible diagnoses, and clinicians later compared these AI suggestions against both their own assessments and the actual diagnoses participants received from healthcare providers. This matters because while language models have shown promise for diagnosis in controlled settings, their real-world performance remained unclear, particularly when everyday patients describe symptoms with varying medical knowledge and incomplete information. The results showed that clinicians preferred the AI-generated diagnoses over those from other clinicians in over half of cases, and the AI recommendations were more likely to include the diagnosis that participants' actual healthcare providers gave them.