The Day My Chatbot Changed: Characterizing the Mental Health Impacts of Social AI App Updates Through Negative User Reviews

Why Read This from an HCI Perspective

This article argues that AI chatbots should be viewed not as ‘high-performing models,’ but as ‘interaction products that keep changing.’ By examining how version updates relate to user ratings and expressions of dissatisfaction through a large-scale review dataset, it offers UX practitioners clues about what signals to watch after a release. For researchers, it prompts thinking about how to detect change, read failure modes, and interpret expectation breakdowns from review text.

CIT's Commentary

A key strength of this study is that it treats updates not as a simple deployment event, but as interaction changes that disrupt user experience. In particular, the finding that negative reviews are framed more as ‘comparisons with the previous version’ than as ‘emotional outbursts’ is especially important. Users tend to react more strongly not to the model itself, but when familiar behaviors suddenly change. That means product improvements can’t rely only on performance metrics; they also need to include design that explains what changed and leaves users with intervention paths that allow them to return to their prior experience. However, because review data alone makes it difficult to separate true causes, follow-up research is needed that combines release notes with real usage logs. While LLMs can assist with this kind of analysis, the criteria for classifying negative experiences and the measurement instruments themselves must be rigorously validated.

Questions to Consider While Reading

• Q.To reduce dissatisfaction caused by version changes, how should release notes and in-app guidance be presented so that users’ expectations don’t collapse?
• Q.To distinguish signals of ‘technical errors’ from ‘psychological risk’ revealed in negative reviews, what additional data or research design would be most useful?
• Q.When using LLMs to automatically classify the sentiment or topics of reviews, how can we design validation procedures that preserve measurement rigor while still being usable for real UX analysis?