AI Psychosis: Does Conversational AI Amplify Delusion-Related Language?

Why Read This from an HCI Perspective

This article shows, from an HCI perspective, how conversational AI can amplify the language and thinking tendencies of vulnerable users—and how such risks can be measured and mitigated. In particular, by covering multi-turn interactions, state tracking, and safety interventions together, it offers direct implications for emotional support, reflective conversations, and agent design. It also prompts both practitioners and researchers to see that ‘safety’ is not a static filter, but a problem that emerges throughout the interaction process.

CIT's Commentary

From a CIT perspective, the core value of this study lies in its attempt to quantify not just ‘what the AI said,’ but ‘how the conversation accumulates and what cognitive pathways it creates.’ While SimUser and DelusionScore do not replace actual clinical diagnoses, they are meaningful as HCI instrumentation tools for tracking risk signals in long-term interactions. That said, you should not directly infer real-world vulnerability or clinical outcomes from Reddit-based simulations and language scores alone. Going forward, the design should also account for uncertainty in user state estimation, the possibility of overprotective interventions, and the cost of false positives when users seek help. Ultimately, this paper well illustrates the need for state-aware safety design that distinguishes when ‘empathetic responses’ become ‘amplifying.’

Questions to Consider While Reading

• Q.In real user conversations, how can state signals like DelusionScore be estimated safely, and how should we calibrate the costs of false positives and false negatives?
• Q.Even if state-aware interventions reduce delusion-related language, how can we evaluate whether users’ perceived empathy, trust, and autonomy are not simultaneously diminished?
• Q.When translating results from Reddit simulations into real product policies, which categories of conversations should be mitigated, and which should remain as more general support?