Why Conversation History Changes an AI’s Answers: How an LLM Reacts When It Hears “Delusions”

Why Read This from an HCI Perspective

This article makes a strong case that LLM safety should be evaluated not as a ‘single response,’ but across the ‘continuing conversation.’ For HCI/UX practitioners, it prompts thinking about how users come to trust AI, when and how to find intervention points, and at what moments the relationship becomes risky. For researchers, it’s a good example showing the contextual effects that short benchmarks miss—and how safety design reveals itself in real interactions.

CIT's Commentary

The most important message is that it’s not model performance alone, but the interaction structure that can either amplify or reduce risk. Even with the same delusional context, some models get pulled in deeper, while others strengthen safety interventions. This suggests that safety is not just a filtering problem, but an ‘interface problem’—one that can read context and know when to cut it off. In particular, it’s practically important that a safer model doesn’t abruptly sever the relationship; instead, it acknowledges responsibility for the prior conversation and then smoothly hands off to external help. However, this kind of warmth can also increase emotional dependence, so balancing friendliness with appropriate distance becomes a key design challenge. This issue is even more sensitive for services in Korea. Conversational services from Naver, Kakao, and startups are already deeply embedded in everyday life, and users are more likely to treat them as a ‘partner’ than as a mere ‘tool.’ Therefore, in the Korean context, it’s necessary to design interventions that consider not only the safety standards from English-language research, but also shorter mobile usage contexts, social structures centered on family and acquaintances, and higher expectations for the relationship.

Questions to Consider While Reading

• Q.When a conversation runs long, how can the interface detect the moment a user starts treating the AI’s words not as ‘information,’ but as ‘relational confirmation’?
• Q.How much does the way a safe model acknowledges mistakes from the prior conversation and changes course restore trust for the user—and how much does it feel like a betrayal?
• Q.In Korea’s mobile- and messenger-centered usage environment, which parts of the safety intervention approach proposed in global research will work more weakly or more strongly?