From Gaze to Guidance: Hinting and Navigating Based on Users’ Cognitive Needs with Multimodal Gaze-Aware AI Assistants

Why Read This from an HCI Perspective

This article treats AI not as a mere answer machine, but as an interactive system that reads the user’s sticking points and helps. Inferring a user’s state from eye-movement cues and then providing support based on that inference raises crucial questions for UX and HCI. In particular, it’s meaningful for both practitioners and researchers because it’s not only about accuracy—it also involves trust, when to intervene, and the possibility of misunderstanding.

CIT's Commentary

The core of this study is not a ‘smarter model,’ but an ‘interface that intervenes better.’ Eye information can be as useful as following a pencil tip during class to see where someone stops, but a pause doesn’t necessarily mean a lack of understanding. So, as much as the system needs to estimate correctly, it also needs to be designed so users can easily correct or skip it when it’s wrong. What’s interesting is that this framework can translate directly to domestic services. Even when applying AI summarization, search, or learning assistance in large platforms like Naver or Kakao, the bigger difference may come less from the ability to ‘get it right’ and more from reducing misunderstandings and shaping a flow where users can actively intervene. In addition, even if discomfort is inferred with LLMs, it naturally leads to meta-level research questions about how to preserve the validity of the measurement tools themselves.

Questions to Consider While Reading

• Q.When gaze-based inference is wrong, what kind of interface would allow users to correct it most easily?
• Q.It seems effective for relatively structured tasks like reading—would the same approach hold up in more complex real work settings?
• Q.If this approach is applied in the context of domestic services, what level should be designed for privacy and trust issues?