I tried to make Gemini a UX researcher that audits websites, results were mixed

Why Read This from an HCI Perspective

This article is meaningful for both HCI practitioners and researchers because it demonstrates a workflow where an AI agent directly explores the web to perform a UX audit. In particular, it shows how well the Gemini + Playwright + Chromium combination can recreate real interaction contexts—and, conversely, under what conditions hallucinations or incorrect inferences occur. You can read it as a balanced look at both the promise and the limitations of automated evaluation.

CIT's Commentary

From a CIT perspective, the key point of this article isn’t whether AI can replace UX evaluation, but rather how far it can assist. Agent-based exploration is useful for quickly scanning screen elements, exploration depth, and repeated paths. However, it can easily lose context in flows with reduced visibility, hover-dependent interactions, or complex branching structures. In other words, the 70% of actionable feedback is valuable for initial diagnosis, but the 30% of hallucinations can actually lead to false positives in research design. Therefore, CIT considers it more appropriate to view such tools as a ‘heuristic screening + human verification’ pipeline rather than ‘evaluation automation.’ In particular, areas like accessibility, state transitions, and exception paths still require human evaluators’ involvement.

Questions to Consider While Reading

• Q.When people validate AI-based UX audit results, what sampling strategy is most efficient?
• Q.How can you improve the exploration reliability of an AI agent in interfaces with many hover interactions, asynchronous loading, and visual rendering failures?
• Q.If you combine these tools with heuristic evaluation or accessibility checks, how should you restructure the workflow in practice?