Statistical Significance Is Not the Same as Practical Significance

Why Read This from an HCI Perspective

This article summarizes common misinterpretation errors in quantitative usability studies, specifically the confusion between 'statistically significant' and 'practically important' from an HCI perspective. It warns against making decisions based solely on p-values and emphasizes considering user perception, task impact, and effect size together. This approach provides practical guidance for UX practitioners and researchers to translate research findings into product priorities.

CIT's Commentary

From a CIT perspective, the core message of this article is to remind us that 'measurable' does not necessarily mean 'valuable.' In HCI, statistical significance supports the reliability of research, but actual design decisions should also account for perceived user experience changes and contextual costs. Especially with large-scale log data, even tiny differences can appear significant; without considering effect size and business context, this can lead to over-optimization. Conversely, strong behavioral patterns observed in small samples can be practically important. Therefore, an interpretive framework that combines quantitative and qualitative evidence is necessary.

Questions to Consider While Reading

• Q.What criteria do we use to distinguish between 'statistically significant differences' and 'differences that genuinely change priorities' in our key product metrics?
• Q.What checklists can we incorporate into our current research process to jointly assess effect size and user perception?
• Q.In small-sample quantitative studies where significance is not found, what qualitative data can we combine to capture practically important signals?