You Are (Probably) Measuring Time Wrong (www.counting-stuff.com)

A recent presentation at Quant UX Con by Michał Chorowski highlighted critical flaws in conventional methods of measuring time metrics in user experience (UX) research. Traditionally, metrics like time-to-onboard or task completion time rely on averages from user data, but this approach often suffers from biases such as Survivorship Bias and Differential Follow-up Bias. These issues lead to systematically optimistic outcomes by excluding users who fail to complete tasks, thus misrepresenting the actual user experience and productivity metrics. For example, if only successful users are considered in time calculations, it can create an inaccurate sense of efficiency, obscuring problems faced by those who drop off during the process. To address these biases, Chorowski advocates for the use of Survival Analysis, a statistical method traditionally employed in fields like medicine that can better capture the dynamics of user behavior over time. By focusing on survival functions and employing tools such as the Kaplan-Meier estimator, researchers can obtain a more accurate view of conversion rates and task completion times, considering not just those who succeed but also those who do not complete the task within the analyzed timeframe. This shift in methodology promises to enhance the precision of UX metrics, providing a clearer understanding of user engagement and helping teams develop more effective strategies.