PostHog watches user sessions with multi-modal LLMs (in 5 not-so-easy steps) (posthog.com)

PostHog has launched a beta feature called Session Summaries, designed to leverage multi-modal LLMs for analyzing user behavior more efficiently. By capturing extensive user interaction data—ranging from clicks to navigation patterns—the tool automatically highlights significant issues across countless user sessions, alleviating the burden of manual reviews. This advancement is particularly vital for companies dealing with large data scales, as it enables quicker identification of user experience problems and improves the overall feedback loop for product development. The technical implementation involves several innovative strategies, including adhering to minimal contextual input for LLM processing to avoid losing relevant details, and utilizing video snapshots to enhance accuracy in issue detection. PostHog strategically combines event data with video analysis to enhance the reliability of the insights provided—thereby reducing the likelihood of parsing errors from LLMs, such as the "crying wolf" effect associated with false positives. Furthermore, the platform employs a multi-phase summary extraction pipeline to identify patterns across sessions, ensuring that issues are well-documented and actionable with verifiable evidence, ultimately aiming to improve user experience and satisfaction.