You don't need an LLM to cluster LLM traces (seldon-ai.com)

Seldon has unveiled Trace Audit v0, a tool designed to analyze and cluster repetitive traces from OpenAI-compatible gateways, ultimately aiming to compile these traces into more efficient and cost-effective pipelines. This initial phase focuses on accurately identifying which traces represent the same underlying extraction program, rather than just similar topics. Through stress-testing on 12,517 traces, Seldon achieved precision, recall, and purity rates all hovering around 1.000, using a structured approach that clusters based on deterministic features, rather than relying on text embeddings or summaries from large language models (LLMs). The significance of this development lies in its potential to streamline the workflow for teams working with LLM products, who often face rising costs as repetitive queries dominate their usage. By focusing on the program identity—categorizing traces that are functionally equivalent despite different data inputs—Seldon allows businesses to map their workflows more effectively. The findings emphasize that traditional methods, which often cluster based on text or topic similarity, can lead to misleading assumptions in program synthesis. The upcoming synthesis phase promises to convert these clusters into actionable, versioned pipelines, ensuring transparency and reducing reliance on opaque LLM models.