Seismograph – open-source early warning for silent LLM API drift (github.com)

Tatiana Radchenko has announced SEISMOGRAPH, an open-source solution designed to provide early warnings for silent drift in large language model (LLM) APIs. As LLM providers often update their models without notification, SEISMOGRAPH addresses the critical risk of undetected semantic drift that could lead to disruptions in applications relying on APIs. By employing a privacy-preserving network that continuously monitors model responses via a fixed canary suite, it detects behavioral changes before they affect users. In a backtest, SEISMOGRAPH identified a significant drift in Anthropic's Claude Sonnet 4 model 38 days prior to the official postmortem, showcasing its potential to preemptively mitigate risks for teams relying on external LLM services. Technically, SEISMOGRAPH utilizes a combination of SHA-256 hashing for response verification and differential privacy techniques to maintain confidentiality while monitoring model outputs. The system employs a Page-CUSUM change-point detection algorithm, facilitating collaboration among different organizations to confirm detected drifts without allowing a single entity to trigger false alarms. This multi-party verification requires a minimum quorum for public alerts, significantly enhancing the reliability of drift detection. As the AI/ML community increasingly relies on third-party LLM APIs, SEISMOGRAPH's innovative approach promises to safeguard applications against unexpected model behaviors, maintaining application integrity and user trust.