Show HN: A/B Test Your LLM Prompts in Production (switchport.ai)

A Show HN demo outlines a light-weight service for A/B testing LLM prompts in production: you register prompt "keys" and then execute them through a simple API (example: client.prompts.execute(prompt_key='welcome-message', user={'user_id':'user_123'}, variables={'name':'Alex'}) returns response.text). The platform handles versioning and traffic routing so different prompt variants can be served to different users or cohorts, and it automatically collects response data for measurement. For AI/ML teams this is effectively feature-flagging + experimentation for prompt engineering: you can iterate on prompt wording, template variables, or model choices and measure outcomes (latency, token usage/cost, accuracy, user engagement, safety metrics) in the wild. Key technical implications include per-user targeting, deterministic variable interpolation, experiment traffic split/canary rollouts, and integration with observability/metrics to evaluate significance and roll back poor variants. This shifts prompt tuning from ad-hoc manual testing to a data-driven MLOps workflow—speeding optimization while helping catch regressions—but it also requires careful metric design, privacy controls, and safety checks when evaluating live model behavior.