Show HN: Pith – a proxy that cuts LLM output tokens, and refuses when it can't (github.com)

A new tool called Pith has been introduced to help optimize the output token usage of large language models (LLMs) without sacrificing information quality. Pith operates as a proxy that monitors LLM outputs, replaying generated responses under various token-efficient profiles while ensuring that these alternatives meet a predetermined quality standard, referred to as the "floor." If the generated responses do not match or exceed the baseline performance, Pith simply refuses to accept them, allowing developers to reduce costs effectively while maintaining data integrity. This development is significant for the AI/ML community as it addresses the increasing costs associated with deploying LLMs by potentially lowering output token consumption. The tool utilizes a systematic approach, employing a mock judge and sampling techniques to evaluate the quality of outputs, thereby confirming that any reductions in token usage are justifiable. By offering a straightforward installation and configuration process, Pith empowers practitioners to easily integrate it into existing LLM workflows, marking a notable advancement in optimizing AI model deployments for efficiency and cost-effectiveness.