Gisting (shopify.engineering)

Shopify has implemented a novel technique called gisting, which reduces the length of system prompts for language models from approximately 6,000 tokens to around 1,500 tokens. This 4:1 compression enables the model to maintain high prediction quality while significantly improving performance metrics. By utilizing a method of knowledge distillation to train special gist tokens, Shopify’s Sidekick GraphQL agent saw a reduction in median time to first token from 438ms to 354ms, end-to-end latency from 6.8s to 4.2s, and an increase in throughput from 20.2 to 23.4 queries per second. This leads to substantial savings in GPU resource allocation, allowing Shopify to operate with 14% fewer GPUs while still improving efficiency. The gisting technique's implications are profound for the AI/ML community, as it addresses the ongoing challenge of balancing system prompt length with computational costs. By efficiently compressing prompts, gisting reduces the burden on memory bandwidth and enhances the model's ability to handle requests concurrently. Additionally, the approach integrates seamlessly with continual learning, enabling ongoing improvements to the model without the extensive computational expense of initial training. This innovation not only optimizes AI operations but also maintains high quality, reflecting the growing demand for efficient and effective AI solutions in an increasingly competitive landscape.