Show HN: AdaptGauge – I found that adding few-shot examples can make LLMs worse (github.com)

AdaptGauge, an open-source evaluation harness, has been unveiled to measure Adaptation Efficiency in large language models (LLMs) when using few-shot examples. The tool quantifies how quickly a model improves with varying shot counts (0, 1, 2, 4, and 8 shots) and identifies instances of few-shot collapse, where performance can degrade as more examples are introduced. Traditional benchmarks typically assess accuracy at a single point, failing to address the nuanced learning processes in real-world applications where few-shot prompting is commonly utilized. This development is significant for the AI/ML community as it shifts focus from static accuracy metrics to dynamic learning efficiency, revealing that some models achieve optimal performance with only a few examples while others can falter with additional prompts. Notably, AdaptGauge showcases that leaderboards can reverse based on sample size, indicating that a model's utility is context-dependent. By automating the evaluation of model performance and resilience against few-shot collapse, AdaptGauge provides critical insights for developers, enabling more effective model deployment strategies in production environments.