Show HN: Catch your local LLM falling back to CPU (github.com)

The launch of Picchio, a new Python tool designed for diagnosing local LLM (Large Language Model) performance, has created a buzz in the AI/ML community. This single-file application allows users to monitor their GPU usage while running LLMs, providing granular insights into the model's processing speed across three key metrics: prefill, decode, and wallclock. By analyzing the engine's logs, Picchio reveals whether the GPU effectively engaged during execution or if the model fell back to using the CPU, which can significantly impact performance. For instance, in tests carried out on an Apple M5, a drop in the number of tokens processed illustrated how models may report misleadingly high speeds without clearly indicating how much time it takes to generate the first response. The significance of Picchio lies in its ability to demystify LLM performance data, allowing developers to pinpoint discrepancies in speed readings typically shared within the community. With its straightforward implementation—requiring no complex dependencies—this tool empowers users to verify performance claims accurately before sharing them. By providing detailed breakdowns of processing times and computational lanes, Picchio encourages a more nuanced understanding of LLM efficiency, ultimately fostering more accurate benchmarks and discussions within the rapidly evolving field of AI.