I know you run LLMs. Will it boot? (mondegreens.github.io)

A new open-source tool called Apron has been developed to streamline the process of running large language models (LLMs) on various GPU configurations. By predicting a model's memory requirements from its configuration and safetensors headers, Apron helps users determine if a model can be run on specific hardware before incurring costs, significantly aiding those in the AI/ML community who regularly experiment with different models. The tool has demonstrated impressive accuracy, with predictions for most models landing within 2% of the actual measurements once run on GPUs. The significance of Apron lies in its ability to mitigate common deployment challenges associated with LLMs, such as rapidly changing model specifications and hardware configurations. As model designs and engine versions evolve frequently, deploying an open model often results in unpredictability regarding performance and memory constraints. Apron addresses these issues by maintaining a record of predicted versus actual performance, allowing users to optimize their GPU usage effectively. This not only saves time and resources but also contributes to a more informed and efficient approach to AI/ML model deployment.