I'm making LLMs work on small and old devices (github.com)

A new initiative, dubbed "smolmaxxing," is enabling the operation of large language models (LLMs) on small and older devices, such as TV boxes and iPads, which typically have underutilized hardware capabilities. By optimizing performance through custom kernels, efficient memory management, and weight streaming techniques, these devices can handle models ranging from 260K to 16 billion parameters. For instance, a third-generation Xiaomi TV Box achieves up to 16.8 tokens per second using the Qwen3.5-0.8B model, while a first-generation iPad Air runs a 1.7B parameter model at an impressive 24-26 tokens per second, demonstrating the surprising potential of these devices. This development is significant for the AI/ML community as it expands access to LLMs, allowing users to leverage existing hardware for advanced AI applications without the need for expensive, high-end devices. The techniques developed are adaptable across various platforms, meaning any ARM-based device can potentially exploit similar gains. Additionally, the optimization strategies employed include notable improvements in processing efficiency and memory usage, which could pave the way for more cost-effective and eco-friendly AI deployment. This breakthrough could democratize AI technology, making it more widely accessible and usable in everyday contexts.