Show HN: Phobos – a tiny kernel language that now runs LLMs on a 2019 GPU (www.joa-ebert.com)

Phobos, a compact kernel language, has successfully transitioned from a learning project to a fully functional inference engine for large language models (LLMs) running on older GPUs, specifically the 2019 RTX 2080 SUPER. This significant achievement allows developers to execute LLMs end-to-end without relying on traditional libraries like cuBLAS or CUDA, making it particularly relevant for users with less powerful hardware. Phobos achieves impressive text generation speeds, performing comparably or even faster than existing models on certain benchmarks, while also demonstrating strong performance in dynamic caching and handling quantized models effectively. With the release of Phobos v0.1.0, users can now explore an innovative alternative for running LLMs that emphasizes real-time kernel compilation and efficiency. Key technical advancements include the successful implementation of FlashAttention and optimizations in kernel execution that boost decoding speeds up to 324 tokens per second. The language excels in processing quantized models and utilizes techniques like dynamic memory management and optimized thread mapping to enhance performance. This development not only addresses the limitations of modern inference frameworks for older GPUs but also opens new avenues for research and optimization in the AI/ML community, especially for those utilizing low-end hardware configurations.