Can a language model run in Linux eBPF? (eunomia.dev)

A recent experiment tested the feasibility of running a real language model, specifically the Qwen3-0.6B, within the Linux extended Berkeley Packet Filter (eBPF) environment. This unique approach shifts the paradigm from eBPF's traditional role—monitoring and influencing system calls—to executing a language model's numerical operations within the kernel. The prototype successfully performs the model's 28 decoder layers and generates tokens directly in the kernel, albeit at a relatively slow pace of about 1.2 seconds per token, which highlights the current limitations of eBPF for intensive tasks typically suited for CPUs or GPUs. The significance of this work for the AI/ML community lies in the exploration of eBPF's capabilities for in-kernel inference, which could minimize latency and overhead associated with user-kernel transitions. Key technical challenges remain, especially concerning the model's memory usage and the trade-offs between different numerical representations, such as using INT4 weights for potential efficiency gains, albeit with accuracy sacrifices. The findings illustrate that while it’s possible to run effective computations in eBPF, further refinements in weight formats and memory handling are necessary for achieving practical performance in real-world applications.