DotLLM – Building an LLM Inference Engine in C# (kokosa.dev)

A new AI inference engine called dotLLM has been launched by a developer aiming to run large language models (LLMs) natively within the .NET ecosystem. Unlike existing options that rely on wrappers or external services, dotLLM provides a complete inference engine written in C#, enabling transformer-based model loading, tokenization, and efficient sampling directly in .NET applications. The first preview release (v0.1.0-preview.2) features SIMD-optimized CPU inference, CUDA GPU acceleration, and an OpenAI-compatible API server, making it a significant addition for .NET developers seeking local LLM deployment. The significance of dotLLM lies in its potential to fill a crucial gap in the .NET ecosystem, allowing developers to run LLMs without transitioning to external languages like Python or C++. The architecture includes performance features such as zero-alloc inference, memory-mapped model loading for rapid access to large models, and extensibility through pluggable backends. While the engine currently demonstrates around 66-88% of the performance of existing solutions like llama.cpp, the developer's journey illustrates the power of AI-assisted development and highlights that substantial systems-level performance can be achieved using C# and .NET tools. This marks a promising step for research and experimentation within the AI/ML community, particularly for those familiar with the .NET framework.