Show HN: GPT-2 inference in pure C#, 0 bytes allocated per token (github.com)

A new project has emerged showcasing GPT-2 inference implemented entirely in pure C#, promising zero memory allocation per token during processing. This breakthrough is significant for the AI and machine learning community as it eliminates the need for dependencies like native binaries, Python runtimes, or ONNX Runtime, thereby advancing the potential of C# in AI application development. By leveraging a managed environment, the engine maintains predictable CPU performance through preallocated buffers and optimized memory management techniques. The project supports loading GPT-2 Small (124 million parameters) from HuggingFace and employs key innovations such as KV-cache decoding, achieving an impressive token generation speed of 71.4 tokens per second. Notably, it replicates PyTorch's output for top logits with minimal deviation, affirming its reliability. The implementation's architecture also facilitates seamless integration with ONNX models through direct imports, maintaining comparable performance metrics while ensuring zero per-token allocation. As such, this development presents an important step forward in creating efficient, resource-conscious AI applications capable of significant inferencing with minimal overhead.