Llama 2 inference from scratch in C++20 (No PyTorch/GGML, ARM NEON) (github.com)

A new repository reveals a high-performance inference engine for the Llama 2 architecture, implemented in C++20 without relying on external frameworks such as PyTorch or TensorFlow. This groundbreaking development is showcased in the paper "Bare-Metal Tensor Virtualization: Overcoming the Memory Wall in Edge-AI Inference." Key enhancements include Zero-Copy Memory Mapping for weight management, a Structure-of-Arrays layout to optimize cache-line utilization, and hand-tuned ARM NEON SIMD kernels. This setup ensures deterministic, jitter-free performance on Apple Silicon, achieving impressively low latencies below 200ms for real-time interactions. The significance of this work lies in its potential to enhance AI inference on ARM64 architectures while exposing the limits of general-purpose CPUs and memory bandwidth. Unlike PyTorch's reliance on exclusive hardware features, this engine runs on any ARM64 device, from Raspberry Pi to AWS Graviton, boosting portability and accessibility. Additionally, the research addresses critical latency issues that impact models using more complex decoding strategies. With thorough technical insights, this implementation opens new pathways for edge-AI adoption, making high-performance inference possible without high-end GPU resources.