Tiny LLM Benchmark: Jetson Orin Nano Super 8GB (www.smolhub.com)

The recent benchmarking of eight models on the NVIDIA Jetson Orin Nano Super 8GB has revealed significant differences in performance and energy efficiency between two backends: llama.cpp and Ollama. Each model was tested at four power modes (7W, 15W, 25W, and MAXN) using a variety of prompt and generation length combinations. A key finding was that the 25W power mode emerged as the "pareto sweet spot," delivering 35-47% more output tokens per second compared to 15W, while also exhibiting improved output tokens per joule across all models. Notably, llama.cpp outperformed Ollama by 36-74% in throughput for sub-1B transformer models, illustrating the effective efficiency of its CUDA backend. The implications of this benchmark are significant for the AI/ML community, particularly in optimizing model deployment for constrained environments. The results suggest that developers and researchers looking to maximize the throughput and energy efficiency of their models should favor llama.cpp, especially at the 25W setting. With detailed telemetry data and extensive performance metrics available for all benchmarks on Hugging Face, researchers have a comprehensive resource to refine their models and applications, paving the way for more efficient AI solutions on compact hardware platforms.