Fastllm: A LLM inference library that runs DeepSeek-V4 with 10GB VRAM (github.com)

Fastllm has introduced a new high-performance inference library for large language models (LLMs), compatible with various GPUs, including NVIDIA and AMD, and requiring only 10GB of VRAM for full model inference. The library, built in C++, replaces PyTorch operators, enabling efficient deployment of models like DeepSeek-V4, Qwen, and Phi. Fastllm's functionality includes support for FP8 inference on any GPU, multi-card tensor parallelism, and a user-friendly installation process. This development is significant as it allows for the efficient inference of large models on a wider range of hardware, including older GPUs, substantially lowering the barrier to entry for deploying sophisticated LLMs. Fastllm supports dynamic quantization, enabling the export of optimized model weights, and offers features like CPU-GPU mixed inference and dynamic batching. By presenting a more accessible solution for utilizing LLMs, Fastllm strengthens the AI/ML community's capacity for innovation and experimentation with large datasets and complex model architectures.