How to run Ministral 3 with an AMD GPU on Windows (www.50-nuances-octets.fr)

A tech enthusiast has successfully run the open-source model Jan on a Windows PC equipped with an AMD RX 9070 XT GPU, marking a significant step forward for the AI/ML community, especially for those looking to leverage smaller language models at home without incurring high cloud costs. After facing compatibility issues with the popular Ollama software, which did not support AMD GPUs adequately, the user turned to Jan, which utilizes the llama.cpp backend. This transition allowed them to capitalize on the potential of the Vulkan API, facilitating efficient model execution directly on their hardware. This achievement not only highlights the growing capabilities of AMD GPUs for generative AI but also empowers users to host their own AI models without relying on expensive cloud services. By setting up remote access via a VPN, the individual can use their home server to serve OpenAI-compatible APIs, expanding possibilities for personal projects and collaborative AI applications. With an average performance of about 50 tokens per usage on the Ministral 3 14B model, this development signals a promising future for accessible AI technologies, allowing enthusiasts to explore and innovate in the realm of generative AI.