A new implementation for LLMs through Nvidia GPUs natively on Apple Silicon (github.com)

🤖 AI Summary
Hawking Systems has announced the Hawking Runtime, a groundbreaking project enabling local inference and GPU computing for Apple Silicon Macs. This implementation utilizes Nvidia GPUs through a native macOS DriverKit driver and CUDA support, allowing developers to leverage the power of their existing hardware. The project features two main components: an extension for local model inference utilizing llama.cpp—compatible with OpenAI's API—and an eGPU setup that enables real-time GPU computation connected via Thunderbolt. Users can easily install and initialize the software with Python, set up connections, and serve models independent of each other. This advancement is significant for the AI/ML community as it paves the way for enhanced computational capabilities on Apple devices, traditionally limited in GPU acceleration for machine learning tasks. By demonstrating successful GPU kernel execution and communication with Nvidia's RTX 4000 SFF Ada, the project opens doors for more powerful local AI applications without relying solely on cloud resources. Developers are encouraged to inspect, adapt, and build upon the open-source components, fostering innovation and potentially augmenting performance and compatibility across differing hardware configurations.
Loading comments...
loading comments...