6.4x faster than llama.cpp, 3.9x faster than MLX (www.basecompute.co)

A new model serving tool called BaseRT has been announced, boasting impressive performance metrics that deliver up to 6.4 times faster results than llama.cpp and 3.9 times faster than MLX when pre-filling tokens, using an Apple M5 Pro chip. This performance improvement exemplifies significant advancements in on-device AI capabilities, promising reduced latency and enhanced user experience for applications relying on local processing. The significance of BaseRT lies in its ability to serve models entirely on user machines without requiring API keys or transmitting data off-device. This not only ensures data privacy but also empowers engineers and developers to work with open-source models more efficiently. By streamlining the deployment of AI applications, BaseRT could foster the growth of on-device AI solutions, especially in environments where data security is a priority, and internet connectivity may be unreliable.