My Mac Mini fits one model. I run five (placona.co.uk)

A tech enthusiast has developed a router solution to effectively manage multiple large language models (LLMs) on an M4 Pro Mac mini, which can only support one large model in RAM at a time due to memory limitations. This router facilitates requests from various devices while ensuring that the active model in use is loaded efficiently without exceeding the memory cap imposed by macOS. The setup utilizes Python to relay requests and ensures a design that isolates model management from client interactions, preventing concurrency issues that could arise when multiple requests are made. This innovation is significant for the AI/ML community as it demonstrates how to optimize resource-constrained environments while managing large models, which are often too memory-intensive for typical setups. With a working budget of approximately 33GB, the router guarantees that the system maintains performance via a well-defined lifecycle management for the models—a crucial capability in production settings. By keeping the other models available without compromising the active one's performance, this solution underscores the importance of router architecture in facilitating seamless AI operations and opens the door for more advanced multi-model applications.