Show HN: QueenMQ – slow LLM calls blocked our Kafka partitions, so we built it (github.com)

QueenMQ, an innovative messaging system designed to address the delays caused by slow LLM (Large Language Model) calls in Kafka, has been launched. Unlike traditional Kafka setups where a slow message can block an entire partition, QueenMQ ensures that each customer, order, and conversation has its own queue. This allows for seamless processing without the bottleneck effect, enabling up to 3 million messages per second across millions of queues, all while maintaining data integrity through single-transaction commits. The significance of QueenMQ lies in its robust performance and open-source framework, built in Rust and available under the Apache-2.0 license. It eliminates the need for Redis locks or cron jobs by implementing a straightforward log system that retains messages while avoiding unnecessary complexity. Technical details highlight its ability to perform 226 Jepsen tests successfully, demonstrating reliability without message loss. Developers benefit from its ease of integration, as QueenMQ works with existing Kafka clients and provides various language support to facilitate implementation across different programming environments. Overall, QueenMQ represents a substantial advancement for the AI/ML community, offering an efficient alternative for managing high-throughput messaging needs in data-intensive applications.