AKM-CLR – pre-inference governance for vLLM-style LLM serving (github.com)

AKM-CLR has introduced a validated governance layer designed for multi-tenant LLM infrastructures, specifically functioning above OpenAI-compatible serving engines like vLLM. This significant development addresses the critical need for pre-inference governance, enabling management of requests to ensure that only authorized and safe interactions reach the backend. With features such as tenant/task authorization, blocking for cross-tenant requests, and comprehensive audit logging, AKM-CLR enhances the security and efficiency of shared LLM services. The prototype has been rigorously tested across multiple controlled experiments, showcasing impressive results: a 100% safe decision rate and zero instances of unsafe backend calls. It is particularly relevant for AI platforms that leverage shared infrastructure, as it mitigates risks associated with rogue or unsafe requests that can lead to data leakage or unauthorized access. As multi-tenant systems become more commonplace, AKM-CLR offers a critical solution, complementing existing serving engines and paving the way for safer AI applications in regulated environments. The team is currently seeking design partners to further develop and integrate this governance layer into commercial applications.