Show HN: YieldOS-Lite – A simulator for LLM inference control-plane governance (github.com)

YieldOS-Lite is a new simulator developed to examine whether utilizing a slow-path resource governance control plane enhances service-level objective (SLO) fulfillment in heterogeneous large language model (LLM) inference scenarios. Unlike traditional mechanistic schedulers, YieldOS-Lite focuses on resource governance policies involving SLO urgency, cache value, and decision-making protocols. This Phase 1 research artifact includes the simulator code, a draft paper, experiment summaries, and replay traces, enabling users to easily explore and experiment with governance strategies before they are applied to production-level serving engines. This project holds significant implications for the AI/ML community as it suggests that slow-path governance may lead to improved performance in responding to workload heterogeneity, particularly in diverse environments characterized by mixed-task demands. The early findings indicate that predictive governance and value-aware key-value accounting can enhance governed goodput, while shape classification requires further refinement. Importantly, YieldOS-Lite is positioned not as a replacement for existing models like vLLM or TensorRT-LLM but as a valuable tool for testing governance mechanisms, paving the way for optimized LLM performance in real-world applications.