The infrastructure behind modern ranking systems (serving, data, MLOps) (www.shaped.ai)

This post launches a series for engineers and ML practitioners on the infrastructure of production ranking systems, opening with the online serving layer and the hard 200 ms / 99th-percentile constraint that shapes design. The key takeaway: ranking is primarily a systems problem, not a modeling one. A monolithic service fails because components (vector DB calls vs. GPU inference) have wildly different performance profiles. The recommended architecture is a decoupled ensemble of purpose-built microservices—a Ranking Gateway to orchestrate fan-out/fan-in and timeouts; multiple Candidate Generation services (vector retrieval, keyword search) that scale independently; a Feature Hydration service that batches and caches lookups from the feature store; and a dedicated Model Inference service that manages accelerators, batching, and model versioning. Operationally, this architecture maps to Kubernetes Deployments with nuanced autoscaling: HPA (CPU/memory) works for compute-bound services but reacts slowly, while KEDA lets you scale proactively off external signals like RPS or queue length. Hardware should be matched to workload—CPU-optimized nodes for retrieval and hydration, high-memory instances for the online feature store, and GPUs (via Triton/TorchServe) for batched transformer/DLRM inference, which can deliver 10–100× throughput improvements. The post closes by noting the next installment will dig into the data layer: feature stores and vector databases that fuel this engine.