Turbocharging LinkedIn's Recommendation Systems with SGLang (www.linkedin.com)

LinkedIn has announced an enhancement to its recommendation systems by integrating SGLang, an open-source LLM serving framework, which significantly improves the efficiency of its ranking and scoring tasks. The new Multi-Item-Scoring (MIS) optimization allows for concatenating multiple candidate items along with user-specific prompts into a single request, drastically reducing latency by 69% when ranking up to 50 items. This advancement is particularly crucial in handling LinkedIn's high-traffic environments, where rapid responses are essential for personalized member experiences. The integration of optimized FlashAttention 3 (FA3) as the default backend further solidifies this improvement, enabling better performance in handling long input contexts. The collaborative enhancements to SGLang not only advance LinkedIn's internal infrastructure but also contribute significantly to the open-source community, setting a new standard for LLMs used in recommendation systems. As a result, LinkedIn can deliver more relevant content to users while minimizing operational complexities, showcasing a substantial leap in utilizing AI for personalization in professional networking.