NEWS · COMPANIES · #3723
LinkedIn engineer details production LLM ranking optimizations including 7B→400M distillation
At QCon AI (covered on InfoQ), Sundara Raman Ramachandran (LinkedIn’s LLM inference lead) described how LinkedIn productionizes LLM-based ranking under tight P99 latency SLAs by distilling a 7B model to ~400M parameters, compressing job contexts into single-token embeddings to boost throughput ~10×, and removing system bottlenecks (custom flash-attention masks, Python GIL workarounds, gc.freeze, non-blocking CUDA memory copies).
KEY POINTS
- At QCon AI (covered on InfoQ), Sundara Raman Ramachandran (LinkedIn’s LLM inference lead) described how LinkedIn productionizes LLM-based ranking under tight P99 latency SLAs by distilling a 7B model to ~400M parameters, compressing job contexts into single-token embeddings to boost throughput ~10×, and removing system bottlenecks (custom flash-attention masks, Python GIL workarounds, gc.freeze, non-blocking CUDA memory copies).
- This matters because it shows concrete, production-proven model compression and system-level techniques that enable LLMs to meet strict latency and throughput requirements for large-scale search and recommendation.
- Presentation: Scaling LLM-Based Ranking Systems for Latency-Critical Search & Recommendation Workloads
WHY IT MATTERS
This matters because it shows concrete, production-proven model compression and system-level techniques that enable LLMs to meet strict latency and throughput requirements for large-scale search and recommendation.