Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

NEWS · COMPANIES · #3723

LinkedIn engineer details production LLM ranking optimizations including 7B→400M distillation

At QCon AI (covered on InfoQ), Sundara Raman Ramachandran (LinkedIn’s LLM inference lead) described how LinkedIn productionizes LLM-based ranking under tight P99 latency SLAs by distilling a 7B model to ~400M parameters, compressing job contexts into single-token embeddings to boost throughput ~10×, and removing system bottlenecks (custom flash-attention masks, Python GIL workarounds, gc.freeze, non-blocking CUDA memory copies).

KEY POINTS

  1. At QCon AI (covered on InfoQ), Sundara Raman Ramachandran (LinkedIn’s LLM inference lead) described how LinkedIn productionizes LLM-based ranking under tight P99 latency SLAs by distilling a 7B model to ~400M parameters, compressing job contexts into single-token embeddings to boost throughput ~10×, and removing system bottlenecks (custom flash-attention masks, Python GIL workarounds, gc.freeze, non-blocking CUDA memory copies).
  2. This matters because it shows concrete, production-proven model compression and system-level techniques that enable LLMs to meet strict latency and throughput requirements for large-scale search and recommendation.
  3. Presentation: Scaling LLM-Based Ranking Systems for Latency-Critical Search & Recommendation Workloads

WHY IT MATTERS

This matters because it shows concrete, production-proven model compression and system-level techniques that enable LLMs to meet strict latency and throughput requirements for large-scale search and recommendation.

SOURCES & TIMELINE

1