LinkedIn engineer details production LLM ranking optimizations including 7B→400M distillation
At QCon AI (covered on InfoQ), Sundara Raman Ramachandran (LinkedIn’s LLM inference lead) described how LinkedIn productionizes LLM-based ranking under tight P99 latency SLAs by distilling a 7B model to ~400M parameters, compressing job contexts into single-token embeddings to boost throughput ~10×, and removing system bottlenecks (custom flash-attention masks, Python GIL workarounds, gc.freeze, non-blocking CUDA memory copies).