AWS launches Ray LLM DLC for multi-node LLM inference on Amazon EKS
AWS published a guide and released a Ray LLM Deep Learning Container (DLC) that supports deploying a single large language model across multiple GPU nodes on Amazon EKS, using a disaggregated prefill/decode setup. The Ray LLM DLC packages vLLM, Ray Serve LLM, NIXL for KV-cache transfer, and the EFA stack, exposes an OpenAI‑compatible endpoint, and includes a helper (build_pd_openai_app) plus example code in a repository.