FUNDING · CODING · #2241
Apple engineers present scaling framework-agnostic AI agent infrastructure with Ray at QCon AI
At QCon AI, Apple engineers Deepak Mohanakumar Chandramouli and Bhumik Vinodkumar Thakkar described moving AI agents from local FastAPI prototypes to resilient production deployments on Kubernetes using Ray, covering RayCluster topology, stateful memory handling, autoscaling for dynamic traffic, control‑plane designs, multi‑region setups, safety guardrails, and an LLM‑as‑a‑judge evaluation pipeline. They illustrated these operational patterns with a trip‑planner agent example and shared production learnings and observability considerations.
KEY POINTS
- At QCon AI, Apple engineers Deepak Mohanakumar Chandramouli and Bhumik Vinodkumar Thakkar described moving AI agents from local FastAPI prototypes to resilient production deployments on Kubernetes using Ray, covering RayCluster topology, stateful memory handling, autoscaling for dynamic traffic, control‑plane designs, multi‑region setups, safety guardrails, and an LLM‑as‑a‑judge evaluation pipeline.
- They illustrated these operational patterns with a trip‑planner agent example and shared production learnings and observability considerations.
- The session surfaces practical, production‑level patterns for running agentic LLM workloads on Ray and Kubernetes — useful playbooks for teams moving beyond prototypes to resilient, observable deployments.
WHY IT MATTERS
The session surfaces practical, production‑level patterns for running agentic LLM workloads on Ray and Kubernetes — useful playbooks for teams moving beyond prototypes to resilient, observable deployments.