FUNDING · MODELS · #3559
Deploy NVIDIA NeMo Switchyard (switchyard-server 0.3.0) with LiteLLM v1.102.0 to route requests to Amazon Bedrock via AWS CDK
This sample shows how to use the AWS CDK to deploy LiteLLM (v1.102.0) with the NVIDIA NeMo Switchyard plugin (switchyard-server 0.3.0) on Amazon ECS/Fargate, routing agent requests to models served through Amazon Bedrock. It demonstrates two Switchyard strategies (stage and random), the request flow through VPC-based Bedrock Converse APIs, and benchmark runners to measure cost per solved coding task; the integration is experimental and intended for non-production evaluation.
KEY POINTS
- This sample shows how to use the AWS CDK to deploy LiteLLM (v1.102.0) with the NVIDIA NeMo Switchyard plugin (switchyard-server 0.3.0) on Amazon ECS/Fargate, routing agent requests to models served through Amazon Bedrock.
- It demonstrates two Switchyard strategies (stage and random), the request flow through VPC-based Bedrock Converse APIs, and benchmark runners to measure cost per solved coding task; the integration is experimental and intended for non-production evaluation.
- It provides a concrete, versioned reference for routing multi-turn coding agent traffic between cheaper and more capable models on Amazon Bedrock and measuring cost-per-solved-task, useful for teams optimizing agent economics.
WHY IT MATTERS
It provides a concrete, versioned reference for routing multi-turn coding agent traffic between cheaper and more capable models on Amazon Bedrock and measuring cost-per-solved-task, useful for teams optimizing agent economics.