NEWS · COMPANIES · #3880
Airbnb uses cost-per-task and CloudWatch metrics to run AirChat coding agents on Amazon Bedrock
Airbnb routes its internal coding agents (AirChat and AirChat Remote) through an LLM gateway into Amazon Bedrock and evaluates models using task-specific evals that report score and list-price cost per task. Production monitoring in Amazon CloudWatch (latency, token counts, throttles, EstimatedTPMQuotaUsage) is compared with eval results to set model defaults, effort ceilings, and to diagnose latency, throttling, and cost changes.
KEY POINTS
- Airbnb routes its internal coding agents (AirChat and AirChat Remote) through an LLM gateway into Amazon Bedrock and evaluates models using task-specific evals that report score and list-price cost per task.
- Production monitoring in Amazon CloudWatch (latency, token counts, throttles, EstimatedTPMQuotaUsage) is compared with eval results to set model defaults, effort ceilings, and to diagnose latency, throttling, and cost changes.
- This shows a practical, production-ready approach to selecting and operating LLMs by combining per-task cost-aware evals with CloudWatch observability on Amazon Bedrock, which many teams can adopt to control latency, capacity, and spend.
WHY IT MATTERS
This shows a practical, production-ready approach to selecting and operating LLMs by combining per-task cost-aware evals with CloudWatch observability on Amazon Bedrock, which many teams can adopt to control latency, capacity, and spend.