FUNDING · COMPANIES · #3727
Datadog adds Online Experiments to Agent Observability to test agents on production traffic
Datadog has extended its Agent Observability platform with an Online Experiments capability that lets teams A/B agent versions on real production traffic and compare evaluation scores alongside operational metrics (latency, token consumption, cost, errors). InfoQ reports Datadog used the feature to split traffic between GPT-5.6 Luna and GPT-6 Luna, finding a reported 44.3% estimated cost reduction for the newer model but also signals of increased execution time and more invalid tool parameters in traces.
KEY POINTS
- Datadog has extended its Agent Observability platform with an Online Experiments capability that lets teams A/B agent versions on real production traffic and compare evaluation scores alongside operational metrics (latency, token consumption, cost, errors).
- InfoQ reports Datadog used the feature to split traffic between GPT-5.6 Luna and GPT-6 Luna, finding a reported 44.3% estimated cost reduction for the newer model but also signals of increased execution time and more invalid tool parameters in traces.
- Bringing controlled A/B experiments, production telemetry, and traces together helps teams detect regressions and trade-offs that offline tests can miss when deploying AI agents in real conditions.
WHY IT MATTERS
Bringing controlled A/B experiments, production telemetry, and traces together helps teams detect regressions and trade-offs that offline tests can miss when deploying AI agents in real conditions.