NEWS · CODING · #878
SWE-Serve benchmark shows many patches that pass local tests fail live serving for SGLang
SWE-Serve is a new benchmark that turns 83 merged SGLang pull requests into 53 executable inference-serving tasks across six families; 19 tasks start a real server and validate patches through the full serving path. Across those live-serving tasks, the same patches pass 45.9% of the time with the complete verifier versus 69.4% when live-serving checks are excluded (roughly one in three otherwise-passing patches fail live checks); multi-domain tasks show a 21.3 percentage-point lower pass rate, and closed-book model pass@1 scores (11 models, mini-swe-agent) ranged from 34.6%–75.5% with Claude Opus 5 and GPT-5.6 Sol among top performers.
KEY POINTS
- SWE-Serve is a new benchmark that turns 83 merged SGLang pull requests into 53 executable inference-serving tasks across six families; 19 tasks start a real server and validate patches through the full serving path.
- Across those live-serving tasks, the same patches pass 45.9% of the time with the complete verifier versus 69.4% when live-serving checks are excluded (roughly one in three otherwise-passing patches fail live checks); multi-domain tasks show a 21.3 percentage-point lower pass rate, and closed-book model pass@1 scores (11 models, mini-swe-agent) ranged from 34.6%–75.5% with Claude Opus 5 and GPT-5.6 Sol among top performers.
- This matters because it demonstrates that repository-level and unit tests can miss failures that only appear when a real model is loaded and the full serving interface handles requests, so end-to-end live-serving verification is necessary for safe inference-engineering changes.
WHY IT MATTERS
This matters because it demonstrates that repository-level and unit tests can miss failures that only appear when a real model is loaded and the full serving interface handles requests, so end-to-end live-serving verification is necessary for safe inference-engineering changes.