SWE-Serve benchmark shows many patches that pass local tests fail live serving for SGLang
SWE-Serve is a new benchmark that turns 83 merged SGLang pull requests into 53 executable inference-serving tasks across six families; 19 tasks start a real server and validate patches through the full serving path. Across those live-serving tasks, the same patches pass 45.9% of the time with the complete verifier versus 69.4% when live-serving checks are excluded (roughly one in three otherwise-passing patches fail live checks); multi-domain tasks show a 21.3 percentage-point lower pass rate, and closed-book model pass@1 scores (11 models, mini-swe-agent) ranged from 34.6%–75.5% with Claude Opus 5 and GPT-5.6 Sol among top performers.