RESEARCH · RESEARCH · #3015
IBM's CUGA harness improves AppWorld consistency with GLM-5.3-flash versus Hermes and Deep Agents
Researchers ran the same open-weight model (GLM-5.3-flash) across three agent harnesses—IBM's CUGA, Hermes (Nous Research), and LangChain's Deep Agents—on 90 AppWorld tasks; task completion varied from about 30% to 81% depending on the harness, and scenario-level consistency (SGC) favored CUGA (20 scenarios) over Hermes (8) and Deep Agents (2). CUGA is open-source, built for enterprise deployment (including sovereign and air-gapped environments), and adds a runtime policy layer (described in the authors' ACM CAIS 2026 paper) that enforces approvals and tool guards rather than relying on prompts; the authors note limits: one benchmark, non-identical setups, and no policy-enabled runs in the 90-task comparison.
KEY POINTS
- Researchers ran the same open-weight model (GLM-5.3-flash) across three agent harnesses—IBM's CUGA, Hermes (Nous Research), and LangChain's Deep Agents—on 90 AppWorld tasks; task completion varied from about 30% to 81% depending on the harness, and scenario-level consistency (SGC) favored CUGA (20 scenarios) over Hermes (8) and Deep Agents (2).
- CUGA is open-source, built for enterprise deployment (including sovereign and air-gapped environments), and adds a runtime policy layer (described in the authors' ACM CAIS 2026 paper) that enforces approvals and tool guards rather than relying on prompts; the authors note limits: one benchmark, non-identical setups, and no policy-enabled runs in the 90-task comparison.
- The result shows that agent harness choice — and runtime policy enforcement — can materially change reliability and safety for enterprise automation even with the same underlying model.
WHY IT MATTERS
The result shows that agent harness choice — and runtime policy enforcement — can materially change reliability and safety for enterprise automation even with the same underlying model.