Microsoft’s ThinkingBox benchmark evaluates agents by backend state and is now on Hugging Face
ThinkingBox, a Microsoft benchmark that grades AI agents by the terminal backend state and side effects they leave (not just generated text), runs 507 stateful business workflows 20 times each and is now available through Hugging Face and runnable via OpenEnv. The paper and benchmark report substantial inconsistency: across 121,680 trials on 12 LLMs, 79,853 failed executable checks and many seemingly successful runs still produced wrong or missing database-side effects.