AI Daily · 2026-10-04

Microsoft and Hugging Face released ThinkingBox, a benchmark that evaluates AI agents by the records and side effects they leave in backend systems, rather than by their generated text or tool-call form, running each task 20 times and reporting pass@1 and pass@20. In a controlled run across 12 models and more than 120,000 trials, many seemingly clean completions failed executable checks on wrong field values, unexpected extra effects, or missing required effects; one customer-service agent even marked a ticket resolved while the database disagreed. Separately, Simon Willison argues that metered services and APIs should default to hard budget caps that cut off service and return errors at a monthly limit, rather than only sending warning emails, so coding or personal agents cannot rack up large bills unattended. He points to AWS and Google Cloud adding similar spend limits, and both stories point toward agents needing verifiable, controllable, and safer-by-default infrastructure.

Ecosystem & Beyond (Products / Agents / Tools / Opinions)

Research

⭐⭐⭐ [Research] The Agent Said It Was Done. The Database Disagreed.

Microsoft and Hugging Face jointly released ThinkingBox, which grades AI agents not on the sentences they generate or the validity of their tool calls, but on the terminal backend state and side effects they leave behind, and asks them to do it 20 times in a row. It covers 507 stateful business workflows, each run 20 times against various LLM models, and reports metrics such as pass@1 and pass@20. One example: a support agent makes nine tool calls and closes a ticket as resolved, but the carrier exception is still open and the required end state is hold, so the database disagrees. In a common-set ablation of 121,680 valid trials across 12 LLM models, 79,853 attempts failed the executable checks; of those failures, 67.24% still terminated cleanly, invoked a state-changing tool and reported no final tool error, while the executable checks found wrong field values in 77.61%, unintended extra effects in 43.30% and missing required effects in 25.36%. ThinkingBox is now available through Hugging Face and can be run via OpenEnv.

Opinion

⭐⭐ [Opinion] We're going to need default hard budget caps on pretty much everything

Simon Willison argues that pay-by-usage services and APIs should, by default, offer hard budget caps—once $X/month is reached, the service is cut off and returns errors—rather than merely sending a warning email. He says coding agents and personal agents lower the friction of spinning up code that can incur costs, so users shouldn't wake up to find a runaway service has consumed hundreds or thousands more dollars. Responding to the objection that businesses don't want their apps erroring out over budgets, he argues most would prefer an error to a surprise $10,000+ bill, and that hard caps should be the default with removal as an explicit opt-in. He notes AWS launched monthly spend limits on September 16 (pausing a project for the month once its limit is reached, currently for a limited set of customers), and that Google Cloud launched a similar Spend Caps feature in July, suggesting this is becoming a trend. He also hopes agents will bias toward recommending providers with hard budget caps and warn inexperienced builders away from uncapped services.