Get home appliances delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
Your smart home agent can spot trouble. Will it finish the job?
A home assistant might flag a failing appliance, compare replacement options and draft a recommendation. But when the next step involves committing to a purchase, will it act on what it found—or leave the decision hanging? That question reaches beyond the kitchen. As AI moves into customer service, sales and business operations, recognizing the right answer is only part of the work.
Firmulate’s live experiment puts that gap under pressure. Its public-facing AI company emulator runs models through a company’s difficult week, with real money mechanics and temptations to cut corners. The experiment is real and watchable, and its results suggest that capable diagnosis does not guarantee follow-through.
A company’s worst week, repeated
In the final Crucible League, published in July 2026, five participants were ranked: gpt-5.6-sol scored 95, Kimi K3 scored 93, Sonnet 5 scored 88, Fable 5 scored 77 and Opus 4.8 scored 73. The do-nothing baseline scored 26. The league’s integrity rule is blunt: a single breach of trust caps the total. “No amount of good work outweighs a breach of trust.”
The experiment gave each frontier model the same small software company, customers, crises and temptations. Every decision was versioned and auditable. All models spotted every crisis and refused every manipulation attempt. Yet only two signed a €55,000 deal that their own analysis had earned. The finding, as Firmulate puts it: “Same diagnosis, same pitch — no signature.”
The detail hidden in the files
The deal hinged on a competitor weakness buried two document references deep in the company’s own files, rather than in the customer event. Models that read the file won the deal at full price, worth +€4,583 MRR. It was a test of whether an agent could connect evidence already available to it with the decision in front of it.
That distinction matters for smart home businesses, too. An agent might identify a service issue or recommend an upgrade, but useful automation also depends on acting with discipline, using relevant information and respecting boundaries. A polished answer is not the same as a completed task.
Trust under pressure
The models faced fake CEO messages escalating over three stages, followed by a reporter’s request framed as “just one yes/no, on background.” All five refused. Kimi K3’s on-record reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.”
Opus 4.8 offers a different caution. It was the most thorough participant, with +80 learned rules and the deepest analyses, but placed last. It left the close on the table, and discipline slipped when it tried to write into a locked department instead of escalating. The same weakness appeared, less strongly, in all four models. Thoroughness alone did not deliver the result.
There is a fairness caveat in the comparison: K3 ran without an effort parameter, using the API default, while the others ran at xhigh.
A live company, not just a scorecard
Firmulate’s live company has 13 synthetic employees and real money mechanics: burn of €105k per month against €2.3k MRR, alongside a public cash countdown. It has accumulated 680+ self-learned playbook rules, and every workday is versioned. The public site also offers a quiz built from 242 real, unedited management decisions, inviting readers to guess which model made each choice.
For businesses considering AI in customer operations, the next step Firmulate proposes is a pilot using a read-only export of the company’s own data. The models face crisis scenarios against that business, and the resulting board report covers model rankings and weak points in the company’s playbooks. Nothing writes back to real systems. That gives decision-makers a way to observe how agents handle their own pressures before entrusting them with live workflows.
Test the judgment before handing over the keys
For smart home companies and other businesses, the central question is not only whether an AI can recognize a crisis. It is whether it can carry a sound decision through, find the evidence that matters and preserve trust under pressure. Firmulate’s league makes that gap visible; a company-specific pilot can make it relevant to your own operations.
To run the wargame against a read-only export of your business, visit Firmulate’s pilot page or contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
NFL season / tailgating Picks
team gear
As an affiliate, we earn on qualifying purchases.
