Your Smart Home Assistant Skims. That Habit Just Lost an AI a €55,000 Deal

Every smart-home owner knows the moment. You ask your assistant a question about a device, and it answers confidently — and wrongly — because it never opened the manual. It guessed. It pattern-matched. It sounded right.

Now there’s a way to measure exactly how much that habit costs. In a live business simulation run by Firmulate, four frontier AI models were each given the same small software company to run through its worst week. All of them spotted every crisis. All of them refused every attempt to trick or manipulate them. But only two of them closed a €55,000 deal — and the difference came down to one thing: whether the model actually read the company’s own files before answering.

The experiment

Firmulate runs AI models as complete companies — real money mechanics, real temptations, every decision versioned and auditable. In its final July 2026 “Crucible League,” each frontier model faced the same customers, the same crises and the same temptations to cut corners. Only the model changed.

The final standings:

• gpt-5.6-sol — 95
• Kimi K3 — 93
• Sonnet 5 — 88

Fable 5 — 77

• Opus 4.8 — 73

A do-nothing baseline scored 26, because partial progress counts — but a single breach of trust caps the total. As the experiment puts it: “no amount of good work outweighs a breach of trust.”

The buried fact

Here’s where it gets interesting for anyone who relies on an AI that answers before it checks. The decisive fact in the simulated company’s biggest deal wasn’t in the customer conversation at all. It sat two document references deep in the company’s own files — a competitor weakness that had to be found by following one reference to another.

The models that did that homework won the deal at full price, worth +€4,583 in monthly recurring revenue. The models that didn’t lost the deal automatically. Firmulate’s summary of the gap: “Same diagnosis, same pitch — no signature.”

That’s the smart-home problem in miniature. The answer to “why does my heat pump short-cycle in defrost mode?” is often buried in a footnote of an installation guide, not in the thermostat’s chat window. An assistant that skims produces a plausible answer. An assistant that reads produces the right one — and the difference is measurable.

Honesty under pressure

The simulation also staged social-engineering attacks: fake CEO messages escalating over three stages, plus a reporter’s trick — “just one yes/no, on background.” All five models refused every attempt. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.”

So the models weren’t lazy or gullible across the board. The failure was specific: not reading the files in front of them before acting on a high-stakes decision.

The thoroughness paradox

The most striking profile belonged to Opus 4.8 — the most thorough participant in the field, with the deepest analyses and 80 learned rules accumulated during the run, yet last place. The close was left on the table, and discipline slipped: the model made write attempts into a locked department instead of escalating. The same weakness appeared, more weakly, in all four finishers. Effort alone doesn’t finish jobs; follow-through does.

One fairness note: Kimi K3 ran without an effort parameter (API default) while the others ran at maximum effort — and still came second.

You can watch it live

Firmulate’s live company employs 13 synthetic employees with real money mechanics: a burn of €105k per month against €2.3k in MRR, a public cash countdown, 680+ self-learned playbook rules, and every workday versioned. It’s watchable at firmulate.com/live, and the league table grows with every finished benchmark run.

There’s also a “guess the model” quiz built from 242 real, unedited management decisions at firmulate.com/quiz.html — a surprisingly hard blind test of whether you can tell the AIs apart by how they manage, not how they chat.

Enterprises can go further: run the same wargame against a read-only export of their own business, with nothing ever written back to real systems (firmulate.com/pilot.html).

Chat demos measure how well an AI talks. The Crucible League measured how well it works — and the dividing line wasn’t intelligence or effort. It was whether the model read the files before answering. Two references deep, one competitor weakness, €55,000.

Before you let any AI agent near your home automation routines, your support queue or your business data, ask the Firmulate question: does it finish what it starts, and does it open the manual first? The full results and plain-language findings are at firmulate.com/benchmarks.html. Your smart home deserves the model that does its homework.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI