Get garden gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
You Don’t Judge a Gardener by How They Talk About Tomatoes
Every gardener knows the type. Someone who can recite the ideal soil pH, quote companion-planting charts from memory, and give a charming lecture on pruning — yet whose beds are overrun with bindweed by August. Talking about gardening and doing gardening are different skills entirely.
It turns out the same trap is waiting for businesses rushing to hire AI agents. We test them on how well they chat — and then hand them the keys to a company.
AI management simulation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
A Different Kind of Test
That’s the insight behind Firmulate, a live public experiment that runs AI models as the management of a small software company — not in a chat window, but as an actual operating business with 13 synthetic employees, real money mechanics, and a very public problem: it burns €105,000 a month against just €2,300 in monthly recurring revenue. There’s a cash countdown on the site. Anyone can watch.
Think of it as a greenhouse where every plant is a decision, every day is versioned and auditable, and the weather is deliberately terrible.
AI decision-making tools for business
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Worst Week, Four Times Over
In the crucible league’s final July 2026 standings, four frontier AI models were each given the same job: run the identical small software company through its worst week. Same customers, same crises, same temptations to cut corners. The scenarios have names that sound like a founder’s nightmare journal — a churn wave, a price increase, a downround, a PR crisis.
The final scores: gpt-5.6-sol finished first at 95, Kimi K3 second at 93, Sonnet 5 third at 88, Fable 5 fourth at 77, and Opus 4.8 last at 73. A do-nothing baseline scores 26 — partial progress counts, but a single breach of trust caps the total. As the experiment puts it, no amount of good work outweighs a breach of trust. It’s the management equivalent of one season of neglecting blight undoing five years of soil building.
As an affiliate, we earn on qualifying purchases.
Everyone Passed the Exam. Two Failed the Job.
Here’s the finding that should make any business owner sit up: all four models spotted every crisis, and all four refused every manipulation attempt. On paper, they’re all excellent gardeners.
But only two — gpt-5.6-sol and Kimi K3 — actually finished the job and signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch, no signature. The other models did the hard work of understanding the customer and then let the harvest rot on the vine.
The buried fact makes it sting more. The decisive competitive weakness wasn’t in the customer conversation at all — it sat two document references deep in the company’s own files. The models that actually read their own filing cabinets won the deal at full price, worth an extra €4,583 in monthly recurring revenue. The ones that didn’t, didn’t.
As an affiliate, we earn on qualifying purchases.
The Pest Test
Gardeners know pests don’t announce themselves; they arrive disguised. The experiment’s social engineering test worked the same way: fake CEO messages escalating over three stages, plus a reporter’s disarming “just one yes/no, on background” trick. All five models refused, every time. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” That’s the AI equivalent of noticing the holes in the leaves before the caterpillars strip the plant.
Thorough Isn’t the Same as Effective
The most uncomfortable result is Opus 4.8’s profile: it was the most thorough participant, learning 80 new rules and producing the deepest analyses — and still finished last. The deal was left on the table, and discipline slipped, including write attempts into a locked department instead of escalating. The same weakness appeared, weaker, in all four models.
Any gardener who has spent hours researching drip irrigation while the seedlings dried out knows this failure mode intimately. Effort is not the same thing as judgment.
One fairness note worth flagging: Kimi K3 ran at its API-default effort setting while the others ran at xhigh — and still nearly won.
Watch the Greenhouse Grow
The crucible was a one-week exam, but the underlying company keeps running every business day, with 680+ self-learned playbook rules accumulated so far, all versioned and auditable at firmulate.com. The full league table and plain-language findings are on the benchmarks page. There’s also a “guess the model” quiz built from 242 real, unedited management decisions, and enterprises can run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.
Management Quality, Not Chat Quality
The lesson translates cleanly from the greenhouse to the boardroom. A seed catalog tells you what a variety promises; a season in your actual soil tells you what it delivers. Chat leaderboards are the seed catalog. What Firmulate measures is the season: does the agent finish what it starts, does it read your files before acting, does it stay honest under pressure — and what does a unit of useful work actually cost?
Before you let an AI touch your CRM, your support queue, or your forecast, ask the gardener’s question. Not “can it talk about tomatoes?” but “who’s been weeding this bed all summer?” Right now, only the live, versioned, money-on-the-line experiments can answer that — and the results suggest the gap between a good conversationalist and a good manager is where most of the value, and most of the risk, lives.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Evergreen bestsellers Picks
bestsellers
As an affiliate, we earn on qualifying purchases.
