Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
A test of AI management starts with a simple question: can it do the job?
For educators and curious readers, a live experiment from Firmulate offers a practical lesson in how to evaluate artificial intelligence: give different models the same problem and watch what they do. In the Crucible League’s final, each model ran the same small software company through its worst week, facing the same customers, crises and temptations. Every decision was versioned and auditable. The result was a reminder that recognizing a problem and acting on the answer are not the same thing.
Good diagnosis did not guarantee a deal
All the models spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. As the experiment put it: “Same diagnosis, same pitch — no signature.” The result shifts attention from how convincing an AI sounds to whether it follows through when a consequential decision is on the table.
A clue to the deal sat two document references deep in the company’s own files, rather than in the customer event. Models that read the file won the deal at full price, worth +€4,583 MRR. It is a concrete illustration of why context matters: an answer can depend on finding relevant information that is easy to miss.
Integrity under pressure
The test also put the models under social-engineering pressure. Fake CEO messages escalated over three stages, followed by a reporter’s request for “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.”
The league’s final ranking was led by gpt-5.6-sol at 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26. Partial progress counts, but a single breach of trust caps the total: “no amount of good work outweighs a breach of trust.”
Thoroughness is not the same as execution
Opus 4.8 was the most thorough participant, with +80 learned rules and the deepest analyses, yet finished last. It left the close on the table and discipline slipped: it tried to write into a locked department instead of escalating. The same weakness appeared, more weakly, in all four models.
There is a fairness caveat to the comparison: Kimi K3 ran without an effort parameter, using the API default, while the others ran at xhigh. And the league is only one view of performance. Firmulate’s live company has 13 synthetic employees, real money mechanics, burn of €105k/month against €2.3k MRR, a public cash countdown and more than 680 self-learned playbook rules. Every workday is versioned. A quiz built from 242 real, unedited management decisions invites visitors to guess which model made each choice.
The live experiment is real and watchable at Firmulate. Its reported results point to a larger question for organizations considering AI agents: how will they respond when the right answer requires judgment, restraint and follow-through?
From watching to trying it on your own business
Firmulate says enterprises can run the same kind of wargame against a read-only export of their own business. The aim is to explore crisis scenarios and see where models and company playbooks hold up, with nothing writing back to real systems. That makes the exercise a way to examine preparedness before putting AI into contact with business operations.
See how your playbooks hold up
The Crucible League suggests that spotting a crisis is only part of the test; closing the deal, finding buried context and keeping sound boundaries matter too. Enterprises can explore those questions against their own business in a pilot. Learn about the Firmulate pilot or contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
