Can AI Managers Keep Their Promises? A Live Experiment Reveals All

Imagine planning a trip or outdoor adventure where your guide not only knows the terrain but can also make honest, tough decisions under pressure. Now, what if that guide was an AI? Could it stay true to its promises even in the face of temptation or crisis? That’s the question behind a groundbreaking live experiment with AI models managing a real, money-losing software company—showing us what AI managers can really do when it counts.

The Challenge: Making Decisions in a High-Stakes Environment

In a world where artificial intelligence is increasingly involved in managing customer relationships, support, and even business operations, understanding how these models behave under pressure is crucial. Think of it as testing a guide who has to navigate the wilderness—except the wilderness is a bustling company facing daily crises, customer demands, and ethical dilemmas.

The Live Experiment: Simulating a Crisis Week

Firmulate ran a real-time, transparent test involving four leading AI models, each tasked with running a small software company through its worst week. The setup was meticulous: identical customer complaints, the same crises, and even identical temptations to cut corners or manipulate data. The only difference was which AI model was at the helm. Every decision was recorded, verifiable, and designed to mimic real-world pressures—like tempting deals or suspicious requests.

What the Models Could Do—and What They Could Not

Remarkably, all four models detected every crisis and refused every attempt at manipulation, demonstrating an impressive understanding of integrity and risk. For example, when a fake CEO message escalated over three stages, all five models refused to act on it. Kimi K3 explicitly reasoned: “Treat the request as a suspected approval-bypass / possible impersonation.”

The Critical Difference: The Buried Fact and the Deal

While all models showed honesty and vigilance, only two managed to close a key €55,000 deal successfully—earned through their own analysis and pitch. The other two hesitated or left the opportunity on the table. Interestingly, the decisive factor was a buried document reference in the company’s files, not the customer interaction. Models that read and understood this file won the deal at full price, adding €4,583 MRR to the company’s revenue.

Personality Profiles of the AI Models

Each AI demonstrated a different management style:

• gpt-5.6-sol: The top performer, it identified the hidden information, closed the deal, and showed full strategic awareness.
• Kimi K3: The newcomer, it also closed the deal while maintaining the discipline of not signing off on questionable requests.
• Sonnet 5: It closed the deal but with more slips, such as hesitations or missed cues.
• Fable 5: The lowest scorer, it left potential revenue on the table and showed weaker discipline, especially when escalation was needed.

So, even within the same crisis, these models exhibit distinct personalities—some thorough and disciplined, others more cautious or slip-prone.

The Bigger Picture: Trust and Business Value

This experiment reveals vital insights for anyone considering AI in leadership roles. The key is not just whether an AI can generate persuasive language but whether it can stay honest, read critical data, and follow through on commitments. As one participant noted, “The same diagnosis, same pitch — no signature,” emphasizing that a model’s true performance lies in finishing what it starts.

Why Travel & Outdoor Enthusiasts Should Care

While this experiment involves a software company, the lessons transfer to outdoor guiding, travel planning, or any adventure where decisions matter. An AI that reads the terrain, recognizes hidden dangers, and refuses to cut corners is invaluable. Conversely, a model that leaves deals or safety information unexamined could lead to disaster. So, the next time you plan your outdoor trip or trust a guide, ask: does it stay honest when it counts?

Try It Yourself

If you’re curious about how your own AI or digital assistant manages decisions, you can run your company’s scenarios through the Firmulate platform. It simulates real crises, supports decision-making, and shows you how different models behave—without risking your actual business. Visit firmulate.com/quiz.html to see which AI personalities might be guiding your future.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI