If you have used ChatGPT, you know the basic rhythm. You type a question, it types an answer. Ask, reply, done. It is useful, but notice what it is doing: it only talks, one turn at a time.

There is a newer kind of AI that does more than talk. It is called an agent. An agent is built on the same kind of model, but instead of just writing a reply it takes a string of actions to reach a goal. It checks something, makes a choice, looks at what happened, then chooses again. A chatbot answers your question. An agent tries to get a job done.

So a fair question is: how good are these agents, really? To answer that you need a benchmark. A benchmark is a fixed test that scores different AI systems the exact same way, so you can compare them honestly. A new one from Princeton, called CEO-Bench and described in the benchmark paper, asks a hard version of the question. Not can an agent answer something, but can an agent run a whole company over a long stretch of time. The short answer the test gives back is sobering. Most of them went broke.

What the test actually does

CEO-Bench drops an AI agent into the boss chair of a fake subscription software company called NovaMind. The agent runs it for 500 simulated days. It starts with $1 million in cash. At the end, it gets one score: how much money it has left.

That single number is the whole point of a benchmark. Cash at day 500 is something you can measure cleanly and compare across every AI that takes the test. No opinions, no judging the writing style. Just, did this agent grow the money or lose it.

The agent is not given a tiny toy job. It controls prices and discounts, marketing and ads, what the company builds next, daily operations like customer support and how much capacity to run, sales negotiations, market research, and even social media posts. To do all that it has 34 tools to act with and a business database of 19 tables to read from. Tools are the agent’s hands. The database is what it can see.

Why this is the hard kind of test

Here is the idea worth slowing down on. Answering one question is a single moment. Running a company is a long game, and the long game is where most AI agents fall apart.

Think about the difference between one decision and a whole career of them. A clever person can make one sharp call. Running a business for years is harder, because every choice changes the next one. Spend on ads today and you have less cash next month. Cut the price to win customers and you might not afford to keep the servers on. The choices stack up, and a move that looked smart on Monday can sink you by Friday.

CEO-Bench builds that pile-up on purpose. The market shifts over the 500 days. Competitors react to what the agent does. There are 26 different customer groups, each with hidden likes the agent has to figure out. Feedback is noisy and indirect, meaning the signals the agent gets are messy and do not point straight at the right answer. The paper sums up the challenge: success demands reading a noisy, tangled business and turning those signals into a sound strategy held together across many decisions.

That phrase, held together across many decisions, is what long-horizon planning means. It is not about being brilliant once. It is about not falling apart over time.

What the scores showed

Now the results, and they are blunt. Out of all the strong, current AI models put through the test, only two finished with more money than the $1 million they started with: Claude Opus 4.8 and GPT-5.5. Everyone else ended up below where they began, and most went fully bankrupt before reaching day 500. They ran the company out of cash and the simulation ended early.

The two that made it through did well. In its best run, Claude Opus 4.8 ended with about $27.8 million. GPT-5.5’s best run ended around $21.3 million. For comparison, a simple rule-based baseline, meaning a fixed set of if-this-then-that instructions with no real thinking, landed around $15.8 million. So the top agents beat a dumb rulebook, but plenty of other models could not.

Two honest cautions sit right next to those numbers. First, even the winners were not reliable. The paper states plainly that neither top performer consistently turns a profit across repeated runs. A good run is not the same as a steady one. Second, the test is far from solved. The paper estimates the best possible final cash is around $2.2 billion, and the top score so far is under $28 million. There is enormous room left.

What separated the winners from the rest

The paper looked at why a few agents survived while most sank, and the reasons line up with common sense about running anything well.

The agents that did best were better at uncovering what actually worked, like which advertising channels were paying off, when that information was hidden in noisy data. They forecast their future cash with smaller errors, so they were less likely to be surprised by running dry. They noticed competitors changing course faster. And they planned in a more careful way, thinking in terms of if this happens, then I will do that, instead of reacting one step at a time.

There was even a telling detail about precision. A strong agent might steer 90 percent of its development effort toward one targeted improvement, while a weaker one spread its effort thin and committed to almost nothing. Sharp aim beat vague effort.

None of that is a trick. It is what a good operator does: find out what is really going on, see trouble coming, and commit to a clear plan.

How to read this if you only use ChatGPT

It is easy to hear that AI ran a company for 500 days and assume the agents are nearly ready to take the wheel. The test says the opposite, and that is the useful part.

Be clear about what this is. CEO-Bench is a simulation built to be measured, not a product you can buy. It deliberately leaves out some of the hardest real-world parts of running a company, like hiring, security, and raising money. So even a top score here is not proof an agent could run a real business.

What it does give you is an honest progress check. The thing that breaks most AI agents is not answering a question. It is holding a sensible plan together over a long stretch where every choice feeds the next one. The single moment is mostly solved. The long game is not.

If you want a simple way to judge any agent you hear about, that is the test to apply. Not how clever it sounds in one reply, but whether it can stay on course when the choices pile up and the easy answers run out.

Sources: huggingface.co