AI agents observability root cause analysis

Why AI agents fail in production and how failure detection fixes it

Corteus Team · Jun 15, 2026 · 7 min read

When an AI agent gets something wrong, the mistake usually hides several steps back. A new AWS tool reads the agent's own logs and tells you which step broke and why.

If you have used ChatGPT, you know the rhythm. You type a question, it types back an answer. You can read the answer and judge it on the spot. If it is wrong, you see it right away.

There is a newer kind of AI that does more than answer. It is called an agent. An agent runs on the same kind of technology, but instead of just writing a reply, it takes actions and works through a whole job in steps. Picture an agent that books travel. It searches flights, picks one, charges a card, and emails you a confirmation. Each step leans on the step before it.

Now picture it going wrong. One Tuesday the agent confidently books a flight to the wrong city. The confirmation email is perfect. The steps all look sensible. And nobody notices until the customer is standing in the wrong airport.

That is the quiet problem with agents. They can fail in a way that looks like success. A chatbot’s mistake is right there on the screen. An agent’s mistake can happen three steps in, get carried forward, and arrive dressed up to look reasonable.

On June 15, 2026, AWS published a post about a tool, called Strands Evals, built to answer the one question a worried team actually has after an agent breaks: why did it fail, and where do I go to fix it. AWS is Amazon’s cloud business, the part of Amazon that rents out computing to other companies.

A score tells you that something broke, not what

Most teams check an agent the way a teacher grades a test. You hand it a hundred jobs and report that it finished 60 of them correctly. That number looks fine on a dashboard. It is useless on the morning something is on fire.

Sixty out of a hundred correct means forty went wrong. But it does not tell you whether all forty broke for the same reason or for forty different reasons. It does not say which step failed. To learn that, an engineer has to open the log, the full recorded record of everything the agent thought and did, and read it line by line.

For one broken job, that is annoying. For an agent that runs thousands of times a day, reading those logs by hand becomes the slow part of every fix. The post calls this hand-checking the bottleneck, the narrow point that holds everything up, between spotting a problem and shipping the repair. It is the unglamorous gap nobody shows in the demo.

The obvious fix usually just moves the problem

The common instinct is to make the AI smarter. Swap in a more powerful model, write it longer instructions, give it more examples, and the failures will go away.

Sometimes they do. Often they just move. A smarter model that invents fewer facts might still pick the wrong tool, because the description of that tool is confusing. Longer instructions might fix one thing and break another simply by being long. Without knowing what kind of mistake you actually had, tinkering with the AI is guessing with extra steps.

The honest answer is less exciting. You need something that can look at a failed run and tell you, in plain words, what kind of mistake it was and what caused it. That is what this tool does, and it does it in two passes.

First pass: name what kind of mistake it was

Think of investigators reading the black box after a plane incident. The first job is not to assign blame. It is to name what happened: the engine, the weather, a misread instrument, a crew action. You cannot investigate until you have named the kind of problem.

The first pass does this for an agent. It walks through each step and sorts any mistake into one of a fixed list of types. This is the detail you can skim. The list has nine main kinds of mistake, and they cover the obvious ways an agent goes wrong: making something up, taking a wrong action, doing steps in the wrong order or skipping one, ignoring its instructions, getting stuck repeating itself, losing track of earlier information, and a couple more.

For every mistake it finds, it reports four things: where in the run it happened, what kind of mistake it was, how sure it is, and the actual evidence that set off the alarm. That “how sure it is” part is a confidence score, a rating of how likely this is a real problem rather than a false alarm. You get to set how jumpy it should be: catch everything, including some false alarms, or only speak up about the things it is most certain of.

Second pass: find the step that started it all

Naming the mistakes is not enough, because one mistake spreads. An agent that misreads something in step two will then take wrong actions in steps four, five, and six. Flag all of those as equal problems and you will spend the morning patching after-effects while the real wound, upstream, goes untouched.

So the second pass follows the chain backward and separates the cause from the effects. It marks one mistake as the one that started the trouble, and marks the rest as later knock-on effects. The first one is what you actually fix.

This is the pass that saves real hours. The output is not “here are six things that look wrong.” It is “fix this one thing in this one place, and most of the rest goes away.” And it tells you where the repair belongs. Often that is the agent’s standing instructions, the fixed orders it works under, or the description of one of its tools.

Where this fits in real, running software

A handful of pieces do the work, and one of them runs both passes at once so a team can get a full diagnosis in a single step.

The important part for a non-technical reader: it does not need a test lab. Live software keeps logs, the running record of what it did. This tool can read the logs from agents already working with real users, not just agents in a sandbox. And it can be told to only investigate the runs that already went wrong, which keeps the cost down, or to check everything when you want a full review.

One more thing that matters. All of this happens after the fact, on logs the agent already wrote. It never sits between the user and the agent. So it adds no waiting time for the person on the other end.

Why this unglamorous layer is the one that counts

Here is the part worth saying to anyone deciding whether to fund or run this kind of software. The demo is never the hard part. Any agent can look brilliant for five minutes on a stage. The hard part is the thousandth run, with nobody watching, when something drifts.

An agent with no failure detection is a self-driving car with no dashboard warning lights. It might be a perfectly good car. You just have no way to know when it isn’t, until you are already in the ditch. This kind of tool is the warning light and the mechanic’s report rolled into one. It does not make the agent smarter. It makes it safe to leave running on its own, which is a different and more valuable thing.

That is the line between a clever prototype and a system you would actually put in front of customers. The clever part gets the funding. The unglamorous part is what survives contact with the real world.

Sources: aws.amazon.com

Frequently asked questions

What is an AI agent in plain terms?

An AI agent is built on the same kind of model as ChatGPT, but instead of only writing an answer it takes actions to finish a job. It might search a database, run a calculation, or call another tool, and it works through several steps to get there rather than replying in one shot.

Why do AI agents go wrong more often than a normal chatbot?

A chatbot answers your question once and stops, so if it is wrong you see it right away. An agent chains many steps together, and a small mistake early on gets carried forward and built upon. More steps means more places to go wrong, and the error can stay hidden until the very end.

What is Strands Evals?

Strands Evals is a tool from AWS that looks at a failed agent run and works out why it failed and how to fix it. It checks each step the agent took, flags the ones that went wrong, and traces the problem back to the step that actually started it.

What kinds of failures does it look for?

It checks each step against a fixed list of nine main types of mistake. These include making something up, taking a wrong action, doing steps in the wrong order, ignoring its instructions, getting stuck repeating itself, losing track of earlier information, and a few others. Sorting the mistake into a type is the first thing it does.

What is the difference between a root cause and a symptom?

A symptom is the problem you can see, like a wrong final answer. The root cause is the earlier step that made that wrong answer unavoidable. Strands Evals labels each problem as the one that started the trouble or as a later knock-on effect, so you fix the real cause instead of chasing the after-effects.

What does the tool actually tell you to do?

After it traces the chain of cause and effect, it suggests fixes and says where each repair belongs. For example, it might point to the agent's standing instructions, or to the description of one of its tools. So the report does not just say something is broken, it says where to go and fix it.

How sure is it that something really failed?

For each problem it finds, it gives a confidence score, which is how sure it is that this is a real mistake and not a false alarm. You can set how cautious it should be, from flagging everything that looks even slightly off, to only flagging the things it is most certain about.

Can it look at agents that are already live with real users?

Yes. Live software keeps logs, which are records of everything it did. The tool can pull those logs from agents already running with real users, and it can be set to only investigate the runs that already went wrong, which keeps the cost down.

Does this slow the agent down for users?

No. It works after the fact on the logs the agent already produced, not while a person is waiting for an answer. So it never sits between the user and the agent, and the user never waits for it.

Why should a non-technical decision maker care about this?

This is what turns an impressive demo into something safe to leave running on its own. Without it, every agent mistake costs an engineer hours of digging through logs by hand, and that slow manual checking is the real thing that holds these systems back at scale.

Related writing

Jul 4, 2026

Get a build memo.

Send us your thesis. A senior engineering team sends back a written assessment — architecture, cost to first production deploy, and the risks most likely to kill it. Five business days.

Request a build memo