How to tell if an AI agent is actually working

Picture asking an AI to plan a trip. It comes back with a tidy itinerary: flights, a hotel, a restaurant near the museum. It reads well. You would book it. The problem is that one of those flights does not exist. The AI made it up, and nothing in the polished reply tells you which part to doubt.

That gap is the subject of a post AWS published on June 11, 2026. It introduces a free toolkit called Agent-EvalKit, and it starts from an uncomfortable point: a right-looking answer can hide a badly broken AI agent.

First, what an agent even is. If you have used ChatGPT, you know the rhythm. You type, it types back. An AI agent is the next step. It is built on the same kind of model, but instead of only writing a reply it takes actions. It searches, looks up records, runs calculations, calls other systems. A chatbot tells you how to plan a trip. An agent goes and tries to plan it. The tools it reaches for are how it gets anything done.

Why a good answer can still be a lie

Here is the trap. When people check whether an agent works, they usually look at the final reply. Does it match what we expected? If yes, ship it.

But the AWS team points out two ways that check fails. An agent can produce a clean, well-structured response while inventing facts that no tool ever returned. It can also reach a correct conclusion through a broken path, calling the wrong tools or feeding them bad inputs, getting lucky this time and not the next.

Think of grading a math test by looking only at the last number on the page. A student writes “42” and you mark it right. You never saw that they added when they should have multiplied and landed on 42 by accident. Next problem, the same wrong habit gives a wrong answer, and you have no idea why. The final number told you nothing about whether the thinking was sound.

That is the difference between checking the answer and checking the work. Agent-EvalKit checks the work.

What it actually looks at

Evaluation is just a fancy word for grading. The toolkit grades an agent by tracing its whole path: which tools it called, what data those tools handed back, and whether the final reply honestly reflects that data.

It boils down to three plain questions, each with a name worth knowing once.

The first is tool parameter accuracy. When the agent reaches for a tool, does it hand over the right inputs? Asking a weather tool about the wrong city gives a confident answer about a place you never asked about.

The second is faithfulness. Does the answer stick to what the data actually said? This is the one that catches the invented flight. A faithful agent says only what its tools support. An unfaithful one fills the silence with guesses.

The third is response quality. Plain coherence and usefulness. Is the reply clear, and does it answer what was asked?

You can think of it as a report card for an AI agent, with separate grades instead of one blurry overall mark. A student can ace neatness and fail honesty. So can an agent, and you want to see both numbers, not their average.

The number that should make you pause

AWS ran the toolkit on a travel research agent and shared the scores. Response quality came back at 83.9 percent. Tool parameter accuracy at 64.5 percent. Both decent. The kind of numbers that would let a demo sail through.

Faithfulness scored 32.3 percent.

Read that again next to the first one. The agent wrote clear, useful-sounding answers more than eight times out of ten, but its answers matched the real data barely one time in three. The cause, the post explains, was specific. When a tool came back empty, the agent did not say “I found nothing.” It invented something to fill the space.

This is exactly the failure a final-answer check would wave through. The replies looked great. Underneath, two thirds of the substance was not earned.

How a non-coder actually uses this

The honest answer is that you probably do not run it yourself. Agent-EvalKit lives inside coding assistants, including Claude Code, and works with a handful of agent-building frameworks. It is built for the people who build agents.

What you do instead is describe the goal in plain language. The post says you trigger steps with simple commands and the assistant handles the heavy lifting. There are six steps in all, and you do not need to memorize them, but the shape is worth seeing. It plans what to test, generates test cases aimed at likely failures, adds tracing so the agent’s steps become visible, runs the agent, scores it, and writes a report. The report does one genuinely useful thing: it ties each problem back to the exact place in the code that caused it. Not “faithfulness is low” in the abstract. Here, this function, this is where it invents things.

For a leader who is paying for an agent rather than building one, that is the part to hold onto. You do not need the toolkit. You need to know it exists, and to ask the people building your agent whether they run something like it.

What this means before you trust an agent with anything

Many AI agents are less reliable than their demos suggest, and the reason is usually this exact gap. The demo shows a smooth final answer. Nobody traced the steps on stage. So the agent that wowed the room is judged on the one thing that hides its worst habits.

Three questions cut through it. Are you grading the steps or only the final answer? What is the faithfulness score, and what does the agent do when a tool returns nothing? And can you point to the exact code behind each failure?

A builder who can answer those has tested the work. One who only shows you a clean reply has shown you a last number on a page and asked you to trust the math you cannot see.

One caveat. The toolkit is new, free, and real, but it is a developer tool tied to particular frameworks, not a dashboard you log into. The deeper point outlasts the tool, though. The next wave of agents will not be judged by how good the answer looks. They will be judged by whether the answer was earned.

Sources: aws.amazon.com