Can AI agents be trusted in drug discovery? What TxBench-PP found

If you have used ChatGPT, you know the basic deal. You type a question, it types back an answer. Useful, but notice what it is doing. It only talks.

There is a newer kind of AI that does more than talk. It is called an agent. An agent runs on the same kind of model as ChatGPT, but instead of just writing a reply, it takes actions. It can open files, read through data, run a few steps, and hand back a result. A chatbot tells you what it knows. An agent goes and works the problem.

That difference is exactly why people are excited about using AI in drug discovery. The hope is that an agent reads the lab data, reaches the right conclusion, and speeds up the slow gap between running an experiment and deciding what it means. That hope has pulled in real money. What nobody had was a fair way to check whether these agents are actually good at the part that counts, which is making the correct scientific call on data they have never seen before.

A group of researchers set out to check. Their paper introduces a test called TxBench-PP. A test like this has a name people use a lot, so it helps to know it: a benchmark is just a fixed set of tasks you run many systems through, so you can compare them fairly on the same questions. TxBench-PP is short for TherapeuticsBench Preclinical Pharmacology. Preclinical means the early lab stage of drug research, before anything is tried in people. And the headline result is blunt. Across every setup they tried, no AI could reliably reach the conclusion a human drug scientist would.

What the test is checking

Most AI tests reward memory. Ask a model a question whose answer sits somewhere in the huge pile of text it was trained on, and a good model digs it out. That tells you the model read a lot. It does not tell you whether the model can think through evidence it has never seen.

TxBench-PP was built to take that crutch away. Instead of quiz questions, the AI is handed real snapshots from actual drug programs. It opens the files in a working environment, looks at the lab results the way a scientist would, and gives back a clear answer. The grading is done by a fixed rule, meaning a set yes-or-no check decides whether the answer is right. No human is in a good or bad mood. No second AI weighs in with an opinion.

Think of the difference between a closed-book and an open-problem exam. A closed-book test asks what you remember. This is closer to handing someone a folder of fresh results from the lab and asking, so what does this actually mean. The questions cover the real judgment calls in early drug work: how a compound behaves, whether it is hitting the target it is supposed to hit, whether it looks safe enough to keep going. These are not trivia. Each one can push a drug forward or kill it, and getting one wrong costs real time and money.

That gap is the whole reason this test exists. A model that has read every paper on a drug target can sound like an expert. It can recite the literature back to you. TxBench-PP asks the harder thing. Given results it has never seen, can the AI figure out what they mean. Sounding right and being right are not the same, and a test like this is built to tell them apart.

The numbers, and what they say

The researchers ran 16 different setups. Here a small piece of jargon matters, so here it is in plain words. A model on its own is just the raw brain. To get it to read files and take steps, you wrap software around it, and that wrapper is called a harness. The same model can do better or worse depending on the harness it is wired into, the way the same driver can do better or worse depending on the car. Those 16 setups came from 11 different models and produced 4,800 attempts in all. One full attempt at working a task from start to finish is called a trajectory.

The strongest setup was Claude Opus 4.8 running on a harness called Pi. It got 59.3 percent of its answers right, which works out to 178 out of 300. GPT-5.5 on the same harness came next at 55.3 percent, or 166 out of 300.

Read those numbers the way a research lead would. A tool that is right on roughly six of every ten high-stakes calls is not a tool you let decide anything by itself. It is a tool you watch closely. The top two were also close enough together that picking a clear winner would oversell the gap between them. What the spread really shows is a field where the best tools available land well short of dependable.

That is the finding the authors put front and center, and it is more useful than a high score would have been. A test that announced some model passing 95 percent of the time would invite an obvious question. Was the test just too easy. This one shows the opposite. The data is real, the questions are hard, and the distance to trustworthy is large.

Why the test itself is the real result

It is tempting to skim a paper like this for the leaderboard and move on. The leaderboard is the least important part.

The thing that lasts is the measuring stick. In serious science, you cannot put a tool into use just because a demo looked good or a salesperson said so. You need proof that holds up when a skeptic re-runs it. Grading by a fixed rule, on real data, is that kind of proof. It turns a vague question, can AI help with early drug work, into a sharp one with a number on it. How often does this exact setup get the call right on these exact tasks.

That changes how decisions get made. Without a test like this, arguing about whether to trust an AI agent in a drug program is just trading impressions. One person finds the output convincing, another does not, and there is no way to settle it. With the test, the argument has a floor. You can say this setup gets 59 percent and that one gets 55, and then argue about whether either is good enough for the job you have in mind. The disagreement turns useful, because it is pinned to something anyone can repeat.

There is also a trust question that reaches past any single program. Drug development is watched by regulators, by funders, and in the end by patients. Putting an AI into that chain raises a fair worry. If an agent helped reach a conclusion that turned out wrong, what was the reason for trusting it in the first place. A test you can re-run gives part of the answer. It does not make the AI flawless, but it lets a team say plainly how the tool does on a set list of tasks, and it lets an outsider go check that claim.

What this means for anyone putting AI into serious work

The drug-research details are narrow. The lesson is not. It applies anywhere people are tempted to hand real judgment to an AI.

First, remembering is not the same as reasoning. An AI that aces a knowledge quiz can still flounder when asked to read fresh evidence. If the job is judgment on new data, the test has to use new data too.

Second, the wrapper counts. The same model gave different results depending on the harness around it. Buying or building an agent means checking the whole package, not just the model name on the box.

Third, a believable test usually reports a humbling number. The best setup here clearing under 60 percent is not a sign the research flopped. It is a sign the test is being honest about a genuinely hard field. Results that look too clean usually mean the test was too soft.

For now, the plain reading of TxBench-PP is this. AI agents can be a useful extra pair of hands in early drug work. They are not yet a replacement for the scientist who checks the answer. The researchers also describe it as the first piece of a larger effort meant to cover more stages of drug research over time. That is the part worth watching, because the tools will get better, and the only way anyone will know they have is a test like this one.

Sources: arxiv.org