AI agents that write their own code to answer questions about company data

If you have used ChatGPT, you know the shape of it. You type a question, it types back an answer. Most of the time the answer reads well. But you have no way to check it from inside the chat. It is words, and words can sound right while being wrong.

Now picture a different question. Not “explain interest rates” but “how much did we refund last quarter, broken down by region.” That answer is not floating in the model’s memory. It is sitting inside your company’s own records. And getting it out is harder than it looks.

This is the problem a new paper takes on. The short version of its idea: stop asking AI to guess answers in words, and start letting it write small programs that fetch the real answer, run them, and fix them when they break.

Why a simple data question is rarely simple

First, a couple of plain definitions, because the rest depends on them.

Enterprise data is just all the information a company keeps to run itself. Sales, customers, payments, stock levels, support tickets. It mostly lives in databases rather than in readable documents. A database is less like a notebook and more like a giant filing cabinet with strict rules about which drawer holds what.

To pull something out of that cabinet, you write a query. A query is a precise question aimed at the data, written in a language the computer follows exactly. The most common one is called SQL. You do not phrase it like a human (“roughly how many big orders did we get?”). You phrase it like a machine, naming the exact table and the exact rule.

Here is where it gets slow. The paper describes how this work normally moves through several people. The data owner knows what is actually stored and what the odd column names mean. An engineer shapes that raw data into something usable. An analyst writes the queries and reads the results. The job gets handed from one to the next.

Think of a relay race where the baton is a little smudged at every exchange. The data owner knows a field called cust_st means customer status, but by the time the analyst sees it, that detail got lost in a hallway conversation. The paper calls these handoffs “repeated, lossy” for exactly this reason. Every pass drops a bit of context, and the final answer can quietly be built on a misunderstanding nobody noticed.

The usual fix, and why it falls short

The obvious move is to point a chatbot at the data and ask it your question in plain English. Type “total refunds last quarter by region” and let it write the query for you.

This works until it doesn’t. The model produces a query that looks correct. It might even run. But if it grabbed the wrong column, or counted refunds twice, or quietly ignored one region because the data was labeled oddly, you get a confident answer that is simply false. And you have no built-in way to tell. The model handed you words. Words do not come with a test attached.

That is the gap the paper aims at. Not “can the AI write a query” but “can the AI tell whether its own query was any good, and fix it if not.”

The actual idea: write code, run it, repair it

The system in the paper is called Data Intelligence Agents, shortened to DIA. The center of it is what the authors call autonomous coding agents. An agent, again, is AI that takes actions instead of only chatting. A coding agent is one whose actions are mostly writing computer code.

The key line in the paper is that these agents, “rather than emitting text,” instead “generate, execute, validate, and repair concrete artifacts.” Strip the jargon and it means four steps the agent does on its own. It writes a small program. It runs the program. It checks whether the result holds up. And if the result is broken, it rewrites the program and tries again.

That loop is the whole point. It is the difference between a cook who reads a recipe aloud and a cook who actually makes the dish, tastes it, and adds salt when it is flat. The first only describes. The second gets feedback from the real world and adjusts. Running the code is the tasting.

DIA splits the work across three agents, each with one job.

• The Data Interpreter reads the raw, messy data and figures out what it actually contains, including the strange column names.
• The Schema Creator builds the structure, a map of how the pieces fit together, so later questions know where to look.
• The Query Generator writes the actual queries that pull out answers.

There is one more piece worth naming. The agents draw on what the paper calls “a shared memory for experience reuse.” In ordinary words, the system remembers what worked before. Solve a tricky question about refunds once, and the next similar question starts from that experience instead of from scratch. It is closer to a person who has done a job a hundred times than to one seeing it fresh every morning.

Did it actually work, or just look good?

This is the question that matters, because plenty of AI systems dazzle in a demo and fall apart in real use.

The paper reports testing on seven SQL benchmarks. A benchmark is a public, standardized test. Many groups run their software against the same set of questions, so the scores can be compared honestly rather than each team grading its own homework. These seven covered four kinds of task and four versions, called dialects, of the database language. Dialects are like regional versions of the same tongue, where the words are mostly shared but a few rules differ.

On those seven tests, the paper says the system “matches or surpasses the best published results on all seven.” Read that carefully. Best published results means the strongest scores anyone had reported before. Matching or beating all seven is a real claim, not a vague gesture at promise.

Two honest cautions. The paper does not lay out a section of limitations, so the rougher edges are not spelled out for you here. And benchmark wins, while meaningful, are still controlled tests. A staged kitchen is not a Friday-night dinner rush.

That said, the authors state the system has run “in production with enterprise customers.” Production means real, live use, not a lab. So this is further along than most research you read about.

What a non-technical reader should take from this

Hold onto one shift. The old way had AI describe an answer. The new way has AI build the answer, run it, see if it stands up, and repair it when it doesn’t. Checking moves from being your job afterward to being part of the work itself.

And notice that a person stays in the loop. The whole design surfaces each result “for review by domain experts,” the people who actually understand the data. The agent does the grinding, repetitive part. A human still looks at it and says yes.

If you run a company sitting on years of data you cannot easily question, this is the direction worth watching. Not AI that talks about your numbers. AI that goes and gets them, then proves it got them right.

Sources: arxiv.org