AI agents computer use training data

Teaching AI to use a computer by letting it practice

Corteus Team · Jun 14, 2026 · 6 min read

Most AI only chats. A new report shows how teaching an AI to click and type its way through a real desktop made it far better at finishing tasks.

If you use AI today, you almost certainly use it by chatting. You open ChatGPT, type a question, and read what comes back. It writes, it explains, it drafts. What it never does is reach over and use your computer for you. It tells you which buttons to press. It does not press them.

There is a kind of AI being built to cross that line. It is called a computer-use agent. The name is literal: it is an AI that uses a computer, by looking at the screen and clicking and typing, the way a person does. A new technical report, the ProCUA-SFT report published on June 14, 2026, is about how to make this kind of AI better at the job. The short version of what it found is worth saying up front: the biggest problem was not the AI, it was the lack of practice, and the fix was to manufacture the practice.

What a computer-use agent actually does

Picture the difference this way. Ask a chatbot to clean up a spreadsheet and it writes you instructions: open the file, select the column, sort it. Useful, but you still have to do every step. A computer-use agent is meant to do those steps itself. It sees the screen as an image, decides where the pointer should go, moves it, clicks, types, then looks again to see what changed.

The report defines it plainly as a model that interacts with desktops through screenshots and keyboard and mouse actions. Think of it as an AI that clicks and types like a person. That single shift, from talking about the work to doing the work, is the whole point of this field.

Why practice is the hard part

Here is the catch. Operating a screen is not one decision, it is a long chain of them. See the file. Click to open it. Wait. Read what appeared. Decide the next click. Each task is dozens of these small moves in a row, and one wrong move can spoil the rest.

To learn a chain like that, an AI needs to watch it done correctly an enormous number of times. That is where this whole area gets stuck. You could record real people doing real tasks and feed those recordings in, but collecting them is slow, expensive, and never produces enough variety. Good worked examples are scarce. The report names this directly as the core problem: a shortage of large-scale training data.

There is a term for the kind of training in play here. Fine-tuning means taking an AI that already exists and giving it extra lessons on one specific skill until it improves at that skill. The version used here is supervised fine-tuning, often shortened to SFT, which simply means showing the model many correct examples and letting it learn the pattern by imitation. No magic. The model gets better at what it is shown a lot of. So the question becomes: where do you get enough to show it?

The fix was to manufacture the practice

Instead of recording people, the report built the examples automatically. A single AI model was put to work generating tasks on a desktop, then carrying them out, producing complete run-throughs from start to finish. It produced 93,000 of these run-throughs across 2,484 different application combinations, meaning different mixes of programs being used together.

Then came a clever, almost obvious trick. Each run-through is a sequence of steps, so each one was sliced into its individual steps, with every slice carrying the full history of what happened before it. A run of thirty steps becomes thirty practice examples, each showing the model a little more of the story. Slicing the 93,000 run-throughs this way produced 3.1 million step-level examples to train on. That is the dataset the report is named after.

One detail keeps the whole thing honest. The desktops were not blank. They were filled with real material: 912 actual spreadsheets, some with more than 100 columns and 20,000 rows, and roughly 10,000 freely-licensed presentation files. So the AI practiced on content that looks like the messy real thing, not on toy examples that would teach it the wrong habits.

Did it work

Yes, and by a clear margin. To measure computer-use AI, researchers use a shared test called OSWorld, which hands the AI real tasks on a real desktop and counts how many it finishes. Because many groups run the same test, the numbers can be compared.

The base model, before this training, finished 26.3 percent of OSWorld tasks. After training on the 3.1 million manufactured examples, it finished 45.0 percent. The report frames that as an 18.7 percentage-point jump, which in this field is large. The model went from failing roughly three tasks in four to finishing nearly half.

The more surprising result sits next to it. The team also tried training on a set of 22,500 real human run-throughs, the kind of authentic data you might assume would be the gold standard. It scored only 8 to 10 percent, and worse, it dragged the model below where it started. The report calls this negative transfer: that particular real data taught the model bad habits and made it worse. Manufactured practice beat human recordings here, which is not what intuition would predict.

Why the manufactured data was better

The report offers a reasonable theory, and it comes down to variety and shape. The made-up tasks were longer, averaging about 29.7 steps against the human set’s 17 to 18, so the model practiced seeing tasks through to the end rather than giving up early. They also leaned harder on the keyboard, 41 percent keyboard actions versus the human set’s 63 percent clicks, which tends to track with getting real work done rather than just poking around.

Most of all, the report points at the range of application combinations as the thing that mattered most. Practicing across 2,484 different mixes of programs taught the model to handle situations it had never seen, which is closer to how messy real work actually is.

Keep this in proportion

One honest caveat. This is a technical report about a method and a dataset, written for people who build AI systems. It is not a product. There is nothing here to download and use this week, and the report itself notes it expects to keep improving the approach as better tools become available. The numbers are real and the method is interesting, but the right way to read it is as a sign of direction, not a finished thing.

The direction is the part worth holding onto. The AI you chat with today is the talking kind. A growing amount of work is going into the doing kind, the AI that operates the screen itself. And this report shows that one of its biggest roadblocks, the shortage of practice, may have a way around it. If a machine can generate its own lessons by the millions and learn from them, the gap between AI that describes the work and AI that does the work gets a little narrower.

Sources: huggingface.co

Frequently asked questions

What is a computer-use agent?

It is an AI that operates a computer by looking at the screen and using a keyboard and mouse, the same way a person does. The report defines it as a model that interacts with a desktop through screenshots and keyboard and mouse actions. Where ChatGPT writes you an answer, this kind of AI clicks and types its way through an actual task.

How is this different from the ChatGPT I use?

ChatGPT reads your message and writes a reply. It does not touch your computer. A computer-use agent takes the next step: it looks at a screen, decides where to click, moves the pointer, and types. It carries out the task instead of only describing how.

What does fine-tuning or SFT mean?

Fine-tuning means taking a model that already exists and giving it extra training on a specific kind of example so it gets better at that thing. SFT, short for supervised fine-tuning, is the plain version of that: you show the model many worked examples and it learns to copy the pattern.

Why does this kind of AI need so much practice data?

Operating a screen is a long chain of small decisions: see this, click here, type that, check the result. To learn the chain, the model needs to see it done correctly many thousands of times. Real recordings of people doing such tasks are slow and costly to gather, so good examples are scarce.

How did the report get 3.1 million practice steps?

It built them automatically instead of recording people. The method generated 93,000 full task run-throughs across 2,484 application combinations, then broke each run-through into its individual steps. That produced 3.1 million step-level examples to train on.

What were the results?

On a standard test called OSWorld, the trained model finished 45.0 percent of tasks. The same base model started at 26.3 percent. The report frames this as an 18.7 percentage-point gain from the new training data.

Is this better than training on real human recordings?

In this report, yes. A set of 22,500 real human run-throughs called AgentNet scored only 8 to 10 percent and actually lowered performance below the starting point. The report calls this negative transfer, meaning that particular real data made the model worse rather than better.

Can a business use this today?

Not as a product. This is a technical report describing a method and a dataset, aimed at people who build AI systems. It is a signal of where computer-use AI is heading, not something to roll out this quarter.

What is OSWorld?

OSWorld is a standard test for computer-use agents. It gives the AI real tasks on a real desktop and measures how many it actually completes. Because many groups use the same test, scores can be compared across different models.

What did the trained model do differently?

Its practice tasks were longer and leaned more on the keyboard. The runs averaged about 29.7 steps, roughly 1.8 times longer than the human-recorded set, and used the keyboard for 41 percent of actions where the human set was 63 percent clicks. The report points to the range of application combinations as the thing that mattered most.

Related writing

Jul 4, 2026

Get a build memo.

Send us your thesis. A senior engineering team sends back a written assessment — architecture, cost to first production deploy, and the risks most likely to kill it. Five business days.

Request a build memo