AI infrastructure AI methods cost of scale

The price of anarchy: why serving AI cheaply is a traffic problem

Corteus Team · Jun 10, 2026 · 6 min read

A new paper studies what happens when an AI service gets crowded. Above a certain load, costs grow more than 280 times. Here is why, in plain words.

Key takeaways

Inference is the moment an AI answers you. Every time you hit send in ChatGPT, a machine does work to write the reply, and that work costs money.
Big AI services split that work across many machines. This is called disaggregated serving, and it is how a service answers millions of people at once without falling over.
When the machines act selfishly, the system gets slow. The paper calls the waste the price of anarchy, the same way traffic jams up when every driver grabs the fastest lane.
The waste stays small until the system fills up, then it explodes. Past a certain load the measured cost jumped from about 19 to 284, more than a 15 times rise.
A simple traffic controller helps. Letting one referee notice the jam and change the rules cut the worst delays by up to 7.6 times in the tests.

Every time you type something into ChatGPT and hit send, a machine somewhere starts working. It reads your words, then writes back one word at a time until the thought is done. That moment, the answering, has a name. It is called inference. And it costs money every single time.

Training a model is the famous expensive part, the one that makes headlines. But training happens once. Inference happens forever, on every question from every user. So the real bill for running an AI service is the inference bill. The question that quietly decides whether an AI company makes money is simple to ask and hard to answer. How do you serve millions of people at once without the costs running away from you?

A new paper takes that question apart using an old idea from a strange place: traffic.

Why one machine is not enough

Start with what answering actually involves. There are two jobs, not one.

The first job is reading your prompt. The machine takes everything you typed and digests it in a single heavy burst. Engineers call this prefill. It is short and demanding, like a chef reading the whole order at once.

The second job is writing the reply. The machine produces the answer one word at a time, each word leaning on the words before it. This is called decode. It is a long slow drip, like the chef plating one dish, then the next.

These two jobs strain the hardware in different ways. Reading wants raw computing muscle. Writing wants fast memory more than muscle. Forcing one machine to do both means it is always good at one and clumsy at the other.

So large services break the work apart. Reading goes to one pool of chips. Writing goes to another. This splitting is what the paper means by disaggregated inference. Disaggregated is just a long word for broken into separate pieces. The pieces share one fixed budget of expensive chips, and they have to divide that budget between them.

Now you have a traffic problem

Here is where it gets interesting. Once you have separate pools fighting over a shared budget, and a flood of requests deciding which machine to land on, you no longer have one tidy system. You have a crowd. Each request wants the fastest machine for itself. Each pool wants more chips for its own job.

Picture rush hour. Every driver picks the lane that looks fastest right now. Each choice is perfectly sensible on its own. The result is a jam that is slower for everybody than if a single traffic controller had assigned lanes. Nobody meant to cause it. Selfish-but-reasonable choices added up to a mess.

Game theory has a name for the size of that mess. It is the price of anarchy. You compare how the system performs when everyone acts selfishly against how it would perform under perfect coordination. A price of anarchy of one means the selfish version is already as good as it gets. A bigger number means coordination would have helped, and the gap is wasted money and wasted time.

This paper is the first to measure that price for a real AI serving system, in this case NVIDIA Dynamo, the software that shuffles requests across machines.

The waste hides until the system fills up

The most useful finding is about timing. The waste does not climb steadily as the service gets busier. It stays flat, then it snaps.

The researchers ran a large model and slowly increased the number of people asking questions at the same time. Below a certain crowd size, the price of anarchy sat calmly around 19 and did not budge no matter how they tuned the routing rules. The system was busy but not full. Choices barely mattered.

Then the machines filled up. Engineers call this point saturation, the moment a system can take no more work without slowing down. Past it, the price of anarchy shot up to 284 at the heaviest load tested. Same software. Same model. The only thing that changed was that the road ran out of room.

The numbers underneath are blunt. The time to get the first word of an answer grew from about 70 milliseconds to 113 seconds as the crowd grew. That is roughly 1,500 times slower. Oddly, the speed of writing each following word barely changed. It was reading the prompts, the prefill job, that choked first and dragged everything down with it.

A referee who watches for the jam

If the trouble only shows up past saturation, then the fix only needs to show up past saturation too.

That is what the researchers built. A small program, about 270 lines of code wrapped around the existing router, that watches the system and waits. While traffic is light it does nothing and lets the normal rules run. The moment it detects the jam coming on, it switches the routing rules to a setting tuned for a crowded road.

It worked. On one model the worst first-word delays dropped by 7.6 times after the switch. On another the overall waste score fell more than threefold. Just as telling, the system stopped swinging wildly. Before the fix, the same setup gave very different results from run to run. After it, the numbers held steady.

There is a cost. Running the crowd-mode rules gives up some raw throughput, between roughly 13 and 36 percent in the tests, because steadier traffic moves a little slower than reckless traffic on a good day. That trade is the whole point. A predictable system that handles its worst moments well beats a fast system that falls apart when it matters.

What to take from this

Be careful about how far you carry these results. This is one paper, by a single author, built on one specific platform and run on a small cluster of three machines. Several of the experiments were done only once, so the exact figures should be read as direction, not gospel. The author says as much.

The idea underneath travels further than the numbers, though. The cost of running AI for a lot of people at once is not a flat line you can plan around. It is calm, then it is a cliff. The cliff sits at the point where the machines fill up, and the system gets dramatically worse right when demand is highest. That is the worst possible time for things to break.

For anyone weighing what it really takes to serve an AI product at scale, that shape is the lesson. The dangerous question is not how cheaply the system runs on a quiet afternoon. It is what happens the moment everybody shows up at once.

Sources: huggingface.co

Frequently asked questions

What does inference mean?

Inference is the moment an AI model actually answers. You type a question, the model runs, and it produces a reply word by word. Training is when the model first learns. Inference is every use after that. When you chat with ChatGPT, you are paying for inference, even if you never see a bill.

What is disaggregated inference?

It means splitting the work of answering across separate machines instead of doing it all in one place. Reading your prompt and writing the reply are two different jobs with different needs, so a large service hands each job to its own pool of chips. Disaggregated just means broken apart.

What is the price of anarchy?

It is the cost of letting everyone act in their own interest with no coordination. The phrase comes from game theory. A price of anarchy of one means selfish behavior is already perfect. A higher number means coordination would have done better, and the gap is pure waste.

Why split the work across machines at all?

Because one machine cannot answer millions of people at once. And because the two parts of answering, reading the question and writing the reply, strain different parts of the hardware. Splitting them lets each part run on chips suited to it, which is cheaper than forcing one machine to do both badly.

What are prefill and decode?

Prefill is reading and digesting your whole prompt in one go. Decode is writing the answer one word at a time. Prefill is a short heavy burst. Decode is a long steady drip. The paper found prefill is usually the part that chokes first when a service gets busy.

What does saturation mean here?

Saturation is the point where the machines are full and cannot take on more work without slowing down. Below it, adding requests barely hurts. Above it, every new request waits in line and drags the whole system down. The paper shows the price of anarchy is calm below saturation and wild above it.

How big was the slowdown the paper measured?

In one test the time to get the first word of an answer grew from about 70 milliseconds to 113 seconds as the system filled up. That is roughly 1,500 times slower. The waste score the paper tracks rose to 284 at the heaviest load, up from a steady 19 below saturation.

What was the fix?

A small program that watches the system, notices when it crosses into the jammed state, and changes the routing rules on the fly. It is about 270 lines of code wrapped around an existing tool. In the tests it cut the worst first-word delays by up to 7.6 times and steadied a wildly swinging system.

Is this something a business can use today?

Not as a product. It is a research paper built on one specific setup, NVIDIA Dynamo, tested on a small three-machine cluster. The findings are a strong signal about how AI serving behaves under load, not a tool to install this quarter. Treat it as a map, not a machine.

Why should a non-engineer care about this?

Because the cost of running AI at scale is the cost of inference, and this paper shows that cost is not steady. It hides until the system fills, then jumps. Anyone planning to serve AI to a lot of users at once is buying into this behavior, whether they understand it or not.

Related writing

Jun 22, 2026

Get a build memo.

Send us your thesis. A senior engineering team sends back a written assessment — architecture, cost to first production deploy, and the risks most likely to kill it. Five business days.

Request a build memo