large language models AI evaluation AI agents

Can AI models remember what they saw? What RNG-Bench found

Corteus Team · Jun 16, 2026 · 7 min read

Researchers built a test to check whether 7 AI models can act on facts that are no longer on the screen. The best one got 62.3 percent of card pairs right, and bigger boards broke them fast.

Key takeaways

The test measures memory on its own. RNG-Bench compares a model against a version that is always shown the hidden answer, so a low score means the model forgot, not that it made a bad call.
The best model still missed a third of an easy task. On the picture version of a card matching game, GPT-5.4 got 62.3 percent of pairs right, Gemini-3.1-Pro 50.0 percent, and Qwen3.5-397B only 25.3 percent.
More to remember breaks them. On a bigger board with the same rules, Qwen3.5-397B fell from 90.6 percent to 0.7 percent.
Pictures are much harder than words. The same task scored 67 to 75 percent when the model read plain text, but dropped to around 20 percent when it had to read messy images.
The models lean on their own notes. Deleting the record of past moves cut their score by roughly 70 to 75 percent, so the memory was mostly in the notes, not in the model.

If you have used ChatGPT, you know it answers whatever is in front of it. You type a question, it types back. But here is something it quietly struggles with: acting on a fact you mentioned a while ago, after that fact has scrolled out of view. Can the model still use what it can no longer see? That is the question this post is about.

Think of the children’s card game Memory. Cards lie face down. You flip two, see a fox and a boat, then flip them back. The whole game is keeping those pictures in your head until you find their twins. The board itself never shows you what you need. You have to carry it in your memory.

That game turns out to be one of the cleanest ways to test an AI model. A group of researchers from Fudan University, Zhejiang University, The Chinese University of Hong Kong, and Shanghai AI Lab built a test around exactly this. The paper introduces a benchmark called RNG-Bench. A benchmark is just a standard test, run the same way for every model so the scores can be compared fairly. The short version of what they found: the best models on the market today are surprisingly bad at the card game, and worse at anything harder.

Why the screen in front of you is not enough

There is a name for situations where whatever you can see right now tells you everything you need to know. It is called Markov. Chess is Markov. Look at the board and the best move is decided by the pieces in front of you. How the game got there does not matter.

Most real work is not like that. A non-Markov situation is one where two moments can look identical on screen but call for different actions, because something earlier changed what the right move is. The face-down cards are the perfect example. The board looks the same whether or not you have already peeked at the fox. Only your memory tells the two apart.

This sounds like a small academic point. It is not. It is the difference between a test a model can pass by spotting surface patterns and a test that forces it to keep a running picture of the world in mind.

What RNG-Bench actually does

The test uses two games, each chosen to stress a different kind of memory.

The first is the card matching game, the children’s game scaled up. Pictures flash for a moment, then hide. To score, the model has to recall where it saw each one and match the pairs. This tests a fixed set of hidden facts that do not move once you have seen them.

The second is a maze seen from the inside. The model walks through a corridor, seeing only what is right in front of it, and has to build a map of the whole place from those narrow glimpses. This is harder, because the thing to remember is not a fixed list. It is a picture that keeps growing as you move.

Two design choices make the test sharp. First, the difficulty can be turned up one dial at a time: how big the board is, how easy the patterns are to tell apart, and whether the model gets its information as text or as a picture. The rules stay the same, so when a score drops you can point at exactly which dial caused it. Second, there is a clever control the researchers call the memory gap.

The trick that tells forgetting apart from bad judgment

Every memory test has a catch. If a model fails, you cannot tell whether it forgot the fox, or remembered the fox perfectly and still made a foolish move. Two very different problems, same low score.

So the researchers built what they call an oracle. It is the same game, except this version is handed the right answer at every single step, for free. It never has to remember anything. So if the oracle scores well and the normal model does not, the gap between them can only be forgetting. Picture giving one card player a perfect photographic memory and watching how far the ordinary player falls behind. That distance is the cost of a leaky memory, measured cleanly.

That one design choice is what makes the numbers worth reading.

The numbers

Seven models went through it: GPT-5.4, Gemini-3.1-Pro, two sizes of Qwen3.5, Kimi-K2.5, and two Seed-2.0 models.

On the card game at a 10 by 10 board, using pictures, GPT-5.4 got 62.3 percent of the pairs right. Gemini-3.1-Pro hit 50.0 percent. Qwen3.5-397B managed 25.3 percent. Read that again. The strongest model on the planet, on a children’s memory game, missed more than a third of the matches.

The maze was worse. On a 13 by 13 layout, Gemini-3.1-Pro reached a 50.0 percent success rate. GPT-5.4 fell to 20.0 percent. Qwen3.5-397B scored zero. Not low. Zero. A model with hundreds of billions of moving parts could not finish a single maze that depended on remembering where it had been.

Then comes the finding that should make anyone building with these tools pay attention. Take Qwen3.5-397B on the card game. At a tiny 4 by 4 board it scored 90.6 percent, nearly perfect. Move to a 12 by 12 board, with the same rules, and it scored 0.7 percent. The model can hold a few facts. Then, past some small point, the whole picture falls apart at once.

Two results that change how you read a flashy demo

The kind of input mattered enormously. When the same fact arrived as a clean text label, the card game scored between 67 and 74.9 percent. When it arrived as a messy picture, scores collapsed to around 20 to 22 percent. The lesson: a model that can handle both words and pictures often has a weak link on the picture side. It has to read the image correctly before it can even try to remember it, and that reading step quietly fails a lot.

The second result is the one worth sitting with. The researchers deleted the model’s own log of its past moves, the running record of “I flipped this, then that.” Performance dropped by roughly 70 to 75 percent. So the model was not holding the board in some inner sense of the world. It was leaning on its own written notes, rebuilding what it knew by rereading its own diary. Take the diary away and the memory mostly disappears. The paper says these notes are load bearing, and that is the right word. Pull them out and the whole thing falls down.

Why a business reader should care

Almost every job people want to hand an AI is non-Markov. Booking travel where the right flight depends on a rule mentioned several steps ago. Working a support ticket where the fix depends on what was already ruled out. Reading a long document and acting on page two while you are on page forty. In all of these, the screen in front of the AI does not hold the answer. The answer lives in what came before.

A short demo never shows this. Demos are quick. There is little to remember, the board is small, the model scores 90 percent, and everyone claps. The trouble shows up later, on the long task, once the load crosses the invisible point that turns 90 percent into nearly zero. RNG-Bench is useful precisely because it makes that point visible, and lets you blame the failure on forgetting rather than guessing.

There is a hopeful note at the end. The researchers trained the smaller Qwen3.5-9B on good example games and lifted its score on a separate memory test by 5.2 percent, without hurting its general ability, which actually nudged up 0.5 percent. So this is teachable. It is not a permanent wall. But right now, today, if you are about to trust an AI with a task that depends on remembering, the honest move is to test it at the size you will actually use, not the size that looks good in a demo.

Sources: huggingface.co

Frequently asked questions

What is a non-Markov game?

It is a game where what you can see right now is not enough to pick the best move. Two moments can look exactly the same on screen but call for different actions, so you have to remember what happened earlier. Chess is the opposite, since the board in front of you tells the whole story.

What is RNG-Bench?

A benchmark, meaning a standard test, built by researchers from Fudan University, Zhejiang University, The Chinese University of Hong Kong, and Shanghai AI Lab. It uses two games to check whether AI models can hold on to information that is no longer on the screen and use it later in a multi-step task.

What two games does it use?

A card matching game where pictures flash for a moment and have to be recalled by their spot later, and a maze the model walks through seeing only what is right in front of it, building a map in its head. One tests fixed facts that stay put, the other tests a picture that keeps changing as you move.

What is the memory gap?

It is the gap between the normal model and a special version handed the right answer at every step for free. The special version never has to remember anything. If it scores well and the normal one does not, the difference is caused by forgetting, not by bad judgment.

Which models were tested?

GPT-5.4, Gemini-3.1-Pro, Qwen3.5-397B and Qwen3.5-9B, Kimi-K2.5, and Seed-2.0-Lite and Seed-2.0-Pro. Seven models in total.

Why did the scores fall as the boards got bigger?

Bigger boards mean more things to hold in mind across more steps. Qwen3.5-397B went from 90.6 percent on a tiny board to under 1 percent on a large one. The models can hold a few facts, then lose track once there is too much to keep.

Why are pictures harder than text for these tasks?

When the same fact came as clean text it scored in the 67 to 75 percent range. When it came as a messy picture, scores dropped to around 20 percent. The model has to read the picture correctly before it can even try to remember it, and that first reading step fails a lot.

What does the action history result mean?

When the researchers deleted the model's own written log of its past moves, its score fell by about 70 to 75 percent. So the model was not holding the picture in its head. It was rereading its own notes to remember. Take the notes away and the memory mostly goes with them.

Why does this matter for AI agents in business?

Most real jobs people want to hand an AI are non-Markov. The right next step depends on something said a few steps ago or a file opened earlier. A system that forgets under load will look fine in a short demo and fail on a long task, which is exactly the failure this test exposes.

Did training help?

Giving the smaller Qwen3.5-9B good example games to learn from raised its score on a separate memory test by 5.2 percent, and its general ability even nudged up 0.5 percent. So the skill can be taught, at least in part.

Related writing

Jun 29, 2026

Get a build memo.

Send us your thesis. A senior engineering team sends back a written assessment — architecture, cost to first production deploy, and the risks most likely to kill it. Five business days.

Request a build memo