An eval is a test that sends real cases to a model, or to a recording of one, and judges what comes back. Unit tests check the plumbing around the model. Both matter, but only an eval tells you whether last week’s prompt change made the product worse. Our July note described what a real suite contains. This time we wanted to see how often popular open-source AI apps ship one.
We picked the apps by rule. On 1 November we searched GitHub for repositories tagged llm with more than 8,000 stars, a push in the last 90 days and no archive flag. In star order we took the first 15 Python and the first 15 TypeScript projects that an end user runs to get a job done and that declare an LLM SDK. Libraries, frameworks and infrastructure stayed out. The 30 apps range from 14,973 to 259,424 stars, and every keep and every skip has a written reason in the data.
Then we ran eval-audit, our open-source scanner that maps the places where code calls a model to the eval files that cover them. Before the run we checked what it could not see. It missed LiteLLM, Python Gemini, Anthropic streaming and a dozen eval frameworks, and it counted ordinary unit tests as evals. We fixed all of that first. Version 0.2.0 has 13 tests where 0.1.0 had 7. Nothing from the 30 repositories was run or installed. We only read their files.
| What the scanner found | Apps |
|---|---|
| No eval files | 17 |
| Eval files covering under 25 percent of model calls | 11 |
| Eval files covering 25 to 75 percent | 1 |
| More than 75 percent covered | 0 |
| Evals run in CI on a pull request | 0 |
| No verdict, because the app calls models over raw HTTP | 1 |
Across the 30 apps the scanner found 318 places that call a model. 18 of them, or 5.7 percent, link to an eval file. 12 apps keep at least one eval file, and in only 5 does an eval file reach the code that calls the model. The one app with an eval command in its workflows, paperclip, runs it weekly or by hand. hermes-agent came out best, with 14 of its 23 model calls covered.
The rule we chose decides the headline. 28 of the 30 apps have unit tests that import their model code. Count those as evals and 19 apps look well covered, with a median of 91.7 percent. The tests we opened replace the model with a mock. That checks that the code runs. It says nothing about whether the answers are any good.
We checked the scanner by opening files in 23 of the repositories. It was right about whether evals exist in 19 of the 22 that got a verdict. It missed two real suites. gitdiagram keeps LLM judges in an experiments folder, and qwen-code runs terminal-bench and publishes SWE-bench scores from tests the scanner does not read as evals. It also counted one suite that is not one, a bundled script in cherry-studio that tests something other than the app. In a random sample of 40 model calls it reported, 36 were real.
A repository without eval files does not prove a team has no evals. AutoGPT points to its scorers in a separate public repo, AgentProbe, and paperclip’s weekly job pulls a repository that is not public. The scanner follows imports one level deep, so suites that drive a whole pipeline, as onyx and dyad do, count for less than they are worth. Seven apps have three or fewer model calls, which makes their percentages thin. Several detector fixes came from this same sample, so we will test them on fresh repositories before we quote the scanner’s precision.
If you build an AI product, three things from this run are cheap. Put the suite in a folder named evals with a short README, so reviewers and new hires find it. Have the evals call the same function the product calls, so a change of model, parameters or parsing shows up in them. And run a small, stable set of cases on every pull request that touches prompts or model code, because a weekly run finds the regression after it has shipped.
Everything is in the study folder of eval-audit: the selection list with a reason for every repository, the commit we scanned for each app, the raw output of both scanner versions, the hand-check notes and the scripts that build every number above. If we got your project wrong, the reason sits next to its row, and we will correct it.