The wire is scheduled for Thursday. The deck was clean, the demo genuinely impressed you, and the founder answered every question without blinking. You have no CTO, no technical friend on call, and a nagging feeling that you just watched a magic show and are about to pay for the theater.
That feeling deserves attention. In August 2025, researchers at MIT’s NANDA initiative reported that about 95 percent of enterprise generative AI pilots were producing no measurable impact on profit and loss. Fortune covered the report and the number rattled a lot of boardrooms. Every one of those failed pilots had a good demo at some point. The gap between a demo and a business is exactly what technical due diligence measures, and you can measure most of it without reading a single line of code. Here is the checklist, ordered so the cheapest checks come first and kill bad deals early.
A free tool automates part of this checklist: eval-audit statically maps every prompt and LLM call in a repository to whether an eval covers it, and reports the coverage gap. It runs locally, with no API keys, alongside wrapper-test and model-rot.
1. Ask for evals, not another demo
A demo is a magic trick. The founder controls the deck of cards, the lighting, and which examples you see. An eval (short for evaluation) is the opposite: an automated exam where hundreds or thousands of real cases run through the product and get graded, producing a score you can track over time. Serious AI teams run evals on every change, the way accountants reconcile books.
So ask this: “Show me your eval suite and how the scores have moved over the past three months.”
A good answer arrives with numbers attached. Something like “we test against 1,200 real support tickets, we pass 91 percent, and we re-run the suite before every release.” A bad answer is another demo, or the phrase “we test constantly” with nothing written down. And there’s a follow-up worth asking: what happened to your scores the last time your model provider shipped an update? Teams with real evals can answer in seconds because they watched it happen on a dashboard. Teams without them found out from angry customers.
2. Find out where the model actually sits
Most AI startups don’t build their own AI. They rent a model from OpenAI, Anthropic, or Google the way a restaurant rents its building, then build product around it. Nothing wrong with renting. The question is what they’ve built that isn’t the building.
Ask: “If OpenAI or Anthropic shipped this exact feature tomorrow, what do you still own?”
A good answer names things a model provider can’t copy. Data no one else has. Integrations wired into customers’ daily workflow. The eval suite from check one. Contracts and distribution. A bad answer is “our prompt engineering is proprietary,” because prompts (the written instructions sent to the model) leak, get reverse-engineered, and can usually be reproduced by a competent engineer in an afternoon. A thin wrapper isn’t automatically a bad investment, but it should be priced like a trading position, not like an asset. There’s a longer companion piece on how to evaluate an AI startup’s technology before investing if the answer here makes you want to dig.
3. Trace the data
If the startup fine-tuned a model or trained anything of its own, the training data has a chain of custody, or it doesn’t. The New York Times sued OpenAI over exactly this in December 2023, and that case reshaped how acquirers and later-stage investors treat data provenance. What was a shrug in 2022 is now a diligence line item.
Ask for the paper trail. Where did the data come from, and can you show me the license or the customer contract that permits this use? A good answer is boring: signed agreements, named datasets, a clause in the terms of service that customers actually agreed to. A bad answer is “it’s publicly available data,” delivered quickly. Publicly visible and legally usable are different things, the way a car parked on the street is publicly visible.
4. Price one query
Every answer the product generates costs real money, because model providers charge per token (a token is roughly three quarters of a word). A restaurant owner who can’t tell you the cost of a plate of pasta doesn’t have a restaurant, they have a hobby. Same test here.
The raw prices are public. Anthropic’s pricing page lists Claude Sonnet 4.6 at $3 per million input tokens and $15 per million output tokens, with its cheapest model, Haiku 4.5, at $1 and $5. OpenAI’s pricing page lists GPT-5.5 at $5 in and $30 out. Anthropic’s own worked example puts 10,000 customer support conversations on Haiku at about $37 total. Fractions of a cent per interaction on a small model. A long agent session on a flagship model can cost dollars.
Ask: “What does one typical user session cost you in model fees, and what do you charge for it?” A good answer comes back to the cent, with a gross margin attached. A bad answer is a shrug, or “costs always come down.” Per-token prices have fallen, true, but newer reasoning models burn far more tokens per task, so cost per session often rises while cost per token drops. The full budget picture lives in how much it costs to build an AI product in 2026.
5. Count the people who understand the system
Engineers call this the bus factor: how many people could get hit by a bus before nobody can maintain the product. At many seed-stage AI companies the honest number is one.
You can test it without reading code. Ask a second engineer, with the founder out of the room, to walk you through how a change gets deployed and how a bad release gets rolled back. If someone other than the founding engineer can explain it calmly, knowledge is shared. If the answer is “that’s really Marko’s area,” you’re not investing in a company, you’re investing in Marko’s continued goodwill and health. Ask what’s written down. Good teams have runbooks. Fragile ones have memories.
6. Check the security posture
AI products carry a new kind of risk on top of ordinary software risk: prompt injection, which means an attacker hides instructions inside content the AI reads, and the AI obeys them. If the product reads customer emails or documents, this is not theoretical. It’s the reason AI agents need an immune system, not just a login page.
Four questions do most of the work. Where do the API keys (the passwords the product uses to call the model provider) live, and who can read them? Can one customer’s data ever appear in another customer’s answers? Has an outside firm ever attempted to break in, and can I see the report? And what’s your SOC 2 status, meaning the standard third-party audit of security controls? At seed stage, “in progress” is a fine answer to the last one. “What’s SOC 2?” is not. A fuller script lives in questions to ask an AI vendor about security.
7. Measure the exit cost from their vendors
Last check, and it loops back to the first. A startup renting one provider’s model has a landlord, and landlords raise rent, renovate, and sometimes tear the building down. Anthropic’s own pricing page lists half a dozen retired or deprecated models. Retirement is routine, not rare.
Ask: “How long would it take you to swap your primary model for a competitor’s?” A good answer is days or a couple of weeks, and the reason is the eval suite: swap the model, re-run the exam, compare scores, ship. A bad answer is “we’re deeply optimized for one provider,” with no fallback ever tested. Lock-in isn’t only about models either. Ask the same question about their cloud vendor and any orchestration platform they’ve built on.
What to do with the answers
You aren’t grading the technology. You’re grading how the team knows things. Across all seven checks, good answers had numbers, documents, a process someone could name and show you. Bad answers had confidence and adjectives. One or two soft spots is normal at seed stage, and the founder response you want is “we don’t have that yet, here’s the plan and the date.” Seven soft spots is a pattern. So run the first four checks before you ever mention a term sheet. They cost you two meetings, and they would have flagged most of that 95 percent while the money was still in your account.