How to evaluate an AI vendor: the twelve questions that separate a supplier from a liability
Model quality is the easiest thing to compare and the least likely to decide whether a vendor hurts you. Here is a practical due-diligence list built from this year's incidents: provenance, patch velocity, pricing cliffs, data paths, financing, and the exit.

Most AI vendor reviews stop at a benchmark table and a price per million tokens. The failures of 2026 happened elsewhere: a sandbox escape that led to $740,000 of fraudulent charges, an inference provider breached twice in a month, coding agents re-installing tools from the public internet daily, and a build-out financed by $680 billion of new debt. The review that covers those is short, and the same for a model API, an agent platform, a coding assistant or a medical scribe.
1. What exactly am I buying, and who else is in it?
Ask for the full list of sub-processors: the clouds, inference providers and tool vendors the product runs on. Many AI products are thin layers over another company's API; when that company has an incident, you have one. Ask which of them see your prompts, outputs and files, and for how long.
2. Where do the weights come from?
For any product that runs an open-weight or fine-tuned model, ask for its model provenance: the base model, the publisher, the hash, and whether the vendor diffs each checkpoint against its declared base before deployment. A backdoor can hide in a handful of neurons and pass every behavioural evaluation. A vendor who cannot answer this is shipping a model it has not inspected.
3. How fast do you fix things?
Ask for the median time from an externally reported vulnerability to a shipped fix, and for the mechanisms the vendor can use before a full patch: feature flags, input filtering, hotpatching. From January 2028 Google's Project Zero will publish bug details 30 days after reporting them, fixed or not, and other researchers will follow. A vendor whose fix cycle is measured in quarters is a vendor whose bugs will be public before they are closed.
4. What does the agent get to do?
If the product includes an agent, ask what tools it can call, what credentials it holds, whether high-impact actions need human approval, and how it is sandboxed. Ask whether its tool servers are pinned to exact versions or resolved from a public registry on every launch, and what it does when a tool returns instructions instead of data.
5. Where are the pricing cliffs?
Read the pricing page for thresholds, not rates: several vendors charge more for the whole request above a prompt-length line, and launch prices carry end dates. Get the price in writing with its expiry and a notice period. Model cost per task on your own workload, not cost per token.
6. What happens to my data?
Training on customer data, retention, deletion on request, residency, and whether staff can read prompts for abuse monitoring. Get the answers as contract terms, not FAQ pages.
7. Can I see the incident history?
Ask for every security incident in the last two years and what customers were told, when. No incidents means very new, or not telling you.
8. Who audits the output?
For regulated uses, ask for the evaluation set, the error rate on your kind of input, and who is accountable when the output is wrong. Clinical AI liability frameworks now assign responsibility by who could have prevented the harm; the contract should match.
9. Is the company going to be here?
Revenue, runway, and in 2026, financing structure: is compute owned, leased or guaranteed, and by whom? Regulators now track the web of debt, leases and special-purpose vehicles behind AI infrastructure. A vendor whose capacity depends on a counterparty refinancing can change prices and availability without a product decision.
10. What is the exit?
Export of prompts, logs, fine-tunes and evaluation data in a usable format, notice before deprecation, and portability of anything built on proprietary features. Prototype on two vendors so the exit is a configuration change.
11. Has the model changed?
Pinned versions, change logs and deprecation schedules. Silent updates break evaluations; ask how long a pinned version is supported.
12. Who is accountable?
A named security contact, a named account owner, and an SLA with credits that bite. If the answer to any question is a marketing page, score it as a no.
A vendor does not need twelve yeses; it needs no unexplained noes on the questions that match your risk. For a chatbot on public data, five and ten matter most. For an agent with credentials, two, three and four decide it.
Common questions
Should I just pick the vendor with the best benchmark scores?
Benchmarks tell you which model is most capable on someone else's test. Run the finalists on a sample of your own tasks and measure cost per correct answer. Then apply the twelve questions; the best model from a vendor that fails questions two and four is the riskier buy.
How often should I re-run the review?
At every renewal and after any incident, acquisition or model change. Pricing, sub-processors and financing change faster in AI than in other software; a review older than twelve months is a guess.
Does this apply to small vendors and open-source tools?
Yes, with the emphasis shifted. A startup or an open-source project may not have an incident history or an SLA; what it must have is provenance, pinned versions and a clear data path. Treat an unmaintained tool the way you would a vendor that has gone out of business.
Related

How to read an AI company's IPO filing: the eight numbers in an S-1 that matter more than the valuation
Anthropic's listing is the largest test yet of whether public investors can price a frontier lab. Here is how to read th…

How AI companies make money: the six revenue lines behind a frontier lab, and why the costs arrive before the cash
Anthropic's listing puts a lab's income statement in public for the first time. Here is where the money comes from (API …

LLM API pricing explained: tokens, caching, batch and the cliffs
How per-million-token pricing works, why output costs more, and the four levers that cut a bill by half or more.…
Get the brief in your inbox
One email every morning: the stories that mattered in AI, each with a one-line "why it matters" for your role. Free, no ads, unsubscribe any time.
Published daily by AI Briefing. We never share your address.