The AI vendor evaluation checklist

The vendor's demo was flawless. Your team was impressed. Six months into the contract, the model was updated without warning and a critical workflow broke overnight. This is not bad luck. It is the predictable result of evaluating a vendor on the wrong criteria.

Why AI vendor evaluations go wrong.

Evaluating software vendors is a known discipline. You have procurement processes, security questionnaires, reference checks, pilot projects. These are not wrong for AI vendors — they are simply insufficient. AI vendor evaluation fails in a specific way: teams evaluate the model's capability in isolation from the operational context that determines whether that capability survives contact with production.

The demo is optimised to show the model at its best. Queries are curated. The knowledge base is freshly indexed. Response time is measured on a single uncongested API call. The evaluator knows what to avoid. None of this resembles production, where queries are messy, the knowledge base is two months stale, API calls are concurrent under load, and nobody is filtering for the edge cases that break things.

The result is a systematic overestimation of vendor quality at the point of contract signing, followed by a disillusioning first quarter in production. Most teams then spend the next six months trying to adapt their use case to the vendor's limitations rather than re-evaluating whether the vendor was the right choice. The re-evaluation never happens because switching costs are real and the sunk cost is significant.

The framework below is designed to surface the production reality during the evaluation period, before you sign. It is not exhaustive — every use case has specific requirements that need specific tests. But it covers the categories where AI vendor evaluations most consistently fail.

Category one: data and privacy.

Before any technical evaluation begins, you need clear answers to a set of data handling questions. These are not negotiable for regulated industries and are becoming less negotiable for everyone else as AI-specific data regulations mature.

The first question is data residency: where are your prompts and completions processed, and where are they stored? For companies handling personal data under PDPA, GDPR, or equivalent frameworks, this question determines whether you can use the vendor at all. "Processed in US-East with optional EU region" is a different answer from "processed in-region, no data leaves Singapore." Get the precise answer in writing, not in a sales slide.

The second question is training data use: do you agree that your prompts, completions, and any uploaded documents will not be used to train future versions of the model? Terms differ by product and tier: business and API tiers of major AI vendors generally offer this commitment, while some consumer products may use data for training unless you opt out. Know which tier you are on and get the guarantee documented in your contract, not just referenced in a help centre article that can be updated without notice.

The third question is retention period: how long does the vendor retain your prompts and completions after each API call? Retention for 30 days for abuse monitoring is a standard and usually acceptable answer. Retention for 12 months for unspecified purposes is not. Ask the question directly and confirm the answer in your data processing agreement.

The fourth question is breach notification: what is the vendor's contractual obligation to notify you of a data breach, and within what timeframe? Well-run vendors commit to a specific, short notification window. Some do not commit to one at all. The answer tells you something about their security posture beyond the specific breach scenario.

Category two: hallucination rate on your use case.

Public benchmarks are useful for comparing models in aggregate. They are not useful for predicting performance on your specific use case. The benchmark data overlaps with training data in ways that are difficult to audit. The query distribution on a benchmark may be entirely unlike your actual query distribution. A model that ranks first on a legal reasoning benchmark may perform mediocrely on the specific type of legal document review your team does.

The only evaluation that matters is the one you conduct yourself, on your data. Before signing any contract, build a test set of at least 50 queries that represent the actual distribution of your production use case. Include straightforward queries where the right answer is unambiguous. Include ambiguous queries where a well-calibrated model should express uncertainty rather than confabulate. Include off-topic queries that the model should decline to answer. Include queries that require the model to distinguish between similar but distinct concepts in your domain.

For each query, document the correct answer, the acceptable answer range, and what an incorrect answer looks like. Run the test set against every vendor you are evaluating. Grade the outputs against your ground truth. Calculate the percentage of responses that are factually correct, the percentage that are clearly wrong, and the percentage that are plausible-sounding but subtly incorrect. That last category — the confident near-miss — is the most dangerous in production and the most underweighted in standard evaluations.

The hallucination rate that matters is the rate on your test set, not the model's published performance on a generic benchmark. If a vendor will not let you run this evaluation before signing, that is itself a significant signal about what production will look like.

Category three: latency and reliability under realistic load.

A single API call to a major AI provider typically responds in one to three seconds under light load. This number appears in demos and documentation. It is not the number that determines your user experience in production.

The number that determines your user experience is the p95 and p99 latency under your peak concurrent load. If your application makes 50 simultaneous API calls at peak — a customer support platform with 50 active conversations — the 95th-percentile response time under that load is what your users experience 5 percent of the time. For many AI APIs, p99 latency under realistic load is several times higher than the median latency shown in documentation.

Run a load test before you sign. Most major vendors allow you to run load tests against their API. Send 20 to 100 concurrent requests at your expected peak volume, with realistic prompt lengths and expected completion lengths. Measure p50, p95, and p99 latency. Measure error rate — the percentage of requests that time out or return a 5xx response. A vendor with 99.9 percent uptime but 3 percent error rate under concurrent load is not actually 99.9 percent available for your application.

Also ask about rate limits and how the vendor handles rate limit errors. Does the API return a 429 with a retry-after header that allows for graceful degradation? Does the vendor offer dedicated throughput for enterprise customers? What happens to queued requests during a provider-side incident? These questions have specific technical answers that vendors with mature infrastructure can give you directly.

Category four: fallback behaviour and graceful degradation.

Every AI system fails sometimes. The question is not whether it fails — it is what it does when it fails. An AI vendor's fallback behaviour is a window into their engineering culture and their understanding of production systems.

When the model is unavailable due to an outage, what does the API return? A clear error code with status page information is good. A timeout after 30 seconds with no status is bad. Ask the vendor for their incident response process, their SLA for partial and full outages, and their historical uptime record with root cause summaries for major incidents. A vendor who cannot produce this information is not operating with production engineering discipline.

When the model's confidence is low, does the vendor's API surface that information? Confidence scores and uncertainty signals are inconsistently implemented across AI APIs. For applications where a low-confidence answer is worse than no answer — medical information, legal advice, compliance guidance — you need a way to detect when the model is uncertain and route accordingly. If the vendor's API does not surface this information, you need to build your own confidence estimation layer on top, which is additional engineering work that should be factored into your evaluation.

When the model should decline to answer, does it decline clearly or does it produce a hedged response that looks like an answer? Test this explicitly. Give the vendor's model a set of questions that fall outside its appropriate scope and evaluate the quality of its refusals. A model that says "I am not able to help with that specific question — here is what I can help with" is more trustworthy than a model that produces a qualified non-answer that looks like engagement.

Category five: contract terms that protect you when things change.

AI vendor contracts have specific risks that standard software procurement checklists do not cover. Three terms require particular attention.

Model version stability is the most important and most frequently overlooked. AI providers update their models continuously. A model update can change output style, output structure, reasoning behaviour, and refusal patterns — often without a versioning system that allows you to pin to a specific behaviour. Ask the vendor: how much advance notice will you give before a model change that affects output behaviour? What is your model versioning policy? Can I pin my application to a specific model version, and for how long? The answers reveal how much the vendor has thought about the downstream impact of model changes on their customers' production applications.

Data deletion on termination is the second critical term. Your contract should specify that the vendor will delete your data within a defined period after contract termination — typically 30 to 90 days — and provide written confirmation that deletion has occurred. Vendors who are vague about this process are signalling that they do not have a clean data deletion capability, which is itself an information security concern.

Liability and indemnification terms are the third area. Most AI vendor contracts cap liability at fees paid in the preceding 12 months. For enterprise applications where an AI error could cause significant downstream harm — a misrouted support case that results in a customer churn, a wrong answer in a compliance context — this cap may be entirely inadequate. Negotiate the cap upward if the use case warrants it, and review your own business insurance coverage for AI-related errors.

• 01
  Build a use-case-specific test set before you evaluate any vendor. Fifty representative queries with documented correct answers will tell you more about production performance than any published benchmark. Run the same test set against every vendor you shortlist to produce a directly comparable result.
• 02
  Run a load test before you sign. Measure p95 and p99 latency under your expected peak concurrent load. Single-call latency in a demo tells you almost nothing about performance under realistic conditions.
• 03
  Get data handling commitments in the contract, not the help centre. Data residency, training use, and retention period commitments in a help centre article can be changed without notice. The same commitments in a signed data processing agreement cannot.
• 04
  Ask specifically about model version stability. The model you evaluated will not be the model you run in production in 18 months. Know the vendor's versioning policy, the notice period for breaking changes, and whether you can pin to a stable version.
• 05
  Test fallback behaviour and refusal quality explicitly. Include off-scope queries, adversarial inputs, and low-information queries in your evaluation. How the model behaves when it should not answer is as important as how it behaves when it should.
• 06
  Compare total cost of ownership, not per-token pricing. Integration cost, monitoring infrastructure, the human review layer most applications still require, and retraining or reconfiguration costs when the model changes all belong in the TCO calculation. The vendor with the cheapest tokens is rarely the cheapest deployment.

Frequently asked questions.