Evaluate an AI vendor against a defined use of its service, then follow the important claims through evidence, testing and the purchase decision. Record which product and configuration you assessed, what your team could verify, and which conditions must be resolved before the proposed use begins.
A completed questionnaire is the start of the evaluation. Its answers need interpretation by the people responsible for the work, information, technology and commercial agreement. The useful output is a recommendation someone can act on, with the remaining uncertainty visible.
This guide is for an enterprise procurement lead working with those owners. If the sourcing choice is still open, first compare buying, building and combining AI capabilities. Here, the question is whether a particular supplier's offer deserves the next commitment.
Define the offer you are evaluating
Name the service precisely enough that someone else could repeat the assessment. A demonstration using the supplier's highest tier may say little about the edition, hosting arrangement or features in your quote.
Record:
- The proposed use. Who does the work, which information may enter the service, and what result they need.
- The configuration. Product, edition, enabled features, connections, permissions and relevant model or service version.
- The commitment. What the supplier will deliver now, what your company must provide, and what remains proposed development.
Suppose Sam Taylor leads procurement at a commercial laundry. Customer service wants an assistant to summarize a customer's request to change a collection time, using approved messages and the current service schedule. A coordinator checks the summary and decides whether to change the schedule. The proposed service may read the agreed sources and draft; it may not alter bookings or send replies.
That scope gives the evaluation a boundary. An impressive demonstration of automatic rescheduling would introduce a different purchase and different approval questions.
Connect important questions to the evidence
Use your existing supplier review process, adding questions that matter for this use. Give each material question an owner who can interpret the answer. Procurement can coordinate the record; it should not silently become the authority on every technical or information risk.
An April 2026 study of vendor AI disclosures examined 39 GovAI FactSheets and interviewed 19 vendor and public-sector participants. The researchers found tensions between marketing, evaluation and dialogue. Disclosures often lacked detail needed to interpret performance claims. This qualitative, coalition-based study does not rank suppliers or establish bad faith. It supports treating a disclosure as something to investigate with the supplier.
For Sam's proposed service, a useful evidence request could look like this:
| Important question | Evidence to request | Who interprets it |
|---|---|---|
Can it produce an acceptable summary? | Relevant test results, failure examples and access to a bounded buyer-run trial | Customer-service owner and reviewers |
Where does the information go? | Data-flow description, involved providers, applicable processing terms and retention details | Information, privacy and security owners |
Does access match the agreed boundary? | Connection permissions and a demonstration with suitable restricted test accounts | Technical and information owners |
What happens after a material change? | Change-notice arrangements, supported configuration and retesting responsibilities | Service owner and commercial team |
Can the company stop or move? | Export scope, usable formats, deletion provisions and transition responsibilities | Operating, technical and commercial owners |
Keep the documents' dates and scope beside the findings. A report about a different service or an answer about a future feature leaves a question open. Ask for clarification rather than turning the existence of an attachment into a pass.
The UK Government's AI procurement guidance recommends multidisciplinary evaluation and defined performance expectations. It also recognizes that an iterative AI project will not arrive with every detail settled. For a private buyer, this is useful guidance: distinguish a prerequisite for the purchase from uncertainty a bounded trial can investigate.
Test the claims you can observe
Use the proposed configuration and approved representative inputs. Let your own reviewers judge the output against agreed requirements, while the supplier helps explain unexpected behavior.
Sam's team can rehearse a concrete case:
- Set the expected result. Prepare an approved sample message with two conflicting collection dates. Agree that the summary must preserve the conflict for the coordinator.
- Run the ordinary route. Use the intended employee account and permitted sample schedule, without a supplier administrator supplying extra context behind the scenes.
- Check the boundary. With authorized test accounts and synthetic records, verify that an account outside the employee's permitted scope cannot be retrieved. Keep the test within the agreed environment.
- Record what happened. Save the configuration, input, output, reviewer finding and correction effort. Distinguish a demonstrated result from a supplier explanation or an unresolved claim.
If the assistant chooses a date without support, the team has found a failure to investigate. If a required connection is missing, it has found a delivery dependency. Those findings call for different responses.
A buyer in a public customer-experience discussion described the difficulty of allocating time to repeated vendor trials, with fixes exposing further issues. The organization and outcomes were not independently verified. The practical constraint is recognizable: resolve obvious evidence and scope gaps before asking employees to spend weeks testing a candidate that cannot yet meet a necessary condition.
For experimental design and comparison with the existing process, use the credible AI pilot guide. A procurement rehearsal can reveal a problem; a small successful sample cannot establish reliability across all future work.
Read the scope behind an assurance
A buyer-run test can establish what happened in that test. It cannot, by itself, establish how a supplier handles every backup, support record or downstream provider. Those questions need applicable documentation, assurance and agreement review by the responsible specialists.
Consider a promise of zero data retention. In documentation checked on October 17, 2026, Anthropic distinguishes eligible products and arrangements, including exceptions. Its API feature table marks the Files API as ineligible for zero data retention and describes retention until deletion or configured expiration. A broad label cannot answer which features a proposed application uses.
Ask the supplier and your reviewers to resolve:
- Applicability. Which agreement, organization, product, feature and data category does the assurance cover?
- Exceptions. What remains stored or accessible, for what purpose, and under whose controls?
- Dependencies. Which other providers process the information, and which commitments apply along that route?
- Your configuration. What must be enabled, disabled or maintained for the proposed arrangement to meet the requirement?
Have the commercial and legal owners reconcile material promises with the applicable agreement. Do not assume that every questionnaire answer becomes a contractual obligation. Equally, an agreement cannot demonstrate that the employee's connection is configured correctly. Keep those findings separate.
Make the next commitment explicit
Write the recommendation around the purchase you can justify now. A useful record identifies the assessed configuration, evidence, unresolved findings, responsible owners and the decision they support.
| Finding | Possible next commitment |
|---|---|
Required conditions are met and the bounded use is supported | Approve that scope, with named operating and review responsibilities |
A material term, permission or delivery dependency remains unresolved | Hold the affected use until its owner resolves the condition |
Fit is uncertain but an approved limited test can answer it | Fund the bounded investigation, with a separate decision before expansion |
The supplier cannot meet an essential requirement | Decline that offer or investigate another arrangement |
Do not average an unresolved prerequisite into a favorable total score. A strong summary does not compensate for access to information the employee may not use.
For Sam, the recommendation might permit a read-only drafting trial with approved sample records while holding production customer messages pending the information owner's review of the proposed processing arrangement. It should name who closes that question and what evidence they need. Approval of the trial would not authorize automatic schedule changes.
Keep a review trigger with the record. In his October 2026 account of evaluating an AI platform, advisor Satish S Murthy described later walkthroughs changing his assessment as the unnamed product developed. His synthetic-data exercise did not validate production integration. The useful lesson is to retain what was actually tested and reconsider it when the relevant service or business conditions change.
Agree how material model, feature, provider, permission or data changes reach the service owner, what must be retested, and how work continues during review. The acceptable-use policy guide explains why approval needs a defined scope. Keep wider operating responsibilities connected to the tools and workflows guide.
Questions about evaluating AI vendors
Where should procurement start?
Start with the proposed use and exact offer: the employee task, permitted information, required output, product configuration and commitment being requested. Ask the workflow, technical, information and commercial owners to identify the requirements that could change the decision. The offer definition provides a starting record before you request evidence.
Can a successful pilot replace contract review?
No. A pilot can supply evidence about behavior under tested conditions. It cannot establish which processing, support, change or exit commitments bind the parties. Have the responsible commercial and legal owners review those terms, while the operating team checks that the proposed configuration matches them. Keep the assurance and observed-test findings separate in the purchase record.
How should we compare vendors' accuracy claims?
Ask what counts as a correct result, which inputs were tested, under which configuration and how failures were assessed. Scores from different tasks or populations may not support a comparison. Test shortlisted candidates against the same relevant acceptance criteria and retain important failure cases. Use the pilot experiment guidance to design a comparison your reviewers can interpret.
What if a supplier cannot disclose proprietary details?
Identify the purchasing question the missing detail prevents you from answering. Ask whether scoped documentation, a suitable independent assurance or a confidential specialist review can resolve it. Your team needs enough evidence to assess the intended use; it does not automatically need the supplier's source code. If an essential requirement remains unassessable, narrow or hold that use rather than recording an unsupported pass.
Does approving a vendor approve every new AI feature?
No. Approval should identify the use, information, permissions and configuration assessed. A new feature or connection can change those conditions, even under an existing supplier contract. Assign an owner to receive material changes and decide what needs review before use expands. Record those triggers with the purchase recommendation, alongside a way to continue work while the question is resolved.
