A useful multi-vendor AI pilot gives shortlisted offers a planned, comparable chance to handle the work. Decide who will use each tool, which tasks they will encounter, what practice and help they will receive, and how the comparison will be interpreted before opening access. Letting people choose a favorite after an unrestricted trial can inform a buying decision, but it does not establish which offer performs better.
For an IT or workflow lead, the difficult part is often the trial schedule. Different participants bring different experience; the same participants learn as they go. A defensible pilot makes those differences visible rather than assuming an identical task description removes them.
Start with the assistant-comparison guide to define the configured offers, accepted output and whether you are comparing supplied records or complete workflows. This article addresses how to run that comparison with employees.
Choose the experience the purchase would include
Suppose Alex Morgan leads estimating at a 450-person packaging manufacturer. Estimators prepare a requirements brief from a customer's carton specification, approved drawing and order notes. The brief identifies dimensions, the grade of board requested and unresolved conflicts. A qualified estimator checks it before pricing. An assistant cannot approve a material substitution or send a binding quote.
Alex has two approved AI offers, A and B, to examine alongside the existing permitted process. Before recruiting participants, Alex needs to distinguish two purchasing questions:
- Introducing an offer. How much setup, training and help will people need before they can prepare acceptable briefs?
- Using an established offer. Once people know the supported route, how does their work compare under the tested conditions?
Both matter, but they need separate observations. Removing every learning cost could misrepresent the first purchase. Giving one tool extensive coaching while another receives only a login could misrepresent their ordinary use.
Record the licensed experience, source access and support that would actually be available. Complete the necessary supplier and information review before company material enters either trial. A pilot is not an exception to those requirements.
Decide whether the same people should try both offers
The same person can reveal useful differences between tools, but their second experience is no longer their first. An estimator who has learned to check a drawing for a conflicting dimension may carry that habit into the next offer. Repeating the exact customer request also lets them remember the answer.
Raluca Budiu's Nielsen Norman Group guidance on comparison designs explains this tradeoff in user research. Separate groups avoid one participant's direct transfer between tested interfaces. Repeated use allows comparisons within the same person, while introducing order and learning concerns. Neither approach is universally preferable.
Choose the arrangement with the person who will analyze the results:
| Arrangement | What it makes possible | What needs attention |
|---|---|---|
Different participants use A or B during the same period | Observe each offer without those participants first learning the other | Group experience, task mix, sharing between colleagues and enough independent participants |
The same participants use both in a planned sequence | Compare each person's experience across the offers | Learning, fatigue, remembered answers, period changes and the relationship between repeated observations |
Participants freely choose or switch tools | Learn preferences, switching behavior and unmet needs | Self-selection prevents a simple performance ranking from the resulting averages |
For Alex, separate groups may be preferable if the first offer teaches an enduring method for interpreting carton requests. Include relevant experience levels in the allocation plan. Assigning the most experienced estimators to A and newcomers to B would test different people as well as different offers.
The NIST guidance on randomized blocks describes accounting for important sources of variation while comparing the factor of interest. Here, an analyst might group participants by relevant experience before random allocation within those groups. That is an application of a general design principle, not a guarantee that a small team will produce a precise result.
Colleagues can still share techniques across separate groups. Record important sharing and consider whether assignment should follow teams that already work together. The credible pilot guide explains why the assignment unit matters. Many completed briefs from a few assigned teams do not create many independently assigned teams.
Make tool order and task exposure explicit
If repeated use is appropriate, avoid having everyone use A first and B second. Then B always receives people who have already practiced the task, while any later workload or source change also coincides with B.
One possible plan for Alex's analyst to examine uses opposite sequences. Allocate participants to the sequences before recorded work begins. In each period, both offers encounter comparable request types across the groups. Each participant receives fresh requests in the second period rather than reproducing a brief whose answer they know.
A proposed schedule for testing both offers
Proposed carton-brief trial. Both offers appear in each period across the groups.
Offer A first, then offer B
First period: A on fresh carton requests. Second period: B on different, comparable requests.
Offer B first, then offer A
First period: B on fresh carton requests. Second period: A on different, comparable requests.
This arrangement makes the planned exposure inspectable. It does not prove that the two request sets are equally difficult or that learning affects both sequences equally. Match important conditions, such as routine specifications and drawing conflicts, and retain the actual case records for review. Do not discard difficult requests after seeing which tool struggled.
A 2025 methodological preprint by Justin Ho and Jonathan Min challenges the assumption that counterbalancing automatically removes carryover. Effects from the first condition can persist differently into the second. The paper is about causal inference, not an AI vendor trial, but its caution applies to the claim being made here: reversing order alone cannot establish a clean product effect.
For the packaging team, a weekend break will not necessarily erase a newly learned checking method. New customer requests prevent exact-answer repetition; they do not erase that learning either. If the analyst cannot justify the repeated-use comparison, change the design or use it for limited usability findings. Do not attach a causal winner to a convenient schedule.
Adding a third vendor increases the allocation, practice and review work. Shortlist first. If the team cannot give each remaining offer a meaningful test, narrow the decision rather than spreading the same attention across more logos.
Separate practice from the recorded comparison
Let participants learn each supported route before asking them to produce the work used for the established-use comparison. Use practice requests outside the recorded case set and agree what they must be able to do, such as find the approved drawing and submit a brief for checking.
Training need not involve identical keystrokes or exactly equal minutes. One interface may require a source-selection step that another handles through an approved connection. Explain the relevant route for each, record the help required and keep the training burden in the purchase estimate.
In a July 2026 workplace discussion, a Reddit user described recent Copilot training that presented it as a personal Canva. The user, who described working in marketing, worried about brand control and about the assistant offering to take over more of a task than requested. Their employer, edition and training arrangement were not established. The account illustrates a useful pilot question: did the introduction prepare people for the specific work being tested, or create a different expectation?
For Alex, practice should teach the permitted task and its limits. It should not supply the answer to a recorded drawing conflict. Keep a request for help visible even when the participant eventually succeeds.
Before recording results:
- Check access. Confirm that the participant can reach the approved sources through the route being tested.
- Complete the agreed practice. Record important prior familiarity and any extra instruction required.
- Start fresh work. Preserve the task, sources, offer configuration and starting conditions.
- Use the agreed help route. Log intervention, retries and fallback work instead of quietly repairing one offer's results.
Keep the support package in the result
A supplier engineer may solve a source problem during the trial. That can reveal useful support, but it changes what the participant received. Record who helped, what they changed, when it happened and whether comparable help would be included after purchase.
If Alex is choosing an offer with implementation support, that support is part of the package to evaluate and price. If Alex is comparing established employee use, an engineer repeatedly rewriting instructions during recorded work needs separate treatment. Do not describe the resulting brief as something an unaided employee produced.
Keep a small conduct log alongside the output review:
- Assigned offer and sequence, actual use and any switching.
- Request type, source version and whether the participant had seen the answer before.
- Practice and help received, including supplier or colleague intervention.
- Abandoned attempts, manual completions and missing outcomes.
- Material changes to access, configuration, source records or the supported route.
These records explain a result that a satisfaction score cannot. Preference remains useful: ask why someone wanted to switch and which work they could not complete. Preserve the total effort through accepted work, including checking and repair, alongside the quality findings.
Allow the pilot to end without a winner
Agree in advance what would make the comparison inconclusive. Examples include unequal access that persisted through the trial, an important unplanned change to one offer, missing difficult cases or a result that depends on unresolved learning between periods. Some defects can be corrected and the affected comparison repeated. Others limit what the available evidence can establish.
A bounded conclusion can still help Alex. One offer may fail a required drawing check. Both may meet the requirements, while the difference in completion effort remains too uncertain to justify switching. Participants may prefer different routes for reasons the next test should examine.
Use the stop, extend or scale guide for a further commitment. An extension should answer a named unresolved question with an improved design. More weeks of the same compromised comparison will not necessarily resolve it.
Before inviting participants, put one proposed request through the allocation and support plan. Ask who sees it, what they already know, which help they can receive and what happens if they abandon it. Fix the first condition that would make the resulting comparison misleading.
Questions about multi-vendor AI pilots
Should every participant test every AI tool?
No. Repeated use lets you compare experiences within a person, but it can transfer knowledge and remembered answers into later conditions. Separate groups may fit better when learning persists. Choose the arrangement with an analyst, considering task sharing, experience and the size of the available population. Start with the comparison options rather than assuming everyone needs every license.
Does reversing the order make a pilot fair?
It helps prevent one offer always occupying the same position, but it does not guarantee an unbiased comparison. Learning, fatigue, source changes and unequal carryover can remain. Use fresh comparable tasks, retain sequence and period records, and have the analysis reflect the design. If those conditions cannot support the intended conclusion, narrow the finding or change the comparison.
Should vendors receive exactly the same training time?
Give participants appropriate preparation for the supported task and record how much preparation each offer requires. Equal minutes may leave one group unable to use its route; unlimited unrecorded help can conceal a purchase cost. Separate onboarding burden from the established-use result. Define the necessary capabilities and help boundary before recorded work begins.
Can employees choose whichever tool they prefer?
Yes, when the question concerns preference or how people work with choice. Record switching and the reasons behind it. People who choose different tools may also have different tasks, experience and motivation, so their average results do not establish which tool caused better performance. Keep that exploratory finding separate from a planned allocation comparison.
How long should a multi-vendor pilot run?
Allow time for the agreed practice, representative work and required outcome follow-up. Duration depends on the decision, task frequency, variation and useful precision. There is no universal number of weeks or participants that makes a vendor comparison conclusive. Have an analyst check the sample and analysis before committing employee time, including the burden of testing additional offers.
What if neither vendor clearly wins?
Report the accepted work, important failures and remaining uncertainty. A tie or inconclusive difference can support retaining the approved incumbent, narrowing the use case or running a specific follow-up. It is not proof that the products are equivalent. Identify what evidence would change the decision before paying for an extension or a switch.
