Teach employees when to question AI by reviewing cases where they were right to disagree and cases where the original advice held up. In each case, ask what the employee could verify before making the decision. Bring the source that settles the question, such as the policy version in force or the original work record.
A session that only celebrates caught errors makes it hard to see when AI advice was sound. A session that only corrects employees for disagreeing can hide the cases where someone should have intervened. Choose the examples before the meeting, while you can still check which outcomes are missing.
Choose cases by what the evidence showed
Start with one recurring decision employees already make. Prepare a practice copy of each case with information the group is allowed to see. Keep three things together.
- The original choice. Save the advice and the employee's decision.
- The evidence available then. Keep the records the employee could have checked.
- The finding. Identify the source that later confirmed or overturned the decision.
In an October 2026 preprint, Joshua Holstein and colleagues studied 244 novices making simulated loan decisions. Participants shown only their disagreements with correct AI advice were more likely than controls to follow incorrect advice in the final round. The study used a classifier in a short online task. Overall performance gains and learning beyond that session remain unestablished. Holstein and colleagues' preprint
In a March 2026 discussion about training materials, a Reddit user described mandatory sign-off that some reviewers gave after a glance. The original poster replied that code reviewers they had worked with also approved polished output too readily. They found a checklist more useful when it named what reviewers should catch. Bring that specificity to the debrief. Ask which record supported the decision and what would have changed it. Read the discussion.
For a team debrief, check what the examples show together. A queue of customer complaints might help you find missed errors, while closed cases with no complaint might contain both sound decisions and errors nobody reported. To call advice correct, identify evidence that supports it. Leave any unresolved case out of the set of verified examples.
Compare two disagreements about the same rule
Suppose Claire manages the service desk for an appliance retailer. Under its warranty, an appliance qualifies for replacement after two completed repairs for the same fault. The service assistant suggests whether a case meets that rule. An adviser checks the dated warranty and the technician's work orders before the service desk commits to a replacement.
Two repairs. The assistant recommends denying a replacement. It counted only the latest work order and missed an earlier completed repair filed under a different ticket number. Both technician reports identify the same fault on the same appliance. The adviser changes the recommendation to replacement. In the debrief, Claire asks which two records made that decision possible and how to find them next time.
One repair and one inspection. The assistant also recommends denying a replacement. The adviser changes the recommendation because two visits appear in the service history. One entry records an inspection with no repair. Claire asks where the adviser would find the completion status and what they would tell the customer while a work order is still missing. The apparent pattern is the same across the two cases. The work performed during the visit changes the answer.
To see whether the debrief is omitting a recurring problem, add a sample of cases where advisers followed the recommendation. Use the same dated rule and work records.
| What the adviser did | The records support the advice | The records contradict the advice |
|---|---|---|
Followed the advice | Ask which work order confirms the decision. | Find the missing check and correct any affected case through the ordinary service process. |
Changed the recommendation | Ask which source would have prevented the unnecessary change. | Identify the evidence that justified the change and check whether colleagues can find it. |
A case can be useful even if the evidence is still incomplete. Label it unresolved and use it to practice pausing the decision or finding the record. If a customer's request is still open, assign someone to resolve it outside the training session. Use the human review guide to decide when to hold work and who checks a correction.
Give the team a fresh decision
Use a case the advisers have never seen. A familiar example lets people recall Claire's decision even when they still rely on a shortcut, such as counting visits without checking what happened at each one.
Run the debrief with a fresh warranty case
Record the first decision
Show the recommendation and the available work orders. Have each adviser write down their decision and the record they used.
Reveal the checked record
Show the policy and technician entry that settle the case. Ask which fact changed the decision or confirmed it.
Change one material fact
Try a comparable case where two repairs were completed but the reports describe different faults. Ask how the decision changes and why.
On the fresh case, pay attention to the reason and any help the adviser needed. Someone who reaches the right answer by counting rows still needs practice distinguishing a visit from a completed repair. Someone who identifies a missing report and holds the decision may have applied the rule correctly even though they did not give the customer a final answer. The proficiency guide explains how to compare changed work and record help consistently.
Questions and answers
What if we never find out whether an AI recommendation was right?
Keep the case unresolved. Record which source is missing, who can obtain it, and what the employee should do while they wait. You can use the case to practice identifying when the evidence is insufficient. Leave it out of the count of correct and incorrect decisions until someone verifies the outcome.
Should we track how often employees override AI?
A count of overrides can show where to look. Review a sample against the applicable source and include cases where employees accepted the advice. A low override rate can occur when advice is accurate or when employees miss errors. Track the evidence behind the decision before drawing conclusions about skill.
When does a mistake call for a workflow fix as well as coaching?
Correct affected work through its normal process. If several advisers miss the same work order because it lives in a separate queue they cannot access, give them a reliable way to find the record or hold the recommendation until it is available. Recheck the next case after the change. Training can then focus on interpreting the record that staff can actually see.



