How should teams maintain an evaluation set of real work?

Maintain an evaluation set as a collection of controlled task records. Each case needs permitted inputs, the evidence needed to judge an answer, explicit acceptance conditions and a history of changes. Keep the set fixed while comparing two versions of an AI workflow, then revise it deliberately as the work changes. A better score is hard to interpret if the test changed too.

For a service owner, the aim is to remember what the workflow must still do correctly after a new instruction, source connection or model is introduced. This is one part of measuring AI value. It does not replace checking live work or investigating complaints.

Give each case a reason to be in the set

Start with one defined task. Suppose Alex Morgan leads customer service at a furniture wholesaler. An approved assistant drafts replies to business customers requesting returns of unopened standard chairs. A colleague checks the draft before sending it; the assistant cannot authorize a refund.

Alex needs cases that distinguish a straightforward request from a missing order date, an excluded product or a request governed by an older policy. Fifty paraphrases of the same easy request would leave those differences untested.

Keep the purpose of each group visible:

• Everyday work. Cases selected to understand the requests the team actually receives, with the selection method recorded.
• Known failures. Cases deliberately retained because a previous version missed a condition or gave an unsupported answer.
• Fresh comparison cases. Relevant examples kept out of instruction tuning until a planned comparison, to check whether improvements extend beyond familiar tasks.

These groups answer different questions. A collection weighted toward past failures is useful for finding regressions, but its overall failure percentage is not automatically the rate customers will experience. A fresh set also loses its unfamiliar status once the team uses its answers to tune the workflow. Record that change and replenish it when the decision requires new evidence.

Anthropic’s January 2026 evaluation guidance recommends starting from manual checks and reported failures, writing unambiguous tasks, and testing situations where a behavior should occur as well as where it should not. Its context is agent engineering. For Alex’s simpler drafting task, the transferable point is to include both a permitted return and a request that needs clarification or falls outside the stated conditions.

Choose coverage before choosing a target count. Record important gaps rather than filling the set with easy variations to reach a number.

Preserve the evidence behind the expected answer

A saved customer question and an old AI response are not enough. Someone revisiting the case must be able to establish why its expected behavior is correct.

For each admitted case, retain:

• Origin and purpose. Where it came from, when it was selected and which behavior it tests.
• Permitted input. The request and necessary attachments, with the approved transformations recorded.
• Governing evidence. The relevant order facts and policy version, including when that policy applies.
• Acceptance conditions. Required facts, prohibited promises and circumstances that should return to a person.
• Review history. Who checked the reference, unresolved disagreement, case version and current status.

For Alex, “mention the returns policy” is too vague. A useful condition is that the draft applies the policy governing this order, explains the next step accurately and makes no refund commitment. A missing order date should produce a request for the information needed to establish the applicable rule.

The output-quality guide explains how to settle unclear acceptance criteria. Do that work before treating a reference answer as authoritative. Several valid phrasings may satisfy the conditions; exact wording is necessary only where the task actually requires it.

The effort is easy to underestimate. In a September 2026 Reddit discussion, a Reddit user building a research system described having actual documents but still needing a data analyst’s help to define acceptable answers. They reported a week spent defining ten answers. The surrounding discussion distinguished production work from a student project without users. This is an unverified account, not a staffing benchmark, but it exposes a real planning question: who has time and authority to establish the reference?

NIST’s AI Risk Management Framework Playbook calls for documenting test sets, measures and testing tools, and assessing performance under conditions similar to deployment. This is voluntary risk-management guidance. A case packet helps retain that context; possession of a packet does not establish that the test is valid.

Protect the information without removing the difficulty

Use the organization’s approved data arrangements for collecting, storing and running cases. Permission to handle a customer request during service work should not be assumed to authorize every later evaluation use or destination.

For the wholesaler, contact details may be unnecessary to test policy selection. Replace them consistently across the request and attachments. Preserve the order date, product category and timing relationship that determine the answer. If the case tests whether two similar order identifiers get confused, replacing both with “order number” destroys the test.

Have someone familiar with the task check the transformed case:

• Difficulty. Does it still present the ambiguity or failure condition the case tests?
• Evidence. Can the reviewer still establish the applicable rule from the supplied material?
• Judgment. Does the transformation preserve what would make an answer acceptable?

Keep access to any sensitive original restricted, and record retention and deletion responsibilities. A historical comparison does not justify keeping information beyond its permitted use.

Synthetic cases can help when real examples cannot be used or a relevant condition is missing. NVIDIA’s December 2025 tutorial demonstrates generating synthetic clinical notes with constraints and quality filtering. It is a vendor demonstration, not evidence that generated examples reproduce your customers’ requests.

For Alex, a constructed missing-date request can test whether the draft asks for the date. It cannot establish how often customers omit it. Label synthetic cases internally, check their expected behavior with a qualified person and keep claims about real-work performance within the evidence available.

Change cases for a stated reason

Suppose the wholesaler’s approved policy permits return requests within 30 days for earlier orders. A revised policy allows 14 days for orders placed from September 1. These are the company’s rules in this example.

An August 20 order with a request 20 days later remains inside its applicable 30-day window. A September 5 order with the same interval falls outside the new 14-day window. Changing the first case’s expected answer to “outside the window” would introduce a bad reference, even though the newer policy exists.

The important fact is which orders the policy governs. Keep that fact with the case rather than replacing every old reference whenever a new policy document appears. If a behavior truly no longer exists in the supported workflow, retire its case from the current set with the reason recorded. Retain only the historical evidence that the organization is permitted to keep.

Use a change record that distinguishes these situations:

Do not rewrite a reference merely because the latest output sounds plausible. Equally, do not preserve a demonstrably wrong label to protect continuity. Correct the evidence and make the break in comparability visible.

LangSmith’s dataset documentation provides one implementation example: case additions, updates and deletions create dataset versions, and evaluations can use a specified version. The bookkeeping principle does not require that product. A controlled file collection can work if the team can identify exactly which inputs, references and rules each run used.

Compare unchanged cases before interpreting a change

An October 2026 X question captures the maintenance dilemma. Ajith Elumalai, describing work on a migration-review product, asked whether a base-model change should prompt rebuilding the trace dataset or first rerunning existing task evaluations. It was a question, not a reported successful method. For a comparison, the existing valid cases provide the continuity that a newly assembled set would lose.

Before interpreting results:

1. Identify the comparison. Record the current and proposed workflow versions, instructions, source material and evaluation conditions.
2. Run the same valid case version. Keep its evidence and acceptance rules constant, and record repeated attempts where variable output matters.
3. Inspect changes by case. Distinguish a missed requirement from an invalid reference, missing test input or a problem in how the test ran.
4. Review new coverage separately. Add recent failures and changed work through the admission process, then rerun both versions on the revised set when feasible.

Report which cases were added, changed or removed. Do not join last month’s score on one collection to this month’s score on another and call the difference an improvement. When both can be rerun under comparable conditions, compare them on the same revised set; otherwise describe the limitation.

Look beyond the aggregate. A new draft may improve ordinary replies while mishandling orders under the earlier policy. The owner needs that failure pattern, not only a rising average. Keep release-blocking conditions distinct from wording preferences, and retain the original output alongside any corrected result.

If the workflow can also change records or send messages, use the multi-step agent evaluation guide. A good return-request draft does not establish that an automated refund or notification worked.

Assign one accountable case-set owner, with service experts supplying and reviewing examples. Review on material task, policy, source or tool changes, and schedule a bounded check for quieter drift in incoming work. The appropriate interval follows the work’s rate of change and consequences. Do not wait for that appointment to capture a consequential failure.

The service-continuity guide explains where maintained cases help with a replacement or recovery decision.

Questions about maintaining evaluation sets

How many examples should an evaluation set contain?

There is no universal count for a real-work evaluation set. A small collection can reveal an omitted condition or an unclear reference, but it cannot establish that rare consequential failures are absent. More examples help only if their selection and coverage support the decision.

Name the task, important variations and unacceptable failures, then choose the evidence needed for that decision. Separate deliberately difficult cases from ordinary sampling and record uncovered conditions. Seek statistical help when a release decision depends on a precise error-rate estimate.

Can we use synthetic examples instead of customer records?

Yes, synthetic examples can test specified conditions when customer records cannot be used or a useful variation is missing. A qualified reviewer still needs to establish that the inputs are coherent and the expected behavior follows from the evidence. Generated answers do not become correct references simply because they accompany generated questions.

Keep synthetic provenance visible internally and avoid treating their frequency as customer demand. For a return request, a constructed missing order date can test clarification behavior; it does not tell you how often real requests lack dates. The information-protection section explains what the case must preserve.

Should we delete a test when its expected answer changes?

First establish why the answer changed. A wrong reference needs a reviewed correction and a new version. A changed policy needs an applicability check: an older order may still be governed by the earlier rule. A genuinely obsolete behavior can leave the current set with a documented reason and permitted historical evidence retained.

Do not delete a case because the new AI version fails it. Preserve the evidence needed to distinguish a regression from an invalid test, using the case-change decisions.

Who should own the evaluation set?

Give one named owner responsibility for case admission, versions, access, review scheduling and unresolved coverage gaps. People who understand the work should confirm the expected behavior and help judge changes. A technical maintainer can run the tests without having authority to decide the service policy.

For the furniture wholesaler, the service lead needs a policy owner’s decision about which orders the revised rule covers. Record that decision before updating references, and keep failures awaiting judgment visible rather than quietly counting them as passes.