AI Evaluations and Safe Model Upgrades

Compare a candidate AI model against what your task really needs, and switch only when the evidence supports it.

Content date: October 15, 2026.

Plain English

An evaluation is a set of test inputs you can run again, along with the behavior you expect and the rules for scoring it. AI answers vary, so one impressive example is weak evidence. Ordinary app tests check fixed, predictable results. Model evaluations check behavior such as answer quality, refusals, use of sources, and choice of tools. You need both.

Use this when

Run one before you change a model, prompt, retrieval setting, tool, or reasoning effort, and again when repeated failures show a missed case. Providers retire models on a schedule, so a forced upgrade needs the same test.

If you are just starting

Start with a small, versioned set of made-up examples. Include ordinary cases, edge cases, and tricky ones meant to cause trouble. Before you look at the candidate's answers, decide what counts as an unacceptable failure and how you will score. If you can't pay for access, score recorded outputs from the instructor, label them honestly, and don't require anyone to buy anything.

When your project grows

Compare the current model and the candidate under the same conditions. Repeat cases that vary, and report how many samples you ran and how uncertain the result is. Weight failures by severity, and track speed, cost per successful task, and new regressions. Roll out to a small group, monitor it, pin the exact model version, and set a rollback trigger. A model judging its own work isn't a safe test until you've checked its scores against human ratings. Note the end date of whatever you pin: OpenAI lists a shutdown date per model, and Anthropic promises at least 60 days' notice for public models. OpenAI's hosted Evals tool closes on November 30, 2026, so keep your test cases in your own files.

Worked example

A candidate model answers a clinic's scheduling questions faster, but in one test it quotes another patient's appointment. Its average score goes up, yet the privacy boundary is a release gate, not part of the average. The team holds the upgrade while they investigate both the app's boundary and the model's behavior. The faster average doesn't cancel the failure.

Practice

Write a model upgrade decision

  1. Create at least twelve different test cases. This is a teaching-sized set, not a statistically sufficient production sample.
  2. Before running any model, write down the required behaviors, the failures that disqualify a model, and your scoring rubric.
  3. Evaluate two configurations you can access, or clearly labeled recorded outputs, and repeat cases whose answers vary.
  4. Read the disagreements yourself, and save every output with its configuration version.
  5. Write an upgrade decision that states limits, rollout scope, monitoring, and rollback rules.

Deliverable: A versioned evaluation set, your raw comparison results, and a decision to hold, roll out, or reject, with reasons.

Common mistakes to watch for

  • A team demos five flattering examples and calls the new model better.
  • A change swaps the model, prompt, and retrieval at once, so nobody knows which caused the new mistakes.
  • A better average hides one case where the model leaks another customer's data.
  • Twelve passing examples get treated as proof of reliability.

Prompting

A weak prompt

The new model sounds better. Switch production to it.

A better prompt

Compare [the candidate model] with my current setup using a versioned evaluation set and pass rules I wrote before testing. Include ordinary, missing-evidence, adversarial, and unauthorized-action cases. Report raw results, sample size, how much repeated runs vary, speed, retries, and cost per successful task. Keep tests that code can check separate from model-graded ones. Recommend hold, limited rollout, or reject, and give a rollback trigger. Don't decide from a benchmark's reputation alone.

A model upgrade becomes a decision backed by evidence, with known tradeoffs and a way back.

Check before you move on

  • Your test cases come from real tasks and failures.
  • You wrote the scoring rubric before running models.
  • Serious failures are reported on their own.
  • You recorded the model, prompt, and retrieval settings for each run.
  • Cost and speed are measured or marked unavailable.
  • You wrote rollback criteria.

You are done when

  • Someone else could repeat your comparison and inspect the raw results.
  • Your decision states the limits of the sample and protects every critical boundary.

Questions to think about

A candidate model raises your average score but fails your customer-isolation test. Should it ship?

No. A required security rule can't be traded for a better average. Treat it as a gate that must pass, separate from quality preferences.

Why record the prompt and retrieval settings along with the model name?

All three shape the answers. Without them you can't reproduce a result or tell which change caused a regression.

Your twelve-case pilot set passes. Does that show the system is reliable in production?

No. It shows how the system behaved on those cases. Add cases from real use and failures, repeat the ones that vary, and watch the rollout.

Official references