AI Prompt Regression Testing: The Five Cases That Catch a Model Update

Your prompt did not get worse. The name you send it to did. AI prompt regression testing is the cheapest insurance a solo builder has: five fixed inputs and one acceptance line will tell you that a model update broke something - before a customer does.

The Model Behind the Name Is Not the One You Tested

What you send is a name; what answers is a version. An API can expose an identifier with a date baked into it, pointing at exactly one version, or accept a short alias that means "the latest" - and the latest moves the day the provider ships something newer. The name is the address of a house: if the tenant changes, your letter still arrives and somebody else reads it. That is a claim about identifiers, not about how a model reasons.

Three Things That Move Without Asking You

The pointer. A dated identifier is a snapshot. A nickname is a pointer, and a pointer can be repointed without anyone asking you.

The shutdown date. Models get a deprecation notice and then an expiry. On October 14, 2026, OpenAI notified developers that gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini would leave the API on January 14, 2027 - a prompt-breaking event with a calendar date. The notice arrives by email, docs and release notes; the same page is the official deprecation index, and the listing endpoint returns an optional shutdown_date per model. Miss all three and a 404 tells you.

The sampling. The same input under the same settings can come back as different text. That is not a bug report.

"It Looks Different" Is Not Evidence

Two runs that differ are normal, not a symptom. Temperature zero reduces variation, it does not remove it; a fixed seed is a request for repeatability, not a promise, and on several reasoning models those parameters do not exist.

Compare a run against a line you already accepted, never A against B.

AI Prompt Regression Testing: Five Cases, One Acceptance Line

Five cases, not fifty, so you actually run them when something changes.

Where the Five Come From

Do not invent them, harvest them. Take the last fifteen or twenty outputs you accepted from real work and keep four that represent the job. Add one that already broke once: it fails first. Fix the success criterion before the cases that measure it, the order Anthropic's evaluation guide asks for.

One Property Per Case

Each case carries one property that must survive, markable with a check or an X:

• "names the policy and does not promise a specific date"
• "returns JSON with the four keys, and nothing outside them"
• "cites the document and does not invent a section number"

"Reads well" is not a property: two people disagree, and the same person disagrees on Tuesday. The acceptance line is written before the rerun, never after - written afterwards it reshapes itself around whatever came out. The Claude App Builder spec-to-code prompt already works that way: criteria before building, failures reported plainly.

What Stays Out of the Set

Cases you cannot judge in thirty seconds. Cases carrying customer names, where the real value gets swapped for a fictional one of the same shape. Cases whose answer depends on someone who is not you. A set that feels like homework does not get run, and one that is not run is a document.

The Five Cases, Ready to Copy

case | input | the one property that must survive | pass/fail
1 | the most common request you get | names the policy, no promised date |
2 | the request with one detail missing | asks for the detail, does not guess |
3 | the request that must be refused | refuses and offers the alternative |
4 | the structured job (JSON, table, list) | parses, four keys, no extra prose |
5 | the one that broke before | fails visibly, never plausibly |

The Rerun: Diff the Decision, Not the Wording

When the model behind the name changes, the text changes too. Do not compare sentences. Compare three things.

Three Questions, Two Minutes

1. Did the decision change? Approved versus refused, chose versus dropped. If the decision flipped, you have a regression, whatever the prose looks like.
2. Does the shape still hold? The JSON parses, the field exists, the length fits. A beautiful reply that will not parse is a broken reply.
3. Did it invent anything? A fact, a figure, a citation that was not in the input. That is the failure that dies in front of a customer.

Three "no" answers and the prompt is fine, whatever the prose says.

Fail Loud, Not Quiet

A prompt that fails quietly is worse than one that fails with an error: it hands you something plausible and lets you ship it. Mark which case must break visibly once the instructions stop binding, and grade it first.

date | model id as the API returned it | prompt file version (commit) | 5/5 or 4/5 | verdict
2026-10-18 | gpt-4o-mini-2024-07-18 | 7c1e2ab | 4/5 | case 2 guessed, one line added

A script pays off once you run the set weekly, and the bash automation script builder prompt covers the parts that matter: a dry run before spending calls, timestamped logging, and exit codes readable at a glance.

Version the Prompt Next to Its Tests

If the prompt lives in a chat window, it has no version and "what changed" has no answer. Keep it in a text file, the cases in another, and save both in the same commit.

One File, One Commit, One Line

A personal habit, not a company process. Two questions answered in ten seconds - which prompt was running when the output was good, and what is different now. Reverting then means copying the previous file back.

Pin the Snapshot, Not the Nickname

Put the full identifier into the log, not the commercial name. If the name has no date, it is a pointer. The Git workflow prompt is a fair model for the rest: plan first, name the steps, keep a path back that does not rewrite history - applied to the repository holding both files.

When a Case Fails After the Update

A failure is information, not an invitation to rewrite.

Find the Line That Stopped Binding

Before touching the text, locate which instruction stopped being obeyed: the output format, the ban on inventing, the length limit, the part that said to ask instead of assume. Change one line and rerun all five. Five out of five after a single edit tells you what fixed it; five edits at once tells you nothing.

What Not to Do

• Do not rewrite the prompt "while you are in there".
• Do not switch the model and change the prompt on the same day.
• Do not accept "it writes better now" as a verdict. That is an impression, not a result.

The Honest Limit

Some regressions are not fixable with one line: a capability is gone, a policy changed, a field no longer exists. The honest verdict is to stay on the previous version while it still answers, write the shutdown date in the log as a deadline, and start the rewrite early. Deprecated and retired are not the same state - one still answers, the other fails - and planning against the wrong one costs you a product.

The Ten-Minute Routine

Once a month, or when something looks off.

1. Read the official deprecation list for the model you use; note any shutdown date.
2. Record the exact identifier that answered today.
3. Run the five cases before changing anything - that is your baseline.
4. Change one thing, the model or the prompt. Rerun all five.
5. Write the log line. If a case failed, note which line fixed it.

If you remember one sentence: a prompt that has never been run twice is a draft.

FAQ

Do I need a testing tool? No. Five inputs in a text file and one line you mark pass or fail is enough.

Does temperature zero make it repeatable? No. It reduces variation and leaves the rest. A seed is a request, not a promise, and on several reasoning models those parameters are not accepted at all.

How often should I rerun? When the model identifier, the prompt file, or the inputs change.

Can I just ask the model whether it still follows the prompt? No. Its account of itself is not evidence. The five cases are evidence.

The Prompt Is Half the Asset

The file is the half you can read; the five cases are the half that tells you it still works. Buy a prompt and you get the first half - a structured instruction that asks for criteria before output, like the spec-to-code builder - but the second half is yours, because it depends on the work you actually do.

Write the acceptance line before the next model update, not after.