Prompt Regression Tester

Detecting when a prompt that previously produced good output starts producing degraded output, usually because the model behind the API changed. New model versions, even minor ones, can shift outputs subtly. A regression tester catches this before users do.

Three checks: (1) string similarity (Levenshtein for short outputs, semantic embedding cosine for long), (2) schema match (does the output still validate against the structured-output spec?), (3) downstream-metric stability (does the agent's eventual answer change?). Any failure flags a regression.

Run your test suite against the current model and store outputs as the golden set. Re-run on schedule (or on every API update notice) and diff. The tool tracks deltas over time so you can see drift trajectory, not just point-in-time changes.

After every confirmed model upgrade in production, plus quarterly if you're tracking ambient drift on a 'static' model version. Do not re-baseline reactively after a regression alert — investigate first.

Use temperature=0 for the regression tests (the tool does by default). For inherently variable outputs (creative tasks), run N samples and check distribution overlap rather than exact match. The methodology page documents both modes.