Playground
Prompt Regression Tester
Against the 65-character baseline, the 2 sample candidates drift 32.9% on average and 65.9% at worst (3-gram distance).
On the documented defaults, computed by this tool's engine · the same answer as JSON
Run one prompt across Claude, GPT and Gemini models with your own API keys and diff the outputs to catch prompt drift before it ships.
01Playground
Runs in your browser with your own Anthropic, OpenAI or Google API key, which is sent only to that provider.
Your own API keys, used only in your browser
You provide your own API keys for each target. Keys stay in your browser's React state for the session; nothing is persisted or sent anywhere except the respective provider's API.
1. Prompt
2. Targets
02Receipt
What the answer above ran on, how it is computed, where the data comes from, and a link that reopens it.
Inputs
baselineQ3 revenue rose 12% to $4.2 billion; net income was $610 million.candidates2 valuesn3
The documented defaults. Change any input above and the tool recomputes.
Method
How the hub's tools are built and checked
Engine /engines/prompt-regression-tester.js, sha256 608777967039. The page, the API and MCP run the same file.
Source
Your inputs / BYO key
Page updated .
Permalink
aifinhub.io/prompt-regression-tester/
The link carries the inputs you set, so it reopens this result.
03For agents
Any agent can run this calculator by URL or MCP and get the same answer as JSON: the inputs that used defaults, the result and a line to cite. The call below follows the inputs you set above. In code, import the same engine; it runs where you import it, with no request or key.
GET https://aifinhub.io/api/run/prompt-regression-tester curl -s 'https://aifinhub.io/api/run/prompt-regression-tester'
# Markdown instead of JSON:
curl -s 'https://aifinhub.io/api/run/prompt-regression-tester?format=md' # MCP over Streamable HTTP, no sign-in: run_calculator
curl -s https://aifinhub.io/mcp \
-H 'content-type: application/json' \
-H 'accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"run_calculator","arguments":{"calculator":"prompt-regression-tester","input":{}}}}' import { compute } from "https://aifinhub.io/engines/prompt-regression-tester.js";
const result = compute(input); {
"calculator": "prompt-regression-tester",
"title": "Prompt Regression Tester",
"site": "AI Fin Hub",
"input": {},
"defaults_used": {
"baseline": "Q3 revenue rose 12% to $4.2 billion; net income was $610 million.",
"candidates": [
{
"label": "new model",
"output": "Revenue grew 12% to $4.2B in Q3, with net income of $610M."
},
{
"label": "prompt v2",
"output": "Q3 revenue rose 12% to $4.2 billion; net income was $610 million."
}
],
"n": 3
},
"result": {
"baselineLength": 65,
"ngramN": 3,
"pairs": [
{
"label": "new model",
"drift": 0.6588235294117647,
"similarity": 0.3411764705882353
},
{
"label": "prompt v2",
"drift": 0,
"similarity": 1
}
],
"meanDrift": 0.32941176470588235,
"maxDrift": 0.6588235294117647,
"worstIndex": 0
},
"provenance": {
"run": "https://aifinhub.io/api/run/prompt-regression-tester",
"page": "https://aifinhub.io/prompt-regression-tester/",
"methodology": "https://aifinhub.io/prompt-regression-tester/",
"contract": "https://aifinhub.io/contracts/prompt-regression-tester.json",
"engine": "https://aifinhub.io/engines/prompt-regression-tester.js",
"engine_sha256": "60877796703934fec63ab86121220652149a202792604ebaee0a1ff4a3eb37c2",
"build": "1a772910d31ef1740892ed0e3a065ed71e8c7406"
},
"disclaimer": "Planning estimates only — not financial, tax, or investment advice.",
"license": "https://creativecommons.org/licenses/by/4.0/",
"cite": "AI Fin Hub, Prompt Regression Tester (engine 608777967039), https://aifinhub.io/api/run/prompt-regression-tester"
} How to use it
- Write or paste the prompt to compare. A balance-sheet summary prompt is preloaded.
- Set up targets, each a provider, model and your own API key, and enable the ones to run.
- Run all enabled targets in parallel at temperature 0 and read each output with its latency and token counts.
- Read the mean pairwise drift (1 − Jaccard similarity of 3-character sequences) and the drift matrix between every pair of successful runs.
- After a model upgrade, re-run the same prompt and read high-drift outputs by hand. The score measures surface text, not meaning.
Questions people ask
What's prompt regression?
Detecting when a prompt that previously produced good output starts producing degraded output, usually because the model behind the API changed. New model versions, even minor ones, can shift outputs subtly. A regression tester catches this before users do.
How does the tester decide outputs are equivalent?
It does not judge equivalence. It scores surface drift between two outputs as 1 − the Jaccard similarity of their 3-character sequences (0 = identical text, 1 = no overlap) and labels the mean below 0.2 tight agreement, 0.2 to 0.5 moderate divergence and above 0.5 high divergence. Schema checks and semantic similarity are not part of the score.
How do I baseline?
Run the prompt on your current model and keep that output. The page does not store runs between sessions; keep the current model as one of the targets, or send your saved output as the baseline through the agent API and compare a new output against it.
How often should I re-baseline?
After every confirmed model upgrade in production, plus quarterly if you're tracking ambient drift on a 'static' model version. Do not re-baseline reactively after a regression alert — investigate first.
What if my prompts produce non-deterministic outputs?
Every run uses temperature 0, but outputs can still differ between calls. Add the same model twice as two targets to see the noise floor before reading drift between different models.
Related tools
- Playgrounds Agent Skill Tester for Markets
Paste a SKILL.md definition + sample input + your Anthropic API key. See structured extraction, token cost, and latency — all in your browser. No signup, key never leaves the page.
- Playgrounds Prompt Injection Tester
Red-team a finance agent against 23 documented prompt-injection attacks — direct override, role confusion, indirect injection via retrieved content.
- Calculators Token-Cost Optimizer
Compute the dollar cost of a trading research loop across Claude, GPT and Gemini: prompt length × model × retries × call volume → cost per idea and per validated trade.
Articles
- 9 min read The 8-Step LLM Research Prompt Template
Free-form prompts yield uncalibrated LLM output. An 8-step template makes research reproducible and better-calibrated across model versions.
- 12 min read Compliance Audit Trails for LLM-Driven Trade Decisions
Schema, append-only log design, and reproducibility patterns for SEC/FINRA-compliant LLM trade audit trails — what regulators expect to see.