AI Fin Hub

Playground

Prompt Regression Tester

Against the 65-character baseline, the 2 sample candidates drift 32.9% on average and 65.9% at worst (3-gram distance).

On the documented defaults, computed by this tool's engine · the same answer as JSON

Run one prompt across Claude, GPT and Gemini models with your own API keys and diff the outputs to catch prompt drift before it ships.

01Playground

Runs in your browser with your own Anthropic, OpenAI or Google API key, which is sent only to that provider.

Education, not investment advice. Past performance does not predict future results. How we check our numbers.

Your own API keys, used only in your browser

You provide your own API keys for each target. Keys stay in your browser's React state for the session; nothing is persisted or sent anywhere except the respective provider's API.

1. Prompt

2. Targets

02Receipt

What the answer above ran on, how it is computed, where the data comes from, and a link that reopens it.

Inputs

  • baselineQ3 revenue rose 12% to $4.2 billion; net income was $610 million.
  • candidates2 values
  • n3

The documented defaults. Change any input above and the tool recomputes.

Method

How the hub's tools are built and checked

Engine /engines/prompt-regression-tester.js, sha256 608777967039. The page, the API and MCP run the same file.

Source

Your inputs / BYO key

Page updated .

Permalink

aifinhub.io/prompt-regression-tester/

The link carries the inputs you set, so it reopens this result.

03For agents

Any agent can run this calculator by URL or MCP and get the same answer as JSON: the inputs that used defaults, the result and a line to cite. The call below follows the inputs you set above. In code, import the same engine; it runs where you import it, with no request or key.

Requestthe documented defaults
GET https://aifinhub.io/api/run/prompt-regression-tester
Response200 application/json · 1.7 KB
{
  "calculator": "prompt-regression-tester",
  "title": "Prompt Regression Tester",
  "site": "AI Fin Hub",
  "input": {},
  "defaults_used": {
    "baseline": "Q3 revenue rose 12% to $4.2 billion; net income was $610 million.",
    "candidates": [
      {
        "label": "new model",
        "output": "Revenue grew 12% to $4.2B in Q3, with net income of $610M."
      },
      {
        "label": "prompt v2",
        "output": "Q3 revenue rose 12% to $4.2 billion; net income was $610 million."
      }
    ],
    "n": 3
  },
  "result": {
    "baselineLength": 65,
    "ngramN": 3,
    "pairs": [
      {
        "label": "new model",
        "drift": 0.6588235294117647,
        "similarity": 0.3411764705882353
      },
      {
        "label": "prompt v2",
        "drift": 0,
        "similarity": 1
      }
    ],
    "meanDrift": 0.32941176470588235,
    "maxDrift": 0.6588235294117647,
    "worstIndex": 0
  },
  "provenance": {
    "run": "https://aifinhub.io/api/run/prompt-regression-tester",
    "page": "https://aifinhub.io/prompt-regression-tester/",
    "methodology": "https://aifinhub.io/prompt-regression-tester/",
    "contract": "https://aifinhub.io/contracts/prompt-regression-tester.json",
    "engine": "https://aifinhub.io/engines/prompt-regression-tester.js",
    "engine_sha256": "60877796703934fec63ab86121220652149a202792604ebaee0a1ff4a3eb37c2",
    "build": "1a772910d31ef1740892ed0e3a065ed71e8c7406"
  },
  "disclaimer": "Planning estimates only — not financial, tax, or investment advice.",
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "cite": "AI Fin Hub, Prompt Regression Tester (engine 608777967039), https://aifinhub.io/api/run/prompt-regression-tester"
}
The response for the defaults, as the run URL returns it. Provenance carries the engine hash and build.

How to use it

  1. Write or paste the prompt to compare. A balance-sheet summary prompt is preloaded.
  2. Set up targets, each a provider, model and your own API key, and enable the ones to run.
  3. Run all enabled targets in parallel at temperature 0 and read each output with its latency and token counts.
  4. Read the mean pairwise drift (1 − Jaccard similarity of 3-character sequences) and the drift matrix between every pair of successful runs.
  5. After a model upgrade, re-run the same prompt and read high-drift outputs by hand. The score measures surface text, not meaning.

Questions people ask

What's prompt regression?

Detecting when a prompt that previously produced good output starts producing degraded output, usually because the model behind the API changed. New model versions, even minor ones, can shift outputs subtly. A regression tester catches this before users do.

How does the tester decide outputs are equivalent?

It does not judge equivalence. It scores surface drift between two outputs as 1 − the Jaccard similarity of their 3-character sequences (0 = identical text, 1 = no overlap) and labels the mean below 0.2 tight agreement, 0.2 to 0.5 moderate divergence and above 0.5 high divergence. Schema checks and semantic similarity are not part of the score.

How do I baseline?

Run the prompt on your current model and keep that output. The page does not store runs between sessions; keep the current model as one of the targets, or send your saved output as the baseline through the agent API and compare a new output against it.

How often should I re-baseline?

After every confirmed model upgrade in production, plus quarterly if you're tracking ambient drift on a 'static' model version. Do not re-baseline reactively after a regression alert — investigate first.

What if my prompts produce non-deterministic outputs?

Every run uses temperature 0, but outputs can still differ between calls. Add the same model twice as two targets to see the noise floor before reading drift between different models.

  • Playgrounds Agent Skill Tester for Markets

    Paste a SKILL.md definition + sample input + your Anthropic API key. See structured extraction, token cost, and latency — all in your browser. No signup, key never leaves the page.

  • Playgrounds Prompt Injection Tester

    Red-team a finance agent against 23 documented prompt-injection attacks — direct override, role confusion, indirect injection via retrieved content.

  • Calculators Token-Cost Optimizer

    Compute the dollar cost of a trading research loop across Claude, GPT and Gemini: prompt length × model × retries × call volume → cost per idea and per validated trade.

All articles