Playground
LLM Finance Error Taxonomy
12 documented LLM-on-finance failure modes (hallucinated ticker, stale price, units, currency, off-by-100, fictional source, more). Paste output, see flags.
Runs in your browser. Nothing you enter is uploaded, and no account or API key is needed.
The demo is a made-up answer with planted errors in 9 of the 12 modes.
When filled, every currency amount, percent, number of 1,000 or more and year in the answer is looked up in this text. A figure the answer computed (a growth rate, a ratio) shows as not found.
Failure modes flagged
9of 12
10 flags raised by surface patterns. Each flag is a lead for a reviewer, not a verdict: a pattern can fire on a correct sentence and miss a wrong one.
Flagged modes
Hallucinated ticker · 1 flag
Fabricated stock symbol or one that does not exist on the implied exchange.
- Low confidence: Symbol "XYZW" is used as a ticker but is not in this tool's list of 60 common US large caps and ETFs — confirm it exists on the exchange.
Fix: Constrain to a verified ticker list and reject unknown symbols.
Stale price · 1 flag
Price quoted appears to come from training data, not a live source. Phrases like 'as of my last update' or specific old-looking dates.
- High confidence: Phrase: "As of my last update"
Fix: Require live data tool calls; reject answers with 'as of my knowledge' caveats.
Ratio mistake · 1 flag
Standard financial ratio formula is mis-stated (e.g. 'P/E = price × earnings').
- High confidence: P/E stated as price × earnings (should be price / earnings)
Fix: Cross-check ratio formulas against a fixed glossary before publishing.
Currency error · 1 flag
Numbers presented in the wrong currency or without currency markers (e.g. EUR for USD).
- Medium confidence: Euro or sterling amount in an answer about a US-listed security — US listings trade in USD
Fix: Force explicit currency annotation; cross-check against ticker exchange.
Time-zone error · 1 flag
Market times reported in wrong timezone (e.g. 09:30 CET for NYSE open).
- High confidence: US cash equities open at 09:30 New York time (ET), not 09:30 CET/GMT/UTC
Fix: Convert all times to UTC before display and require timezone tags.
Off-by-100 magnitude · 1 flag
Decimals shifted: 5% reported as 0.05% or 500%.
- Medium confidence: Three-digit percentage for return/yield (likely off-by-100)
Fix: Sanity-check ranges (return |x| < 100, ratios within plausible bands).
Fictional source · 2 flags
Citation to a paper, page, or filing that doesn't exist or is fabricated.
- Low confidence: Citation "(Smith, 2024)" appears without a verifiable URL nearby.
- Low confidence: Citation "Bloomberg article" appears without a verifiable URL nearby.
Fix: Require URLs that retrieve OK and DOIs that resolve before citing.
Wrong period · 1 flag
Quarterly figures presented as annual (or vice versa) without conversion.
- Medium confidence: Q-period figures presented as annual
Fix: Tag every figure with its period; reject mixed-period aggregations.
Double-counted dividend · 1 flag
Dividend included in both total return and as a separate income line — counted twice.
- Medium confidence: Dividends added on top of a figure that feeds total return — total return already includes them
Fix: Choose one convention (price return vs total return) and stick to it across the answer.
No pattern matched for
Units error, Wrong split-adjustment, Wrong tax bracket / rate
No flag does not mean the answer is right in that respect, only that none of the patterns matched.
How to use it
- Paste a model's finance answer into the first box, or keep the demo answer, which has planted errors in 9 of the 12 modes.
- Optionally paste verified source text into the ground-truth box to check every number in the answer against it.
- Read how many of the 12 failure modes were flagged and, with ground truth, how many numeric claims were found in it.
- Open each flagged mode to see the matched evidence, its confidence and the suggested fix.
- Treat flags as leads: confirm each one, and remember that modes with no flag were only pattern-checked, not verified.
Questions people ask
What categories of LLM error does the taxonomy cover?
Twelve finance-specific failure modes: hallucinated ticker, stale price, ratio mistake, units error (percent vs basis points), currency error, time-zone error, off-by-100 magnitude, fictional source, wrong period (quarterly vs annual), wrong split adjustment, wrong tax bracket or year, and double-counted dividend. Each mode has a description, at least one detection pattern and a suggested fix.
How were the error rates measured?
The tool does not publish error rates. It is a checker, not a benchmark: it scans the text you paste with fixed patterns and lists what matched. How often a given model makes each mistake depends on your prompts, data access and tools, so measure it on your own answers.
What does the ground-truth box check?
If you paste verified text (a filing excerpt, a vendor quote, a press release), every currency amount, percent, number of 1,000 or more and year in the answer is looked up in it, using the same matcher as the hallucination detector. Figures the answer computed itself, such as a growth rate or a ratio, are not in the source and show as not found.
Does a clean result mean the answer is correct?
No. The patterns catch common surface forms, such as 'P/E = price × earnings', '09:30 GMT' for the NYSE open, or '0.5% (5 bps)'. A wrong number in a correct-looking sentence passes unless it contradicts the ground truth, and a correct sentence can still trip a pattern. Read every flag before acting on it.
How do I use the taxonomy in my own evals?
Use the twelve modes as a checklist of what to test. For each mode that matters for your task, write prompts where the mistake is likely (a recent stock split, a non-US listing, a quarterly filing), grade the answers against a reference, and run each answer through this checker or the agent endpoint as a first-pass filter. The checker gives you the structure and a cheap screen, not the grading.
Guides, formulas and worked examples
- Checklist LLM Model Risk Management Checklist
- Checklist Hallucination Guardrails for Finance LLMs
- Checklist LLM Finance Evaluation Design Checklist
- Checklist LLM Output Schema Validation Checklist
Related tools
- Playgrounds Hallucination Detector
Paste a source document + an LLM's extraction. Every numeric claim in the output is checked against the source. Client-side. Catches silent fabrication.
- Playgrounds Structured Schema Validator for Finance
Paste LLM JSON output and validate against four pre-built finance schemas — research output, trade decision, risk snapshot, peer comparison — with sanity.
- Playgrounds Price-Blind Research Auditor
Paste a research prompt or agent context bundle. The auditor flags price numbers, directional words, and outcome-leaking phrases that cause LLMs.
Articles
- 8 min read The 5 Failure Modes of LLM Trading Agents (2026)
The 5 recurring failure modes in retail LLM trading agents: price-blind leaks, numeric fabrication, prompt drift, token runaway, audit amnesia.
- 7 min read LLM Finance Error Taxonomy: The Bond-Yield Trap
A 12-mode taxonomy of LLM finance errors. Coupon-yield conflation is the textbook case, and it passes both pattern checks and a numeric ground-truth check.
- 9 min read Publishing Finance Content with LLMs in the EU
An EU publisher using LLMs sits between BaFin/WpHG, MiCA, EU AI Act, and ESMA. The defensible posture: no advice, disclose AI, publish methodology.
Use it from code
The same calculation as a JavaScript module you can import. It runs where you import it, with no request, key or rate limit.
import { compute } from "https://aifinhub.io/engines/llm-finance-error-taxonomy.js";