Playground
Hallucination Detector — Finance
Paste source + LLM extraction. Every numeric claim is cross-checked against the source; ungrounded claims are flagged.
Runs in your browser. Nothing you enter is uploaded, and no account or API key is needed.
Grounding score
57%
4 / 7 numeric claims found in the source. 3 ungrounded, 4 grounded.
Numeric claims only: currencies, percents, plain numbers of 1,000 or more, and dates. Prose, names and derived figures (growth rates, ratios) are not verified; a computed number shows as not grounded.
3. Claims in the output
shadedfound in the sourcewavynot found: check it
Claims not found in the source
- percent
18.1% - currency
$780 millionnearest in source:615000000 - percent
35%
How grounding is checked
- Numbers in the output are extracted (currencies, plain numbers ≥ 1000, percents, dates).
- Currencies and plain numbers are grounded when the source states the same value within 1%, with scale words applied on both sides ($2,850 million matches $2.85 billion).
- Percents are grounded when they match a source percent up to the output’s rounding (12% matches 12.3%).
- Dates are grounded when their year appears in the source.
- Prose-level fabrication is not detected. This pass catches the numeric class only.
How to use it
- Paste the source document (the filing excerpt or retrieval result the model was supposed to ground its answer in).
- Paste the model's output to be checked. The check runs as you type.
- Read the grounding score: the share of numeric claims (amounts, percents, numbers of 1,000 or more, dates) whose value appears in the source.
- Read the highlighted output and the list of claims not found in the source, with the nearest source number for each. Computed figures such as growth rates are flagged by design, so confirm their derivation.
- For many outputs, call the hallucination-detector engine module with each (source, output) pair and track the grounding rate across your pipeline.
The method, step by step
Detect hallucinations in finance LLM output by checking every numeric claim against the source document, verifying that each cited passage actually supports the statement it backs, validating the output against a structured schema, and routing any unsupported or mismatched claim to a human rather than letting it pass. The detection is mechanical and runs on every output; you do not eyeball it.
-
Verify every numeric claim against the source
Extract each number the model states and check it against the source text it came from. The most common finance hallucination is a transcription error: the right concept with the wrong digits, a units mix-up, or the wrong row of a table. These pass human review because the surrounding prose is correct. A per-number check against the source catches them mechanically, which is the only reliable way given how confident and fluent the wrong number looks.
Pay special attention to figures from tables. Tables are where row, column, and units errors cluster, and where a human reviewer is least likely to catch a transposed value.
-
Check citation faithfulness
For every claim the model attributes to a source, confirm the cited passage actually supports it. Models occasionally cite a real passage that does not contain the claim, which is more insidious than a missing citation because it looks rigorous. A faithfulness check compares the claim to its cited evidence and rejects answers where the evidence does not back the statement. Grounding the model in retrieval is not enough; the citation has to be verified.
An unfaithful citation is worse than no citation, because it manufactures false confidence. Treat a citation that does not support its claim as a hard failure.
-
Recompute derived figures deterministically
Do not trust ratios, totals, growth rates, or projections the model computed itself, since multi-step arithmetic errors compound. Recompute every derived figure with a deterministic engine from the verified inputs, and compare it to the model's stated value. If they disagree beyond a tiny tolerance, surface the mismatch. The model should present checked numbers, not produce them, which is the role it is actually reliable in.
Set a tight numerical tolerance and surface disagreements rather than silently overriding. A silent override can hide a real problem in the inputs you would otherwise catch.
-
Validate the output structure
When the output is structured, validate it against a schema before anything reads it: required fields present, types correct, values within sane ranges, units as expected. A figure outside a plausible range or a missing field is a cheap, mechanical signal that something went wrong. Structural validation will not catch a plausible wrong number, but it catches the malformed and the absurd at near-zero cost, before the more expensive checks run.
Add range sanity checks, not just type checks. A margin of 400 percent or a negative share count passes a type check but fails a sanity check, and that is exactly the kind of error you want to stop.
Structured Schema Validator for Finance -
Flag unsupported claims for human review
Any claim that fails a check, lacks a verifiable citation, or disagrees with the deterministic recomputation should be flagged and routed to a human rather than passed through. The goal is not zero hallucinations, which is unachievable, but zero unreviewed hallucinations reaching a decision. A pipeline that surfaces its own uncertain outputs and gates the rest is trustworthy; one that lets everything through and hopes is not.
Track the flag rate over time. A rising rate is an early warning that the model version, the prompt, or the source data changed.
Sources: Survey of Hallucination in Natural Language Generation (Ziwei Ji et al., ACM Computing Surveys (2023)) ; Evaluating Verifiability in Generative Search Engines (Liu, Zhang, Liang, EMNLP (2023))
Mistakes that undo the result
Relying on human review to catch number errors
A plausible wrong number embedded in correct prose is exactly what human reviewers miss. The fluent, confident presentation defeats spot-checking, which is why numeric verification has to be mechanical and run on every output.
Accepting a citation without checking it supports the claim
Models can cite a real passage that does not contain the stated claim. An unverified citation manufactures false confidence and is more dangerous than no citation at all.
Letting the model compute the numbers that matter
Multi-step arithmetic errors compound and are stated with full confidence. Any figure that feeds a decision must be recomputed deterministically and compared, not trusted because the model produced it.
Questions people ask
How does the detector decide a model output is hallucinated?
It runs one check: numeric grounding. Every currency amount, percent, number of 1,000 or more and date in the output is looked up in the source. Amounts match within 1% after applying scale words ($2,850 million = $2.85 billion), percents match up to the output's rounding, and dates match on the year. Anything not found is flagged, with the nearest source number shown. There is no entity check, no LLM judge and no multi-sample consistency test.
What's the false-positive rate?
It has not been benchmarked, so there is no measured rate. The known false-positive pattern is by design: figures the model computed correctly (a growth rate, a margin, a sum) are flagged because they do not appear in the source. Approximations such as 'about $4 billion' for $4.2 billion are flagged too. Review flagged claims rather than auto-rejecting them.
Does it work for non-English text?
Partly. The number patterns are language-neutral for digits, '%', $ € £ and the scale words million, billion, thousand, bn, mn and k, but they assume a period as the decimal separator and a comma for thousands. Text that writes 1.234,5 or uses words such as Milliarden will be misread, so normalize number formats first.
Can it detect 'plausible but wrong' hallucinations?
Only when the wrong part is a number the source contradicts, such as an EPS of $0.81 when the source says $0.78. It cannot catch an invented company name, a correct number attached to the wrong line item (net income reported as revenue), or a claim the source simply does not address. Pair it with field-level checks against a structured extraction for those.
How many samples does self-consistency need?
The detector does not sample the model: it checks one output against one source, deterministically. Self-consistency checks (asking the model several times and comparing answers) are a separate technique; you can run each sample through the detector and compare grounding rates, but the tool does not do that for you.
Related tools
- Playgrounds Agent Skill Tester for Markets
Paste a SKILL.md definition + sample input + your Anthropic API key. See structured extraction, token cost, and latency — all in your browser. No signup, key never leaves the page.
- Playgrounds Price-Blind Research Auditor
Paste a research prompt or agent context bundle. The auditor flags price numbers, directional words, and outcome-leaking phrases that cause LLMs to retroactively rationalize positions. Builds a price-blind research boundary.
- Playgrounds Prompt Injection Tester
Red-team a finance agent against 23 documented prompt-injection attacks — direct override, role confusion, indirect injection via retrieved content.
Articles
- 7 min read DeepSeek V4 for Finance 2026: SEC Filing Extraction Cost
DeepSeek V4 for finance 2026: V4.1-Flash reads a full 10-K for about $0.041 at $0.30/$1.20 per Mtok, half that off-peak, with a 1M-token window.
- 14 min read Long-Context Financial Question Answering: SEC Filings & RAG 2026
Long-context financial question answering over SEC filings, annual reports & earnings calls with RAG: eight LLM tasks, chunking, and 2026 benchmarks.
- 12 min read Compliance Audit Trails for LLM-Driven Trade Decisions
Schema, append-only log design, and reproducibility patterns for SEC/FINRA-compliant LLM trade audit trails — what regulators expect to see.
Use it from code
The same calculation as a JavaScript module you can import. It runs where you import it, with no request, key or rate limit.
import { compute } from "https://aifinhub.io/engines/hallucination-detector.js";