Playground
Fallback Chain Simulator
LLM fallback chain simulator: Monte Carlo a primary model and two fallbacks across Anthropic, OpenAI and Google for success rate, tail latency and cost.
Rates from aidevhub.io model catalog (each price verified on the vendor page), Anthropic pricing, OpenAI pricing and Google AI / Gemini pricing, checked on October 9, 2026.
Runs in your browser. Nothing you enter is uploaded, and no account or API key is needed.
1. Configure the chain
One budget for the whole chain, fallbacks included.
Success rate
97.7%
977 / 1000 requests answered inside the 3.00s deadline, p95 2.19s, p99 3.00s
p50 811ms (failed requests count at the deadline). Cost $0.012 per request on average, $12.309 across all 1,000. Answered by fallback 1: 85, by fallback 2: 3.
2. Provider utilization (share of successful trials)
Recommendation
On cost per successful call alone, Gemini 3.8 Flash beats the current primary ($0.0144 → $0.0050). Swap only if its output quality is acceptable for the task; quality is not modeled.
How the trial is simulated
for each request:
elapsed = 0
for leg in [primary, fallback1, fallback2?]:
if uniform() < rate_429: # throttled: fast, no tokens billed
elapsed += 50ms; continue
latency ~ LogNormal(median = model p50, 99th pct = your p99)
elapsed += latency; bill tokens
if elapsed <= deadline: return success
return failure # deadline passed, no time for a fallback
return failure # every leg throttledThe model p50 values are rough reference medians, not provider SLAs; set each leg's p99 from your own telemetry. Failures are independent per leg (rate limits and latency tail). Real outages are correlated and bursty, so treat the success rate as an upper bound during an incident.
How to use it
- Add up to three legs in priority order: primary, then fallback 1 and fallback 2.
- For each leg, pick the provider and model, then set its 429 (throttle) rate and p99 latency. The median latency is the model's reference value shown under the model.
- Set the request-level inputs: input and output tokens per call and the end-to-end deadline for the whole chain. Cost is derived from tokens × each model's price; you don't enter a per-call cost.
- Read the success rate, p50/p95/p99 latency, average cost per request, and how many requests each fallback answered. Results update as you type and are reproducible (fixed seed).
- Compare a 2-leg chain against a 3-leg chain; the marginal success-rate gain usually shrinks after the second fallback. Each leg is tried once — there is no retry toggle.
The method, step by step
Design a fallback chain by ordering providers from preferred to backup, setting a timeout and retry policy for each, defining which failures trigger a fallback versus a retry, and simulating rate-limit and latency failures to measure the chain's success rate, latency percentiles, and cost before it goes live. The goal is graceful degradation, not a single point of failure.
-
Order the chain by quality and cost
Decide the order of providers from first choice to last resort, balancing quality, cost, and reliability. The primary is usually your best quality-cost fit; the fallbacks are alternatives that keep the pipeline running when it fails. Be deliberate about the quality drop down the chain: a fallback model that produces noticeably worse output may need extra verification, or may be unacceptable for a high-stakes step where you would rather fail than degrade.
Decide whether a degraded answer or no answer is worse for each step. For some finance steps a wrong-but-cheap fallback is more dangerous than a clean failure.
-
Set timeouts and a retry policy
Give each provider a timeout so a hung call does not stall the whole pipeline, and define how many times to retry before moving to the next provider. Distinguish transient failures worth a brief retry from hard failures that should fall through immediately. A sensible policy retries a couple of times with backoff on a transient error, then falls back. Without timeouts, one slow provider can blow the latency budget for the entire chain.
Use exponential backoff on retries, not immediate hammering. Retrying instantly into a struggling provider often makes its problem and yours worse.
-
Handle rate limits distinctly from errors
A rate-limit response is not the same as a server error and should be handled differently. Hammering a rate-limited provider with retries deepens the limit; the right response is to back off and fall to the next provider while respecting any retry-after signal. Treating rate limits as ordinary errors is a common way to turn a brief throttle into a cascading failure across the chain, especially under the bursty load a finance agent generates.
Respect the retry-after signal when a provider sends one. Ignoring it and retrying immediately is how a short throttle becomes a long outage for your pipeline.
-
Simulate failures and measure the chain
Before relying on the chain, simulate the failures it is meant to survive: rate limits, latency spikes, and provider outages. Measure the resulting success rate, the latency at the median and the tail percentiles, the total cost, and how often degradation events occur. Simulation reveals whether the chain actually delivers the reliability you assumed, and often shows that a chain designed on paper has a worse tail latency or higher cost than expected.
Watch the tail latency, not just the average. A chain that looks fast on average can be unacceptably slow at the 99th percentile when the primary fails and the fallback engages.
-
Monitor and alert on degradation
In production, log every fallback event and alert when the chain is running on a backup, because that usually means the primary is down or throttled. A chain that silently runs on its fallback hides a real problem and may be quietly degrading quality or running up cost. Track the fraction of traffic served by each provider over time, so a creeping shift onto the fallback is visible before it becomes a quality or budget incident.
A rising share of traffic on the fallback is a signal, not a non-event. Alert on it, because it usually means the primary has a problem you need to act on.
Sources: Release It! Design and Deploy Production-Ready Software (Michael T. Nygard, Pragmatic Bookshelf (2018)) ; Rate Limits (Anthropic)
Mistakes that undo the result
Treating rate limits as ordinary errors
Retrying into a rate-limited provider deepens the throttle and can cascade. Rate limits need backoff and a respect for the retry-after signal, distinct from how transient server errors are handled.
Falling back to a model without accounting for quality drop
A backup model can produce noticeably worse output. Silently degrading to it on a high-stakes finance step can be more dangerous than failing cleanly, so the quality drop must be a deliberate, monitored decision.
Running on the fallback silently
If the chain serves traffic from a backup without alerting, a primary outage or a quality degradation goes unnoticed. The fallback engaging is exactly the event you need to know about.
Questions people ask
What's a fallback chain?
A sequence of LLM endpoints to try in order when an upstream call fails: e.g., primary Anthropic, fallback OpenAI, last-resort cache. Production agent systems use chains to maintain availability when individual providers go down or rate-limit. The simulator models cost, latency, and success rate across the chain.
How are failure rates set?
You set them. Each leg has a 429 (throttle) rate from 0 to 50% that you set to your observed provider failure rate, plus a p99 latency; the defaults are 8%, 5% and 3% for the three legs. The simulator does not auto-calibrate from status pages or published reliability targets: the numbers are yours to supply, and each leg's median latency is a rough reference value shown under the model.
What's the cost of a fallback path?
Cumulative cost of every leg that actually runs before success. In the simulator a 429 is fast (about 50ms) and bills no tokens, so if the primary fails 0.5% of the time and the fallback is 3× more expensive, your effective cost is 0.995 × primary + 0.005 × 3·primary ≈ 1.010× primary. A slow answer that misses the end-to-end deadline is still billed (the tokens were generated) and ends the request, because there is no time left for a fallback.
Should the fallback always be a different provider?
Multi-provider fallback gives correlation-of-failure protection (one provider's outage doesn't take down both). Same-provider fallback (different model tier) gives capacity-overflow protection (rate-limit on Opus, retry on Sonnet). The simulator lets you mix both strategies.
Does the simulator model retries?
No. Each leg is tried exactly once, and on failure the request falls through to the next leg — there is no retry-on-timeout or retry-on-error toggle. To approximate a retry against the same provider, add another leg pointing at the same provider and model.
Related tools
- Calculators Agent Cost Envelope Calculator
Model an LLM research loop end-to-end — steps, tool calls, convergence checks, markets per day — and see per-loop, daily, and monthly cost with cost-cap recommendations.
- Generators Trading System Blueprinter
Pick your data source, LLM, broker, storage, risk engine, and logger. Get a Mermaid architecture diagram and a copyable starter file tree — the full stack before you write code.
- Calculators Token-Cost Optimizer
Compute the dollar cost of a trading research loop across Claude, GPT and Gemini: prompt length × model × retries × call volume → cost per idea and per validated trade.
Use it from code
The same calculation as a JavaScript module you can import. It runs where you import it, with no request, key or rate limit.
import { compute } from "https://aifinhub.io/engines/fallback-chain-simulator.js";