TL;DR

RAG looks cheaper at low query volume because the cost is pay-per-call with no upfront training. Fine-tuning looks expensive because the training spend is loaded entirely up front. The break-even tells a different story. Above roughly 100,000 queries per month against a stable knowledge base, fine-tuning a smaller model beats RAG against a larger one on per-call cost, on latency, and often on retrieval quality. The break-even shifts with corpus volatility (if the corpus updates weekly, fine-tuning loses), corpus size (above 10M tokens, RAG is structurally favoured because the corpus does not fit in any practical fine-tune), and capability ceiling (some tasks need the larger base model regardless). This article works the math, identifies the four regimes (low-volume RAG, high-volume fine-tune, hybrid, and "neither, just cache"), and shows the break-even calculation against a concrete 8-million-token financial-research corpus. Run the per-call envelope through the Agent Cost Envelope Calculator and the per-call optimization through the Token-Cost Optimizer.

The two architectures, in one paragraph each

RAG. Chunk a corpus into 200–800-token passages, embed each chunk with a small embedding model, store the embeddings in a vector index. At inference time, embed the query, retrieve the top-k most similar chunks, and inject them into the prompt of a base LLM. Per-call cost: query embedding plus injected-context tokens plus generation tokens. Setup cost: one-time embedding of the corpus plus ongoing index storage. Update cost: re-embed and re-index when the corpus changes (incremental, fast).

Fine-tuning. Curate (input, output) pairs that exemplify the task on the corpus knowledge, typically 5,000–50,000 pairs. Run supervised fine-tuning against a smaller base model. At inference time, call the fine-tuned model directly with no retrieval step; the corpus knowledge is in the model's weights. Per-call cost: just generation against the smaller model, no injected context. Setup cost: data curation plus fine-tuning compute. Update cost: re-run the fine-tuning job whenever the corpus changes meaningfully (slow, expensive).

The two are not mutually exclusive. A common production pattern is to fine-tune for the format and reasoning style of the task, and use RAG to inject the specific facts at inference time. The break-even analysis below is for the pure RAG vs pure fine-tune comparison; the hybrid case is its own optimization.

The cost components

A fair break-even needs every cost line. The components, organized as one-time vs ongoing.

RAG one-time costs:

  • Corpus embedding: corpus_tokens × embedding_cost. For 8M tokens at $0.02/M (text-embedding-3-small), $160. Negligible.
  • Vector index setup: developer time, maybe $5,000 first build, free for incremental.

RAG ongoing per-call costs:

  • Query embedding: query_tokens × $0.02/M ≈ $0.0001/call. Negligible.
  • Context injection: k chunks × chunk_size × generation_input_rate. For k=8, chunk_size=500, rate=$3/M: 8 × 500 × $3/M = $0.012/call.
  • Generation: same as no-RAG generation cost.
  • Vector index storage and query: $50–500/month depending on scale.

RAG ongoing update costs:

  • Re-embed changed chunks: changed_tokens × $0.02/M. Negligible at typical update rates.

Fine-tuning one-time costs:

  • Training data curation: 50–200 engineering hours = $5,000–25,000.
  • Training compute: depends on base model. For Claude 3 Haiku via Anthropic, $25/M training tokens; 50,000 examples × 800 tokens = 40M tokens × $25/M = $1,000. For OpenAI GPT-4o-mini fine-tuning, $3/M training tokens, $120 for the same volume. For self-hosted Llama-3-70B with LoRA, $300–1,000 in cloud GPU rental.
  • Evaluation: 500–2,000 eval examples × $0.02 each = $10–40.

Fine-tuning ongoing per-call costs:

  • Generation only, against the smaller model. For Haiku-class: $0.25/M input + $1.25/M output. For a 1k input + 500 output call: $0.0009/call.

Fine-tuning ongoing update costs:

  • Periodic re-training when corpus drifts. The expensive line if the corpus is volatile.

The asymmetry is structural: RAG loads its cost into per-call inference, fine-tuning loads its cost into setup and updates. The break-even is the call volume at which the ongoing RAG bill exceeds the amortized fine-tune cost.

The break-even formula

For RAG on a Sonnet-class base versus fine-tune on a Haiku-class derived model, the per-call costs differ at:

cost_rag_per_call = injected_context_tokens × input_rate_large + generation_tokens × output_rate_large cost_ft_per_call = generation_tokens × output_rate_small (no injected context)

The break-even on monthly call volume V is:

V × (cost_rag_per_call − cost_ft_per_call) = setup_cost_ft / amortization_months + update_cost_ft / months_between_updates

Plugging the numbers above for a typical financial-research workload (k=8, 500 tokens per chunk, 1k user query, 500 generation tokens, 12-month amortization, quarterly retraining):

cost_rag_per_call = 8 × 500 × $3/M + 500 × $15/M = $0.012 + $0.0075 = $0.0195 cost_ft_per_call = 500 × $1.25/M = $0.000625

Per-call delta = $0.0188. Monthly setup amortization = $20,000 / 12 = $1,667. Quarterly retraining = $5,000 / 3 = $1,667. Total monthly = $3,334.

Break-even monthly volume = $3,334 / $0.0188 = ~177,000 calls/month.

Above 177k/month, fine-tune wins on cost. Below, RAG wins. The number is sensitive to inputs but typically lands in the 100k–300k/month range for a financial-research workload. The Agent Cost Envelope Calculator takes your specific numbers (rates, volumes, corpus size, retraining cadence) and outputs the break-even with sensitivity bands.

What the formula misses

The break-even above is the cost-only calculation. Three other factors shift the answer.

Capability ceiling. Fine-tuning a Haiku-class model gets it to match the larger model on the specific task; it does not lift the model above its base capability. If the task occasionally needs reasoning that only Sonnet-class can do, the fine-tuned Haiku will fail on those calls. The fix is the hybrid pattern: route easy calls to the fine-tuned student, hard calls to the RAG-augmented teacher. The router itself adds engineering cost, and if the easy/hard split is hard to define, the routing fails open and the savings collapse.

Corpus volatility. The break-even assumed quarterly retraining at $5k each. If the corpus updates weekly and the model has to be re-fine-tuned weekly to keep up, the retraining cost rises 13× and the break-even pushes well above 1M calls/month. For volatile corpora, RAG is the structural answer because it absorbs corpus updates at zero training cost.

Latency. A fine-tuned smaller model has lower latency than a RAG-augmented larger model, often by 2–4× at the median. If latency is a binding constraint (it is for trading), the fine-tune wins on latency at any volume, but the cost calculation only matters above the break-even.

Retrieval quality. RAG inherits the quality of the retrieval step. A poorly-indexed corpus, ambiguous chunks, or a query distribution that does not embed well can produce retrieval failures that fine-tuning sidesteps because the knowledge is in weights. The empirical comparison published in Manakul 2023's SelfCheckGPT analysis (and several follow-up evals through 2025) shows fine-tuned models match or exceed RAG on tasks with stable, narrow knowledge, while RAG dominates on tasks with broad, current, or rapidly-changing knowledge.

The four regimes

Mapping volume × volatility × corpus size to architectural choice.

Volume Volatility Corpus size Pick
< 50k/month any any RAG with prompt caching
50k–500k/month low (quarterly) < 5M tokens Fine-tune (break-even is around 150k)
50k–500k/month high (weekly) any RAG
50k–500k/month low > 10M tokens RAG (corpus does not fit fine-tune economically)
> 500k/month low < 5M tokens Fine-tune (decisive)
> 500k/month low 5–20M tokens Hybrid: fine-tune + RAG-on-residual
> 500k/month high any RAG with aggressive prompt caching

The "neither, just cache" entry above is real. For low volumes against stable corpora, the cheapest solution is usually to put the corpus in a long-context call with prompt caching, ride the cache write once per session, and skip both the embedding pipeline and the fine-tuning entirely. This works up to about 100–200k tokens of corpus (the practical ceiling of the cached prefix). Above that, RAG starts to dominate because the prompt cache stops fitting cleanly. Below that and below 50k calls/month, the engineering cost of either RAG or fine-tuning is harder to justify than the per-call cost of cached long-context.

A worked example: 8M-token financial-research corpus

To make the regimes concrete. The corpus is the last five years of 10-K and 10-Q filings for the S&P 500 financial-services tickers, plus their major analyst transcripts, about 8M tokens. The agent answers research questions against this corpus.

Scenario A: 30k queries/month, corpus updated quarterly.

RAG cost: 30k × $0.0195 = $585/month. Fine-tune cost: $20k setup, $5k retraining quarterly. Amortized over 12 months: $1,667 + $1,667 = $3,334/month. Per-call cost on the fine-tuned student: 30k × $0.000625 = $19. Total: $3,353/month.

RAG wins by $2,768/month. Below break-even, decisively.

Scenario B: 250k queries/month, corpus updated quarterly.

RAG cost: 250k × $0.0195 = $4,875/month. Fine-tune cost: $3,334/month amortized + 250k × $0.000625 = $156. Total: $3,490/month.

Fine-tune wins by $1,385/month. Above break-even.

Scenario C: 250k queries/month, corpus updated weekly.

RAG cost: 250k × $0.0195 = $4,875/month. Updates absorbed at near-zero marginal cost. Fine-tune cost: $20k setup over 12 months + 52 retraining cycles at $5k each = $1,667 + $21,667 = $23,334/month amortized. Per-call cost: $156. Total: $23,490/month.

RAG wins by $18,615/month. Volatility crushed the fine-tune economics.

Scenario D: 250k queries/month, corpus stable, hybrid pattern.

Fine-tune the student on the format and the high-frequency reasoning patterns; use RAG to inject specific filing snippets at inference. Per-call cost: 250k × ($0.000625 + $0.0085 retrieval-only) = $2,281. Plus $3,334 amortized fine-tune. Total: $5,615/month. Higher than pure fine-tune (Scenario B) but with capability ceiling raised; the hybrid handles the long-tail questions that pure fine-tune misses.

The hybrid is more expensive in absolute cost and pays back when the marginal call quality matters more than the marginal call cost. For trading specifically, the marginal call quality usually does matter, since a wrong research output that triggers a wrong trade costs more than the cost-saving on the right ones.

Storage and operational tax

The break-even formula above accounts for compute and per-call costs but glosses over operational overhead that compounds at scale.

Vector index storage and query latency. A 100k-chunk index in Pinecone, Qdrant, or pgvector runs $50–500/month at typical scales. Query latency is 20–80ms at p95 for a well-tuned index, more if the corpus is sharded across nodes. This is a real line in the latency budget; for trading pipelines the retrieval step has to fit inside the per-stage budget alongside the model call.

Embedding refresh on corpus changes. When a chunk changes, the embedding has to be recomputed and the index updated. For an 8M-token corpus updated weekly with 5% delta, that is 400k tokens × $0.02/M = $0.008/week, plus the index-update operation. Negligible in cost, non-negligible in operational complexity: the refresh has to be atomic, monitored, and tested.

Fine-tune model hosting. Provider-hosted fine-tunes (Anthropic Bedrock, OpenAI's fine-tune API, Google's Vertex AI) include hosting in the per-call price. Self-hosted fine-tunes on cloud GPU run $300–1,500/month for a continuously-served Llama-3-8B with LoRA, more for larger bases. The self-hosted economics only work above ~200k calls/month against the fine-tuned model.

Versioning and rollback. Both architectures need a story for "the new index/model is worse, roll back." For RAG, this is a versioned index with hot-swap capability. For fine-tune, this is a model-version pin with a documented rollback path. Neither is hard, both are skipped in the first deployment, and the resulting incident is the most expensive cost of either architecture.

The pragmatic accounting: add 15–25% to the calculated cost for the operational tax. The break-even shifts modestly; the four-regime table holds.

When fine-tuning is just the wrong question

A category of teams reach for fine-tuning when the right question is something else.

Output formatting issues. If the model is producing inconsistent JSON, broken markdown, or schema violations, the answer is structured outputs (Anthropic's tool-use, OpenAI's response_format=json_schema, Gemini's response_mime_type), not fine-tuning. Structured outputs solve formatting deterministically; fine-tuning solves it stochastically and reverts under sufficient prompt drift.

Tone or style issues. A few-shot prompt with 3–5 examples covers most tone targets. Fine-tuning for tone is overkill; prompt engineering plus regression testing is the correct surface.

Domain knowledge issues. RAG is the structural answer. The model needs the facts at inference time, not in its weights. Fine-tuning on a knowledge corpus famously underperforms RAG on factual recall benchmarks (Lewis 2020 and the long line of follow-up work) because the fine-tune blurs facts into a probability distribution rather than retrieving them verbatim.

Reasoning issues. Neither RAG nor fine-tuning will rescue a task that is genuinely outside the base model's reasoning capability. The right move is a larger base model, multi-step prompting (chain-of-thought, self-consistency per Wang 2022), or a different decomposition of the task.

Fine-tuning is the right tool for one specific problem: getting a smaller, cheaper, faster model to perform a narrow, stable, repetitive task at the same accuracy as a larger model. When that is the actual problem, the break-even math above tells you when it pays back. When it is not the actual problem, fine-tuning is an expensive distraction.

Connects to

References

  1. Lewis, P., Perez, E., Piktus, A., et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS 2020. The original RAG formulation and the canonical comparison against parametric memory in pretrained models.
  2. Hu, E. J., Shen, Y., Wallis, P., et al. (2021). "LoRA: Low-Rank Adaptation of Large Language Models." arXiv:2106.09685. The fine-tuning method that drives much of the 2026 fine-tuning economics on self-hosted base models.
  3. Manakul, P., Liusie, A., & Gales, M. J. F. (2023). "SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models." EMNLP 2023. Establishes the methodology for measuring factual recall against retrieved versus parametric knowledge.
  4. Wang, X., Wei, J., Schuurmans, D., et al. (2022). "Self-Consistency Improves Chain of Thought Reasoning in Language Models." arXiv:2203.11171. The reasoning-task baseline used in the "neither" regime where the bottleneck is reasoning rather than retrieval.
  5. Anthropic. "Fine-Tuning Claude on Bedrock." docs.anthropic.com, accessed April 2026. Documents the training-token pricing referenced in the break-even.
  6. OpenAI. "Fine-Tuning Guide." platform.openai.com/docs/guides/fine-tuning, accessed April 2026. Documents the GPT-4o-mini fine-tuning pricing referenced above.