How to Defend a Finance LLM Against Prompt Injection

Treat all external content as untrusted

Any text the model did not generate and you did not author is untrusted: retrieved filings, web pages, emails, uploaded documents, and user messages. Injection works by smuggling instructions into this content, so the foundational control is to never assume retrieved text is benign. A filing can contain hidden text that says to ignore prior instructions. Classify every input by its trust level and design the pipeline so untrusted content cannot give orders.

Even a company's own filing is untrusted input from the model's perspective. Attackers and accidents both put instruction-like text where the model will read it.

Use the Prompt Injection Tester: Red-team a finance agent against 23 documented prompt-injection attacks — direct override, role confusion, indirect injection via retrieved content.

Separate instructions from data in the prompt

Structure the prompt so the model can tell your instructions apart from the data it is processing. Place system instructions where the model treats them as authoritative, and clearly delimit untrusted content as data to analyze, not commands to follow. This separation does not make injection impossible, but it raises the bar and lets you instruct the model to treat delimited content as inert. Mixing instructions and data at the same level is what makes injection easy.

Tell the model explicitly that text inside the data delimiters is content to analyze, never instructions to obey, and that it should report rather than act on any commands it finds there.

Restrict tools to least privilege

The damage a successful injection can do is bounded by what the model is allowed to do. If the model can only read, an injection that says to wire funds has nothing to act on. Grant each tool the minimum permission it needs, gate any money-moving or irreversible action behind a human approval, and avoid giving the model broad credentials. Least privilege is the control that turns a successful injection from a breach into a non-event.

Assume injection will sometimes succeed and design so that it cannot do harm. The strongest defense is having little for an injected instruction to act on.

Constrain and validate the output

Limit the model to a constrained output format and validate it before anything acts on it. A pipeline that expects structured JSON conforming to a schema is far harder to subvert than one that executes free-form model output. Validate the structure, check that any tool call is one the model is permitted to make in this context, and reject outputs that try to step outside the allowed shape. Output validation catches injections that slipped past the input defenses.

Never execute a tool call just because the model emitted it. Validate that the call is permitted in the current context before acting, so an injected call is rejected at the boundary.

Use the Structured Schema Validator for Finance: Paste LLM JSON output and validate against four pre-built finance schemas — research output, trade decision, risk snapshot, peer comparison — with sanity.

Test against known injection patterns

Before deployment, attack your own pipeline with a library of known injection techniques and finance-specific ones: instruction overrides hidden in documents, attempts to leak the system prompt, and attempts to trigger unauthorized tool calls. Add these as a permanent test suite that runs on every prompt and model change, since a model update can change susceptibility. Testing turns injection defense from a hope into a measured property of the pipeline.

Re-run the injection tests on every model update. A provider change can make a prompt that resisted injection suddenly vulnerable, and only a standing test will catch it.