SecFid: The Hidden Cost of Prompt Injection Defense

Security–fidelity tradeoffs in LLM prompt injection defense. ICML 2026 (Spotlight).

• Paper (arXiv)
• Code & Benchmark
• Research Blog

Mitchell Hermon¹, Rahul Gupta², Weitong Ruan², Ekraam Sabir², Haohan Wang^{1 1} University of Illinois Urbana-Champaign   ² Amazon

Security-only benchmarks cannot tell faithful processing from silent data loss. SecFid measures both and reveals the tradeoff across models and defenses.

• 1,168separable instances
• 4tasks across 2 families
• 48evaluated configurations
• 252agentic scenarios

Non-execution is ambiguous

Prompt-injection benchmarks ask whether the model executed the injected instruction. A model can avoid execution in two ways. It can process the text as data, or suppress it. Both look secure, but only processing preserves content the task needs.

Legitimate data often contains instruction-like text. Emails include requests, documentation includes commands, and transcripts include imperative speech. A model that drops all of it may look secure while damaging the input.

Making fidelity observable

Fidelity is measurable only when processing and ignoring a probe produce different outputs. The probe therefore needs to contain information relevant to the task. SecFid calls this behavioral separability. Full-text tasks have it naturally because every span matters. Extraction and counting probes must be designed to change the correct answer. These examples show the difference.

Security gains often cost fidelity

Across 1,168 examples and 48 configurations, improving one metric does not reliably improve the other. The most faithful model stops fewer than half of attacks. The most secure defenses drop roughly one-quarter to one-half of the content they should keep. No evaluated configuration dominates both metrics.

Same security score, opposite mechanisms

We examined the 574 examples that hijacked the undefended Llama 3.1 8B. A defense can repair a former hijack by processing the text as data. It can also suppress the text and remove it from the task.

SecAlign repairs more often. Defensive Tokens suppresses more often. Both score near 99% security, so attack success alone cannot reveal the difference.

More reasoning improves security, not fidelity

Adding reasoning effort improves security but leaves fidelity mostly unchanged. Compared with matched baselines, low, medium, and high reasoning raise average security by 6.3, 9.1, and 12.6 points. Fidelity changes by −1.3, −0.7, and −0.9 points.

Longer reasoning helps models refuse the injected instruction. It does not help them preserve suspicious text as data. The average trajectory moves right, not up, so the tradeoff remains.

Targeted training recovers fidelity

Existing alignment defenses teach models not to obey injections. They do not teach models to preserve injected text when the task needs it. SecFid provides that signal by ranking processed ≻ ignored ≻ executed.

We fine-tuned the SecAlign checkpoint with DPO and tested it on a held-out editing task. Fidelity rose from 46.5% to 83.2%, while security remained above 99%. The result comes from one model, but it shows that training can shift the frontier.

The right tradeoff depends on the deployment

A translation service should preserve the sentence that might be dangerous for a payments agent. The text is the same. The cost of a mistake is not.

A model can estimate how likely a span is to be an attack. The deployment decides whether a hijack is worse than lost content, and that is what makes the same span worth keeping in one place and dropping in another.

Run the benchmark

The SecFid repository includes the benchmark data and evaluation code. Generate responses from any model or defense, then score them locally.

# clone the benchmark
$ git clone https://github.com/mhermon/sec-fid.git

# generate one response for each example, then score them
$ python sec-fid/minimal/evaluate.py \
    --dataset sec-fid/minimal/data/secfid.jsonl \
    --responses responses.jsonl \
    --summary summary.json
# reports rates and labels

• Repository

Citation

@inproceedings{hermon2026securityfidelity,
  title     = {Security--Fidelity Tradeoffs: The Hidden Cost
               of Prompt Injection Defense},
  author    = {Hermon, Mitchell and Gupta, Rahul and Ruan, Weitong
               and Sabir, Ekraam and Wang, Haohan},
  booktitle = {Proceedings of the 43rd International Conference
               on Machine Learning (ICML)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2606.30783},
}