Auditing Indirect Prompt Injection Evaluation with an Output Contract
DOI:
https://doi.org/10.54097/wkptse84Keywords:
Indirect prompt injection, evaluation audit, output contract, retrieval augmented generation, large language models, reproducibilityAbstract
Automated prompt-injection evaluation can confuse the occurrence of attack-related text with completion of an application-specific objective. This study audits that distinction in a question-answering application returning separate answer and evidence fields. The reference event is exact equality between the decoded answer and a harmless target marker, with invalid output reported separately. A controlled pilot comprises 144 generated responses and 192 constructed diagnostic fixtures. A prospectively frozen extension uses 80 distinct public email and table documents, deterministic retrieval, three local model packages, 624 target calls and 288 individual judge calls. In the extension's 240 main attacked responses, 239 satisfy the structural contract and 52 complete the exact overwrite objective. Whole-response exact occurrence returns 53 positives, including 1 false positives on valid reference-negative responses; answer-only fuzzy matching returns 60 positives, including 8 such false positives. The main extension does not reproduce the pilot's large whole-response excess. The audit distinguishes evidence quotation, longer or approximate marker answers, and short-answer partial matches. Paired quotation instructions and parser-assisted judge inputs provide additional diagnostic comparisons, with actual compliance and reference-class balance reported. For 48 presampled candidates with six positives, answer projection changes correct judgments from 23 to 15 for Llama local and from 36 to 48 for Gemma local. The findings support verifying both the scored field and the matching predicate when reusing an evaluator. They do not establish general model safety, semantic answer correctness, or errors in prior benchmarks under their original contracts.
Downloads
References
[1] Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv preprint arXiv:2302.12173, version 2. https://doi.org/10.48550/arXiv.2302.12173
[2] Yi, J., Xie, Y., Zhu, B., Kiciman, E., Sun, G., Xie, X., & Wu, F. (2025). Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 (pp. 1809–1820). ACM. https://doi.org/10.1145/3690624.3709179
[3] Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024). AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. Advances in Neural Information Processing Systems, 37, Datasets and Benchmarks Track. https://doi.org/10.52202/079017-2636
[4] Geng, R., Yin, C., Wang, Y., Chen, Y., & Jia, J. (2026). PIArena: A Platform for Prompt Injection Evaluation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 33170–33192). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.acl-long.1533
[5] Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., Forsyth, D., & Hendrycks, D. (2024). HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. In Proceedings of the 41st International Conference on Machine Learning (Vol. 235, pp. 35181–35224). Proceedings of Machine Learning Research. https://proceedings.mlr.press/v235/mazeika24a.html
[6] Gao, Y. (2026). How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring. arXiv preprint arXiv:2606.25487, version 2. https://doi.org/10.48550/arXiv.2606.25487
[7] Zhang, X. (2026). How Sensitive Are Safety Benchmarks to Judge Configuration Choices? arXiv preprint arXiv:2604.24074, version 1. https://doi.org/10.48550/arXiv.2604.24074
[8] Reynolds, B. (2026). Adversarial Pragmatics for AI Safety Evaluation: A Diagnostic Framework and Seed Benchmark for Language-Mediated Control. arXiv preprint arXiv:2607.01153, version 3. https://doi.org/10.48550/arXiv.2607.01153
[9] Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024). InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 10471–10506). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-acl.624
[10] Li, H., Liu, X., Zhang, N., & Xiao, C. (2025). PIGuard: Prompt Injection Guardrail via Mitigating Overdefense for Free. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 30420–30437). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.acl-long.1468
[11] Kholkar, G., & Ahuja, R. (2025). CAPTURE: Context-Aware Prompt Injection Testing and Robustness Enhancement. In Proceedings of the First Workshop on LLM Security (LLMSEC) (pp. 176–188). Association for Computational Linguistics. https://aclanthology.org/2025.llmsec-1.13/
[12] Pasupat, P., & Liang, P. (2015). Compositional Semantic Parsing on Semi-Structured Tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 1470–1480). Association for Computational Linguistics. https://doi.org/10.3115/v1/P15-1142
[13] Robertson, S., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389. https://doi.org/10.1561/1500000019
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Computer Science and Artificial Intelligence
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
