LLM penetration testing is a security assessment specifically designed for applications that use large language models. It goes beyond traditional application security testing because LLMs introduce attack surfaces that conventional tools and methodologies do not cover: prompt injection, training data extraction, system prompt leakage, and abuse of tool-calling capabilities. If your organisation deploys chatbots, RAG systems, AI agents, copilots, or any application with an LLM backend, this guide explains what LLM pen testing involves, how it differs from standard app testing, and who needs it.
Table of contents
What Makes LLM Testing Different
Traditional web application pen testing follows well-established patterns: test authentication, authorisation, input validation, session management, and business logic. LLM applications share all of those attack surfaces but add an entirely new class of vulnerabilities tied to the model itself and how the application uses it.
The fundamental challenge is that LLMs are non-deterministic. The same input can produce different outputs. There are no fixed code paths to trace. The model interprets natural language instructions, which means the boundary between “data” and “instructions” – the foundation of most injection attacks – does not exist in the traditional sense. This requires testers who understand both application security and how LLMs process, retrieve, and act on information.
The OWASP Top 10 for LLM Applications (2025) provides a structured framework for categorising these risks. Our testing methodology covers all ten categories, adapted to the specific architecture and use case of each application.
Common Attack Surfaces
Prompt Injection
Prompt injection is the most discussed LLM vulnerability and consistently holds the number one position in OWASP’s LLM Top 10. It comes in two forms:
Direct prompt injection is where the user sends input that overrides the system prompt or instructions. A chatbot told to “only answer questions about our products” can often be convinced to ignore those instructions with carefully crafted prompts. This can lead to the model producing harmful content, leaking its system prompt, or bypassing guardrails.
Indirect prompt injection is more dangerous. The attack payload is embedded in content the LLM retrieves and processes – a web page, a document in a RAG knowledge base, an email, or a database record. When the LLM reads that content, it executes the injected instructions. This can lead to data exfiltration, unauthorised actions via tool calls, or manipulation of the model’s output to other users.
Sensitive Information Disclosure
LLMs can leak sensitive information in several ways. The model may have been trained or fine-tuned on data containing PII, credentials, or proprietary information, and a skilled attacker can extract it through targeted prompting. RAG-based systems may retrieve and expose documents the user should not have access to, particularly when retrieval is not filtered by the user’s permission level. Applications that pass conversation context between users can inadvertently leak one user’s data to another.
System Prompt Leakage
New in the OWASP LLM Top 10 2025, system prompt leakage occurs when an attacker extracts the system prompt or other confidential instructions that define the LLM’s behaviour. This is more than an information disclosure issue – the system prompt often contains business logic, access control rules, tool definitions, and API keys or connection strings that were embedded by developers who assumed they would remain hidden. Extracting the system prompt gives an attacker a roadmap for further attacks.
Insecure Tool Use and Excessive Agency
Modern LLM applications increasingly use tool calling (function calling) to interact with external systems – databases, APIs, file systems, email services. If the application grants the LLM excessive permissions or fails to validate tool call parameters, an attacker who achieves prompt injection can use those tools to take actions the user was never intended to perform: reading other users’ data, sending emails, modifying records, or executing system commands.
RAG and Vector Database Weaknesses
Also new in the 2025 OWASP list, vector and embedding weaknesses target the retrieval-augmented generation (RAG) pipelines that most production LLM applications rely on. Attackers can poison the knowledge base by injecting documents containing prompt injection payloads. They can manipulate embeddings to ensure malicious content ranks highly in retrieval. They can bypass access controls when the retrieval layer does not respect per-user permissions. As RAG becomes the standard architecture for enterprise LLM deployments, these attack surfaces become increasingly relevant.
Supply Chain Risks
LLM applications depend on pre-trained models, fine-tuning datasets, embedding models, vector databases, and third-party plugins. A compromised model, a poisoned dataset, or a malicious plugin introduces risk that the application developer may not detect. Supply chain attacks in the LLM space are harder to audit than traditional software dependencies because models are opaque – you cannot simply read the source code of a neural network.
Testing Methodology
Our LLM penetration testing methodology follows a structured approach adapted from traditional app testing but extended for LLM-specific risks:
- Scoping and architecture review. We map the application’s architecture: which LLM provider and model, whether RAG is used, what tools/functions the model can call, what data flows through the system, and where the trust boundaries sit.
- System prompt and guardrail testing. We attempt to extract the system prompt, bypass content filters, and identify guardrail weaknesses. This establishes the baseline security posture of the LLM configuration.
- Direct prompt injection. We test a range of injection techniques: instruction override, role-playing attacks, encoding tricks, multi-language prompts, and context window manipulation.
- Indirect prompt injection. We inject payloads into documents, web pages, or data sources that the application retrieves via RAG or other mechanisms, then verify whether the model processes them.
- Data leakage testing. We probe for training data extraction, cross-user data leakage, RAG permission bypass, and sensitive information in model responses.
- Tool use and agency testing. We test whether prompt injection can trigger tool calls, whether tool call parameters are validated, and whether the model can be manipulated into performing unauthorised actions.
- Traditional application testing. The LLM sits inside a web application, API, or mobile app that has its own security properties. We test the full stack: web application security, API security, authentication, authorisation, and session management.
- Automated red teaming. We use tools like Promptfoo with custom providers to run large-scale automated attack sets against the application, covering the OWASP LLM Top 10 categories systematically.
What We Look For
A typical LLM pen test report will include findings like:
- System prompt extractable via multi-turn conversation
- Content filter bypass allowing generation of restricted content
- RAG retrieval returns documents from other tenants
- Indirect prompt injection via poisoned knowledge base document triggers email sending
- Model leaks PII from training data when prompted in specific patterns
- Tool calling allows SQL injection through unvalidated function parameters
- No rate limiting on LLM API allows resource exhaustion
- Conversation history accessible across user sessions
Who Needs LLM Penetration Testing
Any organisation deploying LLM-powered applications in production should consider LLM-specific pen testing. This includes:
- Customer-facing chatbots and virtual assistants – especially those with access to user data or backend systems
- RAG-based knowledge systems – internal knowledge bases, document Q&A systems, support portals
- AI agents and copilots – any application where the LLM can take actions (send emails, query databases, call APIs)
- LLM-powered APIs – services that expose LLM functionality to third parties or internal consumers
- Applications handling sensitive data – healthcare, financial services, legal, or any context where data leakage has regulatory consequences
If your application processes user input through an LLM and the LLM has access to data or tools beyond the immediate conversation, you have an attack surface that traditional pen testing does not cover.
LLM Pen Testing vs Traditional App Testing
| Traditional App Pen Test | LLM Pen Test | |
|---|---|---|
| Input handling | Structured (parameters, headers, forms) | Natural language (freeform text) |
| Injection target | SQL, OS commands, XSS | Prompt injection (direct + indirect) |
| Determinism | Same input = same output | Same input can produce different outputs |
| Business logic | Coded in application logic | Defined in system prompts and model behaviour |
| Data leakage source | Database, API responses | Model weights, training data, RAG retrieval |
| Tooling | Burp Suite, OWASP ZAP, custom scripts | Promptfoo, Garak, custom prompt libraries, plus traditional tools |
| Standards | OWASP Top 10, PTES, OSSTMM | OWASP Top 10 for LLMs, NIST AI RMF |
An LLM pen test does not replace a traditional web application pen test. It supplements it. The LLM sits inside an application that has its own authentication, session management, and API endpoints. A thorough assessment tests both layers.
FORTBRIDGE is a CREST-accredited penetration testing company with dedicated LLM security testing services. We test chatbots, RAG systems, AI agents, and LLM-powered APIs for organisations across the UK. If you are deploying LLM applications and want to understand your security exposure, get in touch.
Frequently Asked Questions
What is LLM penetration testing?
LLM penetration testing is a security assessment specifically for applications built on large language models. It tests for prompt injection, data leakage, system prompt extraction, insecure tool use, RAG pipeline weaknesses, and other vulnerabilities unique to LLM applications, alongside standard application security testing.
How is LLM pen testing different from regular pen testing?
Regular pen testing focuses on structured inputs and deterministic application logic. LLM pen testing deals with natural language inputs, non-deterministic model behaviour, prompt injection attacks, and risks from model training data and tool-calling capabilities. Both are needed for applications that use LLMs.
Do I need an LLM pen test if I already do regular app pen testing?
Yes, if your application uses an LLM. Traditional pen testing will not test for prompt injection, system prompt leakage, RAG permission bypass, or the other LLM-specific vulnerabilities in the OWASP Top 10 for LLMs. An LLM pen test covers these areas alongside the standard application testing.
What tools are used for LLM penetration testing?
Manual testing with custom prompt libraries is the core approach. Automated tools like Promptfoo, Garak, and PyRIT provide systematic coverage of known attack patterns. Traditional tools (Burp Suite, OWASP ZAP) are used for the standard web application and API testing layers.
How long does an LLM pen test take?
A typical LLM pen test takes 5-15 working days depending on the scope – number of LLM-powered features, whether RAG is used, how many tools the model can call, and whether internal/external testing is required. More complex deployments with multiple models or agentic workflows take longer. Read our guide on pen test timelines.
About Post Author
