Deploying an LLM in production is straightforward. Securing it is not. Enterprise LLM deployments introduce attack surfaces that traditional security controls do not cover: prompt injection through user input, data leakage through model responses, uncontrolled tool use, and poisoned knowledge bases. This guide provides a practical checklist for CISOs and security teams looking to secure their LLM deployments, from RAG pipeline hardening to monitoring and incident response.
Table of contents
The LLM Security Checklist
Use this as a starting point for your LLM security programme. Each item is expanded in the sections below.
| Area | Control | Priority |
|---|---|---|
| RAG Pipeline | Per-user access controls on document retrieval | Critical |
| RAG Pipeline | Document ingestion validation and sanitisation | Critical |
| API Gateway | Authentication and rate limiting on LLM endpoints | Critical |
| API Gateway | Per-user and per-session token limits | High |
| I/O Validation | Input classification for malicious prompts | High |
| I/O Validation | Output filtering for PII and sensitive data | Critical |
| DLP | Prevent sensitive data entering prompts | Critical |
| DLP | Monitor and block data exfiltration via responses | High |
| Tool Use | Least privilege for function calling | Critical |
| Tool Use | Human confirmation for sensitive actions | High |
| Supply Chain | Model provenance verification | High |
| Supply Chain | Third-party plugin security review | High |
| Monitoring | Prompt and response logging | Critical |
| Monitoring | Anomaly detection on model behaviour | Medium |
Securing RAG Pipelines
RAG (retrieval-augmented generation) is how most enterprise LLM deployments ground their responses in company data. It is also one of the largest attack surfaces, categorised as LLM08 (Vector and Embedding Weaknesses) in the OWASP Top 10 for LLMs.
Access controls at the retrieval layer. The most common RAG vulnerability is missing per-user access controls. When a user asks a question, the retrieval system should only return documents that user is authorised to see. If the vector database does not enforce permissions, any user can access any document in the knowledge base through careful prompting. Implement metadata-based filtering that checks the user’s permissions before returning search results.
Document ingestion validation. Every document added to the knowledge base is a potential vector for indirect prompt injection. Scan documents for injection patterns before ingestion. Validate document sources. Implement approval workflows for new content. Consider quarantining newly ingested documents until they have been reviewed.
Chunk isolation. When documents are chunked for embedding, ensure that chunks retain their access control metadata. A chunk from a confidential document must carry the same access restrictions as the original document.
API Gateway and Access Controls
Authentication. Every LLM endpoint must require authentication. This sounds obvious, but we regularly find LLM APIs exposed without authentication during API pen tests, especially in development and staging environments that are accidentally reachable.
Rate limiting. LLM calls are expensive. Without rate limiting, an attacker can run up significant costs (wallet-draining) or degrade service for legitimate users. Implement per-user rate limits on requests, token consumption, and tool calls. Set hard ceilings on per-session and per-day usage.
Token budgets. Set maximum input and output token limits per request. This prevents resource exhaustion attacks where adversarial prompts are designed to maximise processing cost. Monitor actual token usage against the budgets and alert on anomalies.
Input and Output Validation
Input classification. Use a separate model or rule-based classifier to flag potentially malicious inputs before they reach the main LLM. This is not a complete defence against prompt injection – sophisticated attacks will bypass classifiers – but it catches the low-hanging fruit and raises the bar for attackers.
Output filtering. Treat all LLM output as untrusted. Scan responses for PII patterns (email addresses, phone numbers, national insurance numbers, credit card numbers), internal URLs, API keys, and other sensitive data patterns. Filter or redact before returning to the user. This is your last line of defence against data leakage.
Content safety. For customer-facing applications, filter model output for harmful, offensive, or off-brand content. The model’s built-in safety training is not sufficient – it can be bypassed through prompt injection. External content filtering provides an additional layer.
Data Loss Prevention for LLMs
LLMs create new data leakage vectors that traditional DLP tools do not cover:
- User input to external models. When employees paste confidential data into ChatGPT, Claude, or other external LLMs, that data leaves your control. Implement DLP policies that monitor and control data flow to external AI services.
- Training data extraction. Fine-tuned models can leak the data they were trained on. Sanitise training and fine-tuning datasets to remove PII and sensitive information before training.
- RAG data exposure. The retrieval pipeline can return documents the user should not see. Access controls at the retrieval layer (covered above) are the primary defence.
- Conversation data. Conversation logs contain user inputs that may include sensitive data. Store, encrypt, and manage conversation data with the same rigour as any other sensitive data store. Implement retention policies and automated deletion.
Tool Use and Function Calling Controls
When the LLM can call tools (databases, APIs, email systems, file systems), a successful prompt injection becomes a code execution primitive. The attacker can use the model’s tools to take actions.
- Least privilege. Grant the LLM only the tools it needs for its intended function. A customer support bot does not need database write access. A document Q&A system does not need email sending capability.
- Parameter validation. Validate all tool call parameters server-side, independently of the model. Do not trust the model’s output as safe input for your APIs. This is standard input validation applied to a new input channel.
- Human confirmation. For sensitive actions (data modification, message sending, payment processing), require explicit user confirmation before execution. Do not let the model autonomously execute high-impact operations.
- Audit trail. Log every tool call with the full context: who triggered it, what the model’s reasoning was, what parameters were passed, and what the result was. This provides the forensic trail for incident response.
Model Supply Chain Security
Model provenance. Know where your models come from. Use models from trusted providers. Verify checksums and signatures. Pin model versions in production. A model downloaded from an unverified source on Hugging Face could contain backdoors or arbitrary code execution payloads in serialised files.
Third-party plugins. Every plugin or tool integration the model can use is an extension of the attack surface. Review plugins for security issues before enabling them. Monitor for supply chain vulnerability disclosures. Apply the same vetting process you would use for any third-party software dependency.
Framework updates. Keep ML frameworks (PyTorch, TensorFlow, LangChain, LlamaIndex) and their dependencies current. Security vulnerabilities in these frameworks can be exploited through the application, just as vulnerabilities in web frameworks can.
Monitoring, Logging, and Incident Response
Log everything. Log all prompts, responses, tool calls, retrieval queries, and error conditions. This is your primary source of truth for detecting attacks, investigating incidents, and understanding model behaviour. Ensure logs are stored securely with appropriate retention.
Anomaly detection. Monitor for patterns that indicate attack attempts: sudden topic shifts, encoding in prompts, system prompt fragments in responses, unusual token consumption spikes, unexpected tool call patterns, and repeated failed injection attempts. Start with rule-based detection and progress to ML-based anomaly detection as your understanding matures.
Incident response. Extend your incident response plan to cover LLM-specific scenarios: successful prompt injection, data leakage through model responses, model abuse (wallet-draining, resource exhaustion), and knowledge base poisoning. Define playbooks for containment (disabling the model, revoking tool access) and investigation (log analysis, response reconstruction).
Testing and Validation
Security controls need testing. An untested defence is an assumed defence.
- Pre-deployment testing. Commission an LLM penetration test before launching any LLM-powered application. Test against the OWASP Top 10 for LLMs.
- Continuous automated testing. Integrate tools like Promptfoo into your CI/CD pipeline to catch prompt injection regressions when prompts, models, or tools change.
- Periodic manual testing. Annual pen testing at minimum, with retesting after significant changes to the model, system prompt, tool integrations, or knowledge base.
- AI red teaming. Periodic adversarial exercises that go beyond automated testing to probe for creative attack chains and business logic abuse.
FORTBRIDGE is a CREST-accredited penetration testing company with dedicated LLM security assessment services. We help organisations secure their LLM deployments through structured testing, architecture review, and actionable recommendations. Get in touch to discuss your LLM security posture.
Frequently Asked Questions
What are the biggest security risks in enterprise LLM deployments?
Prompt injection (enabling most other attacks), data leakage through model responses and RAG retrieval, excessive tool permissions allowing unauthorised actions, and missing access controls on the knowledge base. These are the issues we find most frequently during LLM security assessments.
How do I prevent data leakage through LLMs?
Implement output filtering for PII and sensitive data patterns. Enforce per-user access controls on RAG retrieval. Sanitise training and fine-tuning data. Monitor conversation logs for data exposure. Control employee access to external LLM services through DLP policies.
Is it safe to use open-source LLMs?
Open-source models can be safe if handled correctly. Verify model provenance and integrity. Scan serialised model files for malicious payloads. Pin versions and monitor for security advisories. The risk is not in the model being open-source – it is in skipping the supply chain verification that you would apply to any other dependency.
How often should I test my LLM deployment?
Before initial deployment, after significant changes (model updates, new tools, prompt changes, knowledge base updates), and at least annually for a full security assessment. Automated testing should run continuously as part of your CI/CD pipeline.
Do traditional security controls apply to LLM applications?
Yes. LLM applications still need authentication, authorisation, rate limiting, input validation, logging, and all the standard controls. LLM-specific controls (prompt filtering, output scanning, tool privilege management) are additional layers on top of, not replacements for, traditional application security.
About Post Author
