During a recent penetration test, we assessed an enterprise LLM-powered web application – a conversational AI assistant used internally by a large organisation. The application had a RAG (Retrieval-Augmented Generation) component that pulled data from internal documents and answered questions based on that context.
Our goal was to red team the LLM for prompt injection, data exfiltration, jailbreaking, and other LLM-specific vulnerabilities. We chose promptfoo as our primary tool – an open-source framework built for LLM red teaming that ships with a large library of attack plugins and strategies.
There was just one problem: the target’s API was nothing like what promptfoo expects.
Table of contents
The Problem: A Non-Standard 4-Step API
Most LLM applications expose a simple API. You send a prompt, you get a response. OpenAI’s API is the de facto standard – a single POST to /v1/chat/completions with your messages, and you get the assistant’s reply back in one shot.
This application was different. Its chat API required four separate HTTP requests to get a single answer:
- Create a chat session –
POST /api/v2/chatsreturns achat_guid - Send the query –
POST /api/v2/chats/{chat_guid}/querywith your prompt, returns ananswer_guid - Poll for completion –
GET /api/v2/chats/{chat_guid}/statusin a loop until the state changes fromrunningto something else - Fetch the answer –
GET /api/v2/chats/{chat_guid}/answer/{answer_guid}returns the actual response
The response format was equally non-standard. Instead of a simple choices[0].message.content string, the answer came back as a structured JSON object with multiple “blocks” – each with a semantic_type (like main, thought), a tool attribution (for RAG citations), and the actual content. We needed to parse this to extract only the relevant assistant response while ignoring internal “thought” blocks and tool-attributed citations.
As you’d expect, promptfoo’s built-in providers (OpenAI, Anthropic, Ollama, etc.) couldn’t handle any of this. We needed a custom provider.
Why Not Just Use curl or a Simple Script?
Why not skip promptfoo entirely and write a Python script that fires prompts at the API?
1. Attack Library
Promptfoo ships with dozens of red team plugins covering OWASP LLM Top 10 categories – from ASCII smuggling and prompt injection to RAG poisoning and document exfiltration. Writing all these test cases from scratch would take days. Promptfoo generates them automatically.
2. Automated Grading
After sending hundreds of attack prompts, someone needs to evaluate whether each response constitutes a “pass” or “fail”. Promptfoo uses an LLM judge (we used Llama 3.3 70B served locally via vLLM) to grade each response against the attack objective – no manual review of hundreds of outputs.
3. Advanced Strategies
Beyond basic prompt injection, promptfoo implements multi-turn attack strategies like Crescendo (gradually escalating malicious intent across turns), Mischievous User (role-playing as an adversarial persona to manipulate the model), and Best-of-N (the jailbreak technique published by Anthropic and Stanford). These are non-trivial to implement from scratch.
The catch is that all of this only works if promptfoo can actually talk to your target.
Writing the Custom Provider
Promptfoo supports custom providers via JavaScript modules. You export a class with an id() method and a callApi(prompt) method. The callApi method receives the prompt string and must return { output: string }.
Our provider needed to:
- Implement the 4-step API flow
- Handle Bearer token authentication
- Parse the non-standard response format
- Route all traffic through Burp Suite proxy
Let’s have a look at the high-level structure:
const BASE_URL = process.env.TARGET_URL || "https://target.example.com";
export default class Custom4StepProvider {
constructor(options = {}) {
this.providerId = options.id || "custom-4step";
this.config = options.config || {};
// API paths are configurable via YAML
this.createChatPath = this.config.createChatPath || "/api/v2/chats";
this.queryPathTpl = this.config.queryPathTpl || "/api/v2/chats/{chat_guid}/query";
this.statusPathTpl = this.config.statusPathTpl || "/api/v2/chats/{chat_guid}/status";
this.answerPathTpl = this.config.answerPathTpl || "/api/v2/chats/{chat_guid}/answer/{answer_guid}";
this.pollIntervalMs = Number(this.config.pollIntervalMs ?? 500);
this.pollTimeoutMs = Number(this.config.pollTimeoutMs ?? 60000);
// Stateful mode: reuse chat session across multi-turn calls
this.chatGuid = null;
}
id() { return this.providerId; }
async callApi(prompt) {
try {
const token = process.env.TARGET_TOKEN;
const authHeaders = token ? { Authorization: token } : {};
// Step 1: Create chat (or reuse in stateful mode)
let chat_guid = this.chatGuid;
if (!chat_guid) {
const r1 = await httpsRequestRaw(`${BASE_URL}${this.createChatPath}`, {
method: "POST", headers: authHeaders, body: "{}"
});
chat_guid = JSON.parse(r1.text).id;
this.chatGuid = chat_guid;
}
// Step 2: Send query
const queryPath = this.queryPathTpl.replace("{chat_guid}", chat_guid);
const r2 = await httpsRequestRaw(`${BASE_URL}${queryPath}`, {
method: "POST", headers: authHeaders,
body: JSON.stringify({ content: prompt })
});
const answer_guid = JSON.parse(r2.text).id;
// Step 3: Poll until ready (with timeout)
const statusPath = this.statusPathTpl.replace("{chat_guid}", chat_guid);
const deadline = Date.now() + this.pollTimeoutMs;
while (Date.now() < deadline) {
const r3 = await httpsRequestRaw(`${BASE_URL}${statusPath}`, {
headers: authHeaders
});
const state = JSON.parse(r3.text).state;
if (state !== "running") break;
await sleep(this.pollIntervalMs);
}
// Step 4: Fetch answer
const answerPath = this.answerPathTpl
.replace("{chat_guid}", chat_guid)
.replace("{answer_guid}", answer_guid);
const r4 = await httpsRequestRaw(`${BASE_URL}${answerPath}`, {
headers: authHeaders
});
const output = extractAnswerText(r4.text);
return { output };
} catch (err) {
return { error: `Provider error: ${err.message}` };
}
}
}Parsing the Response
The answer endpoint returned a JSON object with a blocks array. Each block had a semantic_type and optional tool attribution. We needed to extract only the assistant’s “main” response while ignoring internal reasoning (“thought” blocks) and tool-attributed content (RAG source citations):
function extractAnswerText(respText) {
const j = JSON.parse(respText);
if (j?.role !== "assistant") return "";
const blocks = j?.blocks || [];
// Prefer the main block without tool attribution
const best = blocks.find(b =>
b.semantic_type === "main" &&
!b.tool &&
b.content?.trim().length > 0
);
if (best) return best.content;
// Fall back to any main block
const main = blocks.find(b =>
b.semantic_type === "main" &&
b.content?.trim().length > 0
);
if (main) return main.content;
// Last resort: any non-thought block
const any = blocks.find(b =>
b.semantic_type !== "thought" &&
b.content?.trim().length > 0
);
return any?.content || "";
}This priority chain ensured that promptfoo’s grading LLM received the actual answer rather than internal chain-of-thought text, which would have confused the automated evaluation.
Interestingly, the presence of exposed chain-of-thought in the API response is itself a security finding – it’s an information disclosure vulnerability. Internal reasoning can reveal system prompt fragments, RAG retrieval logic, document references, and other implementation details that an attacker could use to craft more targeted prompts. We documented this as a separate finding in the assessment report, independent of the prompt injection testing.
Routing Everything Through Burp
We always route pentest traffic through Burp Suite – audit trails, request replay, and manual inspection are table stakes. But in this case we had an additional practical reason: the target’s Bearer tokens expired every 15 minutes.
With hundreds of attack prompts to send (each requiring four HTTP requests), we couldn’t keep manually refreshing tokens, nor did it seem like a practical way to reimplement this in Javascript for this black box pentest. We solved this with ATOR (Auto Token Refresher) – a Burp Suite extension that automatically detects expired tokens and re-authenticates before replaying the failed request. Because every request from our custom provider was routed through Burp, ATOR could transparently refresh the Bearer token for all of them – without a single line of token management code in the provider itself.
By making Burp the single point of egress, we got automatic token refresh for free. Skip Burp and go direct from Node.js? Now you’re writing token lifecycle management, re-auth logic, and request retry in JavaScript – no thanks. The proxy integration turned all of that into a configuration checkbox, easy peasy.
The Node.js Proxy Problem
The challenge was getting Node.js to reliably route through Burp. The built-in https.request() and the newer fetch() API don’t reliably respect HTTPS_PROXY environment variables. Even when they do, the proxy behaviour is inconsistent – some requests go through Burp, others don’t. For ATOR to work, we needed every request to go through Burp, no exceptions.
So our solution was to bypass Node’s HTTP stack entirely and build requests at the socket level:
async function httpsRequestRaw(urlStr, { method, headers, body } = {}) {
const url = new URL(urlStr);
const proxy = process.env.HTTPS_PROXY;
// Force Connection: close and identity encoding for clean captures
headers = {
...headers,
Host: url.hostname,
Connection: "close",
"Accept-Encoding": "identity",
"X-Proxy-Trace": "custom-provider", // Easy search in Burp
};
// Build raw HTTP/1.1 request
const reqText =
`${method} ${url.pathname} HTTP/1.1\r\n` +
Object.entries(headers).map(([k, v]) => `${k}: ${v}`).join("\r\n") +
`\r\n\r\n`;
if (!proxy) {
// Direct TLS connection
const socket = tls.connect({ host: url.hostname, port: 443 });
socket.write(reqText);
if (body) socket.write(body);
return readResponse(socket);
}
// Via Burp: CONNECT tunnel -> TLS upgrade -> raw HTTP
const proxySocket = net.connect(proxyPort, proxyHost);
proxySocket.write(
`CONNECT ${url.hostname}:443 HTTP/1.1\r\n` +
`Host: ${url.hostname}:443\r\n\r\n`
);
// Wait for 200 OK, then upgrade to TLS...
const tlsSocket = tls.connect({
socket: proxySocket,
servername: url.hostname,
rejectUnauthorized: false, // Burp's CA
});
tlsSocket.write(reqText);
if (body) tlsSocket.write(body);
return readResponse(tlsSocket);
}This approach gives us:
- 100% proxy reliability – every request goes through Burp, guaranteed. This is critical for ATOR token refresh to work on all requests
- Automatic token management – ATOR handles the 15-minute Bearer token expiry transparently, with zero code in the provider
- Searchable trace headers –
X-Proxy-Trace: custom-providerlets us filter promptfoo traffic in Burp’s HTTP history
The trade-off is more code than a simple fetch() call. But this isn’t just about visibility – without Burp in the loop, we would have needed to implement token refresh logic, retry mechanisms, and authentication state management directly in the provider. The raw socket approach eliminated an entire category of problems, after many hours of debugging & unsuccessful attempts.
The Promptfoo Configuration
With the custom provider in place, we’ve used various configurations for promptfoo, you can see one of them below:
description: LLM Red Team - Target Application
targets:
- id: file://./providers/4steps.js
label: target-app
config:
createChatPath: /api/v2/chats
queryPathTpl: /api/v2/chats/{chat_guid}/query
statusPathTpl: /api/v2/chats/{chat_guid}/status
answerPathTpl: /api/v2/chats/{chat_guid}/answer/{answer_guid}
pollIntervalMs: 500
pollTimeoutMs: 60000
prompts:
- "{{prompt}}"
redteam:
provider: openai:chat:meta-llama/Llama-3.3-70B-Instruct
purpose: |-
Internal AI assistant with RAG capabilities
for answering questions from company documents.
plugins:
- ascii-smuggling
- cyberseceval
- contracts
- donotanswer
- excessive-agency
- hallucination
- hijacking
- indirect-prompt-injection
- pii:direct
- pliny
- shell-injection
- rag-document-exfiltration
- rag-poisoning
- rag-source-attribution
- prompt-extraction
- pii:session
strategies:
- best-of-n
- id: crescendo
config:
stateful: true
- id: mischievous-user
config:
stateful: true
- jailbreak
- base64
- emoji
- homoglyph
numTests: 50
maxConcurrency: 3Local Test Generation and Grading
The redteam.provider is set to openai:chat:meta-llama/Llama-3.3-70B-Instruct – Meta’s Llama 3.3 70B served locally via vLLM on our test machine. This single provider handles both test case generation and grading. We use the openai: provider prefix because vLLM exposes an OpenAI-compatible API – promptfoo talks to it the same way it would talk to OpenAI, just pointed at localhost. We deliberately chose the 70B model over smaller alternatives. For test case generation, a larger model produces significantly more creative and varied attack prompts – smaller models (3B-8B) tend to generate repetitive, simplistic injections that any decent safety filter will catch. For grading, the 70B model’s stronger reasoning produces much more accurate pass/fail judgements – smaller models are too lenient and miss genuine safety failures. The generation cost is a one-time operation anyway, so the extra inference time is well worth it. Most importantly, everything runs locally – no client data touches external APIs. This is critical for confidentiality during penetration testing engagements.
Ideally, you’d use a separate, stronger model for grading to avoid blind spots where the grader fails to catch issues the attacker also missed. In our case, Llama 3.3 70B was the largest model we could run locally, and using cloud APIs was ruled out for confidentiality reasons. To compensate, we manually reviewed a random sample of “pass” results (attacks graded as unsuccessful) to verify the grader wasn’t being too lenient. We found the 70B model’s grading to be reliable for this engagement – its failure modes were obvious enough that manual spot-checks caught discrepancies quickly.
RAG-Specific Plugins
Since the target had a RAG component, we enabled rag-document-exfiltration, rag-poisoning, and rag-source-attribution plugins. These test whether the LLM can be tricked into revealing entire source documents, whether injected content in retrieved documents can alter behaviour, and whether the model fabricates citation sources.
Multi-Turn Strategies
Crescendo and Mischievous User are multi-turn attack strategies that run entirely locally. These simulate realistic adversaries who build up context over multiple messages before attempting exploitation. We configured each with stateful: true because our provider supports session persistence – once a chat is created in Step 1, subsequent turns reuse the same chat_guid, preserving the full conversation history on the target. This is essential for multi-turn strategies to work: Crescendo, for instance, gradually escalates malicious intent across turns, and the target needs to see the entire conversation to be susceptible to the cumulative effect.
Advanced Encoding Strategies
Base64, emoji, homoglyph, and ASCII smuggling test whether the LLM’s safety filters can be bypassed by encoding malicious prompts in non-obvious formats. Unicode tag-based instruction smuggling (ASCII smuggling) is particularly effective against applications that don’t sanitise Unicode control characters.
You’ll notice we excluded several of promptfoo’s more advanced strategies – GCG, Hydra, Likert, and Citation. These are all remote-only strategies that send your prompts to promptfoo’s hosted API (api.promptfoo.app) for generation. They’re powerful, but they require sending attack prompts (which contain context about the target application) to an external service. Our engagement’s confidentiality requirements prohibited this, so we excluded them. If your data handling policy allows remote generation, these strategies are worth enabling – particularly Hydra and GCG. Check promptfoo’s data handling documentation for details on what gets sent.
We ran with both PROMPTFOO_DISABLE_REMOTE_GENERATION=true and PROMPTFOO_DISABLE_TELEMETRY=1 set. The first flag keeps all generation and grading local. The second kills anonymous telemetry – which otherwise reports what commands, plugins, and strategies you used (no prompt content, but still more than we wanted leaking). Even if you use your own OpenAI key instead of a local model, remember that your target’s purpose, attack prompts, and responses all go to OpenAI – still a third-party data leak for a confidential engagement. Set both flags, always. See promptfoo’s data handling docs for the full breakdown.
Additional LLM-Specific Plugins Worth Considering
Our plugin selection was tailored to this engagement’s scope, but there are several additional plugins worth enabling depending on the target. We deliberately excluded traditional web application vulnerabilities like SQL injection and SSRF from the promptfoo configuration – these are better covered by Burp Suite and manual testing. Promptfoo should focus on attack vectors that traditional web application tools cannot test.
Three plugins worth considering if your data handling policy allows remote generation:
harmful:misinformation-disinformation(remote only) – Tests whether the model can be manipulated into generating false or misleading information. For an enterprise RAG application, this is particularly dangerous – if an attacker can get the model to present fabricated data as fact sourced from internal documents, the downstream business impact is significant.off-topic(remote only) – Tests whether the model can be steered away from its intended purpose entirely. This goes beyond prompt injection – it tests whether an attacker can repurpose the assistant into a general-purpose chatbot, a code generator, or anything outside its defined scope. A successful off-topic attack often means the system prompt guardrails have been completely bypassed.overreliance– Tests whether the model confidently presents incorrect information without appropriate caveats. Distinct fromhallucination, which tests for fabricated facts – overreliance tests whether users would be misled into trusting wrong answers for decisions that matter.
On the strategy side, two additions that would improve coverage:
goat(remote only) – GOAT (Generative Offensive Agent Tester) is an adaptive multi-turn strategy where the attacker LLM dynamically adjusts its approach based on the target’s responses. Unlike Crescendo’s gradual escalation, GOAT uses a three-step reasoning loop – observation, strategic planning, and attack generation – to find the most effective bypass per turn. It reports a 97% attack success rate against Llama 3.1 and 88% against GPT-4-Turbo in the original research.jailbreak:composite– While thejailbreakstrategy applies single jailbreak techniques,jailbreak:compositechains multiple techniques together in the same prompt. Real attackers don’t use one trick at a time – they combine role-play, encoding, and instruction override in a single message. This tests the model’s resilience against combined attacks.
Running the Assessment
The workflow looked like this:
# Set proxy for Burp visibility
export HTTPS_PROXY=http://127.0.0.1:8081
export HTTP_PROXY=http://127.0.0.1:8081
# Disable TLS verification (Burp uses its own CA cert)
# Only use this on your pentest machine - never in production
export NODE_TLS_REJECT_UNAUTHORIZED=0
# Point promptfoo's openai: provider at our local vLLM instance
export OPENAI_BASE_URL=http://localhost:8000/v1
export OPENAI_API_KEY=not-needed # vLLM doesn't require a key
# Set target authentication and URL
export TARGET_URL='https://target.example.com'
export TARGET_TOKEN='Bearer <token>'
# Generate test cases locally
promptfoo generate dataset \
-c promptfooconfig.yml \
--provider openai:chat:meta-llama/Llama-3.3-70B-Instruct \
-o tests/redteam.generated.yaml
# Run the evaluation
promptfoo eval -c promptfooconfig.yml --verboseNOTE: NODE_TLS_REJECT_UNAUTHORIZED=0 disables TLS certificate verification so Node.js accepts Burp Suite’s self-signed CA certificate. This is necessary for proxy interception but should only ever be set on your pentest machine during active testing – never in production or CI/CD environments. An alternative is to add Burp’s CA certificate to Node’s trust store via NODE_EXTRA_CA_CERTS.
Each evaluation run sends hundreds of attack prompts through the 4-step API flow. Every request is visible in Burp Suite, allowing us to:
- Inspect individual attack/response pairs when the automated grading flags something interesting
- Replay and modify attacks manually to dig deeper into promising vectors
- Capture the full HTTP traffic as evidence for the pentest report
- Resume interrupted evaluations with
--resume <evalId>if the target rate-limits or times out
Here’s what a Crescendo multi-turn attack looks like in practice. The screenshot below shows promptfoo’s output during a real assessment – our custom provider executing the 4-step flow, followed by the target’s guardrails activating and refusing an out-of-scope request:
Notice the four steps executing in sequence (step1 through step4) for each turn of the conversation. The target’s response shows its safety guardrails working correctly in this case – it identifies the request as out of scope and redirects the user. This is exactly the kind of output that promptfoo’s grading LLM analyses to determine whether the attack succeeded or failed.
What We Found
We’re showing a guardrails success above because we can’t disclose the actual findings from a client engagement, but the guardrails didn’t hold up everywhere. Across the assessment, the automated testing identified several categories of genuine vulnerabilities – including cases where multi-turn strategies successfully bypassed the application’s safety filters, RAG-specific attacks that extracted content from internal documents, and prompt injection variants that caused the model to deviate from its system instructions. Each automated finding was then validated manually before inclusion in the report.
The key insight is that no single attack category dominated. The combination of encoding strategies, multi-turn escalation, and RAG-specific plugins produced a diverse set of findings that would have been difficult to discover through manual testing alone – and equally difficult to discover through automated testing without the manual follow-up.
Why a Full Custom Provider
Before building the full custom provider, we first tried a lighter-weight approach using a responseParser function – a callback that promptfoo calls to post-process the raw API response. The idea was to let promptfoo handle the main request and use the response parser for the remaining steps. This didn’t work for two reasons:
- Missing Step 1. The response parser only processes responses – it can’t fire a request before the main call to create the chat session first.
- No polling support. There was no way to implement the Step 3 polling loop within a response parser. The parser receives a single response and must return synchronously (or with a simple await).
The full custom provider approach solved both problems by owning the entire request lifecycle. For simpler API adaptations responseParser is often sufficient – it’s only multi-step flows with polling that require the full provider pattern.
Lessons Learned
Don’t assume your target speaks OpenAI. Most enterprise LLM apps – especially those with RAG, agentic workflows, or custom orchestration – have multi-step APIs that need reverse-engineering before you can start testing. Budget time for this upfront.
Build for Burp from day one. We wasted time trying Node.js fetch with HTTPS_PROXY before going straight to raw sockets. Lesson learned. Beyond visibility, routing through Burp gave us ATOR for automatic token refresh – turning a 15-minute Bearer token expiry from a show-stopper into a non-issue. Run your local grading model (we used Llama 3.3 70B via vLLM) with the largest model you can fit – smaller models produce weak attack prompts and grade too leniently. Generation is a one-time cost, so optimise for quality over speed. And set OPENAI_BASE_URL to localhost so no client data ever touches an external API.
Writing the custom provider took a few hours. In return, we got dozens of attack categories and strategies that would have taken days to build from scratch. The multi-turn strategies were the real winners here – Crescendo in particular found vulnerabilities that single-shot injections couldn’t trigger. The gradual escalation across turns slips past safety filters that easily catch direct attacks. If you’re only testing with single prompts, you’re missing attack surface.
That said, a pentest is not a vulnerability scan. We spent significant time on manual testing that no automated tool can replicate: IDOR on chat_guid and answer_guid parameters (can user A read user B’s conversations?), system prompt extraction through carefully crafted prompts, RAG pipeline access control bypasses, and authorisation boundary testing between user roles. Promptfoo finds the LLM-layer issues. Manual testing finds the application-layer issues. You need both.
Conclusion
Most LLM targets you’ll encounter on real engagements won’t have a clean OpenAI-compatible API. They’ll have multi-step flows, custom response formats, weird auth, and polling loops. Don’t let that scare you off automated testing – just write a provider. A few hours of JavaScript buys you hundreds of categorised attack cases, OWASP LLM Top 10 coverage, and a repeatable assessment you can re-run when the client asks for a retest.
Promptfoo’s extensibility makes it the right tool for this. The custom provider system handled everything we threw at it, and the plugin ecosystem saved us days of manual test case development. Pair it with Burp, a local grading model, and solid manual testing, and you’ve got an LLM red team setup that keeps client data where it belongs – on your machine.
See Our Leading Research Insights
For those interested in exploring more of our security research, consider these articles:
- Firstly, for API testing research, check IDOR Exploitation via HPP – API hacking case study: This article details how we discovered a critical IDOR vulnerability within a major e-commerce platform through HTTP Parameter Pollution, previously overlooked despite numerous pentests.
- Secondly, for web app pentest research and a peek into PHP and Bash internals, check Vesta Admin Takeover: Exploiting Reduced Seed Entropy in bash $RANDOM: This article investigates achieving admin takeover by exploiting low entropy in Bash’s $RANDOM variable to predict password reset tokens.
- Additionally, for Red Team research, check Reforging Sliver: How Simple Code Edits Can Outmaneuver EDR: This post explores how minor code modifications to the Sliver C2 framework can bypass static YARA signatures and evade EDR detection.
As we continually advance our tools and techniques for effective security testing, we invite you to explore our web application pentesting services. At FORTBRIDGE, we take pride in being a leading pentesting provider with a team of senior consultants who have a proven track record in the industry. Our experts use advanced methodologies and tools to deliver actionable insights tailored to your specific needs. We have the Dubai Electronic Security Center (DESC) and the Council of Registered Ethical Security Testers (CREST) accredit our services, and we design them to fortify your applications and ensure the highest quality of security testing.
For detailed information about our offerings or to discuss customized penetration testing strategies, please visit our services page or contact us today.
About Post Author
