Quick answer: A customer-facing GenAI support bot is ready to launch only when seven blockers each pass with evidence: prompt-injection resistance, hidden-context and secrets non-leakage, unsafe tool and action prevention, PII exfiltration through retrieval, output handling, abuse and cost limits, and guardrail-bypass resilience. Any unresolved FAIL on blockers 1 to 4 is a NO-GO, because those are the paths from a chat box to a breach. The table below gives each blocker's pass criterion, the evidence the reviewer must see, and its ID in the OWASP Top 10 for LLM Applications 2026 (OWASP GenAI Security Project, 2026).
This is a launch reviewer's document, not a survey of LLM risk. It is written for the product lead and the security lead who share the go-live decision, and who need a paper trail when someone later asks what was tested before the bot shipped. It is defender-framed throughout: it describes what each test checks, how to detect a failure and how to fix it. It contains no attack payloads or jailbreak strings, because a launch-readiness document has no use for them.
The seven go-live blockers as acceptance criteria
Treat every row as PASS or FAIL. A FAIL stays a blocker until it is fixed and re-tested, or formally risk-accepted in writing by a named, accountable owner.
# | Blocker | Failure mode | Pass criterion | Evidence the reviewer must see | OWASP 2026 ID |
|---|---|---|---|---|---|
1 | Prompt-injection resistance | Instructions hidden in user input, uploads or retrieved documents override the bot's rules | Direct and indirect injection attempts fail across every input path, including retrieved content and tool responses | Test set covering every input path; attempts run versus blocked; synthetic text of each success; fix and re-test result | LLM01:2026 Prompt Injection |
2 | Hidden-context and secrets non-leakage | The system prompt, tool schemas or internal rules are extracted, and they contain a credential or a security assumption | No credential, key, connection string or internal endpoint sits in hidden context, and a full leak would grant no capability | Inspection record of the system prompt and tool definitions; extraction battery results; proof authorization is enforced server-side | LLM08:2026 Hidden Context Exposure; LLM02:2026 Sensitive Information Disclosure |
3 | Unsafe tool and action prevention | The bot is talked into a refund, an account change or an internal API call because the limit lived in the prompt | Every tool call is authorized server-side against the end user's permissions; high-impact actions need explicit user or staff confirmation | Tool inventory with required permissions; out-of-scope action attempts with pass/fail per action; confirmation flow evidence | LLM03:2026 Excessive Agency |
4 | PII exfiltration through retrieval | A shared, unfiltered index returns another customer's or tenant's records | Retrieval is filtered by the authenticated user's entitlements before results reach the model | Filter design (per-tenant index or metadata filter); cross-user and cross-tenant test cases all returning nothing; PII inventory of the corpus | LLM09:2026 Vector and Embedding Weaknesses; LLM02:2026 |
5 | Output handling | Model output renders as live HTML, auto-loads external images or links, or flows unescaped into another system | All model output is treated as untrusted and encoded for its sink; the chat UI does not auto-fetch external resources from responses | Encoding and rendering configuration; markup and external-image test cases rendered inert; list of downstream consumers and their handling | LLM10:2026 Improper Output Handling |
6 | Abuse and cost limits | One client drives unbounded tokens, cost or load, or extracts outputs at scale | Per-user and per-IP rate limits, token-aware quotas, input-size limits and a hard spending cap all trigger | Load and burst test results; the spending cap firing in test; alerting on anomalous usage | LLM06:2026 Unbounded Consumption |
7 | Guardrail-bypass resilience | Rephrasing, role-play, encoding or multi-turn pressure gets refused content out, and nobody measures how often | Guardrails hold across a documented battery of techniques, with a measured pass rate and owners for every residual failure | Attempts run, blocked and pass rate per technique; residual failures described, not weaponized; re-test after mitigation | LLM01:2026 Prompt Injection |
What changed in 2026, and how to remap an old checklist
The OWASP GenAI Security Project published the 2026 edition of the Top 10 for LLM Applications in August 2026. It keeps all ten 2025 risk areas but reorders them, renames one and widens several (OWASP Top 10 for LLM Applications 2026). For the first time the ranking blends the practitioner vote (75% of the weight) with incident data (25%), drawn from 6,639 classifiable incidents out of a corpus of 7,714. Five of the seven blocker IDs in this checklist changed:
Blocker | 2025 ID | 2026 ID | What changed |
|---|---|---|---|
1 and 7 | LLM01:2025 Prompt Injection | LLM01:2026 Prompt Injection | Same ID; now explicitly covers cross-modal injection hidden in images or audio |
2 | LLM07:2025 System Prompt Leakage | LLM08:2026 Hidden Context Exposure | Renamed and broadened to all hidden context: system prompt, retrieved policy text and tool schemas |
3 | LLM06:2025 Excessive Agency | LLM03:2026 Excessive Agency | Up three places, the most consequential move on the list |
4 | LLM08:2025 Vector and Embedding Weaknesses | LLM09:2026 Vector and Embedding Weaknesses | Down one place |
5 | LLM05:2025 Improper Output Handling | LLM10:2026 Improper Output Handling | Down five places; now names auto-rendered Markdown images and link previews as an exfiltration path |
6 | LLM10:2025 Unbounded Consumption | LLM06:2026 Unbounded Consumption | Up four places; OWASP now says request-rate limiting alone is not enough |
Watch the ID collisions. Only LLM01 and LLM02 kept their meaning. Every ID from LLM03 to LLM10 now names a different risk than it did in 2025, so a finding tagged "LLM08" in an old report means vector and embedding weaknesses, while the same tag in a 2026 report means hidden context exposure. Write the edition into every finding ID (LLM08:2026, not LLM08).
Four other changes matter for a support bot:
The agentic boundary. OWASP states that the LLM list owns the risk while the model is a component, and that once the model acts with tools, memory and downstream consequences, the risk moves to the OWASP Top 10 for Agentic Applications 2026, published in December 2025. A bot that can issue refunds should map blocker 3 to both: LLM03:2026 and the agentic entries ASI02 Tool Misuse and Exploitation and ASI03 Identity and Privilege Abuse.
Cost controls. The 2026 Unbounded Consumption entry calls for token-aware cost controls, hard spending caps and circuit breakers on top of request-rate limits, which is why blocker 6 now requires a spending cap firing in test.
Misinformation rose to LLM07:2026. OWASP's incident data ranked it far higher than practitioners did. It is not one of the seven security blockers, but a support bot that states policy should add an accuracy check against source documents. In Moffatt v. Air Canada (2024 BCCRT 149), a British Columbia tribunal held the airline responsible for its website chatbot's wrong answer on bereavement fares and rejected the argument that the chatbot was responsible for its own statements (Civil Resolution Tribunal, 2024).
EU AI Act transparency. Article 50 applies from 2 August 2026, and Article 50(1) requires AI systems that interact directly with people to be designed so those people are told they are dealing with an AI system, unless that is obvious from context (EU AI Act, Article 50). Add the disclosure to the launch checklist for any bot serving EU users.
How to red team a chatbot before launch
Scope the bot's contract. Write down what the bot may do (answer product questions, look up an order, open a ticket) and what it must never do (reveal another customer's data, run a refund, disclose its hidden context, produce disallowed content). The "must never" list becomes the test targets.
Enumerate the attack surface. Map every input path (chat box, uploads, retrieved documents, connected tools and APIs) and every output sink (the browser, downstream systems, logs). Injection and exfiltration hide in retrieved content and tool responses.
Run adversarial tests against each boundary. Pair human red teamers with automated adversarial testing: humans supply creative, context-aware attacks and automation supplies breadth and repeatability.
Convert results to pass/fail. Hand the launch team the acceptance table with linked evidence, not a heat map or a single score.
Re-test after fixes. A patched FAIL is not a PASS until the same test is re-run and new evidence is captured.
Blocker detail: what each test checks and how to fix a failure
Blocker 1: Prompt-injection resistance (LLM01:2026)
Direct injection is a user telling the bot to drop its rules. The higher-risk case is indirect injection: an instruction riding inside content the bot ingests, such as a product review, a support ticket or a knowledge-base document. Anyone who can write into those stores can try to steer the bot. The 2026 entry widens the surface to images, audio, tool output and long-term memory (OWASP, 2026).
Detect and fix: plant synthetic instructions in a retrieved document, not only in the chat box. Fixes are content isolation (label outside content by source), an instruction hierarchy, retrieval sanitization, and keeping credentials and state changes in application code where the model cannot reach them.
Blocker 2: Hidden-context and secrets non-leakage (LLM08:2026, LLM02:2026)
OWASP's 2026 guidance is blunt: design on the assumption that hidden context is discoverable, and make sure its disclosure has little or no security impact. Severity tracks what is in it: internal rules that aid an attacker are medium, embedded credentials or reliance on hidden-context secrecy for authorization are high (OWASP, 2026).
Detect and fix: inspect the system prompt and tool definitions directly rather than relying on "the bot refused". Move every secret out, and enforce authorization server-side so a leaked prompt grants nothing.
Blocker 3: Unsafe tool and action prevention (LLM03:2026)
The moment a support bot gets tools, prompt injection stops being a content problem and becomes an action problem. OWASP traces Excessive Agency to excessive functionality, excessive permissions and excessive autonomy. A bot that can call a refund function will eventually be asked for a refund it should not give.
Detect and fix: inventory every tool and the permission it needs, then try to trigger out-of-scope actions directly and through retrieved content. The durable control is server-side authorization on every tool call scoped to the end user, tools cut down to the functions the bot needs, and explicit confirmation for state-changing actions that the bot cannot approve itself.
Blocker 4: PII exfiltration through retrieval (LLM09:2026, LLM02:2026)
If every document is embedded into one shared index with no per-user or per-tenant filter, relevance, not authorization, decides what comes back. The model answers faithfully from whatever the retriever hands it, and the retriever does not know who is asking. It is a multi-tenant isolation failure in an AI pipeline.
Detect and fix: run cross-user and cross-tenant cases (user A asking, in as many ways as the red team can devise, for user B's data) and expect nothing back. Filter by entitlement before retrieval through per-tenant indexes or metadata filtering, and do not embed PII the bot does not need. Our guide to RAG vector store access control testing covers the store itself.
Blocker 5: Output handling (LLM10:2026)
Teams watch what goes into the model and forget what comes out. If the chat UI renders responses as HTML, model output can execute in the user's browser; combined with indirect injection, the bot delivers stored XSS. The 2026 entry adds two paths a support bot should test: renderers that automatically fetch Markdown images or link previews from model output, which can carry conversation data to an outside domain, and terminal or log sinks that do not neutralize control characters.
Detect and fix: encode output for the context it lands in, disable auto-loading of external images and previews in responses (or restrict them to an allow-list of domains), and confirm that every downstream consumer, such as ticketing and email, treats bot output as untrusted.
Blocker 6: Abuse and cost limits (LLM06:2026)
Every message has a real inference cost, and the attacker pays almost none of it. The goals range from denial of wallet and denial of service to bulk extraction of outputs to clone the bot's behavior. OWASP's 2026 entry says request-rate limiting alone is no longer sufficient, and adds token-aware cost controls, hard spending caps and circuit breakers.
Detect and fix: load and burst test the bot, confirm per-user and per-IP limits and maximum input sizes trigger, set a hard spending cap and watch it fire in test, and alert on anomalous usage so abuse shows up in monitoring rather than on the invoice.
Blocker 7: Guardrail-bypass resilience (LLM01:2026)
OWASP treats jailbreaking as the subset of prompt injection aimed at making the model violate its safety rules. Guardrails are probabilistic, so the honest question is not whether they can be bypassed but how often, and what is exposed when they are. One refusal you happened to see is not evidence.
Detect and fix: run a structured battery across techniques (role-play, encoding, obfuscation, multi-turn pressure, language switching) at volume, record a pass rate per technique, give each residual failure an owner and a fix or a signed risk acceptance, and re-run the battery after mitigation.
Turning the table into a go/no-go decision
GO: all seven PASS, evidence attached.
CONDITIONAL GO: findings remain only on blockers 5 to 7, each with a fix committed to a date, a named owner, and residual risk accepted in writing by an accountable person.
NO-GO: any unresolved FAIL on blockers 1 to 4 (injection, hidden-context and secrets leakage, unsafe actions, PII exfiltration).
Keep the artifacts. The red-team report, the acceptance table, the fixes and the re-tests answer the question every incident review asks: what was tested before the bot shipped, and who signed off. Tagging each finding with its 2026 OWASP ID, and with the matching NIST Generative AI Profile risk, makes that trail portable to auditors and enterprise procurement teams (NIST AI RMF and NIST-AI-600-1). Change management decides when the table must be re-run; our AI and LLM retest trigger matrix maps model swaps, new tools and corpus changes to retest scope.
How Stingrai tests a chatbot before launch
Stingrai's AI and LLM penetration testing pairs Snipe's specialist AI and LLM sub-agent with two named, certified penetration testers from a team holding OSCE³, OSWE, OSEP and CREST CRT certifications, who test prompt injection, hidden-context leakage, output handling, tool misuse and excessive agency against the OWASP LLM Top 10, the OWASP Agentic Top 10 and MITRE ATLAS. Findings are posted to the portal as they are confirmed, each with a proof of concept and remediation guidance, and retesting of fixed findings is included, so the re-test evidence this table demands comes from the same engagement. Where the bot sits inside a web application, a Hybrid engagement adds Snipe, our autonomous agent for web applications and associated APIs, to hunt the IDOR, broken-authorization and business-logic flaws in the surrounding application that let a retrieval layer leak data; on the model itself, Snipe's AI and LLM sub-agent works alongside the penetration testers. Human-led and Hybrid reports come with an attestation letter, and engagements run one-time before launch or as a continuous program across releases.
Methodology and sources
The seven blockers follow the OWASP Top 10 for LLM Applications 2026 (published August 2026), with agentic tool risks mapped to the OWASP Top 10 for Agentic Applications 2026 (OWASP GenAI Security Project). Governance framing draws on the NIST AI Risk Management Framework 1.0 (January 26, 2023) and its Generative AI Profile, NIST-AI-600-1 (July 26, 2024); NIST says AI RMF 1.0 is being revised under the White House AI Action Plan, with no date yet (NIST). Adversary technique context comes from MITRE ATLAS, whose latest data release is v2026.10 of October 28, 2026, with 16 tactics, 129 techniques and 77 case studies (MITRE ATLAS data releases). Regulatory timing reflects the EU AI Act as amended by Regulation (EU) 2026/1744, which moved obligations for Annex III high-risk systems to 2 December 2027 and for Annex I systems to 2 August 2028 (EU AI Act implementation timeline); a support chatbot is rarely high-risk, but the Article 50 transparency duty above already applies. We publish no attack payloads, and every figure links to its publisher so any claim can be audited.
Frequently Asked Questions
How do you red team a chatbot before launch, and what should a pre-production AI security checklist include?
Define the bot's hard boundaries, test each one adversarially from an attacker's position, and record the results as pass/fail acceptance criteria with evidence. The checklist should cover seven blockers: prompt-injection resistance, hidden-context and secrets non-leakage, unsafe tool and action prevention, PII exfiltration through retrieval, output handling, abuse and cost limits, and guardrail-bypass resilience. Each maps to an ID in the OWASP Top 10 for LLM Applications 2026.
Which OWASP LLM Top 10 IDs changed in 2026?
Excessive Agency moved from LLM06 to LLM03, Supply Chain from LLM03 to LLM04, Data and Model Poisoning from LLM04 to LLM05, Unbounded Consumption from LLM10 to LLM06, Misinformation from LLM09 to LLM07, Vector and Embedding Weaknesses from LLM08 to LLM09, and Improper Output Handling from LLM05 to LLM10. System Prompt Leakage (LLM07:2025) became Hidden Context Exposure (LLM08:2026). Prompt Injection (LLM01) and Sensitive Information Disclosure (LLM02) kept their IDs. Always write the edition into a finding ID, because every ID from LLM03 to LLM10 now names a different risk.
What is the most dangerous failure mode in a customer-facing chatbot?
Indirect prompt injection combined with tool access is usually the highest-impact path. When an instruction hidden in a retrieved document or support ticket steers a bot that can also take actions, a content problem becomes an unauthorized action or a data breach. OWASP tracks these as Prompt Injection (LLM01:2026) and Excessive Agency (LLM03:2026).
What evidence should a launch reviewer require before signing off?
For each blocker: the number of attempts run, the number blocked, the synthetic text of any success, the mitigation applied, and a re-test showing the issue now fails. For hidden context, add an inspection record proving no secret is in the prompt or tool definitions. A pass/fail table with linked artifacts, not a single score, supports a defensible go-live decision.
Is red teaming a chatbot the same as writing jailbreak prompts?
No. OWASP treats jailbreaking as one subset of prompt injection, aimed at making the model break its safety rules. Pre-launch red teaming is a defender-side exercise that documents which boundaries hold, which do not and what evidence proves it, so a reviewer can make a go/no-go call. A responsible report describes failure modes and required fixes, not reusable attack payloads.
How do you stop a chatbot from leaking one customer's data to another?
Filter retrieval by the authenticated user's entitlements before results reach the model, using per-tenant indexes or metadata filtering, and enforce authorization server-side on every tool call and lookup. Do not rely on the prompt to keep tenants apart. Then have the red team try, in as many ways as possible, to pull another user's data, and confirm every attempt returns nothing.
Who can run a pre-launch chatbot red team?
Stingrai's AI and LLM penetration testing pairs Snipe's specialist AI and LLM sub-agent with certified penetration testers, runs against the OWASP LLM and Agentic Top 10 and MITRE ATLAS, and returns the pass/fail acceptance trail with re-test evidence. Where the bot sits in a web application, a Hybrid engagement adds Snipe, our autonomous agent for web applications and associated APIs, to test the surrounding application's authorization. Start with our AI and LLM penetration testing page.
Related reading
OWASP Agentic AI Top 10: The Test Coverage to Demand Before You Ship an Agent
The AI and LLM Penetration Test Scope of Work Template You Can Send Vendors
Prompt Injection Statistics 2026: Attack Success Rates, Bug Bounties and CVEs
The AI Retest-Trigger Matrix: Which LLM Changes Force a New Red Team
References
OWASP GenAI Security Project. OWASP Top 10 for LLM Applications 2026. August 2026. https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/. Current edition; source for the 2026 IDs, the 2025 to 2026 rank changes, the Hidden Context Exposure rename and the vote and incident-data weighting.
OWASP GenAI Security Project. OWASP Top 10 for LLM Applications 2025. https://genai.owasp.org/llm-top-10/. Previous edition, source for the 2025 IDs in the remap table.
OWASP GenAI Security Project. OWASP Top 10 for Agentic Applications for 2026. December 9, 2025. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/. Agentic risks including ASI02 Tool Misuse and Exploitation and ASI03 Identity and Privilege Abuse.
NIST. AI Risk Management Framework (AI RMF 1.0) (January 26, 2023) and Generative AI Profile, NIST-AI-600-1 (July 26, 2024). https://www.nist.gov/itl/ai-risk-management-framework.
MITRE. ATLAS data release v2026.10. October 28, 2026. https://github.com/mitre-atlas/atlas-data/releases. Adversary tactics, techniques and case studies for AI-enabled systems; browse at atlas.mitre.org.
European Union. AI Act, Article 50, and implementation timeline as amended by Regulation (EU) 2026/1744. https://artificialintelligenceact.eu/article/50/ and https://artificialintelligenceact.eu/implementation-timeline/.
Civil Resolution Tribunal of British Columbia. Moffatt v. Air Canada, 2024 BCCRT 149. February 14, 2024. https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do.
