Healthcare carried the most expensive data breaches of any industry for the fourteenth year running in 2025, at an average of US$7.42M per incident, per IBM's Cost of a Data Breach 2025. Those breaches also took the longest to catch: healthcare organizations needed 279 days on average to identify and contain an incident, well above the global figure. Now put a clinical large language model (LLM) in the middle of that data, drafting notes from a live patient conversation and pulling history straight from the electronic health record (EHR), and you have added a new, non-deterministic component to the most breach-prone surface in the economy.
Healthcare AI penetration testing is how you get ahead of that. This guide is written for the healthcare and health-tech buyer who is scoping independent security testing of a clinical AI feature, whether that is an ambient scribe, a triage assistant, a patient-facing chatbot, or retrieval-augmented generation (RAG) over the EHR.
How do you penetration test a clinical AI scribe or healthcare LLM feature for PHI and HIPAA risk?
You test the whole clinical surface, not just the model. A credible healthcare AI penetration test starts by mapping where protected health information (PHI) flows through the feature, sets a clear in and out-of-scope boundary, and then attacks four threat categories in combination: PHI leakage, indirect prompt injection, clinical-model manipulation, and insecure direct object reference (IDOR) or broken authorization across patient records. The engagement measures whether PHI stays contained, whether the model ignores instructions hidden in its inputs, whether clinical outputs stay inside safe bounds, and whether one patient can ever reach another patient's data. The written findings become evidence for your HIPAA Security Rule risk analysis, your Canadian PHIPA and PIPEDA obligations, and your SOC 2 program.
TL;DR
Scope the surface, not the scribe. An ambient scribe is one node in a chain that includes capture, an LLM, RAG-over-EHR retrieval, a draft note, and write-back. Test the chain.
Map PHI first. Identify every hop where PHI is present, every trust boundary it crosses, and every place it is stored, logged, or sent to a third party.
Four threat categories. PHI leakage (OWASP LLM02), indirect prompt injection (OWASP LLM01), clinical-model manipulation, and IDOR or broken authorization on the app and API layer.
A clean result is defined up front. PHI stays contained, injected instructions are ignored, outputs stay inside clinical guardrails, and cross-patient access is impossible.
Testing supports compliance. Findings feed your HIPAA Security Rule risk analysis, PHIPA and PIPEDA, and SOC 2 evidence.
Why a clinical AI feature needs its own test scope
A standard web application penetration test of your portal will not exercise the parts that make a clinical AI feature risky. The scribe listens to a consultation, an LLM turns that into a structured note, a retrieval layer fetches the patient's prior history, and the draft is written back into the chart. Each of those steps introduces failure modes that a traditional test never touches: a model that repeats another patient's data, an instruction smuggled in through a document the model reads, or a summary that drops a critical allergy.
The proposed HIPAA Security Rule update, published in the Federal Register on January 6, 2025, points in the same direction. If finalized as drafted, it would make annual penetration testing and six-monthly vulnerability scanning explicit expectations rather than implied ones. Testing frameworks have caught up too: the OWASP Top 10 for LLM Applications 2025 and the NIST AI Risk Management Framework Generative AI Profile both give testers a shared vocabulary for the LLM-specific classes that a clinical AI assessment has to cover.
Map the PHI data flow before you scope
Scoping starts with a data-flow map, because you cannot protect PHI you have not located. Walk the clinical AI feature end to end and mark every hop where PHI is present, every trust boundary it crosses, and every store, log, cache, or third-party endpoint it touches.
A typical ambient scribe flow has at least five PHI-bearing hops:
Capture. Audio or a live transcript of the consultation. PHI is present the moment recording starts.
Transcription and pre-processing. Speech to text, often through a third-party service. Ask where the audio and transcript are stored and for how long.
The LLM and retrieval layer. The prompt sent to the model, plus any patient history pulled from the EHR through RAG. This is where a request for one patient can accidentally surface another patient's context.
Draft output. The generated note, held in application state or a queue before a clinician signs off.
Write-back. The signed note committed to the EHR, plus every log line, analytics event, and error report produced along the way.
For each hop, record three things: is PHI present, does it cross a trust boundary (for example, from your tenant to a model provider), and is it retained anywhere it does not need to be. That map drives everything downstream, from the scope boundary to the test cases.
What is in scope and what is out of scope
A clinical AI assessment works best with a boundary agreed in writing before testing starts. The table below is a starting template. The guiding rule: test everything you or your integration control, and exclude the internals of systems you do not own, while still testing how your feature uses them.
Area | In scope | Out of scope |
|---|---|---|
Ambient scribe and capture | Transcript handling, retention, redaction, tenant isolation | The device microphone firmware |
Clinical LLM feature | Prompt handling, output guardrails, PHI containment, injection resilience | Retraining the foundation model's base weights |
RAG over EHR | Retrieval scoping, per-patient access checks, embedding store isolation | The EHR vendor's internal source code |
Application and API layer | Authentication, authorization, IDOR, business logic, session handling | Third-party payment processor internals |
Integrations and logs | Data sent to third parties, log content, error reporting, webhooks | The third party's own infrastructure |
Model provider | How your feature calls, filters, and constrains the provider API | The provider's model internals and data center |
Two boundary notes are worth stating explicitly. First, the clinical model's medical accuracy, whether the summary is clinically correct, is a job for clinical governance and validation, not a security test. A penetration test checks whether the model can be manipulated into unsafe behavior, not whether an un-manipulated model gives good medical advice. Second, testing does not touch real patient records. A responsible engagement runs against synthetic PHI in a staging environment that mirrors production.
The four threat categories a clinical AI assessment must cover
PHI leakage (OWASP LLM02: Sensitive Information Disclosure)
The core risk is that the feature discloses PHI to someone who should not see it: another patient, an unauthenticated user, a log file, an analytics pipeline, or the model provider. Testers probe whether the model repeats data from its context that belongs to a different patient, whether prior-session PHI bleeds into a new conversation, and whether transcripts or drafts land in logs and error reports in the clear. A clean result: PHI appears only where it is authorized, never in logs, analytics, or another user's session.
Indirect prompt injection (OWASP LLM01: Prompt Injection)
Clinical AI features read untrusted text. A referral letter, an uploaded document, a prior note, or a patient-supplied message can carry instructions aimed at the model rather than at a human reader. Indirect prompt injection tests whether that hidden text can steer the model into exfiltrating context, ignoring its guardrails, or taking an unintended action. Testing uses synthetic, controlled inputs to confirm the boundary holds. A clean result: content the model reads is treated strictly as data, and instructions embedded in it are never executed.
Clinical-model manipulation
Beyond injection, testers assess whether an adversary can push model outputs outside safe clinical bounds: coaxing it to drop a documented allergy from a summary, invent a medication, or alter a dosage in a draft note. This is scoped as a security concern, whether the output can be adversarially manipulated, and works alongside, not instead of, the clinical validation your medical team runs. A clean result: adversarial inputs cannot force the model to produce clinically unsafe or altered content that a clinician would reasonably trust.
IDOR and broken authorization across patient records
This is the classic web and API weakness in a clinical wrapper, and it is where the most damaging cross-patient exposure usually lives. If patient A's request can be manipulated to fetch, summarize, or write to patient B's record by changing an identifier, tampering with a request, or exploiting a flaw in how the retrieval layer scopes access, the AI feature has become a PHI disclosure engine. Testers work every object reference, every retrieval parameter, and every authorization check across the app and API layer. A clean result: no identifier, parameter, or request can be manipulated to reach another patient's data.
What the engagement measures
Define outcomes before testing begins so the report answers questions your risk analysis actually asks. A clinical AI penetration test should measure and report on:
PHI containment. Whether protected health information stays inside its authorized boundary at every hop, with zero leakage to logs, analytics, other users, or third parties.
Prompt-injection resilience. Whether the feature treats all read content as data and refuses embedded instructions, across every input channel that reaches the model.
Model-output safety. Whether adversarial inputs can push outputs outside safe clinical bounds, reported as a security finding for your clinical governance team to act on.
Access control and IDOR. Whether authorization holds across every patient object and retrieval path, with no cross-patient access possible.
De-identification and retention. Whether redaction, minimization, and retention behave as designed, and whether PHI is retained anywhere it should not be.
Audit logging. Whether security-relevant events are logged usefully, without the logs themselves becoming a PHI leak.
Remediation guidance. Each finding rated by severity with a concrete fix, plus a retest to confirm closure.
Every one of those outcomes maps to evidence your compliance program needs, from the HIPAA Security Rule risk analysis to SOC 2 and, for Canadian providers, PHIPA and PIPEDA.
How Stingrai scopes a clinical AI assessment
Stingrai is a CREST-accredited offensive-security firm founded in 2021, with headquarters in Toronto and an office in London, which puts a testing team on both sides of the Canada, US, and UK health-tech corridor. A clinical AI assessment combines two complementary approaches.
Senior pentesters run the human red team against the model-facing threats: PHI leakage, indirect prompt injection, and clinical-model manipulation. These require clinical context, careful synthetic test design, and judgment about what counts as unsafe behavior, which is human work. Our team's methodology is the same one used across our red teaming and AI red teaming for LLM and agentic apps engagements, and the RAG-over-EHR retrieval layer is tested with the same access-control rigor described in our guide to RAG vector store access-control testing.
On the application and API layer, Snipe, our autonomous AI agent for web application penetration testing, hunts the IDOR, business logic, and broken-authorization flaws that let one patient's record surface to another. Snipe is purpose-built for exactly the complex, high-impact classes that generic AI scanners miss, and it runs black-box dynamic testing plus white-box source review, generates AutoFix pull requests, and can gate every pull request as a merge check. Pairing an autonomous agent on the access-control surface with a senior human red team on the model surface is the same hybrid model we apply when red teaming AI agents that move money in fintech. Both feed one report through our web application penetration testing and PTaaS delivery.
The output supports your HIPAA Security Rule risk analysis, your PHIPA and PIPEDA obligations, and your SOC 2 evidence. For scope sizing and package options, see the Stingrai pricing page.
Frequently Asked Questions
How do you penetration test a clinical AI scribe or healthcare LLM feature for PHI and HIPAA risk?
You test the whole clinical surface, not just the model. Start by mapping where protected health information flows through the feature, set an in and out-of-scope boundary in writing, then attack four threat categories together: PHI leakage, indirect prompt injection, clinical-model manipulation, and IDOR or broken authorization across patient records. The findings become evidence for your HIPAA Security Rule risk analysis, your Canadian PHIPA and PIPEDA obligations, and your SOC 2 program.
What is healthcare AI penetration testing?
Healthcare AI penetration testing is independent security testing of a clinical AI feature such as an ambient scribe, triage assistant, patient chatbot, or RAG-over-EHR system. It covers the model-facing risks (PHI leakage, prompt injection, model manipulation) and the application and API risks (IDOR, broken authorization, business logic) as one connected surface, because PHI moves across all of them.
What should be in scope for a clinical LLM or ambient scribe assessment?
In scope: transcript handling and retention, prompt handling and output guardrails, PHI containment, injection resilience, RAG retrieval scoping and per-patient access checks, and the authentication, authorization, and business logic of the app and API layer. Out of scope: the internals of systems you do not own, such as the foundation model's base weights, the EHR vendor's source code, and the model provider's infrastructure. Testing runs against synthetic PHI in staging, never real patient records.
What is PHI leakage in a clinical AI feature, and how is it tested?
PHI leakage is when the feature discloses protected health information to someone or something that should not receive it, such as another patient's session, a log file, an analytics pipeline, or the model provider. It maps to OWASP LLM02, Sensitive Information Disclosure. Testers check whether the model repeats data from a different patient's context, whether prior-session PHI bleeds into a new conversation, and whether transcripts or drafts appear in logs and error reports. A clean result is PHI appearing only where it is authorized.
What is indirect prompt injection in a healthcare AI scribe?
Indirect prompt injection is when instructions hidden inside content the model reads, such as an uploaded referral, a prior note, or a patient message, try to steer the model instead of informing it. It maps to OWASP LLM01, Prompt Injection. A healthcare AI penetration test uses synthetic, controlled inputs to confirm the model treats read content strictly as data and never executes instructions embedded in it.
Does a healthcare AI penetration test cover IDOR and broken access control across patient records?
Yes, and it is one of the highest-impact parts of the engagement. IDOR and broken authorization are where a manipulated request can fetch, summarize, or write to another patient's record by changing an identifier or tampering with a retrieval parameter. At Stingrai, the autonomous agent Snipe hunts these access-control and business-logic flaws on the app and API layer, while senior pentesters cover the model-facing threats.
Does healthcare AI penetration testing help with HIPAA compliance?
Yes. The findings produce evidence that supports your HIPAA Security Rule risk analysis, and, for Canadian providers, your PHIPA and PIPEDA obligations and SOC 2 program. The proposed 2025 HIPAA Security Rule update would make annual penetration testing an explicit expectation. Independent testing gives your compliance program concrete, tested evidence to draw on.
How long does a clinical AI penetration test take and what does it cost?
Duration and cost depend on the number of PHI-bearing hops, whether RAG-over-EHR is in scope, the size of the app and API surface, and whether you want a one-time assessment or continuous coverage. The most reliable way to size a clinical AI engagement is a short scoping call that walks the PHI data-flow map. See the Stingrai pricing page for package options.
References
IBM. Cost of a Data Breach Report 2025. July 2025. https://www.ibm.com/reports/data-breach. Global breach-cost benchmark; healthcare remained the costliest industry at US$7.42M average with a 279-day identify-and-contain lifecycle.
OWASP. OWASP Top 10 for LLM Applications 2025. 2025. https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/. Reference taxonomy for LLM risks, including LLM01 Prompt Injection and LLM02 Sensitive Information Disclosure.
U.S. Department of Health and Human Services, Office for Civil Rights. HIPAA Security Rule to Strengthen the Cybersecurity of Electronic Protected Health Information (NPRM). Federal Register, January 6, 2025. https://www.federalregister.gov/documents/2025/01/06/2024-30983/hipaa-security-rule-to-strengthen-the-cybersecurity-of-electronic-protected-health-information. Proposed rule adding explicit written risk analysis, six-monthly vulnerability scanning, and annual penetration testing.
U.S. Code of Federal Regulations. 45 CFR 164.308: Administrative safeguards (Security Rule risk analysis). https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-C/part-164/subpart-C/section-164.308. Current binding requirement to conduct a risk analysis of ePHI.
NIST. Artificial Intelligence Risk Management Framework: Generative AI Profile (NIST-AI-600-1). July 2024. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf. Framework of generative-AI risks and suggested actions used to structure LLM testing.
Government of Ontario. Personal Health Information Protection Act, 2004 (PHIPA). https://www.ontario.ca/laws/statute/04p03. Ontario health-information privacy statute governing custodians of personal health information.
Office of the Privacy Commissioner of Canada. Personal Information Protection and Electronic Documents Act (PIPEDA). https://www.priv.gc.ca/en/privacy-topics/privacy-laws-in-canada/the-personal-information-protection-and-electronic-documents-act-pipeda/. Canadian federal private-sector privacy law.
