CISA's Microsoft Entra ID SCuBA baseline states a limit that belongs in every enterprise identity test plan: ScubaGear does not assess Conditional Access condition values beyond the conditions specified in its policy implementations. CISA warns that conflicting or misaligned conditions can leave users or devices outside intended protection. The baseline is a useful configuration reference, but a buyer still needs an observed access decision for the identity paths that protect its own data.
Quick answer. Validate a SCuBA-aligned Conditional Access policy by pairing its configuration result with controlled sign-in tests across the relevant user, device, client application and resource combinations. Record the intended decision, the result actually enforced, and the corresponding redacted sign-in evidence. The matrix below is Stingrai's test-design synthesis; CISA does not prescribe this exact matrix or require a penetration test for each policy.
The distinction matters for a bank, health plan or enterprise SaaS vendor that has a correctly written policy but an unexpected exclusion. A tenant may show a policy configured for broad coverage while a guest, emergency account, legacy client, unmanaged device or newly added application takes a path the owner did not model. The question for a security review is not whether a setting exists. It is whether the setting reaches the person and resource it is meant to govern.
This reference is for security leaders, identity architects, procurement teams and testers commissioning an authorized review of their own Microsoft Entra tenant. It turns CISA's source policies and its stated automation limits into buyer acceptance criteria. Examples are test ideas, not assertions that every organization must use the same Conditional Access policy or take the same risk decision.
What SCuBA measures and where judgment begins
The SCuBA Entra baseline is designed around the federal government's threats and risk tolerance; CISA says organizations outside the federal government may also use it as a reference. Its policy entries include an identifier, desired configuration, rationale, implementation guidance and indicators that distinguish automated checks from manual verification. ScubaGear assesses Microsoft 365 tenant configuration against those policies.
Some of the baseline language is unambiguous. MS.AAD.1.1v1 says legacy authentication shall be blocked. MS.AAD.3.1v1 says phishing-resistant MFA shall be enforced for all users. MS.AAD.3.2v2 says MFA shall be enforced for all users, including cases where the stronger method cannot be applied as intended. These are CISA policy statements, not claims that a particular customer's tenant passes them. The exact current policy version should be checked at the time an engagement begins, because the repository can change.
Conditional Access evaluation adds a second layer. CISA recommends putting a newly created policy in Report-only while the owner tests it, then turning it On. Its baseline states that a policy is enforced only in the On state. It also explains that ScubaGear does not inspect every condition value and warns about exclusions caused by overlapping policies, new cloud applications, unmanaged devices and evolving network ranges. That is the opening for a behavior-focused test: take the organization's intended rule, choose representative identities and contexts, and observe whether access is actually blocked or allowed.
Figure 1: A configuration check, a scoped identity, a controlled sign-in and its observed decision form one evidence chain. Source for the configuration boundary: CISA, Microsoft Entra ID SCuBA baseline. The evidence chain is Stingrai's synthesis.
Passing a baseline check is valuable evidence about configuration. A manual result asks for a human review of the specified setting. Neither result by itself proves that the account, application and device combination relevant to a high-value workflow receives the intended decision. That last proposition requires a test. Conversely, one blocked sign-in does not prove universal coverage. The report must identify which combinations were actually exercised and which were inferred from configuration.
The original Conditional Access validation matrix
Use the table as a set of candidate acceptance tests. The buyer first records the policy it wants enforced. The tester then uses synthetic identities and approved client contexts to check a permitted case and a denied case. A condition shown here is a test dimension, not a claim that CISA requires every listed scenario in every tenant.
Boundary to validate | Baseline anchor or policy intent | Authorized test pair | Evidence to retain |
|---|---|---|---|
Policy state | CISA: a policy enforces only when On | Compare a Report-only simulation with an On-policy decision in the agreed test environment | Policy state at test time and sign-in result |
Legacy client access | MS.AAD.1.1v1 | Supported modern client succeeds; approved legacy-client attempt is blocked | Client type, policy applied and denied sign-in |
Phishing-resistant factor | MS.AAD.3.1v1 | Authorized user with an approved strong method succeeds; weaker method is denied where policy requires resistance | User cohort, authentication method and result |
General MFA coverage | MS.AAD.3.2v2 | Included user succeeds with MFA; selected user cannot enter through a single-factor path | User assignment, grant control and outcome |
Guest identity | Organization's guest policy | Invited guest receives intended restriction; unrelated guest has no route to protected resource | Guest type, resource, decision and policy trace |
Privileged role | Organization's administrator policy | Named privileged test account receives the specified stronger control; ordinary user cannot use a privilege exception | Role assignment snapshot and sign-in decision |
Device context | Organization's managed-device rule | Managed test device reaches permitted resource; unmanaged test device is refused where required | Device state, resource and decision |
Client application | Organization's app inclusion rule | Approved client reaches its resource; alternative client cannot bypass intended grant control | Client identity and matched policy |
Newly added resource | Organization's app coverage rule | Existing application and new test application receive the intended control | Resource identifiers and policy evaluation |
Network context | Organization's location rule | Approved location behaves as intended; a second authorized context does not fall outside the rule | Location classification and observed result |
Break-glass exception | Organization's emergency-access procedure | Documented emergency account follows its restricted exception path; an ordinary account cannot enter via it | Exception owner, monitoring record and decision |
Downstream application session | Product access policy | Entra sign-in creates only the allowed application role and tenant session | Identity, app role, tenant and protected-resource denial |
The first three source-backed identifiers are the starting points for a SCuBA review. The remaining rows translate the baseline's warning about Conditions and exclusions into a test design. The organization owns the policy choices. The tester's job is to verify those choices in a safe, repeatable way and to distinguish an unexpected allow from an approved exception.
Two kinds of proof should be kept separate. A configuration artifact shows the Conditional Access policy definition, assignment, state and exceptions as it existed during the test. A behavior artifact shows what happened during a particular sign-in and, where relevant, what the application allowed after authentication. An executive summary can then say precisely which policy intent held, which failed and which could not be tested.
Build a test inventory before sending the RFP
A vendor cannot test a tenant's identity policy by guessing how it is supposed to work. The buyer should provide a small, approved inventory: policy names and owners; protected applications; required authentication methods; expected behavior for employees, guests, administrators and emergency accounts; device compliance definitions; and the known exceptions. Any legitimate exception should have a business owner and a review date. A missing inventory is a scoping limitation, not an invitation to classify every unusual policy as a vulnerability.
For each protected resource, name the data or operation it guards. A customer portal's administrative screen, an internal document store and an enterprise billing console do not carry the same impact. If a policy is intended to apply to all resources, test a representative existing resource and a newly introduced test resource. If the tenant uses separate production and staging applications, make clear whether the staging result can be generalized to production. Often it cannot, because app assignments and conditions differ.
Create synthetic identities that reflect actual policy cohorts. At minimum, an employee, a privileged user and a guest may reveal different assignment paths. Add a device state or location only if that condition is in scope and can be represented lawfully. Test accounts should carry no real customer data, and the owner should approve every sign-in context. This is an assessment of decision enforcement, not a credential attack or an attempt to trigger risk detections at scale.
The test plan should also name an application owner. A Conditional Access decision is one boundary; the application still has its own roles, sessions and tenant mapping. A sign-in that meets a strong-factor policy could still land in the wrong application role. The enterprise identity test matrix addresses that lifecycle more broadly. Keep this SCuBA matrix focused on whether a stated Conditional Access intent actually covers the contexts in question.
Design safe allow and deny cases
Each meaningful test needs a positive control. If every attempted sign-in fails, the team cannot tell whether the expected policy blocked access, an account was misconfigured, or a test application was unavailable. Begin with a permitted identity and approved method, record the successful decision, then change one relevant dimension. Use a different user cohort, client, device state or resource. That single-variable structure makes the denial interpretable.
The test should not rely on a screenshot of a sign-in page alone. Request the redacted sign-in event, the policy evaluation result and the application outcome. Keep timestamps and pseudonymous test-account IDs aligned so the assessor can show that an event belongs to the exact test case. Remove tokens, secrets, personal identifiers and real tenant data before the evidence package leaves the customer environment.
For Report-only policies, a simulated result describes what would have happened under the proposed configuration. It is a planning artifact. The owner's acceptance decision should separately confirm that the corresponding policy was set to On and that a subsequent authorized test produced the intended enforced decision. This distinction follows CISA's own baseline guidance.
For exceptions, do not erase them to make a dashboard green. An emergency account can be a legitimate operational necessity. The test asks whether the exception is specific, monitored and confined to the intended path. It also asks whether an ordinary identity can accidentally inherit it. Describe both the permitted emergency behavior and the denied ordinary behavior in the same evidence row.
For downstream access, stop at synthetic records and approved resources. The useful result is a controlled statement such as: the sign-in succeeded for the correct user, but the application's protected action was denied for the unauthorized tenant. It does not require access to real files, financial records or patient data. A safe negative result is often stronger audit evidence than a broad claim of successful access.
What to put in the evidence record
An external reviewer should be able to read one row without opening a stack of unrelated screenshots. The following record format is Stingrai's proposed acceptance worksheet. It is not a CISA form.
Field | What the reviewer needs |
|---|---|
Policy and owner | Stable name, baseline ID if applicable, owner and approved intent |
Version and state | Baseline version, policy revision, Report-only or On, test date |
Target | Application, resource and tenant boundary, described without sensitive data |
Subject | Synthetic account cohort, role, guest status and approved device context |
Expected result | Permit, challenge, block or specific documented exception |
Observed result | Actual sign-in and application decision, including timing |
Correlation | Redacted event identifier and policy evaluation trace |
Verdict | Pass, fail, not applicable or blocked by a stated test limitation |
Retest | Configuration change, owner approval and a fresh observed decision |
Do not treat not applicable as a quiet pass. It needs a reason: for example, the tenant does not allow guest accounts, or the customer has no legacy client to exercise. Do not treat blocked as failure either when the tester lacked permission to perform the test. Record the missing access or fixture and the risk accepted by the owner. Precise limitations protect the credibility of the report.
For failed rows, prioritize by the protected resource and the reachable identity, not solely by whether a CISA policy ID appears beside the test. An unintended path to a high-value administrator function is different from an unusual but approved access route to a non-sensitive test application. The remediation note should identify the condition or assignment that produced the decision, the policy owner who can change it and the retest needed to demonstrate closure.
Interpret an unexpected allow before calling it a failure
An unexpected allow is a finding candidate, not a finished finding. First confirm that the sign-in belongs to the intended synthetic account, device and application. Check the policy evaluation trace against the saved policy state and assignment. Then ask the policy owner whether an exception was approved for that exact cohort and resource. If it was, the report should describe the exception and test whether its scope is confined to what was approved. If it was not, the report should name the path that escaped the intended control.
This distinction is important in a tenant with several overlapping policies. A denied sign-in might have been blocked by a different policy than the one being evaluated, while a successful sign-in might be permitted by an exception that is fully documented. Report the observed decision and the matched policy path separately. Avoid concluding that a named SCuBA policy passed merely because some control denied access. Equally, avoid calling every unusual permitted sign-in a baseline failure before checking which rule applied.
Sampled sign-ins also have a defined limit. A test of an employee on one managed device cannot prove how every employee, guest, location and application combination behaves. The final report should list the combinations exercised, the configuration evidence used to infer broader coverage and any paths that remain untested. This lets a buyer decide whether to add cases in the next assessment instead of relying on a vague claim of universal protection.
Procurement language that produces a usable report
A short statement of work can make the distinction between configuration and behavior explicit: “Assess the applicable CISA SCuBA Microsoft Entra policies; identify automated and manual configuration evidence; execute owner-approved positive and negative sign-in tests for the agreed user, device, client and resource contexts; and report expected versus observed decisions with redacted sign-in evidence.” The quoted text is suggested contract language, not language from CISA.
Ask bidders to mark each matrix row as included, excluded or dependent on a customer fixture. Confirm who supplies test identities, devices and application access. Confirm whether they will review the Conditional Access policy definitions, execute actual sign-ins, test the post-login application role and return a retest record. A service that only runs ScubaGear may be a useful baseline assessment, but its report should be priced and described as such. A human-led identity assessment should connect the settings to observed decisions.
The organization should keep control of the test window. Changes to a production Conditional Access policy can lock out legitimate users if handled carelessly. A good assessor begins with observation and approved synthetic accounts, identifies any proposed changes, and leaves policy activation to the owner's change process. CISA itself recommends hands-on testing before switching a policy from Report-only to On. That is a sound operational boundary to preserve during the engagement.
The report should identify its applicability date. The baseline lives in an actively maintained repository and policy identifiers can change. If the buyer later quotes the report in a customer security review, it should be able to say which baseline revision was examined and when the behavior was tested. A timeless sentence claiming “SCuBA compliant” is too vague to carry the evidence.
Where Stingrai fits
Stingrai is a proud CREST-accredited penetration testing service provider. Its cloud penetration testing service expressly includes Microsoft Entra application registrations, service principals, consent grants and Conditional Access gaps. For an enterprise identity assessment, human-led testers can work from the tenant's documented access policy, test approved identity contexts and produce a report that separates settings, observed decisions and remediation evidence. The engagement can be scoped as a one-time assessment or a continuous program where policy and application changes warrant repeated checks.
The buyer should still decide which policy is correct for its users and resources. Stingrai's role is to test whether the configured system enforces that policy, describe a reproducible gap and retest a fix. The resulting report can support a compliance program and supply evidence for an audit without overstating what a configuration check or a single sign-in proves. Share the protected applications, identity cohorts and policy inventory through Get a Quote.
How to cite this matrix
Citation: Arafat Afzalzada, “Microsoft Entra Conditional Access Validation Matrix: SCuBA Checks and Test Evidence,” Stingrai, 16 October 2026. The table is an original buyer-facing test design. CISA's Microsoft Entra baseline remains the authority for its policy statements and ScubaGear's documented check boundary.
Frequently Asked Questions
Does a passing ScubaGear result prove Conditional Access protects every user?
No. CISA says ScubaGear does not assess all Conditional Access condition values. A passing configuration result should be paired with controlled sign-in decisions for the user, device, client and resource contexts that matter to the organization.
What does this SCuBA validation matrix add to the CISA baseline?
It adds proposed test cases and evidence fields for a buyer to verify how its own policy behaves. The matrix is Stingrai's editorial synthesis, not a CISA control set or a mandatory CISA assessment method.
Which CISA Entra policies anchor the first tests?
The matrix starts with MS.AAD.1.1v1 for legacy authentication, MS.AAD.3.1v1 for phishing-resistant MFA and MS.AAD.3.2v2 for MFA coverage. Confirm the currently published policy IDs and versions before using them in a new engagement.
Is a Report-only Conditional Access policy enforced?
No. CISA states that a Conditional Access policy is enforced only when set to On. A Report-only result is useful for planning, but the final evidence should show the authorized policy state and observed enforced decision.
Why test a permitted sign-in as well as a denied one?
A permitted control confirms the test identity, device and application work as expected. The denied case then changes one relevant condition, making it possible to attribute the decision to the intended policy rather than to a broken account or unavailable application.
Can the tester use real employee accounts and records?
Synthetic accounts and records are the cleaner starting point. The customer should approve every identity, resource and sign-in context, and the evidence package should redact sensitive identifiers, tokens and tenant data.
What should the buyer ask for in the final report?
Ask for the applicable baseline revision, policy state, test cohort, target resource, expected decision, observed sign-in and application result, redacted event correlation, limitations and a retest decision for each material gap.
Does Stingrai's testing replace the organization's access policy decision?
No. The organization sets the required access behavior and owns exceptions. Stingrai can test the approved policy in an authorized human-led cloud identity engagement and document whether the observed decisions meet it.
Related reading
References
CISA. Microsoft Entra ID Secure Configuration Baseline. Accessed 16 October 2026. https://github.com/cisagov/ScubaGear/blob/main/PowerShell/ScubaGear/baselines/aad.md. Source policy text, check types and Conditional Access limitations.
CISA. ScubaGear. Accessed 16 October 2026. https://github.com/cisagov/ScubaGear. CISA's tenant-configuration assessment tool.
Stingrai. Cloud Penetration Testing. Accessed 16 October 2026. https://www.stingrai.io/services/cloud-penetration-testing. Published cloud and Microsoft Entra testing scope.
Talk to Stingrai
Bring the policies, applications and synthetic identity paths you want tested to Get a Quote, or book a free discovery call to define the evidence and retest scope.
