New Risk-based Access Policies Workbook in Entra ID

As an admin, you’re no stranger to the challenges posed by password spray attacks, anonymous browser logins, leaked credentials, and more. Unfortunately, these threats only seem to be on the rise, rather than down! Luckily, Microsoft has a game-changing solution: the “Risk-based Conditional Access policy in Entra ID.” You can easily configure risk-based CA policies in Microsoft 365 based on the risk level and give secure access control!

But that’s not a thing! Admins need to assess the impact of the risk-based policies in Microsoft 365 before implementing them. This process involves creating the policy, setting it to report-only mode, monitoring its effects, and deleting it if the impact isn’t significant. That’s take a lot of time!

Fortunately, the new Entra workbook – “Impact analysis of risk-based access policies” comes in! You can easily analyze risk policies’ impacts without having or creating such policies in your tenant. Let’s see that in detail, after having a short description of risk-based policies.

What are Risk-based Conditional Access Policies?

To know about risk-based CA policies, you need to know two essential features in Microsoft Entra Identity protection – User risk and sign-in risk.

User risk– A user risk is risk detected when there is potential account compromise, leaked credentials, etc.,
Sign-in risk – These risks are detected when users try to authenticate from malicious IP addresses, geographically distant locations, and suspicious browsers.

To address these risks, Microsoft 365 offers a must-needed Entra ID security setting – Risk-based Conditional Access policies. These policies allow organizations to implement security measures based on the detected user and sign-in risk level.

To facilitate the analysis and implementation of risk-based access policies, the Entra workbook comes in!

What are Entra Workbooks?

Microsoft Entra workbooks serve as tools for generating personalized interactive dashboards based on Entra ID sign-in and audit logs. These workbooks enable users to create visually compelling reports and charts, offering a comprehensive overview of critical Entra data.

Microsoft offers many ready-to use workbook templates in Entra ID, including

One particularly valuable workbook, especially in assessing the effectiveness of risk-based policies, is the “Impact analysis of risk-based Conditional Access policies”.

What is Risk-based Access Policies Impact Analysis Workbook?

The impact analysis of the risk-based Conditional Access workbook shows what will happen if you apply risk-based policies. These policies include multi-factor authentication (MFA), blocking sign-ins, or changing passwords. So, this workbook helps you identify gaps and effects in your planned policies before implementing them.

This workbook enables you to identify users and sign-ins that will be impacted by your planned Conditional Access policy with ease. Once you audit the impact, you can enable the new risk-based CA policy with confidence. Let’s see how to use this workbook now.

Prerequisite: To access this workbook, you should configure the diagnostic setting to send SignInLogs to your Log Analytics workspace.

Once done, you can view this workbook to monitor Conditional Access policy impacts by using the steps below.

Log in to Microsoft Entra admin center.
Navigate to the Monitoring & health section and select “Workbooks” options.
Select the “Impact analysis of risk-based access policies” workbook from the ID Protection section.

This workbook contains two main sections. Let’s see them in detail.

Summary of Risky Users Impacted by Risk Policies

This “Impact summary” section provides an overview of the potential impact on users, and trusted IP addresses if risk-based policies are enabled in the tenant. This is sub-divided into two four sections.

User risk scenarios
Sign-in risk & trusted network scenarios
Federated sign-in risk scenarios
Legacy Identity Protection policies

1. Monitor Risky Users in Microsoft 365

In this section, you can get all the necessary details that will drive you whether to implement the planned user risk-based Azure Conditional Access policies or not. Here are the insights you can get.

High-risk users not being blocked: This metric provides the total number of users who have not been blocked by any of the risk-based Conditional Access policies. These high risks are detected due to anonymous IP addresses, leaked credentials, and more.

High-risk users not prompted for password change: Here, you can observe the total count of users who have not been prompted to change their passwords as a result of any risk-based policies.

User that changed password: This tile highlights the number of users who have changed their passwords due to being affected by risk-based CA policies, shedding light on the proactive measures taken.

High-risk users not successfully signing in: From here, you can gain insights into the users’ count for not successfully signing in due to risk-based access policies. This indicates the potential access challenges faced by users under the current policy framework.

User risk remediated by on-premises password reset: The count of risky users remediated by prompting for on-premises password reset.

User risk remediated by password reset: Total count of all users remediated by password resets due to applied risk-based CA policies.

You can also use the “Time Range” parameter to get the risk-based CA policy impacts over a specific period. This filter applies to all the sections of the workbook. But you can tweak the workbook to set section-based filters.

What to do after analyzing the results of the above section?

Ensure multi-factor authentication is required for risky sign-ins (By observing case 1).
Risk-based access policy prompting for password resets (By observing case 2).

This section provides a summary of various risky sign-in insights that can assist you in refining your planned sign-in risk Conditional Access policies and enabling them.

Sign-in risks: Like the above, you can get the total count of users with high-risk sign-ins that are,

Not blocked by risk-based policy
Not prompted for MFA
Remediated by prompting for MFA
Not successfully signed-in

IP address not trusted: This tile provides the count of unique IP addresses categorized as untrusted (IP location that’s not included in trusted location is considered as untrusted), from which multiple users have signed into Microsoft 365. This might impact sign-in risk in Microsoft 365.

What will you do after analyzing the sign-in risks insights?

Configure sign-in risk-based MFA, password resets using Conditional Access.
When a user signs in from an IP with a “not trusted” user, additional verification steps like two-factor authentication can be enforced via risk-based policies.

Microsoft allows federated sign-in to access M365 resources using login credentials from trusted external providers like Google, Facebook, or others. From this section, you can analyze federated sign-in risk impacts. Here are the two tiles available.

Count of Federated Risky Sign-in Users Redirected for MFA: This metric indicates the number of users with risky sign-in behavior who are redirected to their external identity provider for MFA. It highlights instances where additional authentication measures are triggered as part of federated sign-in processes to mitigate security risks.
Count of Users Remediated by MFA from External Provider: Gain insights into the number of users with sign-in risks who have been successfully remediated by completing MFA through their external identity provider.

4. Get Risky Sign-ins Report Impacted by Entra ID Protection Policies

Microsoft Entra offers a user-risk policy and sign-in risk policy. You can enable these Azure AD identity protection policies for risky users and sign-ins and take control like password changes, allow/block access, etc.,

But Microsoft announced that these policies are being retired and replaced with Conditional Access policies, which offer more granular control and functionality.

So, from this section, you can easily

Identify the count of risky users that are impacted by Entra ID protection policies
Get the count of users with high-risk sign ins that are affected by legacy risk-based policies

Takeaway: Move to Risk-based Access policies

This helps us understand how well the legacy Entra ID protection policies help to identify and stop risky users. It further helps you to,

Enable Conditional Access based user risk policy for password change
Enable sign-in risk policy for MFA with Conditional Access

View and Investigate Risky Users in Microsoft 365

This “Impact details” section offers a drilled down detail of the “Impact summary” section. Let’s delve into this section.

In the previous section, you can get only the total count of users/sign-ins. But this ‘Impact details’ section gives you a detailed report of all the counts you’ve witnessed in the previous sections. For example, you can,

Identify high risk users without MFA
Lists the risky user sign-ins resolved after a secure password change
Users impacted by Azure AD Identity Protection user risk policies
Users Blocked due to Risk-based Access policies
Federated sign-in users remediated with password reset and more.

These reports not only identify users but also provide additional details such as their risk level, accessed applications, device status, and more. This granular information enables a deeper understanding of user behaviors and policy effectiveness, facilitating informed decision-making regarding risk policy implementation.

Create or Edit New Workbook in Entra ID

If you find that the impact analysis of risk-based access policies workbook doesn’t cover all the insights you require, Microsoft offers flexibility for customization. You can tailor the risk-based access workbook to better suit your needs. This customization can range from simple adjustments to completely personalized workbooks.

For instance, if you need to completely create your own Entra workbook, Microsoft provides five main components: text, query, links, group, and parameter. These components enable you to refine the workbook according to your specific requirements.

For example, suppose you want to export a list of all risky users in Microsoft 365 to implement a risk-based access policy with customized conditions based on user risk. In that case, you can utilize the Query component. Below is a sample Kusto Query for this requirement.

let risklevel = pack_array("high");  
SigninLogs  
| where RiskLevelAggregated in (risklevel) 
| project UserPrincipalName

Like Query component, you can also combine other components to make descriptions, workbook navigations, grouping similar insights together, etc.

Better Analyze and Implement with Risk-based Access Policies Workbook!

So, Entra workbook gives you what you need before implementing risk-based access policies. So, analyze it, configure risk-based CA policies with MFA, and password requirements accordingly and monitor risky sign-ins in Entra effectively. Granular access control and security will be achieved with a single workbook!

I hope this blog brings you more information about the impact analysis of Conditional Access policy workbook, which helps you monitor Conditional Access policies easily. If you have doubts or need clarification, please contact us in the comments section. We are here to assist! Also, stay tuned for the next workbook in the series!