How to Monitor Conditional Access Policy Changes

As the COVID-19 situation happened, remote work culture stepped in, so Microsoft strengthened the Conditional Access policies in Azure AD for double-checking security. As part of the Azure AD Premium license, the Azure AD Conditional Access policy gives enterprises better control over corporate applications and systems.

Conditional access policies allow IT admins to define and enforce policies for all the incoming signals and ensure it meets the level-set requirements to access Office 365 resources.

• The policies can be based on many factors, such as who the user is, where the sign-in should take place, what they can do, etc.

One of the most important things is to understand CA policies rightly and ensure they are configured correctly. Otherwise, it will drastically impact your Microsoft 365 work environment. (It’s like digging your own graveyard, like the below meme 😶‍🌫️)

These conditional access policies are considered a big boon to Azure AD services and a key driving force behind the Office 365 zero-trust architecture. Therefore, it is important to often catch insights on the conditional access policy changes and actions taken by them around your organization.

Rightly utilizing the ‘conditional access policy’ tool can have a great deal of impact. But what if a minor misconfiguration locks out the organization or a subset of users inaccessible? 😱 That seems like a pretty risky proposition, doesn’t it?

Therefore, to witness the successive results, we must keenly monitor CA policy strategies and keep tabs on them. Fortunately, Microsoft has provided a handful of ways to monitor conditional access policy changes. Now, let’s look at the different ways to spy on actions triggered by CA policies.

Note: Listed here are reports of varying levels of complexity.

1. Analyze Sign-in behaviors due to CA policies using Azure AD Sign-in logs.

2. Discover What’s Happening in CA policies using Audit logs.

3. Use Azure AD Conditional Access insights and reporting workbook.

4. Get CA policies using Azure log analytics query auditing.

First, before getting started to hunt through the logs, ensure that you comply with the prerequisites for viewing the Azure sign-in logs. Follow the below steps to view the sign-in logs along with the conditional access details.

Microsoft Azure → Azure Active Directory → Security → Conditional Access → Sign-in logs (Under ‘Monitoring’ tab)

Now, filter by ‘Conditional Access’ and set the status (Success/Failure) as required. Administrators can use this report to troubleshoot sign-in issues and evaluate the security posture of their tenants.

On a final note, this report will assist you in obtaining the overall details, as well as the failure reason, in the mentioned contexts.

• Sign-ins passed through Conditional Access successfully.
• User Sign-ins Successfully Blocked by Conditional Access Policy.
• Users Failed to Fulfill the Conditional Access Requirement during Sign-in Process.

This log is your essential resource for tracking the detailed implementation of web content filtering for Microsoft Entra Internet Access via CA policies. By examining this log, you can acquire valuable insights into how the policy shapes and regulates users’ website access.

Let’s say you have multiple CA policy administrators and someone updates or deletes a setting; then how will you find out who made the change? You will have a hard time identifying who posed such modifications, and in the meantime, the work environment will have deteriorated completely!

So, do we have any way to find out the conditional access policy changes that happened in your workspace? Yes, we do. A thief always leaves a footprint! Hence, Conditional Access Audit Logs allow you to determine who caused a security disaster among your tenants. Follow the path below to further fine & tune further conditional access reports.

Microsoft Azure → Azure Active Directory → Security → Conditional Access → Audit logs (Under ‘Monitoring’ tab)

This report helps to find all the changes made to the CA policies configured within your organization. Here is a brief summary of what is logged in the audit logs of CA policies.

• Add conditional access policy – Lists who created a new policy.
• Update conditional access policy – Shows who updated which CA policy and what value was updated.
• Evaluate conditional access policy – Lists the evaluations carried out on CA policies.
• Delete conditional access policy – Lists who deleted which policy and when.

Hereafter, no more escapes from doing suspicious activities! You can also use Microsoft365DSC to compare tenant settings and spot Conditional Access settings changes.

Microsoft introduced a dashboard dedicated to conditional access policies called Conditional Access insights and reporting. It is an effective solution to analyze and monitor the impact of conditional access policies with a visualized frame. Additionally, admins can use the Conditional Access What If tool to simulate policy effects before enforcing them, ensuring there are no unintended consequences.

Note: Your organization should have an Azure subscription to view the log analytics workbook.

First, you must set up an Azure monitor to stream data to this workbook! For the workbook to load, you need to integrate Azure AD logs with Azure Monitor logs. The steps below should be completed before viewing the workbook.

1. Create a Log Analytics workspace in Azure Monitor.

2. Integrate Azure AD logs with Azure Monitor logs.

Scratching your head on what this workbook provides than other sign-in logs? No worries, let’s look at the answers to your query now!

1. Considering that an organization has multiple conditional access policies, this allows us to examine the impact of an individual policy or a subset of all policies.

2. Admins can view the impact of a policy by time, set of applications, and users.

3. Reports have information per condition [success, failure, not applied], so it is possible to catch on to the impact of each policy.

Navigate to the below path to view the conditional access insights and reporting workbook.

Microsoft Azure → Azure Active Directory → Security → Conditional Access → Insights and reporting.

Added to this Entra pre-built workbook, you can also make use of the insight rich Conditional Access gap analyzer workbook, which will help you to spot missed configurations like named locations, risky sign-ins, and others easily. You can also to edit the existing workbook or create a new Entra workbook for Conditional Access reporting in Entra ID.

With log analytics, you can run long queries against Azure monitor logs and analyze their results via the Azure portal. This can be used to find records that meet specific standards, spot trends, evaluate patterns, and offer a wide array of insights.

Querying allows you to granularly apply complex combinations and discover various insights about conditional access policies.

Navigate to the below path to run KQL (Kusto Query Language) and retrieve various CA reports:

Microsoft Azure → Azure Active Directory → Log Analytics (Under the ‘Monitoring’ tab)

Here, I’ve attached a KQL query that will create a pivot table showing all the conditional access policy outcomes over the last 30 days.

Didn’t we make it easy now? 🤗So, the rest is in your hands! Make tweaks and twists as needed and generate more granular reports you desire!

Your organization has been alarmed by a sudden threatening activity, but the admin has no idea about it! Because the suspicious activity gets logged in audit logs but isn’t notified to the admin.

Why does it always work this way? Are there any possible ways to get notifications for any conditional access policy events?

Yes, there are multiple ways within Office 365. The only downside is that you have to loop through Log Analytics, Defender for Cloud Apps, Azure logic apps, and set alerts for any activity. Therefore, it is hard for administrators to stay updated on conditional access policies.

So, to ease the load, try using AdminDroid, an Office 365 Reporter tool that lets you set alerts and schedule all the reports all at once.

AdminDroid includes a dedicated section of reports for conditional access analytics, which provides you with an overall view of all CA actions. With these reports, it is possible to set alerts and get notified about the sign-ins blocked/allowed by the conditional access policy, policy assignment details, etc. Everything is in one place!

👋Say adieu to searching for various reports in several places and get complete information in one place.

AdminDroid is an intuitive Office 365 reporting, auditing, management, alerting, and monitoring tool. Free Office 365 Reporting tool by AdminDroid is a web-based analytical tool that provides free reports on Azure AD management like group memberships, group membership changes, users, licenses, user logins, password changes, etc. There are over 100 reports and dashboards in the free edition. AdminDroid provides you with over 1600 all-in-one reports for Exchange Online, SharePoint Online, Microsoft Teams, OneDrive for Business, and every other Office 365 service in an easy-to-use interface with sleek graphical dashboards.

Download a 15-day free trial and start exploring its features!

On a final note, there you go! Here are all possible ways in which administrators can obtain conditional access reports in Office 365. And along with this, I would like to suggest the 5 must-know device-based conditional access policies that my fellow admins can make use of it. These policies have been serving me a lot!

Finally, now is the time to start monitoring your Office 365 user access and prevent suspicious attacks threatening your Office 365 workspace. Hope this blog helped you understand different ways to analyze and monitor conditional policies. Further, if you have any ideas to monitor, do share them in the comment section!