Monitoring Azure AD Sign-in Logs and Risky Sign-In Activities

On Day 20 of Cybersecurity awareness month, learn to safeguard risky users from threats and attacks by monitoring Azure AD sign-ins in Office 365. Stay tuned for more blogs in the Office 365 Cybersecurity blog series.

There are multiple ways for attackers to gain access to your Microsoft 365 organization. Frequent login failures and disabled account login attempts may indicate that your organization is prone to cyber threats. Isn’t it suspicious when a login attempt occurs from an unusual location rather than the user’s current location?

“Only the insecure strive for Security”

– Wayne Dyer

Absolutely! However, proactive and constant monitoring of Azure AD Sign-ins has the capability of preventing severe security risks like password spray attacks, logging in with malicious IP addresses, and more such threats. Also, admins should have an up-to-date authentication mechanism and knowledge about recent security attacks.

1. To access the sign-in logs, admins should be assigned with any one of the following roles.

• Global Administrator
• Security Admin
• Security Reader
• Global Reader
• Reports Reader

2. The sign-in activity report in the AAD admin center is available in all the editions of Azure AD. However, you’ll require an AAD P1 or P2 license if you wish to access the report through Microsoft Graph API.

3. Users can also view their sign-ins through https://mysignins.microsoft.com/.

By reviewing Azure AD sign-in logs, admins can keep an eye and take required actions on the users’ sign-ins with the help of the below critical information that the report provides.

• The sign-in pattern of the user.
• User sign-in location.
• Signed-in device details.
• Identify users’ sign-ins using legacy protocols.
• Sign-in that passes MFA challenges.
• Client app that uses an access token to sign-in
• Any sign-in occurred from disabled accounts or expired passwords.
• The number of users, apps or services that were signed in recently.
• Is there any conditional access defined? If yes, the status of the conditional access.
• The authentication methods used for sign-ins, and more such details.

To reduce the burden for admins, Microsoft has classified the O365 sign-in reports as follows.

Interactive sign-ins – Sign-ins performed by users where the authentication factors such as passwords, passing MFA challenges, biometric factors, or QR codes are used.

Non – Interactive sign-ins – Sign-ins performed by the client apps on behalf of the users without the need of any authentication factors or interaction from users

Service principal sign-ins – Sign-ins performed by the applications or service principal using its own certificate or client app secret for accessing the resources.

Managed identity sign-ins – Sign-ins performed by the resources that have their secrets managed by Azure in the Key Vault service.

However, other sign-ins except interactive sign-ins are currently in preview mode and will generally be available soon.

Non-interactive sign-ins play a major role in the organization’s malicious security attacks. This sign-in requires neither a username nor a password. Instead, an OAuth authentication code or MFA token is used for authorization.

Also, security spoilers perform phishing attacks by stealing passwords that are hard-coded in shared scripts or files and compromise security within minutes.

So, it is advisable to check the non-interactive sign-ins in the Azure Active Directory to minimize the blind spot in Office 365. To make it easier for admins, Microsoft groups and shows all the non-interactive logins when attempted from the same client application.

Admins can expand the rows to see all the different sign-ins and their different time stamps.

Besides reviewing the non-interactive sign-ins, it is also vital to keep track of potential risky sign-ins and take necessary actions.

An Office 365 user who failed to pass MFA, configured basic authentication, logged in with an expired password, and more is considered a risky sign-in user. The attacker steals the login credentials of those users and attempts to log in with the stolen data. This makes the sign-in process riskier and more malicious to the organization.

“Understand what data you hold, how you are using it, and make sure that you are practicing good data hygiene.”

Thus, monitoring risky sign-in detections is very important for admins to protect their organizations from severe threats. To view the risky sign-ins in Azure AD, admins should

• Investigate Risks and Risky sign-in users
• Categorize the risky sign-ins based on Risk event types
• Automate the risk detection and protection

To help admins, Azure Active Directory provides 3 key reports to analyze the severity of attacks and determine how to respond to the risk and future threats. Admins can navigate to Azure Active Directory > Security > Identity Protection to get the following reports.

1. Risky users Report

This report provides info such as risky users, risk levels, details of detections, and so on. This report lets admins take action such as resetting the user password, dismissing user risk, blocking user sign-ins, etc.

2. Risky sign-ins Report

MFA details, device details, location info, application information, and more will be available in this report. This report contains filterable data for up to 30 days. Admins can let users to confirm the sign-in compromise and confirm sign-in safe using this report.

3. Risk detections Report

Risks based on risk event types, sign-in attempt location, and more are listed under this report. It consists of risk detections for up to 90 days. Admins can choose to take action as suggested for the above reports in a more detailed way.

Each report above allows additions and removals of columns based on admins’ reporting requirements.

Risks can be real-time or offline. Therefore, admins should know the way to check all types of risks based on risk events to better understand and identify the weak points and safeguard the organization. Some crucial risk events that cause severe impact are,

• Impossible travel – It occurs when logging in to Office 365 from distant locations within a small interval of time.
• Password Spray – This attack happens when an attacker attempts to guess a user’s password multiple times in a short period.
• Malware linked IP address – It depicts that a user sign-in with a malicious IP address.
• Leaked Credentials – It occurs due to the theft of user credentials.
• Unfamiliar sign-in properties – This risk is detected when a sign-in occurs from a different and unfamiliar sign-in property in azure ad compared to the previously saved sign-in history.

Admins can view all the risk sign-in detections based on risk event types from their Azure AD Identity Protection portal. However, some risk event types are marked as the premium detections and will be available for only Azure AD Premium P2 users.

Users without Azure AD P2 license would still receive the premium detections and are titled as “additional risk detected” that can’t be viewed.

One step further, admins can enable two types of risk policies to automate the risk response and allow users to self-remediate upon detecting any risk. They are,

1. User Risk Policy – This policy requires a secure password reset when the user risk level is High. It will ask for MFA before creating a new password with SSPR to remediate the risk.

2. Sign-in Risk Policy – This policy requires users to prove that he is legitimate users using any one of their authentication methods (Azure AD MFA) when a user’s sign-in risk level is Medium or High.

Additionally, admins can create custom user risk and sign-in risk policies using the Azure AD Conditional Access policies. With these risk-based access policies, admins can restrict access to vulnerable and compromised devices and protect data inside applications.

Though native sign-in reports include sign-in details, you will get only comprehensive reports and can’t foresee the exact info such as unmanaged device logins, suspicious location attempts, and so on.

With AdminDroid’s Azure AD sign-in analytics tool, managing your Microsoft 365 environment is a breeze! It reforms your ability to monitor sign-ins, offering detailed reports on sign-in devices, sign-in locations & details, etc.

So, let’s delve into some key statistics and see how AdminDroid can help streamline your Azure AD sign-in analysis:

Risky Sign-ins Reports: AdminDroid aids in detecting potential security breaches by generating reports that contain information on risk levels, risk confirmed time, risk event types, and users involved in risky sign-ins. These are further categorized based on factors like risk detection timing, status, and resolution methods.

Detailed Sign-in Reports: Gain granular insight into user sign-ins with metrics like location, application, device details, authentication protocol, and more. These reports also cover crucial scenarios including first-time sign-ins, failed sign-in attempts, guest sign-ins, and sign-ins from external tenants.

M365 Sign-in Summary Reports: These reports offer a complete snapshot of individual users and organization-wide sign-ins across all applications. They help in identifying unusual activities and a sudden spike in login failures by tracking the count of successful sign-ins, failures, and interruptions, along with the last logon info.

Beyond the sign-in reports, AdminDroid provides an array of other reports like device sign-ins, MFA analysis, and more, ensuring you have all the insights you need for your Azure AD sign-in logs.

What Can You Gain from AdminDroid’s Free Azure AD Reporting Tool?

AdminDroid’s free Azure AD login reports encapsulate all aspects of user logins – successful attempts, failed attempts, and user login summary reports. You can customize these audit reports with filters and save them as “Views” for later use.

In addition to this, AdminDroid offers more than 100 free Azure AD reports, enabling you to track users, licenses, subscriptions, groups, applications, directory details, MFA, CA policies, etc.

AdminDroid’s Azure AD management tool enhances your Azure AD administration, making it more efficient and effective than ever before.

Explore AdminDroid now and elevate your Microsoft 365 environment to the next level.

Even though we have created several risk policies using Microsoft 365, these policies will not be effective until a security breach is detected. So, admins should precisely monitor the O365 user accesses and their activities to avoid harmful security attacks.

Ultimately, security is what matters most!

I hope this blog will help you to know how to analyze Azure AD sign-ins and remediate risk sign-ins using Azure AD. Furthermore, for visually engaging Entra sign-in logs, you can go for creating a new workbook in Entra ID. Feel free to post your thoughts and suggestions in the comment section.