Microsoft Azure
Microsoft Azure is a complete cloud platform with infrastructure, software, and applications available as services. Azure can complement an on-premises infrastructure as an extension of your organization’s technical assets. When using Azure in your environment, whether you opt for the cloud or on-premises option, security and monitoring are still an essential part of your daily operations.
To provide flexibility and customer choice in security operations, Microsoft offers Azure Event Hubs as a centralized service to collect data and logs from other Azure services. You can integrate InsightIDR with Azure Event Hubs to access and ingest all applicable Azure logs and data. This combines Microsoft’s data ingestion service with the incident detection and response system of InsightIDR.
When you configure Azure Event Hubs and consume data and logs through the Microsoft Azure event source, InsightIDR will:
- Parse Microsoft Defender for Cloud events as third-party alert detections.
- Parse Azure Active Directory events to offer ingress authentication, single sign-on (SSO), cloud service activity, and cloud service admin activity detections.
- Parse Microsoft Defender for Endpoint Advanced hunting events as third-party alert detections.
Azure detections trigger legacy detection rules in InsightIDR
Because user behaviors are monitored from the event sources and Insight Agents deployed in your environment, InsightIDR continues to offer and track additional Azure detections over time. Legacy detection rules (formerly known as User Behavior Detection Rules) treat Azure Cloud Services like an extension of your environment.
New Azure alerts for Exchange and SharePoint audit logs
InsightIDR can now produce alerts for Microsoft’s exchange and SharePoint audit logs. These alerts generate cloud service activity and ingress authentication events. Read more about the event log format here: https://learn.microsoft.com/en-gb/microsoft-365/compliance/audit-log-search?view=o365-worldwide.
To set up Microsoft Azure:
- Complete the prerequisite steps.
- Configure Microsoft Azure Hub to:
- Communicate with InsightIDR.
- Send data to InsightIDR.
- Configure InsightIDR to receive data from the event source.
- Test the configuration.
- Troubleshoot common issues.
Requirements
To successfully configure the Microsoft Azure event source, you must:
- Have a Premium P1 or Premium P2 license for Azure Active Directory.
- Have a license for Azure Monitor, Microsoft Defender for Cloud, or Microsoft Defender for Endpoint, depending on which data you want to send to InsightIDR.
- Have access to a standard pricing tier event hub or higher. The pricing tier selected will also affect the Retention time (hrs).
- Allocate and open an outbound connection over TCP port 9093 on the InsightIDR Collector. If you do not open this port, your event source configuration will fail.
- Ensure that InsightIDR has an adequate data ingestion rate. Your throughput must equal the number of partitions. Read more about Event Hub Scalability here: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-scalability#number-of-partitions.
Restrictions when editing configured Event Hubs
To edit configured Event Hubs, you must have a premium tier event hub. Read more about the premium tier event hub here: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-premium-overview. You can find more information about the event hub tiers here: https://learn.microsoft.com/en-us/azure/event-hubs/compare-tiers.
Configure Microsoft Azure Event Hub to send data to InsightIDR
To establish communication between InsightIDR and Microsoft Azure, you need to complete two phases consisting of several tasks.
Configure a Microsoft Azure Event Hub
To enable communication between Microsoft Azure and InsightIDR, you must first create an Event Hub.
Task 1: Create a new Event Hub
To send the right information to InsightIDR, you must create a new Azure Event Hub, named insights-operational-logs. You can configure the Microsoft Azure event source only if you have access to the standard tier subscription or above.
To create a new Event Hub, follow Microsoft’s documentation at: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create.
Configure the Microsoft Azure Event Hub to communicate with InsightIDR
With the Microsoft Azure event source, you can send logs from multiple Microsoft Azure products. Depending on which product you want to use, ensure that you:
Configure the Azure Monitor
You must configure the Azure Monitor to send its logs to the Event Hub. During configuration, ensure that you:
- Select the Administrative, Security, and Alert checkboxes, at a minimum.
- Select the Stream to an Event Hub checkbox.
To configure the Azure Monitor, follow Microsoft’s documentation at: https://learn.microsoft.com/en-us/training/modules/configure-azure-monitor.
For more information, read Microsoft’s documentation at: https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/stream-monitoring-data-event-hubs.
Configure Azure Active Directory
You can configure Azure Active Directory to stream sign-in and audit events to the Event Hub for ingestion into InsightIDR. During configuration, ensure that you:
- Select the subscription that you specified earlier.
- Select the Event Hub namespace that you created earlier, for example insights-operational-logs.
- Select the
RootManageSharedAccessKey
policy name.
To configure Azure Active Directory, follow Microsoft's documentation at: https://docs.microsoft.com/en-us/azure/active-directory/reports-monitoring/tutorial-azure-monitor-stream-logs-to-event-hub#stream-logs-to-an-event-hub.
Configure Microsoft Defender for Cloud
You can configure Microsoft Defender for Cloud to send its logs to the Event Hub.
To configure Microsoft Defender for Cloud, follow Microsoft's documentation at: https://learn.microsoft.com/en-us/azure/defender-for-cloud/continuous-export?tabs=azure-portal#set-up-a-continuous-export.
For more information, read Microsoft’s documentation at: https://docs.microsoft.com/en-us/azure/defender-for-cloud/continuous-export?tabs=azure-portal.
Configure Microsoft Defender for Endpoint
You can configure Microsoft Defender for Endpoint to send its logs to the Event Hub.
To configure Microsoft Defender for Endpoint, follow Microsoft's documentation at: https://learn.microsoft.com/en-us/microsoft-365/security/defender-endpoint/raw-data-export-event-hub?view=o365-worldwide.
Configure InsightIDR to collect data from the event source
Once you have created a Microsoft Azure Event Hub and configured the data that you want to send to InsightIDR, you can set up the Microsoft Azure event source.
To configure the new event source in InsightIDR:
- From the left menu, go to Data Collection and click Setup Event Source > Add Event Source.
- Do one of the following:
- Search for Microsoft Azure in the event sources search bar.
- In the Product Type filter, select Cloud Service.
- Select the Microsoft Azure event source tile.
- Name the event source. This name will be used to name the log that contains the event data in Log Search.
- Select a collector.
- Optionally, choose to send unparsed logs.
- Select your LDAP account attribution preference.
- In the Event Hub Name field, enter the name of the event hub as it is displayed in the Azure portal. Note that you'll need the name only, not the namespace.
- In the Endpoint field, enter the URL of the event hub namespace (excluding the protocol and the trailing slash).
- Under Credential, select Create New.
- Give the new credential a name that clearly identifies it.
- Enter the SharedAccessKeyName and Shared Access Key in their respective fields
- Click Save.
Test the configuration
To test that event data is flowing into InsightIDR through the Collector:
- Verify that data is flowing to the Collector:
- From the Data Collection Management page, click the Event Sources tab.
- Find the event source you created and click View raw log. If the Raw Logs modal displays raw log entries, logs are successfully flowing to the Collector.
- Wait approximately seven minutes, then open the Log Search page in InsightIDR.
- Verify that log entries are appearing in Log Search:
- From the left menu, go to Log Search.
- Select the applicable Log Sets and the Log Names within them. The Log Name is the event source name. The EventSource logs flow into these Log Sets:
- Ingress Authentication
- SSO Authentication
- Third-Party Alerts (Azure Security Alerts)
- Unparsed Data
- Cloud Service Activity
- Cloud Service Admin Activity
- Set the time range to Last 10 minutes, and click Run.
The Results table displays all log entries that flowed into InsightIDR in the last 10 minutes. The keys and values that are displayed are helpful to know when you want to build a query and search your logs.
Some log formats are incompatible
If you see raw log entries when you select View raw log, but you do not see any log entries in Log Search, then your logs do not match the recommended format and type for this event source.
Sample logs
There are multiple ways to generate sample audit events in Azure to send to the Event Hub:
- Start and stop virtual machines. If you have a test or spare virtual machine, you can generate sample audit events by starting and stopping those machines.
- List shared access policies. Open the Event Hub Namespace. Under Settings, select Shared Access Policies for
RootManageSharedAccessKey
.
It might take several minutes for events to be available in InsightIDR.
Troubleshoot common issues
This section covers some common troubleshooting scenarios.
A connection has been established, but no data is flowing to IDR
If a connection has been established, but there is no data flowing to InsightIDR, verify that you are logged into the correct Event Hub Topic.
There is an error in the connection
If there is an error in the connection, check the following:
- Verify that you have selected the Standard tier for Azure Security Center, as stated in the Requirements.
- Verify that you are logged into the correct Event Hub Instance.
- Check your firewall to verify that you have configured an outbound connection over TCP port 9093 on your InsightIDR Collector.
- Check your credentials. Ensure that you are using the Connection String Primary Key and the correct connection string as described in Configure a Microsoft Azure Event Hub.
Invalid SASL mechanism response error
If you are seeing an error that says Invalid SASL mechanism response, server may be expecting a different protocol
, update your Connection String Primary Key in InsightIDR. To do this, complete Task 3 of Configure a Microsoft Azure Event Hub to copy the key and Configure InsightIDR again.
Create or update activity log profilesFailure error
When configuring the Azure Monitor, you may try to save your changes but see an error on the top right of the UI saying Create or update activity log profilesFailure
.
To fix this error:
- Search for Subscriptions in all services.
- Select your subscription and click on Resource Providers in the left hand panel.
- Search for
microsoft.insights
. - Ensure that it is registered by clicking on either Register or Re-Register. Wait for the process to complete.
- Click Refresh.
- Repeat the steps in Configure the Azure Monitor to ensure the activity log saves without error.