Microsoft Azure

Microsoft Azure is a complete cloud platform with infrastructure, software, and applications available as services. Azure can complement an on-premises infrastructure as an extension of your organization’s technical assets. When using Azure in your environment, whether you opt for the cloud or on-premises option, security and monitoring are still an essential part of your daily operations.

To provide flexibility and customer choice in security operations, Microsoft offers Azure Event Hubs as a centralized service to collect data and logs from other Azure services. You can integrate InsightIDR with Azure Event Hubs to access and ingest all applicable Azure logs and data. This combines Microsoft’s data ingestion service with the incident detection and response system of InsightIDR.

When you configure Azure Event Hubs and consume data and logs through the Microsoft Azure event source, InsightIDR will:

  • Parse Microsoft Defender for Cloud events as third-party alert detections.
  • Parse Azure Active Directory events to offer ingress authentication, single sign-on (SSO), cloud service activity, and cloud service admin activity detections.
  • Parse Microsoft Defender for Endpoint Advanced hunting events as third-party alert detections.

Azure detections trigger legacy detection rules in InsightIDR

Because user behaviors are monitored from the event sources and Insight Agents deployed in your environment, InsightIDR continues to offer and track additional Azure detections over time. Legacy detection rules (formerly known as User Behavior Detection Rules) treat Azure Cloud Services like an extension of your environment.

New Azure alerts for Exchange and SharePoint audit logs

InsightIDR can now produce alerts for Microsoft’s exchange and SharePoint audit logs. These alerts generate cloud service activity and ingress authentication events. Read more about the event log format here: https://learn.microsoft.com/en-gb/microsoft-365/compliance/audit-log-search?view=o365-worldwide.

To set up Microsoft Azure:

  1. Complete the prerequisite steps.
  2. Configure Microsoft Azure Hub to:
    1. Communicate with InsightIDR.
    2. Send data to InsightIDR.
  3. Configure InsightIDR to receive data from the event source.
  4. Test the configuration.
  5. Troubleshoot common issues.

Requirements

To successfully configure the Microsoft Azure event source, you must:

  • Have a Premium P1 or Premium P2 license for Azure Active Directory.
  • Have a license for Azure Monitor, Microsoft Defender for Cloud, or Microsoft Defender for Endpoint, depending on which data you want to send to InsightIDR.
  • Have access to a standard pricing tier event hub or higher. The pricing tier selected will also affect the Retention time (hrs).
  • Allocate and open an outbound connection over TCP port 9093 on the InsightIDR Collector. If you do not open this port, your event source configuration will fail.
  • Ensure that InsightIDR has an adequate data ingestion rate. Your throughput must equal the number of partitions. Read more about Event Hub Scalability here: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-scalability#number-of-partitions.

Restrictions when editing configured Event Hubs

To edit configured Event Hubs, you must have a premium tier event hub. Read more about the premium tier event hub here: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-premium-overview. You can find more information about the event hub tiers here: https://learn.microsoft.com/en-us/azure/event-hubs/compare-tiers.

Configure Microsoft Azure Event Hub to send data to InsightIDR

To establish communication between InsightIDR and Microsoft Azure, you need to complete two phases consisting of several tasks.

Configure a Microsoft Azure Event Hub

To enable communication between Microsoft Azure and InsightIDR, you must first create an Event Hub.

Task 1: Create a new Event Hub

To send the right information to InsightIDR, you must create a new Azure Event Hub, named insights-operational-logs. You can configure the Microsoft Azure event source only if you have access to the standard tier subscription or above.

To create a new Event Hub, follow Microsoft’s documentation at: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create.

Task 2: Create a Shared Access Authorization Policy for the Event Hub Namespace

A Shared Access Authorization Policy allows InsightIDR to access the messages that Azure publishes to the Event Hub. When you create a Shared Access Authorization Policy, ensure that you:

  • Make a note of the policy name. We recommend a name such as R7InsightIDR, for example.
  • Grant your policy Send and Listen permissions.

To create a Shared Access Policy, follow Microsoft’s documentation at: https://docs.microsoft.com/en-us/azure/event-hubs/authorize-access-shared-access-signature#shared-access-authorization-policies.

Task 3: Copy the Shared Access Policy Key

To configure the event source in InsightIDR, you will need to copy or record the Connection String Primary Key policy key from the newly created Event Hub.

The connection string for a namespace contains these components:

  • Fully qualified domain name of the Event Hubs namespace you created
  • Name of the shared access key
  • Value of the shared access key

Each of the three components are presented as key-value pairs, and are separated by semi-colons:

Endpoint=sb://<NamespaceName>.servicebus.windows.net/;SharedAccessKeyName=<KeyName>;SharedAccessKey=<KeyValue>

For example, your Connection String Primary Key might look like this:

Endpoint=sb://rapid7idreventhub.servicebus.windows.net/;SharedAccessKeyName=foobar;SharedAccessKey=password

You must copy each value into its respective field on the InsightIDR event source configuration screen.

Note: You don't need to include the trailing forward slash or protocol when you enter the Endpoint URL.

To use the earlier example, you would enter the values in InsightIDR as follows:

  • Endpoint is rapid7idreventhub.servicebus.windows.net
  • AccessKeyName is foobar
  • SharedAccessKey is password

Important: The SharedAccessKey itself can often contain an equals sign (=) at the end of the string. This is part of the password and you must include it in the event source configuration screen.

For instructions on how to copy the policy key, follow Microsoft’s documentation at: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string#connection-string-for-a-specific-event-hub-in-a-namespace.

Configure the Microsoft Azure Event Hub to communicate with InsightIDR

With the Microsoft Azure event source, you can send logs from multiple Microsoft Azure products. Depending on which product you want to use, ensure that you:

Configure the Azure Activity Log

You must configure the Azure Monitor to send its logs to the Event Hub. During configuration, ensure that you:

  • Select the Administrative, Resource Health, and Policy checkboxes, at a minimum.
  • Select the Stream to an Event Hub checkbox.

Do not select the Security checkbox. To gather this data, follow the instructions in the Configure Microsoft Defender for Cloud section.

To configure the Azure Monitor, follow Microsoft’s documentation at: https://learn.microsoft.com/en-us/training/modules/configure-azure-monitor.

For more information, read Microsoft’s documentation at: https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/stream-monitoring-data-event-hubs.

Configure Azure Active Directory

You can configure Azure Active Directory to stream sign-in and audit events to the Event Hub for ingestion into InsightIDR. During configuration, ensure that you:

  • Select the subscription that you specified earlier.
  • Select the Event Hub namespace that you created earlier, for example insights-operational-logs.
  • Select the RootManageSharedAccessKey policy name.

To configure Azure Active Directory, follow Microsoft's documentation at: https://docs.microsoft.com/en-us/azure/active-directory/reports-monitoring/tutorial-azure-monitor-stream-logs-to-event-hub#stream-logs-to-an-event-hub.

Configure Microsoft Defender for Cloud

You can configure Microsoft Defender for Cloud to send its logs to the Event Hub.

To configure Microsoft Defender for Cloud, follow Microsoft's documentation at: https://learn.microsoft.com/en-us/azure/defender-for-cloud/continuous-export?tabs=azure-portal#set-up-a-continuous-export.

For more information, read Microsoft’s documentation at: https://docs.microsoft.com/en-us/azure/defender-for-cloud/continuous-export?tabs=azure-portal.

Configure Microsoft Defender for Endpoint

You can configure Microsoft Defender for Endpoint to send its logs to the Event Hub.

To configure Microsoft Defender for Endpoint, follow Microsoft's documentation at: https://learn.microsoft.com/en-us/microsoft-365/security/defender-endpoint/raw-data-export-event-hub?view=o365-worldwide.

Configure InsightIDR to collect data from the event source

Once you have created a Microsoft Azure Event Hub and configured the data that you want to send to InsightIDR, you can set up the Microsoft Azure event source.

To configure the new event source in InsightIDR:

  1. From the left menu, go to Data Collection and click Setup Event Source > Add Event Source.
  2. Do one of the following:
    • Search for Microsoft Azure in the event sources search bar.
    • In the Product Type filter, select Cloud Service.
  3. Select the Microsoft Azure event source tile.
  4. Name the event source. This name will be used to name the log that contains the event data in Log Search.
  5. Select a collector.
  6. Optionally, choose to send unparsed logs.
  7. Select your LDAP account attribution preference.
  8. In the Event Hub Name field, enter the name of the event hub as it is displayed in the Azure portal. Note that you'll need the name only, not the namespace.
  9. In the Endpoint field, enter the URL of the event hub namespace (excluding the protocol and the trailing slash).
  10. Under Credential, select Create New.
  11. Give the new credential a name that clearly identifies it.
  12. Enter the SharedAccessKeyName and Shared Access Key in their respective fields
  13. Click Save.

Test the configuration

To test that event data is flowing into InsightIDR through the Collector:

  1. Verify that data is flowing to the Collector:
    1. From the Data Collection Management page, click the Event Sources tab.
    2. Find the event source you created and click View raw log. If the Raw Logs modal displays raw log entries, logs are successfully flowing to the Collector.
    3. Wait approximately seven minutes, then open the Log Search page in InsightIDR.
  2. Verify that log entries are appearing in Log Search:
    1. From the left menu, go to Log Search.
    2. Select the applicable Log Sets and the Log Names within them. The Log Name is the event source name. The EventSource logs flow into these Log Sets:
      • Ingress Authentication
      • SSO Authentication
      • Third-Party Alerts (Azure Security Alerts)
      • Unparsed Data
      • Cloud Service Activity
      • Cloud Service Admin Activity
  3. Set the time range to Last 10 minutes, and click Run.

The Results table displays all log entries that flowed into InsightIDR in the last 10 minutes. The keys and values that are displayed are helpful to know when you want to build a query and search your logs.

Some log formats are incompatible

If you see raw log entries when you select View raw log, but you do not see any log entries in Log Search, then your logs do not match the recommended format and type for this event source.

Sample logs

There are multiple ways to generate sample audit events in Azure to send to the Event Hub:

  • Start and stop virtual machines. If you have a test or spare virtual machine, you can generate sample audit events by starting and stopping those machines.
  • List shared access policies. Open the Event Hub Namespace. Under Settings, select Shared Access Policies for RootManageSharedAccessKey.

It might take several minutes for events to be available in InsightIDR.

Troubleshoot common issues

This section covers some common troubleshooting scenarios.

A connection has been established, but no data is flowing to IDR

If a connection has been established, but there is no data flowing to InsightIDR, verify that you are logged into the correct Event Hub Topic.

There is an error in the connection

If there is an error in the connection, check the following:

  • Verify that you have selected the Standard tier for Azure Security Center, as stated in the Requirements.
  • Verify that you are logged into the correct Event Hub Instance.
  • Check your firewall to verify that you have configured an outbound connection over TCP port 9093 on your InsightIDR Collector.
  • Check your credentials. Ensure that you are using the Connection String Primary Key and the correct connection string as described in Configure a Microsoft Azure Event Hub.

Invalid SASL mechanism response error

If you are seeing an error that says Invalid SASL mechanism response, server may be expecting a different protocol, update your Connection String Primary Key in InsightIDR. To do this, complete Task 3 of Configure a Microsoft Azure Event Hub to copy the key and Configure InsightIDR again.

Create or update activity log profilesFailure error

When configuring the Azure Monitor, you may try to save your changes but see an error on the top right of the UI saying Create or update activity log profilesFailure.

To fix this error:

  1. Search for Subscriptions in all services.
  2. Select your subscription and click on Resource Providers in the left hand panel.
  3. Search for microsoft.insights.
  4. Ensure that it is registered by clicking on either Register or Re-Register. Wait for the process to complete.
  5. Click Refresh.
  6. Repeat the steps in Configure the Azure Monitor to ensure the activity log saves without error.