Microsoft Azure

Microsoft Azure is a complete cloud platform with infrastructure, software, and applications available as services. Azure can complement an on-premises infrastructure as an extension of your organization’s technical assets. When using Azure in your environment, whether you opt for the cloud or on-premises option, security and monitoring are still an essential part of your daily operations.

To provide flexibility and customer choice in security operations, Microsoft offers Azure Event Hubs as a centralized service to collect data and logs from other Azure services. You can integrate InsightIDR with Azure Event Hubs to access and ingest all applicable Azure logs and data. This combines Microsoft’s data ingestion service with the incident detection and response system of InsightIDR.

When you configure Azure Event Hubs and consume data and logs through InsightIDR's Microsoft Azure event source, InsightIDR will:

  • Parse Microsoft Defender for Cloud events as third-party alert detections.
  • Parse Azure Active Directory events to offer ingress authentication, single sign-on (SSO), cloud service activity, and cloud service admin activity detections.
  • Parse Microsoft Defender for Endpoint Advanced hunting events as third-party alert detections.

Azure detections trigger legacy detection rules in InsightIDR

Because user behaviors are monitored from the event sources and Insight Agents deployed in your environment, InsightIDR continues to offer and track additional Azure detections over time. Legacy detection rules (formerly known as User Behavior Detection Rules) treat Azure Cloud Services like an extension of your environment.

New Azure alerts for Exchange and SharePoint audit logs

InsightIDR can now produce alerts for Microsoft’s exchange and SharePoint audit logs. These alerts generate cloud service activity and ingress authentication events. Read more about the event log format here: https://learn.microsoft.com/en-gb/microsoft-365/compliance/audit-log-search?view=o365-worldwide.

To set up Microsoft Azure:

  1. Complete the prerequisite steps.
  2. Configure Microsoft Azure to send data to InsightIDR.
  3. Configure InsightIDR to receive data from the event source.
  4. Test the configuration.
  5. Troubleshoot common issues.

Requirements

To successfully configure the Microsoft Azure event source, you must:

  • Have a Premium P1 or Premium P2 license for Azure Active Directory.
  • Have a license for Azure Monitor, Microsoft Defender for Cloud, or Microsoft Defender for Endpoint, depending on which data you want to send to InsightIDR.
  • Have access to a Standard pricing tier or higher. The pricing tier also affects the value you can set for retention time (hrs).
  • Allocate and open an outbound connection over TCP port 9093 on the InsightIDR Collector. If you do not open this port, your event source configuration will fail.
  • Ensure that InsightIDR has an adequate data ingestion rate. Your throughput must equal the number of partitions. Read more about event hub scalability here: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-scalability#number-of-partitions.

Restrictions when editing configured event hubs

To edit configured event hubs, you must have a Premium tier license. Read more about the Premium tier here: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-premium-overview. You can find more information about tiers here: https://learn.microsoft.com/en-us/azure/event-hubs/compare-tiers.

Configure Microsoft Azure to send data to InsightIDR

Complete these tasks to establish communication between InsightIDR and Microsoft Azure.

Something not quite right?

Sometimes we're not able to provide the most current information about other vendors. For the most up-to-date information, refer to Microsoft’s documentation, including:

Task 1: Create a new event hub

To send the right information to InsightIDR, create a new Azure event hub with a unique name, such as insights-operational-logs. You can configure the Microsoft Azure event source only if you have access to the Standard tier subscription or above.

To create an event hub:
  1. Search for Event Hubs in the search bar.
  2. Select the name of an active Azure Subscription.
  3. Create a namespace:
    1. Create a new Resource group.
    2. Enter a unique Namespace name. The system checks whether the name is available.
    3. Select a Location (Region) for the namespace.
    4. Select Standard for the Pricing tier.
    5. Set Throughput Units to 1. Each throughput unit provides 1MB/s ingress and 2MB/s egress.
    6. Optionally, Enable Auto-Inflate if you want the event hub namespace to automatically allocate more throughput units during times of heavy traffic. You can also configure this setting in the event hub namespace, in Settings > Scale.
    7. Optionally, set the Auto-Inflate Maximum Throughput Units to 2. Each node costs $73 per month, per throughput unit.
    8. Select Review + Create.
    9. After the Validation succeeded message displays, select Create. A Deployment in Process message displays. Azure automatically provisions and activates the new namespace in about 5 minutes. An Activated notification displays when complete, and the namespace is created.
  4. Create the event hub:
    1. Select + Event Hub on the Overview page.
    2. Enter a unique Event Hub name, such as insights-operational-logs.
    3. Ensure that the Partition Count is set to 1.
    4. (Recommended) Set the Retention time (hrs) to the maximum value allowed by your tier. Standards tiers support up to 7 days (168 hours), and Premium and Dedicated tiers support up to 90 days (2160 hours).
      • For existing event hubs, configure this Message Retention setting in Event Hub > Properties.
    5. Select Review + Create.
    6. After the Validation succeeded message displays, select Create. The event hub is created.

Task 2: Create a shared access policy from the event hub

A shared access policy allows InsightIDR to access the messages that Azure publishes to the event hub. When you create a shared access policy, ensure that you make a note of the policy name (we recommend a name such as R7InsightIDR) and grant your policy Listen permission only.

To create a shared access policy:
  1. In the namespace, go to Event Hubs.
  2. Select the event hub.
  3. Select Shared access policies.
  4. Select + Add.
  5. Specify a policy name, such as R7InsightIDR.
  6. Grant the policy Listen permission only.
  7. Select Create. The policy is created and added to the Azure event source credentials.
  8. Optionally, you can use the RootManageSharedAccessKey policy with all activity selected for Namespace activity logs and Azure Activity Directory Logs.

Task 3: Configure activity logs

Configure Azure Monitor to send its logs to the event hub.

To configure activity logs:
  1. From the namespace, select Activity Logs.
  2. Select Export Activity Logs.
  3. Select + Add diagnostic setting.
  4. Select at least Administrative, Security, and Alert.
  5. Select Stream to an Event Hub.
  6. Ensure that your Azure Subscription is selected.
  7. Set the Event hub name (optional) to insights-operational-logs, which is the event hub you created in task 1.
  8. Ensure that the Event hub policy name is set to RootManagedSharedAccessKey.
  9. Select Save.

Task 4: Retrieve the shared access policy key

To configure the event source in InsightIDR, you will need the primary key from the R7InsightIDR shared access policy (created in task 2), associated with the insights-operational-logs event source (created in task 1).

To retrieve the shared access policy key:
  1. From Event Hubs, go to Shared Access Policy.
  2. Select the shared access policy that you created in task 2 (for example, R7InsightIDR).
  3. Record the Primary Key for later use in InsightIDR.
Shared access policy key format

The connection string for a namespace contains these components:

  • Fully qualified domain name of the Event Hubs namespace you created
  • Name of the shared access key
  • Value of the shared access key

Each of the three components are presented as key-value pairs, and are separated by semi-colons: Endpoint=sb://<NamespaceName>.servicebus.windows.net/;SharedAccessKeyName=<KeyName>;SharedAccessKey=<KeyValue>. For example, your Connection String Primary Key might look like this: Endpoint=sb://rapid7idreventhub.servicebus.windows.net/;SharedAccessKeyName=foobar;SharedAccessKey=password.

You must copy each value into its respective field on the InsightIDR event source configuration screen. You don't need to include the trailing forward slash or protocol when you enter the Endpoint URL.

To use the earlier example, you would enter the values in InsightIDR as follows:

  • Endpoint is rapid7idreventhub.servicebus.windows.net
  • AccessKeyName is foobar
  • SharedAccessKey is password

The SharedAccessKey itself can often contain an equals sign (=) at the end of the string. This is part of the password and you must include it in the event source configuration screen.

Task 5 (Optional): Configure other Microsoft solutions

To configure Azure Active Directory:

Configure Azure Active Directory to send sign-in and audit events to the event hub for ingestion into InsightIDR.

  1. Search for Azure Active Directory in the search bar.
  2. Select Audit Logs from the Monitoring section.
  3. Select Export Data Settings.
  4. Select Add diagnostic setting.
  5. Select all Categories that you want to ingest logs for.
  6. Ensure that the active Azure Subscription is selected.
  7. Set the Event hub name (optional) to insights-operational-logs, which is the event hub you created in task 1.
  8. Ensure that the Event hub policy name is set to RootManagedSharedAccessKey.
  9. Select Save.
To configure Microsoft Defender for Cloud:

Configure Microsoft Defender for Cloud to send its logs to the event hub for ingestion into InsightIDR. You can configure Microsoft Defender for Cloud using one of two options.

Option 1: Configure with the event hub policy name

  1. Go to Microsoft Defender for Cloud > Environment settings.
  2. Select a Subscription.
  3. Select Continuous export.
  4. Select all checkboxes.
  5. Select Resource Group.
  6. Select Namespace.
  7. Select Event hub.
  8. Select Event hub policy name.
  9. Save the configuration.

Option 2: Configure as a trusted service

  1. If your event hub is behind a firewall, grant the Continuous Export service access to the event hub:
    1. Go to the event hub > Access Control.
    2. Select Add role assignment for the Azure Event Hubs Data Sender role.
    3. In Select members, select Windows Azure Security Resource Provider.
  2. Configure continuous export:
    1. Go to Microsoft Defender for Cloud > Environment settings.
    2. Select a Subscription.
    3. Select Continuous export.
    4. Select all checkboxes.
    5. Select Resource Group.
    6. Select Namespace.
    7. Select Event hub.
    8. Select Export as a trusted service.
  3. Add access controls:
    1. Go to Event hubs > Access control (IAM).
    2. Select Add > Add role assignment.
    3. Select Azure Event Hubs Data Sender, and go to Members.
    4. Select Select members.
    5. Search for and select Windows Azure Security Resource Provider.
    6. Select Select to save the changes.
    7. Select Review + assign.
  4. Return to the Environment settings > Subscription > Continuous export settings page.
  5. Save the configuration.
To configure Microsoft Defender for Endpoint:

Configure Microsoft Defender for Endpoint to send its logs to the event hub for ingestion into InsightIDR.

  1. Go to https://security.microsoft.com.
  2. Go to Settings.
  3. Select Microsoft Defender XDR.
  4. Select Streaming API.
  5. Select Add.
  6. Name the API.
  7. Select Forward events to Event Hub.
  8. Retrieve the Event-Hub Resource ID:
    1. Go to the event hub Overview page.
    2. Select JSON View on the right.
    3. Copy the Resource ID.
  9. Enter the Event-Hub name.
  10. Select Events-Types.
  11. Select Submit.

Configure InsightIDR to collect data from the event source

Once you have created a Microsoft Azure event hub and configured the data that you want to send to InsightIDR, you can set up the Microsoft Azure event source.

To configure the new event source in InsightIDR:

  1. From the left menu, go to Data Collection and click Setup Event Source > Add Event Source.
  2. Do one of the following:
    • Search for Microsoft Azure in the event sources search bar.
    • In the Product Type filter, select Cloud Service.
  3. Select the Microsoft Azure event source tile.
  4. Name the event source. This name will be used to name the log that contains the event data in Log Search.
  5. Select a collector.
  6. Optionally, choose to send unparsed logs.
  7. Select your Account Attribution preference:
    • Use short name attribution: The system first attempts to attribute data by email address, for example, jsmith@myorg.example.com. If the first attempt is unsuccessful, attribution is attempted by short name, for example, jsmith. If the short name is unsuccessful, attribution is attempted by a user’s first and last name, for example, John Smith.
    • Use fully qualified domain name attribution: The system first attempts to attribute data by email address, for example, jsmith@myorg.example.com. If the first attempt is unsuccessful, attribution is attempted by a user’s first and last name, for example, John Smith. This option is best if your environment has collisions with short names.
  8. In the Event Hub Name field, enter the name of the event hub as it is displayed in the Azure portal. Note that you'll need the name only, not the namespace.
  9. In the Endpoint field, enter the URL of the event hub namespace (excluding the protocol and the trailing slash).
  10. Under Credential, select Create New.
  11. Give the new credential a name that clearly identifies it.
  12. Enter the SharedAccessKeyName and Shared Access Key in their respective fields.
  13. Click Save.

Test the configuration

To test that event data is flowing into InsightIDR through the Collector:

  1. Verify that data is flowing to the Collector:
    1. From the Data Collection Management page, click the Event Sources tab.
    2. Find the event source you created and click View raw log. If the Raw Logs modal displays raw log entries, logs are successfully flowing to the Collector.
    3. Wait approximately seven minutes, then open the Log Search page in InsightIDR.
  2. Verify that log entries are appearing in Log Search:
    1. From the left menu, go to Log Search.
    2. Select the applicable Log Sets and the Log Names within them. The Log Name is the event source name. The EventSource logs flow into these Log Sets:
      • Ingress Authentication
      • SSO Authentication
      • Third-Party Alerts (Azure Security Alerts)
      • Unparsed Data
      • Cloud Service Activity
      • Cloud Service Admin Activity
  3. Set the time range to Last 10 minutes, and click Run.

The Results table displays all log entries that flowed into InsightIDR in the last 10 minutes. The keys and values that are displayed are helpful to know when you want to build a query and search your logs.

Some log formats are incompatible

If you see raw log entries when you select View raw log, but you do not see any log entries in Log Search, then your logs do not match the recommended format and type for this event source.

Sample logs

There are multiple ways to generate sample audit events in Azure to send to the Event Hub:

  • Start and stop virtual machines. If you have a test or spare virtual machine, you can generate sample audit events by starting and stopping those machines.
  • List shared access policies. Open the Event Hub Namespace. Under Settings, select Shared Access Policies for RootManageSharedAccessKey.

It might take several minutes for events to be available in InsightIDR.

Troubleshoot common issues

This section covers some common troubleshooting scenarios.

A connection has been established, but no data is flowing to InsightIDR

If a connection has been established, but there is no data flowing to InsightIDR, verify that you are logged into the correct Event Hub Topic.

A connection has been established, but the data flowing into InsightIDR is delayed

If there's a delay between when Microsoft generates a log and when the log appears in InsightIDR, you might need to increase the number of throughput units (TUs) for your Azure namespace or partitions for your Azure event hub.

Consider these factors to determine the number of TUs and partitions required for your environment:

Understand TUs and partitions
  • Throughput units (TUs) - Each TU has a limit on how much data can be ingested into or sent out of an Azure namespace. If the number of events per second flowing from your Azure services into the namespaces exceeds the recommended ingress limit of 1 TU, then you must increase the number of TUs to accommodate your expected data volume.
  • Partitions - Partitions enable parallel processing of data from an Azure event hub. Each partition supports up to 1 MB/s of data, so you might need to create several partitions if you need to send more than 1 MB/s of data to InsightIDR.
Determine your data requirements

To determine the data volume that your event hub must support, we recommend sending data from Microsoft Azure to InsightIDR, and measuring the performance.

To determine your data requirements:

  1. Configure the Azure event hub and namespace using the recommended number of TUs and partitions, and configure the InsightIDR event source.
  2. Let data flow from Azure to InsightIDR for several days.
  3. In the Azure portal, go to Event Hubs, and select the namespace you created to use with InsightIDR.
  4. In the three graphs, adjust the time horizon to see activity for the duration of time data has been flowing through the namespace.
  5. Select any of the graphs to open a detailed view.
  6. Select Add Metric, and then select Throttled Requests from the dropdown, ensuring the aggregation is set to Sum.
  7. Remove all other metrics.
  8. If you don't see any throttled requests, that means you have enough TUs and partitions to handle the volume of data you're sending to InsightIDR. You can expect your connection between Azure and InsightIDR to function as expected.
  9. If you see throttled requests, complete these steps to determine the number of TUs and partitions required for your environment:
    1. Confirm that Azure is sending only data to InsightIDR that is relevant for security use cases:
      1. In InsightIDR, go to Log Search.
      2. Select the Unparsed Data log set, and review the logs that Azure sends.
      3. If Azure is sending data that likely isn't relevant for security use cases, consider turning off these logs in Azure to decrease the amount of data send through the event hub and namespace. Read Microsoft's documentation about resource logs to learn more.
    2. Review the amount of data flowing through your namespace and event hub:
      1. In the Azure portal, repeat steps 2-8.
      2. Change the metric to Incoming Bytes, and review the data.
      3. Change the metric to Incoming Messages, and review the data.
      4. Use the number of Incoming Bytes and Incoming Messages to determine how many TUs to allocate for your namespace:
        • If the number of incoming bytes consistently exceeds the amount that your namespace can handle, increase the number of TUs to accomodate it. For example, if your namespace currently has 1 TU (which supports up to 1 MB/s), and the namespace consistently receives more than 2 MB/s, increase your namespace's TUs to 3, so that it can handle up to 3 MB/s.
        • If the number of incoming messages (events per second) consistently exceeds the amount that your namespace can handle, increase the number of TUs to accomodate it. For example, if your namespace currently has 1 TU (which supports up to 1,000 events per second), and the namespace consistently receives more than 2,000 events per second, increase your namespace's TUs to 3, so that it can handle up to 3,000 events per second.
        • If both the number of incoming bytes and incoming messages (events per second) consistently exceed the amount that your namespace can handle, increase the number of TUs based on the higher value. For example, if your namespace currently has 1 TU (which supports up to 1 MB/s or 1,000 events per second), and the namespace consistently receives more than 2 MB/s and more than 3,000 events per second, increase your namespace's TUs to 4, so that it can handle up to 4,000 events per second.
      5. Use the number of Incoming Bytes to confirm whether the event hub has enough partitions to support the incoming data volume:
        • If the number of incoming bytes consistently exceeds the amount that your event hub can handle, allocate more partitions. Each partition supports up to 1 MB/s.
        • Because you can't add partitions to an event hub after it's created, you must create a new event hub within the namespace, and give it the appropriate number of partitions. You must also update your Azure services to export their logs to the new event hub, and update the InsightIDR event source to connect to the new event hub.

There is an error in the connection

If there is an error in the connection, check the following:

  • Verify that you have selected the Standard tier for Azure Security Center, as stated in the Requirements.
  • Verify that you are logged into the correct Event Hub Instance.
  • Check your firewall to verify that you have configured an outbound connection over TCP port 9093 on your InsightIDR Collector.
  • Check your credentials. Ensure that you are using the Connection String Primary Key and the correct connection string as described in Configure a Microsoft Azure Event Hub.

Invalid SASL mechanism response error

If you are seeing an error that says Invalid SASL mechanism response, server may be expecting a different protocol, update your Connection String Primary Key in InsightIDR. To do this, complete Task 3 of Configure a Microsoft Azure Event Hub to copy the key and Configure InsightIDR again.

Create or update activity log profilesFailure error

When configuring the Azure Monitor, you may try to save your changes but see an error on the top right of the UI saying Create or update activity log profilesFailure.

To fix this error:

  1. Search for Subscriptions in all services.
  2. Select your subscription and click on Resource Providers in the left hand panel.
  3. Search for microsoft.insights.
  4. Ensure that it is registered by clicking on either Register or Re-Register. Wait for the process to complete.
  5. Click Refresh.
  6. Repeat the steps in Configure the Azure Monitor to ensure the activity log saves without error.