Microsoft Azure

Microsoft Azure is a complete cloud platform with infrastructure, software, and applications available as services. Azure can complement an on-premises infrastructure as an extension of your organization’s technical assets. When using Azure in your environment, whether you opt for the cloud or on-premises option, security and monitoring are still an essential part of your daily operations.

To provide flexibility and customer choice in security operations, Microsoft offers Azure Event Hubs as a centralized service to collect data and logs from other Azure services. You can integrate InsightIDR with Azure Event Hubs to access and ingest all applicable Azure data and logs. This combines Microsoft’s data ingestion service with the powerful incident detection and response system of InsightIDR.

When you configure Azure Event Hubs and consume data and logs through the Microsoft Azure event source, InsightIDR will:

  • Collect Azure Monitor events to offer Azure Security Center alerts as third-party alert detections. Read more about Azure Security Center here: https://azure.microsoft.com/en-us/services/security-center/
  • Collect Azure Active Directory events to offer ingress authentication, single sign-on (SSO), cloud service activity, and cloud service admin activity detections.
  • Collect Microsoft Defender for Cloud events to generate third-party alert detections.

Azure detections trigger behavioral alerts in InsightIDR

InsightIDR will continue to offer additional Azure detections over time and track them, because user behaviors are monitored from the event sources and the Insight Agents that are deployed in your environment.

Behavioral alerts will be triggered using Azure detections and treat Azure Cloud Services like an extension of your own environment.

To set up Microsoft Azure, you’ll need to:

  1. Review the requirements.
  2. Configure a Microsoft Azure Event Hub.
  3. Configure Microsoft Azure data to send to InsightIDR.
  4. Set up Microsoft Azure in InsightIDR
  5. Troubleshoot common issues.

Requirements

Ensure that your system meets the following requirements:

  • You must have a license for Azure Monitor, Azure Active Directory, or Defender for Cloud, depending on what data you would like to send to InsightIDR.
  • You must select the Standard tier for Azure Security Center to send third-party alerts from Azure Security Center to InsightIDR.
  • You must configure a Microsoft Azure Event Hub to send data to InsightIDR.

Open an outbound connection over TCP port 9093 on your InsightIDR Collector

The Microsoft Azure event source can only connect to Azure through an outbound connection on TCP port 9093. If you do not open this port, your event source configuration will fail.

Configure a Microsoft Azure Event Hub

To enable communication between Microsoft Azure and InsightIDR, you must first create an Event Hub.

Task 1: Create a New Event Hub

To send the right information to InsightIDR, you must create a new Azure Event Hub. You should name your Event Hub insights-operational-logs.

To create a new Event Hub, follow Microsoft’s documentation: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create

Standard tier required

The Microsoft Azure event source can only be successfully configured if you have access to the Standard tier or above.

Task 2: Create a Shared Access Policy for the Event Hub

A Shared Access Policy is used to allow InsightIDR access to read the messages Azure will publish to your Event Hub. To create a Shared Access Policy, follow these steps:

  1. In Microsoft Azure, navigate to Shared access policies and add an SAS policy.
  2. Enter the name of your policy, for example, R7InsightIDR.
  3. Grant your policy Listen permissions.

For more information, read Microsoft’s documentation: https://docs.microsoft.com/en-us/azure/event-hubs/authorize-access-shared-access-signature#shared-access-authorization-policies

Task 3: Copy Shared Access Policy Key

You will need to copy a specific policy key from your Event Hub for configuration in InsightIDR.

To copy the key:

  1. Select your Event Hub.
  2. Select the Shared Access Policy link.
  3. Click on the Policy you created.
  4. Copy the Connection String Primary Key for later use in InsightIDR.

Configure Microsoft Azure data to send to InsightIDR

With the Microsoft Azure event source, you can send logs from multiple Microsoft Azure products. Depending on which product you would like to configure, follow these steps to:

Configure the Azure Monitor

You can configure the Azure Monitor to send its logs to your Event Hub by following these steps:

  1. From the Monitor page, click Activity logs.
  2. Select Export Activity Logs.
  3. Confirm your subscription and add diagnostic settings. At a minimum, you should check the Administrative, Security and Alert checkboxes.
  4. Check the Stream to an Event Hub checkbox and configure your subscription, namespace, Event Hub, and policy.

For more information, read Microsoft’s documentation at: https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/stream-monitoring-data-event-hubs

Configure Azure Active Directory

You can configure Azure Active Directory to stream sign-in and audit events to your Event Hub for ingestion into InsightIDR.

To configure Azure Active Directory, follow Microsoft's documentation at: https://docs.microsoft.com/en-us/azure/active-directory/reports-monitoring/tutorial-azure-monitor-stream-logs-to-event-hub#stream-logs-to-an-event-hub

You should ensure that you:

  • Select the Subscription you named earlier.
  • Select the Event Hub namespace (insights-operational-logs) you created earlier.
  • Select the RootManageSharedAccessKey policy name.
Configure Microsoft Defender for Cloud

You can configure Microsoft Defender for Cloud to send its logs to your Event Hub by following these steps:

  1. In the Defender for Cloud menu, open Environment settings.
  2. Select the subscription for which you want to configure the data export.
  3. Select Continuous export. Here, you can configure which data you would like to export, where it should be saved and at what frequency.

For more information, read Microsoft’s documentation at: https://docs.microsoft.com/en-us/azure/defender-for-cloud/continuous-export?tabs=azure-portal

Set up Microsoft Azure in InsightIDR

Once you have created your Microsoft Azure Event Hub and configured the data you’d like to send to InsightIDR, you can set up the Microsoft Azure event source.

Ensure you have opened an outbound connection over TCP port 9093 on your InsightIDR Collector

As previously called out here, ensure you have opened this port, or your event source configuration will fail.

Task 1: Configure the event source
  1. From the left menu, go to Data Collection > Setup Event Source > Add Event Source.
  2. From the Security Data section, click the Cloud Service icon. The Add Event Source panel appears.
  3. Select your collector and Microsoft Azure from the event source dropdown menu.
  4. Enter the name of your event source.
  5. Optionally choose to send unparsed logs to make additional Azure events searchable in Log Search.
  6. Select your LDAP account attribution preference.
  7. In the “Host Name” field, enter the hostname of the Azure Service bus namespace that you documented in earlier steps. For example, rapid7idr.servicebus.windows.net.
  8. In the “Event Hub Name” field, enter insights-operational-logs as the topic name.
  9. Select your Microsoft Azure credentials, or optionally create a new credential that matches the Shared Access Key Name you created in previous steps. The Shared Access Key is a part of the key you copied earlier.
  1. Click Save.
Task 2: Verify the Configuration
  1. From the left menu, click Log Search to view your raw logs to ensure events are making it to the Collector.
  2. Select the applicable Log Sets and the Log Names within them. The Log Name will be the event source name or Microsoft Azure if you didn’t name the event source. Microsoft Azure logs flow into these Log Sets:
    • Ingress Authentication
    • SSO Authentication
    • Third-Party Alerts
    • Unparsed Data
    • Cloud Service Activity
    • Cloud Service Admin Activity
  3. Perform a Log Search to ensure Microsoft Azure events are coming through.

Here is an example of what the Microsoft Azure log search data looks like:

Verify the configuration

Generate Sample Events

There are a couple of ways to generate sample audit events in Azure to send over to your Event Hub.

  • Start/Stop VMs. If you have a test or spare VM, you can generate sample audit events by simply starting and stopping those machines.
  • List Shared Access Policies. Open the Event Hub Namespace. Under Settings, select Shared Access Policies for RootManageSharedAccessKey.

By completing either of these steps, you will generate audit logs. It may take several minutes for events to be available in InsightIDR.

Troubleshoot Common Issues

This section covers some common troubleshooting scenarios.

A connection has been established, but no data is flowing to IDR

If a connection has been established, but there is no data flowing to InsightIDR, verify that you are logged into the correct Event Hub Topic.

There is an error in the connection

If there is an error in the connection, check the following:

  • Verify that you have selected the Standard tier for Azure Security Center, as stated in the Requirements.
  • Verify that you are logged into the correct Event Hub Instance.
  • Check your firewall to verify that you have configured an outbound connection over TCP port 9093 on your InsightIDR Collector.
  • Check your credentials. Verify that you are using the Connection String Primary Key and the correct connection string. For more information, see Task 1, Step 3: Copy Shared Access Policy Key

Invalid SASL mechanism response error

If you are seeing an error that says Invalid SASL mechanism response, server may be expecting a different protocol, update your Shared Access Key in InsightIDR. To do this, complete Step 10, here: and Task 3, Step 10: Set up Microsoft Azure in InsightIDR again.

Create or update activity log profilesFailure error

When configuring the Azure Monitor, you may try to save your changes but see an error on the top right of the UI saying Create or update activity log profilesFailure.

To fix this error:

  1. Search for Subscriptions in all services.
  2. Select your subscription and click on Resource Providers in the left hand panel.
  3. Search for microsoft.insights.
  4. Ensure that it is registered by clicking on either Register or Re-Register. Wait for the process to complete.
  5. Click Refresh.
  6. Repeat the steps in Configure the Azure Monitor to ensure the activity log saves without error.