Microsoft Azure

Microsoft Azure is a complete cloud platform with infrastructure, software, and applications available as services. Azure can complement an on-premises infrastructure as an extension of your organization’s technical assets. When using Azure in your environment, whether you opt for the cloud or on-premises option, security and monitoring are still an essential part of your daily operations.

To provide flexibility and customer choice in security operations, Microsoft offers Azure Event Hubs as a centralized service to collect data and logs from other Azure services. You can integrate InsightIDR with Azure Event Hubs to access and ingest all applicable Azure data and logs. This combines Microsoft’s data ingestion service with the powerful incident detection and response system of InsightIDR.

When you configure Azure Event Hubs and consume data and logs through the Microsoft Azure event source, InsightIDR will:

  • Collect Azure Active Directory events to offer ingress authentication and Single Sign-On (SSO) detections.
  • Collect Azure Active Directory events to offer cloud service activity and cloud service admin activity.
  • Collect Azure Monitor events to offer Azure Security Center alerts as a third-party alert. Read more about Azure Security Center here: https://azure.microsoft.com/en-us/services/security-center/

Azure detections trigger behavioral alerts in InsightIDR

InsightIDR will continue to offer additional Azure detections over time and track them, because user behaviors are monitored from the event sources and the Insight Agents that are deployed in your environment.

Behavioral alerts will be triggered using Azure detections and treat Azure Cloud Services like an extension of your own environment.

Requirements

Ensure that your system meets the following requirements:

  • You must have a Premium P1 or P2 license to Azure Active Directory.
  • You must select the Standard tier for Azure Security Center to send third-party alerts from Azure Security Center to InsightIDR.

Open an outbound connection over TCP port 9093 on your InsightIDR Collector

The Microsoft Azure event source can only connect to Azure through an outbound connection on TCP port 9093. If you do not open this port, your event source configuration will fail.

Configure Microsoft Azure

To enable communication between Microsoft Azure and InsightIDR, complete these steps:

  1. Create a New Event Hub
  2. Create a Shared Access Policy for the Event Hub
  3. Configure the Azure Monitor to the Event Hub
  4. Configure Azure Active Directory to the Event Hub
  5. Copy the Shared Access Policy Key from the Event Hub
  6. Add Microsoft Azure Event Source in InsightIDR

Task 1: Create a New Event Hub

To provide the right information to InsightIDR, you must create a new Azure Event Hub. You should name your Event Hub "insights-operational-logs".

To create a new Event Hub, follow Microsoft’s documentation at: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create

Standard tier required

The Microsoft Azure event source can only be successfully configured through the Standard tier subscription.

Task 2: Create a Shared Access Policy for the Event Hub

A Shared Access Policy is used to allow InsightIDR access to read the messages Azure will publish to your Event Hub. To create a Shared Access Policy, follow these steps:

  1. In Microsoft Azure, navigate to Shared access policies and add a SAS policy.
  2. Enter the name of your policy, for example, "R7InsightIDR”.
  3. Grant your policy Listen permissions.

For more information, read Microsoft’s documentation at: https://docs.microsoft.com/en-us/azure/event-hubs/authorize-access-shared-access-signature#shared-access-authorization-policies

Task 3: Configure the Azure Monitor

Configure the Azure Monitor to send its logs to the Event Hub by following these steps:

  1. From the Monitor page, click Activity logs.
  2. Select Export Activity Logs.
  3. Confirm your subscription and add diagnostic settings. At a minimum, you should check the “Administrative”, “Security” and “Alert” checkboxes.
  4. Check the “Stream to an Event Hub” checkbox and configure your subscription, namespace, Event Hub, and policy.

For more information, read Microsoft’s documentation at: https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/stream-monitoring-data-event-hubs

Task 4: Configure Azure Active Directory

Azure Active Directory sign-in and audit events can also be streamed to an event hub for ingestion into InsightIDR.

Follow Microsoft's documentation at: https://docs.microsoft.com/en-us/azure/active-directory/reports-monitoring/tutorial-azure-monitor-stream-logs-to-event-hub#stream-logs-to-an-event-hub

You should ensure that you:

  • Select the Subscription you named earlier.
  • Select the Event Hub namespace (insights-operational-logs) you created earlier.
  • Select the RootManageSharedAccessKey policy name.

Task 5: Copy Shared Access Policy Key

You will need to copy a specific policy key from your Event Hub for configuration in InsightIDR.

To copy the key:

  1. Select your Event Hub to see its details.
  2. Select the Shared Access Policy link.
  3. Click on the Policy you created.
  4. Copy the Connection String Primary Key for later use in InsightIDR.

Task 6: Add Microsoft Azure Event Source in InsightIDR

Once your setup of Azure is complete, you can connect it to InsightIDR.

Open an outbound connection over TCP port 9093 on your InsightIDR Collector

The Microsoft Azure event source can only connect to Azure through an outbound connection on TCP port 9093. If you do not open this port, your event source configuration will fail.

  1. From the left menu, go to Data Collection.
  2. From the Data Collection screen, click the Setup Event Source dropdown menu and select Add Event Source.
  3. From the Security Data section, click the Cloud Service icon. The Add Event Source panel appears.
  4. Select your collector and Microsoft Azure from the event source dropdown menu.
  5. Enter the name of your event source.
  6. Optionally choose to send unfiltered logs to make additional Azure events searchable in Log Search.
  7. Select your LDAP account attribution preference.
  8. In the “Server” field, enter the hostname of the Azure Service bus namespace that you documented in earlier steps. For example, rapid7idr.servicebus.windows.net.
  9. In the “Topic” field, enter insights-operational-logs as the topic name.
  10. Select your Microsoft Azure credentials, or optionally create a new credential that matches the Shared Access Key Name you created in previous steps. The Shared Access Key is a part of the key you copied earlier.
  1. Click Save.

Verify the Configuration

  1. From the left menu, click Log Search to view your raw logs to ensure events are making it to the Collector.
  2. Select the applicable Log Sets and the Log Names within them. The Log Name will be the event source name or Microsoft Azure if you didn’t name the event source. Microsoft Azure logs flow into these Log Sets:
    • Ingress Authentication
    • SSO Authentication
    • Third-Party Alerts
    • Unparsed Data
    • Cloud Service Activity
    • Cloud Service Admin Activity
  3. Perform a Log Search to ensure Microsoft Azure events are coming through.

Here is an example of what the Microsoft Azure log search data looks like:

Verify the configuration

Generate Sample Events

There are a couple of ways to generate sample audit events in Azure to send over to your Event Hub.

  • Start/Stop VMs. If you have a test or spare VM, you can generate sample audit events by simply starting and stopping those machines.
  • List Shared Access Policies. Open the Event Hub Namespace, under Settings, select Shared Access Policies for RootManageSharedAccessKey. By completing either of those steps, you will generate audit logs. It may take several minutes for events to be available in InsightIDR.

Troubleshoot Common Issues

This section covers some common troubleshooting scenarios.

A connection has been established, but no data is flowing to IDR

If a connection has been established, but there is no data flowing to InsightIDR, verify that you are logged into the correct Event Hub Topic.

There is an error in the connection

If there is an error in the connection, check the following:

  • Verify that you have selected the Standard tier for Azure Security Center, as stated in the Requirements.
  • Verify that you are logged into the correct Event Hub Instance.
  • Check your firewall to verify that you have configured an outbound connection over TCP port 9093 on your InsightIDR Collector.
  • Check your credentials. Verify that you are using the Connection String Primary Key and the correct connection string. For more information, see Task 5: Copy Shared Access Policy Key

Invalid SASL mechanism response error

If you are seeing an error that says Invalid SASL mechanism response, server may be expecting a different protocol, update your Shared Access Key in InsightIDR. To do this, complete Task 5: Copy Shared Access Policy Key and step 9 of Task 6: Add Microsoft Azure Event Source in InsightIDR again.

Create or update activity log profilesFailure error

During Task 3, you may try to save your changes but see an error on the top right of the UI saying “Create or update activity log profilesFailure”.

To fix this error:

  1. Search for “Subscriptions” in all services.
  2. Select your subscription and click on Resource Providers in the left hand panel.
  3. Search for “microsoft.insights”.
  4. Ensure that it is registered by clicking on either Register or Re-Register. Wait for the process to complete.
  5. Click Refresh.
  6. Repeat the steps in Task 3 to ensure the activity log saves without error.