Microsoft Azure

Microsoft Azure is a complete cloud platform with infrastructure, software, and applications available as services. Azure can complement an on-premises infrastructure as an extension of an organization’s technical assets. When using Azure in your environment, whether through the cloud or on-premises, security and monitoring are required for the daily operation of any organization.

To provide flexibility and customer choice in security operations, Microsoft offers Azure Event Hubs as a centralized service to collect data and logs from other Azure services. You can integrate InsightIDR with Azure Event Hubs to access and ingest all applicable Azure data and logs and combine Microsoft’s data ingestion service with the powerful intrusion detection and response system of InsightIDR.

When you configure Azure Event Hubs and consume data and logs via the Microsoft Azure event source, InsightIDR will:

InsightIDR will continue to offer additional Azure detections over time and track them as user behaviors monitored across event sources and the Insight Agents deployed in your environment.

Behavioral alerts will fire using Azure detections and treat Azure Cloud Services like an extension of your environment.

To use Microsoft Azure Event Hubs with InsightIDR:

  1. Create a New Event Hub
  2. Create a Shared Access Policy for the Event Hub
  3. Configure the Azure Monitor to the Event Hub
  4. Configure Azure Active Directory to the Event Hub
  5. Copy the Shared Access Policy Key from the Event Hub
  6. Add Microsoft Azure Event Source in InsightIDR

Before You Begin

Ensure that your system meets the following requirements:

  • To complete the procedures described in this article, you must have a Premium P1 or P2 license to Azure Active Directory.
  • If you want to send third party alerts from Azure Security Center to InsightIDR, select the Standard tier for Azure Security Center.
  • If you want to use an existing Azure Event Hub, verify that Kafka Messages are enabled. (See step 7 of “Create a New Event Hub” for details) If your existing Azure Event Hub does not have Kafka enabled, you must create a new Event Hub and enable it there.

Open an outbound connection over TCP port 9093 on your InsightIDR Collector

The Microsoft Azure event source can only connect to Azure through an outbound connection on TCP port 9093. If you do not open this port, your event source configuration will fail.

Task 1: Create a New Event Hub

To provide precisely the right information for InsightIDR, create a new Azure Event Hub.

To create a new Event Hub:

  1. Navigate to http://portal.azure.com and sign in.
  2. From the left menu, select **All services > everything and search for “Event Hubs.”
  3. Click the star button to add the Event Hubs resource to the left menu under “Favorites” for easier access.
  4. In the “Event Hubs” page, click the +Add button.
  5. Name your Event Hub.
  6. From the “Pricing Tier” dropdown, select the Standard option.
  7. Check on the Enable Kafka box. Leave the other box unchecked.
  8. From the “Subscription” dropdown, select a subscription that can later be used with the Azure Monitor. In future steps, ensure that you use this same subscription.
  9. From the “Resource Group” dropdown, choose any value you want. This field is required for Azure, but InsightIDR does not reference this field in any configuration.
  10. From the “Location” dropdown, select the location where your Azure environment resources are located.
  1. Leave the “Throughput Units” slider at 1.
  2. Leave the “Enable Auto-Inflate” box unchecked.
  3. Click the Create button.
    • Azure will automatically provision and activate the new Event Hub after a few minutes. You will see an “Activated” notification.
  1. Click the Refresh button.
  2. Select your newly created Event Hub to see the details page.

Task 2: Create a Shared Access Policy for the Event Hub

A Shared Access Policy is used to allow InsightIDR access to read the messages Azure will publish to your Event Hub.

To create a Shared Access Policy:

  1. In your new Event Hub details page, select the Shared access policies page from the left menu.
  1. Click the +Add button. The “Add SAS Policy” panel appears.
  2. Name your policy something recognizable for later use in InsightIDR, such as "R7InsightIDR."
  3. Check on the Listen box.
  4. Click the Create button.

Your Shared Access Policy is now created. The policy name and keys will always be in this location. Remember the name of this policy for later use.

Task 3: Configure the Azure Monitor

Now you must configure the Azure Monitor to send its logs to the Event Hub.

To do so, complete the following steps:

  1. To open the Azure Monitor, enter Monitor in the Search bar, and select it from the search results.
  1. From the Left menu, click Activity Log.
  2. Select the subscription you configured in Task 1.
  3. Click Diagnostic Settings.
  4. Click the banner to launch the Export Activity Log.
  1. Select one or more regions. Microsoft recommends selecting all regions.
  2. Choose your desired storage account.
  3. Optionally configure retention in days.
  4. Select the “Azure Event Hub” option; a panel will appear.
  5. In the “Service bus namespace,” choose the Event Hub you created earlier. Make note of this name for later use in InsightIDR.
  6. Choose the RootManageSharedAccessKey policy name, which can write data directly to the Event Hub.
  7. Click the OK button to collapse the panel.

(Optional) Task 4: Add the Azure Active Directory to the Event Hub

To configure your Azure Active Directory to send its Audit logs to the Event Hub, you must have a Premium P1 or P2 license to Azure Active Directory. You can read more about this process here: https://docs.microsoft.com/en-us/azure/devops/project/navigation/set-favorites?view=azure-devops

To send audit logs to the Event Hub, complete the following steps:

  1. From the left menu, select All services > everything and search for “Azure Active Directory.”
  2. Click the star button to add this page to the left menu under “Favorites” for easier access.
  3. From the "Azure Active Directory” page, select the Audit Logs page under the “Monitoring” section.
  4. Click the Export Data Settings button at the top of the page.
  1. On the “Diagnostics Settings” page, click the Turn on diagnostics link. A panel will appear on the right.
  2. Name the diagnostic setting.
  3. Check on the Stream to an event hub box.
  1. Click the Configure button under Event hub. A second panel will appear on the right.
  2. Choose the subscription you named from previous steps.
  3. Choose the Event Hub namespace you created from previous steps.
  4. Leave the “Event Hub name” box empty.

Once Azure is successfully sending logs to InsightIDR, return to this screen to configure the “Event Hub name” box. Send data to “insights-operational-logs” which Azure automatically creates after the Event Hub processes its first log event.

  1. Choose the RootManageSharedAccessKey policy name.
  2. Click the OK button to collapse the panel.
  3. Check on the AuditLogs and SignInLogs boxes.
  4. Click the Save button at the top of the “Diagnostics” panel to save the configuration and start streaming data.

Task 5: Copy Shared Access Policy Key

You will need to copy a specific policy key from your Event Hub for configuration in InsightIDR.

To copy the key:

  1. Return to the “Event Hubs” page and select your Event Hub to see details.
  2. Select the Shared Access Policy link.
  3. Click on the policy you created during these instructions. A panel will appear.
  4. You will see four different keys you can copy. Copy the Connection string - primary key for later use in InsightIDR.

Task 6: Add Microsoft Azure Event Source in InsightIDR

Once your setup of Azure is complete, you can connect it to InsightIDR.

Open an outbound connection over TCP port 9093 on your InsightIDR Collector

The Microsoft Azure event source can only connect to Azure through an outbound connection on TCP port 9093. If you do not open this port, your event source configuration will fail.

  1. From your dashboard, select Data Collection on the left hand menu.
  2. When the Data Collection page appears, click the Setup Event Source dropdown and choose Add Event Source.
  3. From the “Security” section, click the Cloud Service icon. The “Add Event Source” panel appears.
  4. Choose your collector and select Microsoft Azure as the event source. Give your event source the same name you selected for the Event Hub.
  5. Check on the unfiltered logs box in order to make additional Azure events searchable in Log Search.
  6. In the “Server” field, enter the hostname of the Azure Service bus namespace that you documented in earlier steps. For example, rapid7idr.servicebus.windows.net.
  7. In the “Topic” field, enter insights-operational-logs as the topic name.

By default, every Azure service that is configured to send data to the Event Hub has a pre-defined topic name. When you set up your event source in InsightIDR, either configure all services to send logs to the same topic, or create multiple event sources in InsightIDR with different topics.

  1. Choose or create a new credential for the Azure account that matches the Shared Access Key Name you created in previous steps.
  2. Enter the Shared Access Key you copied earlier, “Connection string - primary key.”
  1. Configure your default domain.
  2. Click the Save button.

Verify the Configuration

  1. From the left menu, click Log Search to view your raw logs to ensure events are making it to the Collector. Select the applicable Log Sets and the Log Names within them. The Log Name will be the event source name or Microsoft Azure if you didn’t name the event source. Microsoft Azure logs flow into these Log Sets:
    • Ingress Authentication
    • SSO Authentication
    • Third Party Alerts
    • Unparsed Data
  2. Next, perform a Log Search to make sure Microsoft Azure events are coming through.

Here is an example of what the Microsoft Azure log search data looks like:

Generate Sample Events

There are a couple of ways to generate sample audit events in Azure to send over to your Event Hub.

  • Start/Stop VMs. If you have a test or spare VM, you can generate sample audit events by simply starting and stopping those machines.
  • List Shared Access Policies. Open the Event Hub Namespace, under Settings, select Shared Access Policies for RootManageSharedAccessKey. By completing either of those steps, you will generate audit logs. It may take several minutes for events to be available in InsightIDR.

Troubleshoot Common Issues

This section covers some common troubleshooting scenarios.

A connection has been established, but no data is flowing to IDR

If a connection has been established, but there is no data flowing to InsightIDR, verify that you are logged into the correct Event Hub Topic.

There is an error in the connection

If there is an error in the connection, check the following:

  • Verify that you have the correct subscription level/licensing.
  • Verify that you are logged into the correct Event Hub Topic and that Kafka is enabled.
  • Check your firewall/proxy permissions to verify that you have configured an outbound connection over TCP port 9093 on your InsightIDR Collector.
  • Check your credentials. Verify that you are using the correct primary key and the correct connection string. For more information, see Task 5: Copy Shared Access Policy Key

Invalid SASL mechanism response error

If you are seeing an error that says Invalid SASL mechanism response, server may be expecting a different protocol, update your Shared Access Key in InsightIDR. To do this, complete Task 5: Copy Shared Access Policy Key and step 9 of Task 6: Add Microsoft Azure Event Source in InsightIDR again.