Microsoft Azure

Microsoft Azure is a complete cloud platform with infrastructure, software, and applications available as services. Azure can complement an on-premises infrastructure as an extension of an organization’s technical assets. When using Azure in your environment, whether through the cloud or on-premises, security and monitoring are required for the daily operation of any organization.

To provide flexibility and customer choice in security operations, Microsoft offers Azure Event Hubs as a centralized service to collect data and logs from other Azure services. You can integrate InsightIDR with Azure Event Hubs to access and ingest all applicable Azure data and logs and combine Microsoft’s data ingestion service with the powerful intrusion detection and response system of InsightIDR.

When you configure Azure Event Hubs and consume data and logs via the Microsoft Azure event source, InsightIDR will:

Azure detections trigger behaviorial alerts in InsightIDR

InsightIDR will continue to offer additional Azure detections over time and track them as user behaviors monitored across event sources and the Insight Agents deployed in your environment.

Behavioral alerts will fire using Azure detections and treat Azure Cloud Services like an extension of your environment.

To use Microsoft Azure Event Hubs with InsightIDR:

  1. Create a New Event Hub
  2. Create a Shared Access Policy for the Event Hub
  3. Configure the Azure Monitor to the Event Hub
  4. Configure Azure Active Directory to the Event Hub
  5. Copy the Shared Access Policy Key from the Event Hub
  6. Add Microsoft Azure Event Source in InsightIDR

Before You Begin

Ensure that your system meets the following requirements:

  • To complete the procedures described in this article, you must have a Premium P1 or P2 license to Azure Active Directory.
  • If you want to send third party alerts from Azure Security Center to InsightIDR, you must select the Standard tier for Azure Security Center.

Open an outbound connection over TCP port 9093 on your InsightIDR Collector

The Microsoft Azure event source can only connect to Azure through an outbound connection on TCP port 9093. If you do not open this port, your event source configuration will fail.

Task 1: Create a New Event Hub

To provide precisely the right information for InsightIDR, create a new Azure Event Hub.

To create a new Event Hub:

  1. Navigate to http://portal.azure.com and sign in.
  2. From the left menu, select All services > everything and search for “Event Hubs.”
  3. Optionally, click the pin button to add the Event Hubs resource to a dashboard for easy access.
  4. On the Event Hubs page, click the +Add button.
  5. On the Create Namespace screen, from the Subscription dropdown, select a subscription that can later be used with the Azure Monitor. In future steps, ensure that you use this same subscription.
  6. From the Resource Group dropdown, choose any value you want. This field is required for Azure, but InsightIDR does not reference this field in any configuration.
  7. Enter a name for your Event Hub under Namespace name.
  8. From the Location dropdown, select the location where your Azure environment resources are located.
  9. From the Pricing Tier dropdown, select the Standard option.
  10. Leave the Throughput Units slider at 1.
  11. Click Next: Features >. Azure Task 1
  12. Leave the Enable Availability Zones and Enable Auto-Inflate boxes unchecked. Azure Task 1
  13. Click the Next: Tags > button. Make no changes.
  14. Click the Review + Create > button. Wait for “Validation succeeded” to appear at the top of the screen.
  15. Click Create and you will see a “Deployment in Process” message.
    • Azure will automatically provision and activate the new Event Hub after a 5-10 minutes. You will see an “Activated” notification. Azure Task 1
  16. Go back to the Events Hubs page and click the Refresh button to see your new Event Hub. It will show the status “Activating” initially. Wait for it to show as “Active”. Azure Task 1
  17. Select your newly created Event Hub to see the details page. Azure Task 1

Task 2: Create a Shared Access Policy for the Event Hub

A Shared Access Policy is used to allow InsightIDR access to read the messages Azure will publish to your Event Hub.

To create a Shared Access Policy:

  1. In your new Event Hub details page, select the Shared access policies page from the left menu. Azure Task 2
  2. Click the +Add button. The “Add SAS Policy” panel appears.
  3. Name your policy something recognizable for later use in InsightIDR, such as "R7InsightIDR."
  4. Check on the Listen box. Azure Task 2
  5. Click the Create button. Azure Task 2

Your Shared Access Policy is now created. The policy name and keys will always be in this location. You will use both the RootManageSharedAccessKey policy and new policy in the following steps.

Task 3: Configure the Azure Monitor

Next, configure the Azure Monitor to send its logs to the Event Hub.

  1. To open the Azure Monitor, enter Monitor in the Search bar, and select it from the search results. You can also navigate to the home screen and select Monitor under Azure Services. Azure Task 3
  2. From the Left menu, click Activity Log.
  3. In the Subscription field, select the subscription that you configured in Task 1. Azure Task 3
  4. Click on Diagnostics settings.
  5. On the following page, click on the Looking for the legacy experience? purple banner. Azure Task 3
  6. Confirm your Subscription.
  7. In the Regions dropdown list, it’s recommended to select all regions.
  8. Optionally check the box to Export to a storage account and choose a Storage Account to export to.
  9. Optionally select Retention (days).
  10. Check the box for Export to Event Hub.
  11. Select Service Bus Namespace and in the pop out section to the right select the Subscription, the Event Hub Namespace and the Event Hub Policy name, which is RootManageSharedAccessKey.
  12. Click OK to close the panel.
  13. Click Save to confirm the settings. Azure Task 3

You should see the following message appear in the top right corner: Azure Task 3

If you see an error message “Create or update activity log profilesFailure” after clicking on Save, refer to Create or update activity log profilesFailure error.

Azure Task 3

Task 4: Add the Azure Active Directory to the Event Hub (Optional)

To configure your Azure Active Directory to send its Audit logs to the Event Hub, you must have a Premium P1 or P2 license to Azure Active Directory. For more information, see: https://docs.microsoft.com/en-us/azure/devops/project/navigation/set-favorites?view=azure-devops

To send audit logs to the Event Hub:

  1. From the left menu, select All services > everything and search for “Azure Active Directory.”
  2. Optionally, click the pin button to add this page to a dashboard for easier access.
  3. From the Azure Active Directory page, select the Audit Logs page under the “Monitoring” section.
  4. Click on Diagnostic Settings.
  5. On the “Diagnostics Settings” page, click the +Add diagnostic setting button.
  6. On the Diagnostics Setting page, name the diagnostic setting.
  7. Click the checkboxes to enable AuditLogs and SignInLogs.
  8. Check on the Stream to an event hub box. Azure Task 4
  9. Choose the Subscription you named from previous steps.
  10. Choose the Event Hub namespace you created from previous steps.
  11. Leave the “Event Hub name” box empty.

Once Azure is successfully sending logs to InsightIDR, return to this screen to configure the “Event Hub name” box. Send data to “insights-operational-logs” which Azure automatically creates after the Event Hub processes its first log event.

  1. Choose the RootManageSharedAccessKey policy name.
  2. Click the Save button at the top of the page to save the configuration and start streaming data.

Task 5: Copy Shared Access Policy Key

You will need to copy a specific policy key from your Event Hub for configuration in InsightIDR.

To copy the key:

  1. Return to the “Event Hubs” page and select your Event Hub to see details.
  2. Select the Shared Access Policy link.
  3. Click on the policy you created during these instructions. A panel will appear.
  4. You will see four different keys you can copy. Copy the Primary key for later use in InsightIDR.

Task 6: Add Microsoft Azure Event Source in InsightIDR

Once your setup of Azure is complete, you can connect it to InsightIDR.

Open an outbound connection over TCP port 9093 on your InsightIDR Collector

The Microsoft Azure event source can only connect to Azure through an outbound connection on TCP port 9093. If you do not open this port, your event source configuration will fail.

  1. From your dashboard, select Data Collection on the left hand menu.
  2. When the Data Collection page appears, click the Setup Event Source dropdown and choose Add Event Source.
  3. From the Security section, click the Cloud Service icon. The Add Event Source panel appears.
  4. Choose your collector and select Microsoft Azure as the event source. Give your event source the same name you selected for the Event Hub.
  5. Check on the unfiltered logs box in order to make additional Azure events searchable in Log Search.
  6. In the Server field, enter the hostname of the Azure Service bus namespace that you documented in earlier steps. For example, rapid7idr.servicebus.windows.net.
  7. In the Topic field, enter insights-operational-logs as the topic name.

By default, every Azure service that is configured to send data to the Event Hub has a pre-defined topic name. When you set up your event source in InsightIDR, either configure all services to send logs to the same topic, or create multiple event sources in InsightIDR with different topics.

  1. Choose or create a new credential for the Azure account that matches the Shared Access Key Name you created in previous steps.
  2. Enter the Shared Access Key you copied earlier, “Connection string - primary key.” Note that it is everything after SharedAccessKey= and up to and including the final ‘=’.
  1. Configure your default domain.
  2. Click the Save button.

Azure Task 6Azure Task 6

Verify the Configuration

  1. From the left menu, click Log Search to view your raw logs to ensure events are making it to the Collector. Select the applicable Log Sets and the Log Names within them. The Log Name will be the event source name or Microsoft Azure if you didn’t name the event source. Microsoft Azure logs flow into these Log Sets:
    • Ingress Authentication
    • SSO Authentication
    • Third Party Alerts
    • Unparsed Data
  2. Next, perform a Log Search to make sure Microsoft Azure events are coming through.

Here is an example of what the Microsoft Azure log search data looks like:

Generate Sample Events

There are a couple of ways to generate sample audit events in Azure to send over to your Event Hub.

  • Start/Stop VMs. If you have a test or spare VM, you can generate sample audit events by simply starting and stopping those machines.
  • List Shared Access Policies. Open the Event Hub Namespace, under Settings, select Shared Access Policies for RootManageSharedAccessKey. By completing either of those steps, you will generate audit logs. It may take several minutes for events to be available in InsightIDR.

Troubleshoot Common Issues

This section covers some common troubleshooting scenarios.

A connection has been established, but no data is flowing to IDR

If a connection has been established, but there is no data flowing to InsightIDR, verify that you are logged into the correct Event Hub Topic.

There is an error in the connection

If there is an error in the connection, check the following:

  • Verify that you have the correct subscription level/licensing.
  • Verify that you are logged into the correct Event Hub Topic.
  • Check your firewall/proxy permissions to verify that you have configured an outbound connection over TCP port 9093 on your InsightIDR Collector.
  • Check your credentials. Verify that you are using the correct primary key and the correct connection string. For more information, see Task 5: Copy Shared Access Policy Key

Invalid SASL mechanism response error

If you are seeing an error that says Invalid SASL mechanism response, server may be expecting a different protocol, update your Shared Access Key in InsightIDR. To do this, complete Task 5: Copy Shared Access Policy Key and step 9 of Task 6: Add Microsoft Azure Event Source in InsightIDR again.

Create or update activity log profilesFailure error

During Task 3, you may try to save your changes but see an error on the top right of the UI saying “Create or update activity log profilesFailure”.

To fix this error:

  1. Search for “Subscriptions” in all services.
  2. Select your subscription and click on Resource Providers in the left hand panel.
  3. Search for “microsoft.insights”.
  4. Ensure that it is registered by clicking on either Register or Re-Register. Wait for the process to complete.
  5. Click Refresh. Azure Troubleshooting
  6. Repeat the steps in Task 3 to ensure the activity log saves without error. Azure Troubleshooting