Custom Alert Details

When you configure a custom alert, there are several different settings to consider to optimize alerts and make them efficient.

Set Trigger Notification Settings

For Pattern Match Alerts, an alert triggers every time the event occurs on a rolling basis by default. If you choose Custom Match Settings, you can specify how many times an event must occur within a given time frame before it can trigger an alert.

For example, if the pattern match threshold is 100 times in the throttle window of 60 minutes, 100 alerts must occur within the next hour before you receive the first alert. The alert is triggered when our alert counter reaches this limit. However, note that it does not trigger again if the pattern is continually matched above the threshold: the counter must drop again below the limit, and then again over the threshold to be re-triggered.

For Change Detection alerts, new queries require that you specify a calculation to use, and a key to apply the calculation. Any changes of the key based off of the calculation will trigger an alert.

Define Notification Methods

For Change Detection and Inactivity Detection alerts, define one or more communication methods. You can choose PagerDuty, Slack, Webhook, and email.

Select existing recipients or create new ones and provide the appropriate details for the communication method.

If you are using email, select existing recipients from the dropdown or create new ones. Use commas to separate email addresses if you are sending to multiple recipients.

Define what log information you’d like included in your notifications. By default, the matching log line and context will be included. To change this, open the edit alert modal, click Alert Notifications > Notification(email/integrations) and select the communication method tab you'd like to edit:

  • Email: Click the Recipients dropdown and select Add & select recipient. Click the Log Context dropdown to view options.
  • PagerDuty: Click the Service key for Pagerduty account dropdown and select Add & select key. Click the Log Context dropdown to view options.
  • Slack: Click the Slack Webhook URL dropdown and select Add & select Webhook. Click the Log Context dropdown to view options.
  • Webhook: Click the Webhook URL and select Add & select Webhook. Click the Log Context dropdown to view options.
  • InsightConnect Workflow: Click the InsightConnect Workflow dropdown and select Add & select InsightConnect workflow. If you want log line information to be included in your notifications, select Send Log Line.

Alternatively, you can also select Detection Rules > Labels & Notifications > Notification Target to define the log information you'd like to be included.

Create Log Labels with an Alert

In addition to triggering notifications such as emails, you can configure custom Pattern Detection alerts to apply color-coded labels to log entries. This helps you to identify the log entries that were flagged in specific alerts or to filter the list of log entries.

If you often use the same queries to identify logs of interest, applying a label can save you time. Labels provide a visual cue in the list of log entries to help you identify important events.

After a pattern detection alert is saved, the label is displayed on log entries that match the search criteria–both on the Entries tab and the Table tab. The label appears at the top of the tab and also inline with the log entries that it’s applicable to.

Log Labels as Filters

Label counts are applied at ingestion only

The count (the number in parentheses) in a label is calculated at the time of log ingestion. If you change the time frame of the search query to exclude the ingestion time, the count may become ineffective or the label won't be displayed.

To filter log entries by the alert’s search query, click a label. When a filter is applied, the results appear in the list and the selected label is filled with its designated color.

To remove the filter, click the label again. You can click either instance of the label to switch the filter on and off.

Tip for creating labels

Before you create a custom alert that includes a label, we recommend that you fine-tune your search query in Log Search. By trying out the search query and checking the results, you can identify exactly what query syntax to enter in the alert creation panel.

To create a label in a custom pattern detection alert:

  1. In Log Search, click Add Alert and select Pattern Detection Alert from the dropdown menu.
  2. Configure the alert as described in Manually Create a Pattern Detection Alert.
  3. In the Alert Notification section, select a severity and a label or create your own custom label.
  4. Optionally add a notification to the alert by clicking the Notification (emails/integrations) tab and selecting a notification type.
  5. Click Create Alert.

Default Labels

Default labels are applied to provide visual cues that indicate potential issues with your logs. Within InsightIDR, there are currently two default labels:

Exception

These labels are applied to log entries that contain the word “exception”, which indicates a deviation from the product’s normal flow. You may notice Exception labels on log entries when you search for environment or code issues.

Example:

 
1
31 Aug 2022 19:48:18.304<14>1 2022-08-31T19:48:18.269729Z ip-1-2-3-4 lerest - - - 	at java.base/java.lang.NumberFormatException.forCharSequence(NumberFormatException.java:81)

OutOfOrder

These labels are applied to logs entries that appear out of time sequence in terms of IDR time.

Example:

 
1
idr-timestamp=1 "my first message"
2
idr-timestamp=3 "my third message"
3
idr-timestamp=4 "my fourth message"
4
idr-timestamp=4 "my second message" OutOfOrder

OutOfOrder indicates that time has been mutated to maintain the order of arrival, but the original IDR time was older than 4.

If you want to customize the appearance of log entries by applying different labels, you can create a custom alert.

Create an Investigation

Once an alert fires based on your query, you have the option for InsightIDR to automatically open an Investigation.

When creating an alert, turn on the Create an Investigation button to enable this action. Then define your notification throttle.

Create New Email Recipients

When you using email to send alerts to your team, you can create a new email recipient.

To add new email recipients:

  1. From the Recipients dropdown, click Add & select recipient.
  2. Name the recipient, and then enter one or more email addresses separated by commas.
  3. Optionally check the box to include other entries logged when the alert was generated.
  4. Click the Add & select recipient button.

The new email will populate the Recipients field.

Configure Third-Party Integrations

If you are using one of our partner integrations, you can send alerts to the appropriate teams to remediate issues as they arise. You can configure alerts for the following integrations:

Slack

In order to connect Slack and InsightIDR, you must be a Slack administrator.

To configure this integration:

  1. Log in to Slack.
  2. Expand your Profile and go to Administration.
  3. From the Administration menu, select Manage apps.

A list of your current Slack app integrations appears. 4. Search for "Logentries" in the App Directory and select the matching result. Logentries is the previous name for InsightIDR. 5. Click Install. 6. Choose the channel you want to integrate with the Logentries app. 7. Click the Add Logentries Integration button.

  1. Scroll down the page until you find the webhook URL.
  2. Copy the Webhook URL.
  3. In InsightIDR, create an alert. Under "Alert Notifications," choose Notifications (email/integrations) > Slack.
  4. Paste the Webhook URL you copied from the Slack App directory in the "Slack Webhook URL" field.
  5. Configure the rest of your alert.
  6. Press Create Alert.

PagerDuty

Any user can complete this PagerDuty Integration.

To configure this integration:

  1. Log in to your PagerDuty account.
  2. Select the Configuration tab and select the Services option.
  3. Click the + New Service button.
  4. Enter a name for the service.
  5. From the "Integration Type" dropdown, find "Logentries."
  1. Name the integration.
  2. Select an Escalation Policy.
  3. Optionally check the Response Play box.
  4. Choose how responders should be notified.
  5. Click the Add Service button. This will generate a service key for InsightIDR.
  6. From the "Services" page, copy that key for Logentries and go back to InsightIDR.
  7. You will see an Integration Key. Copy that key and return to InsightIDR.
  8. In InsightIDR, create an alert. From "Alert Notifications," choose Notifications (email/integrations) > PagerDuty.
  9. Paste the service key you copied from PagerDuty into the "Service Key" field.
  1. Configure the rest of your alert.
  2. Click Create Alert.

Webhook

A Webhook URL allows you to send a POST request to an API as events happen, instead of requiring that you poll for updates. Learn how to use and create a Webhook here: https://developer.github.com/webhooks/

You can use a Webhook URL to notify your team of InsightIDR alerts.

To use a Webhook with InsightIDR alerts:

  1. In InsightIDR, create an alert.
  2. Under "Alert Notifications," choose Notifications (email/integrations) > Webhook.
  3. Name your Webhook.
  4. Paste your Webhook URL.
  5. Configure the rest of your alert.
  6. Press Create Alert.