Custom Logs

Like other raw data, custom logs contextualize information throughout InsightIDR and are helpful during log search. Any text-based log can be ingested through InsightIDR. However, Rapid7 recommends using JSON or KVP format for logging as data is presented in Log Search in this form and allows for keyword search. Sending an unstructured string will yield an unstructured log entry in InsightIDR, meaning you can search for any text in the event but lose the benefit of keyword search.

Collecting logs prior to event source setup

After you have turned on an event source, Rapid7 can collect some logs that were created prior to setup:

  • Cloud event source - custom logs created five minutes prior to setup
  • Collector event source - custom logs created 24 hours prior to setup

If an event source is paused, logs will only be collected for a maximum of 72 hours. For example, if the event source was paused for 24 hours and then turned on again, Rapid7 will collect the logs created 24 hours prior to unpausing the event source.

Configure InsightIDR to collect data from the event source

After you complete the prerequisite steps and configure the event source to send data, you must add the event source in InsightIDR.

To configure the new event source in InsightIDR:

  1. From the left menu, go to Data Collection and click Setup Event Source > Add Event Source.
    • Alternatively, from the left menu, select Data Collection > Event Sources then click Add Event Source.
  2. Click Add Raw Data > Custom Logs.
    • Alternatively, you can search for Custom Logs or filter by the Rapid7 Product Type, and then select the Rapid7 Custom Logs event source tile.
  3. Follow the instructions below for the collection method of your choice:

Cloud

Amazon S3

Maximum file size

The maximum file size supported for Amazon S3 cloud event sources using a compressed .gzip file is 10MB. Files larger than 10MB will not be processed.

You can configure InsightIDR to read logs that are stored in an Amazon S3 bucket. For information about setting up an Amazon S3 bucket, read the AWS documentation on creating S3 buckets: https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html.

Amazon S3 requirements

To allow InsightIDR to receive data, you must configure the relevant Amazon account to provide access to its data:

Access Keys in AWS

In AWS, access keys consist of two parts: an access key ID, for example, AKIAIOSFODNN7EXAMPLE and a secret access key, for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY. You must use both the access key ID and secret access key together to authenticate your requests.

Amazon S3 log formatting

The Amazon S3 option supports reading files stored in an S3 bucket that contain newline-delimited plain text (.txt, .log), CSV (.csv), and JSON (.json). This option also supports reading supported file types when they have been compressed using gzip (for example: filename.json.gz or filename.csv.gz). In either case, the files should have new line characters separating each log entry or event.

CSV formatting

If your logs are contained in a .csv file, it is recommended to remove the header row prior to sending the logs to InsightIDR.

If the files contain log events spread across multiple lines, each line will be interpreted as a separate event. Amazon S3 does not support reading files with encrypted contents, binary data, or files that have been compressed using a method other than gzip.

`.gzip` formatting

If you are using the gzip format, the file metadata must contain the term gzip, or alternatively the file extension must be set to .gz.

To configure Amazon S3:

  1. Name your event source.
  2. Amazon S3 will be selected for the collection method. Specify your Amazon S3 Bucket Name.
    • Optionally, you can enter an Amazon S3 Key Prefix.
  3. Select an Amazon S3 Bucket from the dropdown or click Add a New Connection to create one.
  4. Enter the name of the Amazon S3 Bucket that you created, but remove the s3:// prefix. For example, if your bucket is s3://your.bucket.url, enter only your.bucket.url.
  5. Optionally, enter an Amazon S3 Key Prefix. A key prefix allows you to specify the folder the logs are stored in. For example, if your logs are stored in a folder named abcd1234, you would enter abcd1234/. If the logs are not stored in a folder, leave this field empty.
  6. Click Add a New Connection.
  7. In the Create a Cloud Connection screen, enter a name for the new connection.
  8. In the Bucket field, enter the name of a bucket you want to check for modified files.
  9. In the Region field, enter the region of the Amazon S3 bucket. For the precise format of this value, review the table of supported regions.
  10. In the AWS Access Key ID field, add a new credential:
    1. Name your credential.
    2. Describe your credential.
    3. Select the credential type.
    4. Enter the Access Key, which is the Access Key you obtained previously.
    5. Specify the product access for this credential.
  11. In the AWS Secret Access Key field, add a new credential:
    1. Name your credential.
    2. Describe your credential.
    3. Select the credential type.
    4. Enter the Secret Key, which is the Secret Key you obtained previously.
    5. Specify the product access for this credential.
  12. Click Save Connection.
  13. Click Save.

Webhook

InsightIDR allows data collection from products and systems that can send events through webhook requests (HTTP POST body method). A webhook event source can have one or more URLs associated with it. These unique URLs are used by the third party products as the destination of the webhook requests.

Do not share unique URLs

The URLs associated with webhook event sources are unique and should be protected to prevent unauthorised users from sending data to the event source.

Webhook requirements

InsightIDR supports the followwing data formats for webhook event sources:

Webhook data formatting

The type of data sent should be expressed by the Content-Type header used by the requests. If no header is included in the request, the system will attempt to determine whether the contents are JSON or plain text.

  • Plain text - Webhook requests containing plain text are processed with new line characters separating individual event lines.

  • JSON - Webhook requests containing JSON data are supported natively. If the contents are an array of events, the elements of the array are treated as individual events. Otherwise, the entire JSON request body will be treated as a single event.

    • For example, this JSON data will be interpreted as two events automatically:
     
    1
    [
    2
    { "log": "event 1"},
    3
    { "log": "event 2"}
    4
    ]
    • If the events are contained within a field in the JSON object, the JSON Events Key field can be configured to indicate which field contains the events. In this example, the JSON Events Key is set to resources to identify that the contents should be read from that field within the JSON. Periods can be used to identify a nested field, for example path.to.events.
     
    1
    {
    2
      "type": "AuditLogEntry",
    3
      "size": 2,
    4
      "resources": [
    5
         { "log": "event 1" },
    6
         { "log": "event 2" }
    7
      ]
    8
    }

  • NDJSON - Webhook requests containing newline delimited JSON will have multiple JSON objects, separated by new line characters. Each line will be treated as individual events. The JSON Events Key may be specified if needed, as described in the JSON example.

  • URL encoded form values - Webhook requests containing form values will translate the form data's key plus value pairs into a JSON presentation.

To configure webhook collection:

  1. Name your event source.
  2. Click Copy to copy the Webhook URL for use in the product you wish to configure to send events to InsightDR.
    • Click Generate a new Webhook URL if you want to add another URL. This can also be used if the existing URL needs to be replaced.
  3. Optionally, configure the JSON Events Key if needed.
  4. Click Save.

Test the configuration

You can test if logs can be sent to the event source using a curl command. The following examples work for Windows, Mac and Linux operating systems. Ensure you replace the placeholder with your new Webhook URL:

bash
1
curl --verbose <your-webhook-URL> --header "Content-type:application/json" --data "{\"message\":\"Something else happened\",\"user\":\"jsmith\",\"hostname\":\"server1\"}"
2


3
curl --verbose <your-webhook-URL> --header "Content-type:application/text" --data "Raw text message"

Collector

Listen on Network Port

You can configure your application to forward log events to a syslog server, and then configure the InsightIDR Collector to listen on a network port for syslog data on a unique port in order to receive it.

To configure Listen on Network Port:

  1. Name your event source.
  2. Choose your collector from the dropdown list.
  3. Choose the timezone that matches the location of your event source logs.
  4. Optionally, select Parse RFC 3164 syslog headers to parse logs with syslog headers and format them in JSON. Do not select this option if you want to process those logs as unstructured raw data.
  5. Under Collection Method, select Listen on Network Port.
  6. Follow the instructions to configure the Listen on Network Port collection method for your event source.
    • Optionally, choose to Encrypt the event source if choosing the TCP Protocol by downloading the Rapid7 Certificate.
  7. Click Save.
Log Aggregator

If you want to collect logs that have already been collected by a SIEM or a Log Aggregator, you can send raw logs to the Collector using a unique port.

To configure Log Aggregator:

  1. Name your event source.
  2. Choose your collector from the dropdown list.
  3. Choose the timezone that matches the location of your event source logs.
  4. Optionally, select Parse RFC 3164 syslog headers to parse logs with syslog headers and format them in JSON. Do not select this option if you want to process those logs as unstructured raw data.
  5. Under Collection Method, select Log Aggregator.
  6. Follow the instructions to configure the Log Aggregator collection method for your event source.
    • Optionally, choose to Encrypt the event source if choosing the TCP Protocol by downloading the Rapid7 Certificate.
  7. Click Save.
SQS Messages

AWS SQS, or Amazon Simple Queue Services, is a managed queuing service that works with InsightIDR when sending messages as events.

To configure SQS Messages:

  1. Name your event source.
  2. Choose your collector from the dropdown list.
  3. Choose the timezone that matches the location of your event source logs.
  4. Optionally, select Parse RFC 3164 syslog headers to parse logs with syslog headers and format them in JSON. Do not select this option if you want to process those logs as unstructured raw data.
  5. Under Collection Method, select SQS Messages.
  6. Follow the instructions to configure the SQS Messages collection method for your event source.
  7. Click Save.
Watch Directory

To configure Watch Directory:

You can monitor a network location that hosts log files copied from a specified directory on a local or remote host with Watch Directory.

  1. Name your event source.
  2. Choose your collector from the dropdown list.
  3. Choose the timezone that matches the location of your event source logs.
  4. Optionally, select Parse RFC 3164 syslog headers to parse logs with syslog headers and format them in JSON. Do not select this option if you want to process those logs as unstructured raw data.
  5. Under Collection Method, select Watch Directory.
    • Optionally, choose to Encrypt the event source if choosing the TCP Protocol by downloading the Rapid7 Certificate.
  6. Follow the instructions to configure the Watch Directory collection method for your event source.
  7. Click Save.
Tail File

You can configure InsightIDR to watch the network location where a host stores log data, and ingest any new data added to the log file on a local or remote host. Using the equivalent of the Unix tail command, InsightIDR will collect data written to the host disk every 20 seconds.

To configure Tail File:

  1. Name your event source.
  2. Choose your collector from the dropdown list.
  3. Choose the timezone that matches the location of your event source logs.
  4. Optionally, select Parse RFC 3164 syslog headers to parse logs with syslog headers and format them in JSON. Do not select this option if you want to process those logs as unstructured raw data.
  5. Under Collection Method, select Tail File.
  6. Follow the instructions to configure the Tail File collection method for your event source.
  7. Click Save.
Amazon S3

You can configure InsightIDR to read logs that are stored in an Amazon S3 bucket. For information about setting up an Amazon S3 bucket, visit the third-party vendor's documentation.

To configure Amazon S3:

  1. Name your event source.
  2. Choose your collector from the dropdown list.
  3. Choose the timezone that matches the location of your event source logs.
  4. Optionally, select Parse RFC 3164 syslog headers to parse logs with syslog headers and format them in JSON. Do not select this option if you want to process those logs as unstructured raw data.
  5. Under Collection Method, select Amazon S3.
  6. Follow the instructions to configure the Amazon S3 collection method for your event source.
  7. Click Save.