Custom Logs

Like other raw data, custom logs contextualize information throughout InsightIDR and are helpful during log search. Any text-based log can be ingested through InsightIDR. However, Rapid7 recommends using JSON or KVP format for logging as data is presented in Log Search in this form and allows for keyword search. Sending an unstructured string will yield an unstructured log entry in InsightIDR, meaning you can search for any text in the event but lose the benefit of keyword search.

Collecting logs prior to event source setup

After you have turned on an event source, Rapid7 can collect some logs that were created prior to setup:

  • Cloud event source - custom logs created five minutes prior to setup
  • Collector event source - custom logs created 24 hours prior to setup

If an event source is paused, logs will only be collected for a maximum of 72 hours. For example, if the event source was paused for 24 hours and then turned on again, Rapid7 will collect the logs created 24 hours prior to unpausing the event source.

Configure InsightIDR to collect data from the event source

After you complete the prerequisite steps and configure the event source to send data, you must add the event source in InsightIDR.

To configure the new event source in InsightIDR:

  1. From the left menu, go to Data Collection and click Setup Event Source > Add Event Source.
    • Alternatively, from the left menu, select Data Collection > Event Sources then click Add Event Source.
  2. Click Add Raw Data > Custom Logs.
    • Alternatively, you can search for Custom Logs or filter by the Rapid7 Product Type, and then select the Rapid7 Custom Logs event source tile.
  3. Follow the instructions below for the collection method of your choice:

Cloud

Cloud event sources are being phased in from February 2025

InsightIDR is adding cloud event collection capabilities to a select number of supported event sources; this one is included. This will be a phased release, so if your environment is not yet displaying the Cloud option, please be patient – your environment will update shortly.

Amazon S3

You can configure InsightIDR to read logs that are stored in an Amazon S3 bucket. For information about setting up an Amazon S3 bucket, visit the third-party vendor's documentation.

CSV formatting

If your logs are contained in a .csv file, it is recommended to remove the header row prior to sending the logs to InsightIDR. Read more about Amazon S3 log formatting.

To configure Amazon S3:

  1. Name your event source.
  2. Amazon S3 will be selected for the collection method. Specify your Amazon S3 Bucket Name.
    • Optionally, you can enter an Amazon S3 Key Prefix.
  3. Select an Amazon S3 Bucket from the dropdown or click Add a New Connection to create one.
  4. Click Save.

Collector

Listen on Network Port

You can configure your application to forward log events to a syslog server, and then configure the InsightIDR Collector to listen on a network port for syslog data on a unique port in order to receive it.

To configure Listen on Network Port:

  1. Name your event source.
  2. Choose your collector from the dropdown list.
  3. Choose the timezone that matches the location of your event source logs.
  4. Optionally, select Parse RFC 3164 syslog headers to parse logs with syslog headers and format them in JSON. Do not select this option if you want to process those logs as unstructured raw data.
  5. Under Collection Method, select Listen on Network Port.
  6. Follow the instructions to configure the Listen on Network Port collection method for your event source.
    • Optionally, choose to Encrypt the event source if choosing the TCP Protocol by downloading the Rapid7 Certificate.
  7. Click Save.
Log Aggregator

If you want to collect logs that have already been collected by a SIEM or a Log Aggregator, you can send raw logs to the Collector using a unique port.

To configure Log Aggregator:

  1. Name your event source.
  2. Choose your collector from the dropdown list.
  3. Choose the timezone that matches the location of your event source logs.
  4. Optionally, select Parse RFC 3164 syslog headers to parse logs with syslog headers and format them in JSON. Do not select this option if you want to process those logs as unstructured raw data.
  5. Under Collection Method, select Log Aggregator.
  6. Follow the instructions to configure the Log Aggregator collection method for your event source.
    • Optionally, choose to Encrypt the event source if choosing the TCP Protocol by downloading the Rapid7 Certificate.
  7. Click Save.
SQS Messages

AWS SQS, or Amazon Simple Queue Services, is a managed queuing service that works with InsightIDR when sending messages as events.

To configure SQS Messages:

  1. Name your event source.
  2. Choose your collector from the dropdown list.
  3. Choose the timezone that matches the location of your event source logs.
  4. Optionally, select Parse RFC 3164 syslog headers to parse logs with syslog headers and format them in JSON. Do not select this option if you want to process those logs as unstructured raw data.
  5. Under Collection Method, select SQS Messages.
  6. Follow the instructions to configure the SQS Messages collection method for your event source.
  7. Click Save.
Watch Directory

To configure Watch Directory:

You can monitor a network location that hosts log files copied from a specified directory on a local or remote host with Watch Directory.

  1. Name your event source.
  2. Choose your collector from the dropdown list.
  3. Choose the timezone that matches the location of your event source logs.
  4. Optionally, select Parse RFC 3164 syslog headers to parse logs with syslog headers and format them in JSON. Do not select this option if you want to process those logs as unstructured raw data.
  5. Under Collection Method, select Watch Directory.
    • Optionally, choose to Encrypt the event source if choosing the TCP Protocol by downloading the Rapid7 Certificate.
  6. Follow the instructions to configure the Watch Directory collection method for your event source.
  7. Click Save.
Tail File

You can configure InsightIDR to watch the network location where a host stores log data, and ingest any new data added to the log file on a local or remote host. Using the equivalent of the Unix tail command, InsightIDR will collect data written to the host disk every 20 seconds.

To configure Tail File:

  1. Name your event source.
  2. Choose your collector from the dropdown list.
  3. Choose the timezone that matches the location of your event source logs.
  4. Optionally, select Parse RFC 3164 syslog headers to parse logs with syslog headers and format them in JSON. Do not select this option if you want to process those logs as unstructured raw data.
  5. Under Collection Method, select Tail File.
  6. Follow the instructions to configure the Tail File collection method for your event source.
  7. Click Save.
Amazon S3

You can configure InsightIDR to read logs that are stored in an Amazon S3 bucket. For information about setting up an Amazon S3 bucket, visit the third-party vendor's documentation.

To configure Amazon S3:

  1. Name your event source.
  2. Choose your collector from the dropdown list.
  3. Choose the timezone that matches the location of your event source logs.
  4. Optionally, select Parse RFC 3164 syslog headers to parse logs with syslog headers and format them in JSON. Do not select this option if you want to process those logs as unstructured raw data.
  5. Under Collection Method, select Amazon S3.
  6. Follow the instructions to configure the Amazon S3 collection method for your event source.
  7. Click Save.