Configure the Insight Agent to Send Additional Logs

The Insight Agent performs default event log collection and process monitoring with InsightIDR. However, if you have compliance or operational needs that require additional log monitoring, you can configure the Insight Agent to run another job to send additional data to Log Search using a configuration file named logging.json. The logging.json file instructs the Insight Agent to send logs directly to the data.logs.insight.rapid7.com endpoint according to your region. See log details for more information.

Log forwarding use cases

Additional log forwarding produced by the Insight Agent's logging.json file can potentially send an enormous amount of data to the platform, so it’s important to understand when to use this feature to get the most value from it.

Monitor your logs for compliance

Using the Insight Agent to forward logs can help you meet certain compliance requirements because it will give you a more complete picture of all activity happening on a group of your devices.

For example, you can configure the Insight Agent to forward logs from all machines in your environment that store or process payment data. This can help you meet PCI compliance requirements.

Monitor CPU use and memory

The Insight Agent can also collect hardware metrics and usage details like CPU use and disk memory. This can be used to build dashboards for monitoring, reports, and basic detection rules for low disk space for example.

Unsupported collection methods in InsightIDR

The Insight Agent does not support the collection of Windows event logs from assets acting as domain controllers using the logging.json configuration file. If you need to deploy a collection method for this use case, consider using an alternative configuration.

Before you begin

Before you get started, there are a few things to keep in mind:

• Log files collected by the Insight Agent using the logging.json configuration file bypass our InsightIDR threat detection engine. They are sent directly to our log ingestion endpoint and cannot be parsed using the Custom Parsing Tool because the Custom Parsing tool requires the data be ingested through an event source.
• Log entries collected by the logging.json method will not trigger any detection rules, or be attributed to legacy detection rules, as the data bypasses our detection engine.
• The Insight Agent will not automatically route data collected using this method through a Collector. If you want to route data through a Collector, you must configure it as a proxy.

Prerequisites

To use log forwarding through the logging.json file, you must have access to an API key that allows you to create logs and logsets.

Ensure you have the correct permissions:

1. Create a new user account.
2. Assign the user to only InsightIDR.
3. Restrict access to Log Search operations by selecting the Log Search Insight Agent Service Role permission.
4. Generate an API key.

Create your logging.json file

The following examples show a few different ways that you can create your logging.json file. Follow the instructions for Mac and Linux or Windows and select the configuration option that best suits your needs.

Mac and Linux

To create a logging.json file for Mac or Linux:

1. Copy and paste the template in a new file. Be sure to update the following values:
   • Region code: Set <region-code> to your region. You can check your region from the Platform Home.
   • Platform API Key: Replace <platform-api-key> with your API key.
   • Path: Configure the “path” key to tail specific files on the system.
   • Destination: Configure the destination to send your data to the desired Log Set and Log. In Log Search, you can view the default Log Sets generated by your InsightIDR Collectors. We do not recommend using these Log Sets for this data. Instead, we recommend setting the “destination” to be a new Log Set such as “Data Center 1”. For a log named “Syslog”, your destination would be "destination": "Data Center 1/Syslog",. You may reuse the same destination across multiple machines and all data will be transmitted to the same Log Set\Log.
   • Name: Use a static name.
2. When ready, save the logging.json file in the following directory: /opt/rapid7/ir_agent/components/insight_agent/common/config.

3. After saving your file, restart the agent service.

Mac and Linux Template

If you use this template, be sure to change the values as specified in the previous section.

json
1
{
2
  "config": {
3
    "name": "Linux Agent",
4
    "endpoint": "<region-code>.data.logs.insight.rapid7.com",
5
    "region": "<region-code>",
6
    "api-key": "<platform-api-key>",
7
    "state-file": "/opt/rapid7/ir_agent/components/insight_agent/common/config/logs.state",
8
    "logs": [
9
      {
10
        "name": "Syslog",
11
        "destination": "Linux Logs/Syslog",
12
        "path": "/var/log/syslog",
13
        "enabled": true
14
      },
15
      {
16
        "name": "Audit log",
17
        "destination": "Linux Logs/audit.log",
18
        "path": "/var/log/audit/audit.log",
19
        "enabled": true
20
      },
21
      {
22
        "name": "My Log Name",
23
        "destination": "Enter Desired Log Set Name Here/Enter Desired Log Name Here",
24
        "path": "/path/to/log/file.log",
25
        "enabled": true
26
      }
27
    ]
28
  }
29
}

Windows

When you configure the logging.json file to collect Windows event logs, it will collect logs from the following channels:

• Application
• System
• Security

All entries in these channels are collected and cannot be modified to collect a subset or a superset of these event logs. On certain machines, depending on their audit policy, this may be a large amount of data to constantly transmit. You may not want to configure this on all of the machines in your environment.

Refactored Windows log forwarding for Log Search

On an opt-in basis, you can use an alternative component of the Insight Agent to monitor logs on Windows assets and send them to Log Search. This component has been refactored with more features and stability.

To opt in to this refactored Windows log collection capability, add the windows_following: true key-value pair to the log's JSON object in your logging.json file. The refactored component includes the following functionality:

• Log rotation support for Windows assets.
• An Insight Agent restart is no longer required to move, rename, or delete a file that is being followed by the le_realtime Insight Agent job.

Domain controller log collection alternatives

While you can utilize the Insight Agent’s IDR event collection on domain controllers, additional event collection will not run on servers with a role of domain controllers or backup domain controllers using the logging.json method. If you need to deploy a collection method for this use case, consider implementing one of the following alternatives for the collection of the Security event log:

• You can enable the Insight Agent to collect some Security event log events. For more information, see Alternatives to Domain Admin Accounts.
• As an alternative to running the Insight Agent to collect Active Directory events for InsightIDR as mentioned, Create an Active Directory event source in InsightIDR using WMI and select the Optional box to send unparsed logs using a Collector. This will instruct the event source to query the entire security log and send any security event codes not listed here to an Unparsed Log Set in Log Search. See the Active Directory article for instructions on how to do this.
• Create an NXLog event source. See the NXLog page for instructions.

Option 1: Send data to Log Search as a single log

By sending your data to Log Search as a single log, you can group logs to reflect your unique environment.

To send data as a single log:

In the following example, log data is grouped into a single “Windows Log” grouping, but you can also create a log for a different subset of your machines, like your dev servers.

1. Create a new logging.json file using the template provided below.
2. Make the following changes:
   • Set <region-code> to your correct region. You can check your region from the Platform Home.
   • Replace <platform-api-key> with your selected API key.
3. Save the logging.json file in the following directory: C:\Program Files\Rapid7\Insight Agent\components\insight_agent\common\config.

4. Once saved, restart the agent service.

json
1
{
2
  "config": {
3
    "name": "Windows Agent",
4
    "endpoint": "<region-code>.data.logs.insight.rapid7.com",
5
    "region": "<region-code>",
6
    "api-key": "<platform-api-key>",
7
    "state-file": "C:\\Program Files\\Rapid7\\Insight Agent\\components\\insight_agent\\common\\state.file",
8
    "formatter" : "plain",
9
    "windows-eventlog": {
10
      "enabled": true,
11
      "destination": "Windows Servers/Event Logs"
12
      },
13
     "logs": []
14
  }
15
}

Option 2: Configure a logging.json file to tail several different logs

You can configure one logging.json file to tail several different logs. This allows you to configure the Insight Agent to send data to Log Search from multiple logs in multiple locations.

The following example shows IIS Logs for forensic value in determining who is accessing your web server and where they are coming from.

Configure logging.json with an array of logs:

1. Create a new logging.json file using the template below.
2. Make the following changes to the template:
   • Set <region-code> to your correct region. You can check your region from the Platform Home.
   • Provide your API key.
   • Save the logging.json file in the following directory: C:\Program Files\Rapid7\Insight Agent\components\insight_agent\common\config.

3. Once saved, restart the agent service.

json
1
{
2
  "config": {
3
    "name": "Windows Agent",
4
    "endpoint": "<region-code>.data.logs.insight.rapid7.com",
5
    "region": "<region-code>",
6
    "api-key": "<platform-api-key>",
7
    "state-file": "C:\\Program Files\\Rapid7\\Insight Agent\\components\\insight_agent\\common\\state.file",
8
    "formatter" : "plain",
9
    "windows-eventlog": {
10
      "enabled": true,
11
      "destination": "Windows Event Logs/IdOPs"
12
      },
13
     "logs": [
14
     {
15
       "destination": "Windows Logs/IIS Logs",
16
       "path": "c:\\inetpub\\logs\\LogFiles\\W3SVC2\\u*.log",
17
       "enabled": true
18
     },
19
     {
20
       "destination": "Windows Logs/Application X Logs",
21
       "path": "d:\\LogFiles\\myLOG.log",
22
       "enabled": true
23
     },
24
     {
25
       "destination": "Enter Desired Log Set Name Here/Enter Desired Log Name Here",
26
       "path": "c:\\local\\path\\to\\log\\file.log",
27
       "enabled": true
28
     }]
29
  }
30
}

Configure a proxy

Once you create your logging.json file, you may need to configure a proxy if your machine cannot directly reach <region>.data.logs.insight.rapid7.com.

To configure a proxy, add the following key-value pairs in the overall config object of your logging.json file:

• proxy-type: Specifies the application protocol. At this time, only HTTP is supported.
• proxy-url: Specifies the address of your proxy server. This can be a hostname or IP address.
• proxy-port: Specifies the port that the Insight Agent will use to communicate with your proxy server.

This example shows how you should format your proxy definition in logging.json:

json
1
{
2
  "config": {
3
    "proxy-type": "HTTP",
4
    "proxy-url": "10.7.1.219",
5
    "proxy-port": "8037",
6
    "name": "Windows Agent",
7
    "endpoint": "<region-code>.data.logs.insight.rapid7.com",
8
    "region": "<region-code>",
9
    "api-key": "<platform-api-key>",
10
    ...followed by the rest of your objects
11
   }
12
}

Log Details

Troubleshoot

If you don’t see any logs coming in, open the agent log and search for Logentries and logging.json to look for errors.

Example of an Error:

 
1
2020-07-20 16:22:38,301 [ERROR] [agent.jobs.le_realtime]: 
2
Error: C:\Program Files\Rapid7\InsightAgent\components\insight_agent\common\config\logging.json 
3
JSON configuration not formatted correctly 
4
Expecting property name enclosed in double quotes: line 13 column 3 (char 414)