Configure the Insight Agent to Send Additional Logs

The Insight Agent performs default event log collection and process monitoring with InsightIDR. However, if you have compliance or operational needs that require additional log monitoring, you can configure the Insight Agent to run another job to send additional data to Log Search using a configuration file named logging.json. The logging.json file instructs the Insight Agent to send logs directly to the data.logs.insight.rapid7.com endpoint according to your region. See log details for more information.

JSON object definitions

When you create your logging.json file, you will need to define several different objects such as, path, name, and destination. For a complete list of object descriptions, check out log details.

Log forwarding use cases

Additional log forwarding produced by the Insight Agent's logging.json file can potentially send an enormous amount of data to the platform, so it’s important to understand when to use this feature to get the most value from it.

Monitor your logs for compliance

Using the Insight Agent to forward logs can help you meet certain compliance requirements because it will give you a more complete picture of all activity happening on a group of your devices.

For example, you can configure the Insight Agent to forward logs from all machines in your environment that store or process payment data. This can help you meet PCI compliance requirements.

Monitor CPU use and memory

The Insight Agent can also collect hardware metrics and usage details like CPU use and disk memory. This can be used to build dashboards for monitoring, reports, and basic detection rules for low disk space for example.

Unsupported collection methods in InsightIDR

The Insight Agent does not support the collection of Windows event logs from assets acting as domain controllers using the logging.json configuration file. If you need to deploy a collection method for this use case, consider using an alternative configuration.

Before you begin

Before you get started, there are a few things to keep in mind:

  • Log files collected by the Insight Agent using the logging.json configuration file bypass our InsightIDR threat detection engine. They are sent directly to our log ingestion endpoint and cannot be parsed using the Custom Parsing Tool because the Custom Parsing tool requires the data be ingested through an event source.
  • Log entries collected by the logging.json method will not trigger any detection rules, or be attributed to legacy detection rules, as the data bypasses our detection engine.
  • The Insight Agent will not automatically route data collected using this method through a Collector. If you want to route data through a Collector, you must configure it as a proxy.

Prerequisites

To use log forwarding through the logging.json file, you must have access to an API key that allows you to create logs and logsets.

Ensure you have the correct permissions:

  1. Create a new user account.
  2. Assign the user to only InsightIDR.
  3. Restrict access to Log Search operations by selecting the Log Search Insight Agent Service Role permission.
  4. Generate an API key.

Do not use an organization key or an administrator’s API key

If you do so, the selected API key inherits that user's permissions and will provide access to other APIs such as user management.

Create your logging.json file

The following examples show a few different ways that you can create your logging.json file. Follow the instructions for Mac and Linux or Windows and select the configuration option that best suits your needs.

Logs must be in a text file format

Source log files to be tailed must be in a UTF-8 text file format. EVTX files or other encoded file formats that are not text will not work with the logging.json method.

Mac and Linux

To create a logging.json file for Mac or Linux:

  1. Copy and paste the template in a new file. Be sure to update the following values:
    • Region code: Set <region-code> to your region. You can check your region from the Platform Home.
    • Platform API Key: Replace <platform-api-key> with your API key.
    • Path: Configure the “path” key to tail specific files on the system.
    • Destination: Configure the destination to send your data to the desired Log Set and Log. In Log Search, you can view the default Log Sets generated by your InsightIDR Collectors. We do not recommend using these Log Sets for this data. Instead, we recommend setting the “destination” to be a new Log Set such as “Data Center 1”. For a log named “Syslog”, your destination would be "destination": "Data Center 1/Syslog",. You may reuse the same destination across multiple machines and all data will be transmitted to the same Log Set\Log.
    • Name: Use a static name.
  2. When ready, save the logging.json file in the following directory: /opt/rapid7/ir_agent/components/insight_agent/common/config.

Protect the logging.json file

Restrict file system permissions to read access by using chmod 0600 to ensure your selected API key is hidden from those who should not have access to the Insight Platform.

  1. After saving your file, restart the agent service.

Mac and Linux Template

If you use this template, be sure to change the values as specified in the previous section.

json
1
{
2
"config": {
3
"name": "Linux Agent",
4
"endpoint": "<region-code>.data.logs.insight.rapid7.com",
5
"region": "<region-code>",
6
"api-key": "<platform-api-key>",
7
"state-file": "/opt/rapid7/ir_agent/components/insight_agent/common/config/logs.state",
8
"logs": [
9
{
10
"name": "Syslog",
11
"destination": "Linux Logs/Syslog",
12
"path": "/var/log/syslog",
13
"enabled": true
14
},
15
{
16
"name": "Audit log",
17
"destination": "Linux Logs/audit.log",
18
"path": "/var/log/audit/audit.log",
19
"enabled": true
20
},
21
{
22
"name": "My Log Name",
23
"destination": "Enter Desired Log Set Name Here/Enter Desired Log Name Here",
24
"path": "/path/to/log/file.log",
25
"enabled": true
26
}
27
]
28
}
29
}

Do not use unanchored wildcards

When specifying a wildcard rule, your wildcard must be anchored to a specific log filename base. Do not specify unanchored wildcards, such as "path": "c:\\logs\\*.log", because this will pull any and all log files being written to your specified directory and result in an inordinate amount of data transmission.

Windows

When you configure the logging.json file to collect Windows event logs, it will collect logs from the following channels:

  • Application
  • System
  • Security

All entries in these channels are collected and cannot be modified to collect a subset or a superset of these event logs. On certain machines, depending on their audit policy, this may be a large amount of data to constantly transmit. You may not want to configure this on all of the machines in your environment.

Domain controller log collection alternatives

While you can utilize the Insight Agent’s IDR event collection on domain controllers, additional event collection will not run on servers with a role of domain controllers or backup domain controllers using the logging.json method. If you need to deploy a collection method for this use case, consider implementing one of the following alternatives for the collection of the Security event log:

  • You can enable the Insight Agent to collect some Security event log events. For more information, see Alternatives to Domain Admin Accounts.
  • As an alternative to running the Insight Agent to collect Active Directory events for InsightIDR as mentioned, Create an Active Directory event source in InsightIDR using WMI and select the Optional box to send unparsed logs using a Collector. This will instruct the event source to query the entire security log and send any security event codes not listed here to an Unparsed Log Set in Log Search. See the Active Directory article for instructions on how to do this.
  • Create an NXLog event source. See the NXLog page for instructions.

Option 1: Send data to Log Search as a single log

By sending your data to Log Search as a single log, you can group logs to reflect your unique environment.

To send data as a single log:

In the following example, log data is grouped into a single “Windows Log” grouping, but you can also create a log for a different subset of your machines, like your dev servers.

  1. Create a new logging.json file using the template provided below.
  2. Make the following changes:
    • Set <region-code> to your correct region. You can check your region from the Platform Home.
    • Replace <platform-api-key> with your selected API key.
  3. Save the logging.json file in the following directory: C:\Program Files\Rapid7\Insight Agent\components\insight_agent\common\config.

Protect the logging.json file

Verify that only the LocalSystem has read and write access to the file to ensure your selected API key is hidden from those who should not have access to the Insight Platform.

  1. Once saved, restart the agent service.
json
1
{
2
"config": {
3
"name": "Windows Agent",
4
"endpoint": "<region-code>.data.logs.insight.rapid7.com",
5
"region": "<region-code>",
6
"api-key": "<platform-api-key>",
7
"state-file": "C:\\Program Files\\Rapid7\\Insight Agent\\components\\insight_agent\\common\\state.file",
8
"formatter" : "plain",
9
"windows-eventlog": {
10
"enabled": true,
11
"destination": "Windows Servers/Event Logs"
12
},
13
"logs": []
14
}
15
}

Option 2: Configure a logging.json file to tail several different logs

You can configure one logging.json file to tail several different logs. This allows you to configure the Insight Agent to send data to Log Search from multiple logs in multiple locations.

The following example shows IIS Logs for forensic value in determining who is accessing your web server and where they are coming from.

Configure logging.json with an array of logs:

  1. Create a new logging.json file using the template below.
  2. Make the following changes to the template:
    • Set <region-code> to your correct region. You can check your region from the Platform Home.
    • Provide your API key.
    • Save the logging.json file in the following directory: C:\Program Files\Rapid7\Insight Agent\components\insight_agent\common\config.

Protect the logging.json file

Verify that only the LocalSystem has read and write access to the file to ensure your selected API key is hidden from those who should not have access to the Insight Platform.

  1. Once saved, restart the agent service.
json
1
{
2
"config": {
3
"name": "Windows Agent",
4
"endpoint": "<region-code>.data.logs.insight.rapid7.com",
5
"region": "<region-code>",
6
"api-key": "<platform-api-key>",
7
"state-file": "C:\\Program Files\\Rapid7\\Insight Agent\\components\\insight_agent\\common\\state.file",
8
"formatter" : "plain",
9
"windows-eventlog": {
10
"enabled": true,
11
"destination": "Windows Event Logs/IdOPs"
12
},
13
"logs": [
14
{
15
"destination": "Windows Logs/IIS Logs",
16
"path": "c:\\inetpub\\logs\\LogFiles\\W3SVC2\\u*.log",
17
"enabled": true
18
},
19
{
20
"destination": "Windows Logs/Application X Logs",
21
"path": "d:\\LogFiles\\myLOG.log",
22
"enabled": true
23
},
24
{
25
"destination": "Enter Desired Log Set Name Here/Enter Desired Log Name Here",
26
"path": "c:\\local\\path\\to\\log\\file.log",
27
"enabled": true
28
}]
29
}
30
}

Do not use unanchored wildcards

When specifying a wildcard rule, your wildcard must be anchored to a specific log filename base. Do not specify unanchored wildcards, such as "path": "c:\\logs\\*.log", because this will pull any and all log files being written to your specified directory and result in an inordinate amount of data transmission.

Configure a proxy

Once you create your logging.json file, you may need to configure a proxy if your machine cannot directly reach <region>.data.logs.insight.rapid7.com.

To configure a proxy, add the following key-value pairs in the overall config object of your logging.json file:

  • proxy-type: Specifies the application protocol. At this time, only HTTP is supported.
  • proxy-url: Specifies the address of your proxy server. This can be a hostname or IP address.
  • proxy-port: Specifies the port that the Insight Agent will use to communicate with your proxy server.

Configure the Collector as a proxy

If you’ve already deployed one or more Collectors they will be listening for inbound proxy requests on port 8037 by default. You can target one of your Collectors as a proxy and streamline your setup by using your existing infrastructure.

This example shows how you should format your proxy definition in logging.json:

json
1
{
2
"config": {
3
"proxy-type": "HTTP",
4
"proxy-url": "10.7.1.219",
5
"proxy-port": "8037",
6
"name": "Windows Agent",
7
"endpoint": "<region-code>.data.logs.insight.rapid7.com",
8
"region": "<region-code>",
9
"api-key": "<platform-api-key>",
10
...followed by the rest of your objects
11
}
12
}

Log Details

ObjectDescription
pathThis is the path to the file you wish to follow on the host running the agent. For files on Windows machines, you will need to escape the backslash. For example:
"path": "c:\\logs\\mylogfile.log"

InsightIDR supports log file rotation for certain log rollover policies that do not allow you to specify an absolute destination file name. This is typical when log files are timestamped or assigned a sequential number. You can specify a wildcard in the filename so the contents of any active log file matching the pattern in the specified directory will be forwarded to the same log in InsightIDR. Be aware that the Insight Agent is not capable of watching a directory itself. Rather, it's designed to watch logs that rotate on dates and sequential numbers, or a syslog rotation.

For example:

"path": "c:\\logs\\mylogfile-*.log"


When specifying a wildcard rule, make sure that your wildcard is anchored to a specific log filename base. Do not specify unanchored wildcards, such as
"path": "c:\\logs\\*.log"
because this will pull any and all log files being written to your specified directory and result in an inordinate amount of data transmission.
nameThe label for the configuration for your own reference. This name will appear in the syslog header, which is prepended to each log entry if the “formatter”:”plain”, line is omitted.
endpointThe region specific log data endpoint that the agent should forward log data to. Check your region.
Australia: au.data.logs.insight.rapid7.com
Canada: ca.data.logs.insight.rapid7.com
Europe: eu.data.logs.insight.rapid7.com
Japan: ap.data.logs.insight.rapid7.com
United States - Region 1: us.data.logs.insight.rapid7.com
United States - Region 2: us2.data.logs.insight.rapid7.com
United States - Region 3: us3.data.logs.insight.rapid7.com
regionThe specific region where your deployment of InsightIDR is based. For example:
au, ca, ap, eu, us, us2, us3
api-keyThe selected API key that is generated and fetched from Platform Home > API Key management.

Do not use an organization key or an administrator’s API key. To use log forwarding through the logging.json file, create a new user account and generate an API key.
state-fileThis is the path to a local file that tracks the last entry that was sent to Log Search. When the agent restarts, it will send logs from that point to minimize missing log data while the agent is offline.
formatterSet formatter to "plain" to send the entries to InsightIDR exactly as they were written in the local log file.

Remove this field if you want the log entries to be sent in syslog format RFC 5424, which will add a timestamp, host and other information to the log.

WARNING: If your logs are natively in JSON and the formatter line is omitted, the logs will no longer be valid JSON as the syslog header will break the format.
metricsThis section allows you to send the host's resource utilization metrics to InsightIDR.
system-stat-enabledRequired to be set to true to collect configured metrics.
windows-eventlogThis section defines if you want to automatically collect entries from the Windows Event log. These entries will be converted to JSON for easy search analysis.

This does not apply to Domain Controllers.
logsThis is a mandatory field for the logging.json file.

This section is a list of log files on the host that you want to follow. As new lines are written to these logs, updates will be sent to InsightIDR in real time. The agent will only follow logs in UTF-8 text format. No encoded log formats (such as .evtx) are supported.
destinationThis field is used to specify which logset and log the data should be sent to. If a matching log already exists, the agent will transmit data to that same log. If not, it will create a new Log Set and Log. The Log referred to here is the name given to this set of data in Log Search. For example:

“destination": "RedHat Servers/Secure Log"

destination": "Windows Workstations/Event Logs”

IMPORTANT: You can use %hostname% as a variable to create a distinct logset or log for each host (for example, “RedHat Servers/%hostname%”), however we do not recommend this configuration if you have more than 150 agents because it can make searching your logs difficult. This method is only recommended to be deployed to a small number of hosts, since there is a limit of 10000 unique logs in log search.
enabledThis is a boolean value of true or false and is used to specify if the file should be followed.

If you do not want to follow a log we recommend that you omit it entirely from the logs:[] object rather than setting the value to false.
proxy-typeSpecifies the application protocol. At this time, only HTTP is supported.
proxy-urlSpecifies the address of your proxy server. This can be a hostname or IP address.
proxy-portSpecifies the port that the Insight Agent will use to communicate with your proxy server. If you're specifying a Collector in your proxy definition, this must be set to 8037.

Troubleshoot

If you don’t see any logs coming in, open the agent log and search for Logentries and logging.json to look for errors.

Example of an Error:

1
2020-07-20 16:22:38,301 [ERROR] [agent.jobs.le_realtime]:
2
Error: C:\Program Files\Rapid7\InsightAgent\components\insight_agent\common\config\logging.json
3
JSON configuration not formatted correctly
4
Expecting property name enclosed in double quotes: line 13 column 3 (char 414)