Configure the Insight Agent to Send Logs

The Insight Agent sends asset log data to the Insight platform using a special configuration file called logging.json. By default, the logging.json file instructs the agent to send logs directly to the data.logs.insight.rapid7.com endpoint according to your region, but you can configure a proxy destination if necessary.

This article explains how to generate a logging.json file for InsightIDR, provides example formatting by operating system, and defines the JSON objects that the file contains.

Log forwarding characteristics

  • Log files that are collected with the Insight Agent go directly to Log Search and cannot be parsed with the Custom Parsing Tool.
  • There is no behavioral analytics provided for log entries collected by the Insight Agent.
  • The agent will not automatically route log data through a collector - you must explicitly specify this.
  • The agent will follow all System/Application/Security events if you utilize the Windows log follower.

The configuration file requires that a Platform API Key be added into the "api-key" field. Use the instructions in the Managing Platform API Keys article to generate a key to use in the file.

Proxy configuration

If you prefer to send asset log data through a proxy instead of directly to the Insight platform, you can do so by adding the following additional key-value pairs in the overall config object of your logging.json file:

  • proxy-type - Specifies the application protocol. At this time, only HTTP is supported.
  • proxy-url - Specifies the address of your proxy server. This can be a hostname or IP address.
  • proxy-port - Specifies the port that the Insight Agent will use to communicate with your proxy server.

You can specify a Collector as your proxy

If you’ve already deployed one or more Collectors in your environment, you can target one of them to use as a proxy for this use case. The Insight Agent’s collection tasks related to InsightIDR do not distinguish between a Collector or another proxy type. Specifying either achieves the same goal of reaching the internet.

Object configuration in logging.json for a Collector address remains the same, except that proxy-port must be set to 8037 in accordance with Collector requirements.

This example shows how you should format your proxy definition in logging.json:

json
1
{
2
"config": {
3
"proxy-type": "HTTP",
4
"proxy-url": "10.7.1.219",
5
"proxy-port": "3128",
6
...followed by the rest of your objects
7
}
8
}

Logsets and logs

To create your logging.json file manually, see the following templates.

Mac and Linux

Do not use unanchored wildcards

When specifying a wildcard rule, your wildcard must be anchored to a specific log filename base. Do not specify unanchored wildcards, such as "path": "c:\\logs\\*.log", because this will pull any and all log files being written to your specified directory and result in an inordinate amount of data transmission.

The Mac and Linux version of logging.json has the following structure. Remember to set <region-code> to your correct region, provide your API key, and configure any log names and paths as needed.

The logging.json file should be saved in the following directory: /opt/rapid7/ir_agent/components/insight_agent/common/config. Once saved, restart the agent.

json
1
{
2
"config": {
3
"name": "Linux Agent",
4
"endpoint": "<region-code>.data.logs.insight.rapid7.com",
5
"region": "<region-code>",
6
"api-key": "<platform-api-key>",
7
"state-file": "/opt/rapid7/ir_agent/components/insight_agent/common/config/logs.state",
8
"formatter" : "plain",
9
"logs": [
10
{
11
"destination": "Enter Desired Log Set Name Here/Enter Desired Log Name Here",
12
"path": "/path/to/log/file.log",
13
"enabled": true
14
},
15
{
16
"destination": "Syslog/%hostname%",
17
"path": "/var/log/syslog",
18
"enabled": true
19
},
20
{
21
"destination": "Auth Log/%hostname%",
22
"path": "/var/log/auth.log",
23
"enabled": true
24
}]
25
}

Windows

NOTE - Windows event log collection channels

Windows event logs are collected from the following channels:

  • Application
  • System
  • Security

All entries in these channels are collected, and it is not customizable at this time.

NOTE - Event log collection from domain controllers is not supported

The InsightIDR component of the Insight Agent does not currently support the collection of event logs from assets acting as domain controllers. If you need to deploy a collection method for this use case, consider implementing one of the following alternatives:

Do not use unanchored wildcards

When specifying a wildcard rule, your wildcard must be anchored to a specific log filename base. Do not specify unanchored wildcards, such as "path": "c:\\logs\\*.log", because this will pull any and all log files being written to your specified directory and result in an inordinate amount of data transmission.

The Windows version of logging.json has the following structure. Remember to set <region-code> to your correct region and provide your API key.

The logging.json file should be saved in the following directory: C:\Program Files\Rapid7\Insight Agent\components\insight_agent\common\config. Once saved, restart the agent.

json
1
{
2
"config": {
3
"name": "Windows Agent",
4
"endpoint": "<region-code>.data.logs.insight.rapid7.com",
5
"region": "<region-code>",
6
"api-key": "<platform-api-key>",
7
"state-file": "C:\\Program Files\\Rapid7\\Insight Agent\\components\\insight_agent\\common\\state.file",
8
"formatter" : "plain",
9
"windows-eventlog": {
10
"enabled": true,
11
"destination": "Windows Event Logs/%hostname%"
12
},
13
"logs": []
14
}
15
}

Windows version of logging.json with an array of logs:

The following example shows IIS Logs for forensic value in determining who is accessing your web server and where they are coming in from. Remember to set <region-code> to your correct region and provide your API key:

json
1
{
2
"config": {
3
"name": "Windows Agent",
4
"endpoint": "<region-code>.data.logs.insight.rapid7.com",
5
"region": "<region-code>",
6
"api-key": "<platform-api-key>",
7
"state-file": "C:\\Program Files\\Rapid7\\Insight Agent\\components\\insight_agent\\common\\state.file",
8
"formatter" : "plain",
9
"windows-eventlog": {
10
"enabled": true,
11
"destination": "Windows Event Logs/IdOPs"
12
},
13
"logs": [
14
{
15
"destination": "Windows Logs/IIS Logs",
16
"path": "c:\\inetpub\\logs\\LogFiles\\W3SVC2\\u*.log",
17
"enabled": true
18
},
19
{
20
"destination": "Windows Logs/Application X Logs",
21
"path": "d:\\LogFiles\\myLOG.log",
22
"enabled": true
23
},
24
{
25
"destination": "Application Logs/%hostname% - app",
26
"path": "c:\\local\path\\to\\file-*.log",
27
"enabled": true
28
},
29
{
30
"destination": "%hostname% Security Log/Security-Tool",
31
"path": "c:\\local\path\\to\\file.log",
32
"enabled": true
33
}]
34
}
35
}

Object legend

Object

Description

path

This is the path to the file you wish to follow on the host running the agent. For files on Windows machines, you will need to escape the backslash. For example:
"path": "c:\\logs\\mylogfile.log"

InsightIDR supports log file rotation for certain log rollover policies that do not allow you to specify an absolute destination file name. This is typical when log files are timestamped or assigned a sequential number. You can specify a wildcard in the filename so the contents of any active log file matching the pattern in the specified directory will be forwarded to the same log in InsightIDR. Be aware that the Insight Agent is not capable of watching a directory itself. Rather, it's designed to watch logs that rotate on dates and sequential numbers, or a syslog rotation.

For example:

"path": "c:\\logs\\mylogfile-*.log"


When specifying a wildcard rule, make sure that your wildcard is anchored to a specific log filename base. Do not specify unanchored wildcards, such as "path": "c:\\logs\\*.log", because this will pull any and all log files being written to your specified directory and result in an inordinate amount of data transmission.


Note that wildcards are not necessary in typical syslog rotation schemes. The typical scheme will perform this rotation by renaming the original logname.log to logname.log.0 before creating a new logname.log to replace it. In this situation, you only need to specify logname.log by itself without any wildcard character.

name

The label for the configuration for your own reference.

endpoint

The log data endpoint that the agent should forward log data to. For example:
eu.data.logs.insight.rapid7.com, us.data.logs.insight.rapid7.com, au.data.logs.insight.rapid7.com, ca.data.logs.insight.rapid7.com

region

The specific region where your deployment of InsightIDR is based. For example:
eu, us, ca, au

api-key

This is your Read Write API key that can be found in Settings > API Keys.

state-file

This is the path to a local file that tracks the last entry that was sent to Log Search. When the agent restarts, it will send logs from that point to minimize missing log data while the agent is offline.

formatter

Setting formatter to "plain" will send the entries to InsightIDR exactly as they were written to in the local log file.

Remove this field if you want the log lines to be sent in Syslog format RFC 5424 - adding a timestamp, host and other information.

metrics

This section allows you to send the host's resource utilization metrics to InsightIDR.

system-stat-enabled

Required to be set to true but no stats are currently collected.

windows-eventlog

This section defines if you want to automatically collect entries from the Windows Event log. These entries will be converted to JSON for easy search analysis.

This does not apply to Domain Controllers.

logs

This section is a list of log files on the host that you want to follow. As new lines are written to these logs, updates will be sent to InsightIDR in real time.

This section is a mandatory field for the logging.json file.

destination

When used, this field will automatically create a Log Set and a Log if a matching one does not exist. The Log referred to here is the name given to this set of data in Log Search. For example: destination": "Linux Logs/%hostname%-Secure Log", destination": "Windows Event Logs/%hostname%-Event Logs”

enabled

This is a boolean value of true or false and is used to specify if the file should be followed or not.

proxy-type

Specifies the application protocol. At this time, only HTTP is supported.

proxy-url

Specifies the address of your proxy server. This can be a hostname or IP address.

proxy-port

Specifies the port that the Insight Agent will use to communicate with your proxy server. If you're specifying a Collector in your proxy definition, this must be set to 8037.

Troubleshoot

If you don’t see any logs coming in, open the agent log and search for Logentries and logging.json to look for errors.

Example of an Error:

1
2020-07-20 16:22:38,301 [ERROR] [agent.jobs.le_realtime]:
2
Error: C:\Program Files\Rapid7\InsightAgent\components\insight_agent\common\config\logging.json
3
JSON configuration not formatted correctly
4
Expecting property name enclosed in double quotes: line 13 column 3 (char 414)