InsightOps - Configure the Insight Agent to Send Logs

The Insight Agent sends asset log data to the Insight platform using a special configuration file called logging.json. By default, the logging.json file instructs the agent to send logs directly to the data.logs.insight.rapid7.com endpoint according to your region, but you can configure a proxy destination if necessary.

This article explains how to generate a logging.json file in InsightOps, provides example formatting by operating system, and defines the JSON objects that the file contains.

Proxy Configuration

If you prefer to send asset log data through a proxy instead of directly to the Insight platform, you can do so by adding the following additional key-value pairs in the overall config object of your logging.json file:

  • proxy-type - Specifies the application protocol. At this time, only HTTP is supported.
  • proxy-url - Specifies the address of your proxy server. This can be a hostname or IP address.
  • proxy-port - Specifies the port that the Insight Agent will use to communicate with your proxy server.

You can specify a Collector as your proxy

If you’ve already deployed one or more Collectors in your environment, you can target one of them to use as a proxy for this use case. The Insight Agent’s collection tasks related to InsightOps do not distinguish between a Collector or another proxy type. Specifying either achieves the same goal of reaching the internet.

Object configuration in logging.json for a Collector address remains the same, except that proxy-port must be set to 8037 in accordance with Collector requirements.

This example shows how you should format your proxy definition in logging.json:

json
1
{
2
"config": {
3
"proxy-type": "HTTP",
4
"proxy-url": "10.7.1.219",
5
"proxy-port": "3128",
6
...followed by the rest of your objects
7
}
8
}

logging.json Creation Wizard

Use the Insight Agent data source wizard in InsightOps to automatically create and format a logging.json file that includes your API key and data region. You can download the complete file directly from the wizard afterwards.

You must create proxy objects manually

The logging.json creation wizard in InsightOps does not currently include the proxy-related objects described previously. If you want the agent to send logs through a proxy, you must edit your logging.json file after downloading it and add the proxy objects manually.

To create your logging.json file with the wizard:

  1. In InsightOps, click the Add Data button.
    • Alternatively, you can click the Add Data link located on either the Log Search or Data Collection tabs.
  2. In the "Install the Insight Agent" section, click the operating system of the asset that you intend to install the agent on.

TIP

The wizard will assume that you have not yet installed the Insight Agent on your assets, so feel free to skip the installation step if you have done so already.

  1. On the "Send Logs via Insight Agent" panel, choose to use an existing API key or to generate a new API key:
    • If you are using an existing API key, paste the key into the field and click the Apply button.
    • If you are generating a new API key, click the Generate API Key button and then click the Apply button. Mac Agent Logging
  2. Verify that your formatted logging.json file includes your API key, your data region endpoint, and your region code. Note that the hostnames in the 'destination' fields can be changed as per requiremnents. Click Download to retrieve your file.
  3. Place your logging.json file in one of the following directories on your asset according to its operating system:
    • Windows - C:\Program Files\Rapid7\Insight Agent\components\insight_agent\common\config
    • Mac and Linux - /opt/rapid7/ir_agent/components/insight_agent/common/config
  4. Finally, restart the Insight Agent service.
    • On Windows, this service is called "Rapid7 Insight Agent" in your service manager.
    • On Mac and Linux, you can restart the service by running the sudo service ir_agent restart command in a terminal.

logging.json Formatting Templates

If you prefer to create your logging.json file manually, see the following templates.

Mac and Linux

The Mac and Linux version of logging.json has the following structure:

logging.json
1
{
2
"config": {
3
"name": "A label for your log following configuration",
4
"endpoint": "eu.data.logs.insight.rapid7.com",
5
"region": "eu",
6
"api-key": "Your API key",
7
"state-file": "/opt/rapid7/ir_agent/components/insight_agent/common/config/logs.state",
8
"formatter" : "plain",
9
"metrics": {
10
"system-stat-token": "your log token here",
11
"system-stat-enabled": true,
12
"metrics-interval": "60s",
13
"metrics-cpu": "system",
14
"metrics-vcpu": "core",
15
"metrics-mem": "system",
16
"metrics-swap": "system",
17
"metrics-net": "sum eth0",
18
"metrics-disk": "sum sda4 sda5",
19
"metrics-space": "/"
20
},
21
"logs": [
22
{
23
"name": "System log",
24
"destination": "your-server-name/syslog",
25
"path": "/var/log/syslog",
26
"enabled": true
27
},
28
{
29
"name": "Kernel",
30
"destination": "your-server-name/kernel",
31
"path": "/var/log/kern.log",
32
"enabled": true
33
},
34
{
35
"name": "Authentication",
36
"destination": "your-server-name/auth",
37
"path": "/var/log/auth.log",
38
"enabled": true
39
},
40
{
41
"name": "Insight Agent log",
42
"token": "your log token here",
43
"path": "/opt/rapid7/ir_agent/components/insight_agent/common/agent.log",
44
"enabled": false
45
}]
46
}
47
}

Windows

NOTE - Event log collection from domain controllers is not supported

The InsightOps component of the Insight Agent does not currently support the collection of event logs from assets acting as domain controllers. If you need to deploy a collection method for this use case, consider implementing one of the following alternatives:

The Windows version of logging.json has the following structure:

logging.json
1
{
2
"config": {
3
"name": "A label for your log following configuration",
4
"endpoint": "eu.data.logs.insight.rapid7.com",
5
"region": "eu",
6
"api-key": "Your API key",
7
"state-file": "C:\\Program Files\\Rapid7\\Insight Agent\\components\\insight_agent\\common\\state.file",
8
"formatter" : "plain",
9
"metrics": {
10
"system-stat-token": "your log token here",
11
"system-stat-enabled": true,
12
"metrics-interval": "60s",
13
"metrics-cpu": "system",
14
"metrics-vcpu": "core",
15
"metrics-mem": "system",
16
"metrics-swap": "system",
17
"metrics-net": "sum eth0",
18
"metrics-disk": "sum sda4 sda5",
19
"metrics-space": "/"
20
},
21
"windows-eventlog": {
22
"enabled": true,
23
"token": "cfe50cc8-7651-42f3-ac23-866bf87a6aad"
24
},
25
"logs": [
26
{
27
"name": "Custom log",
28
"destination": "your-server-name/custom log",
29
"path": "c:\\logs\\mylogfile*.log",
30
"enabled": true
31
},
32
{
33
"name": "Insight Agent log",
34
"token": "your log token here",
35
"path": "C:\\Program Files\\Rapid7\\Insight Agent\\components\\insight_agent\\common\\agent.log",
36
"enabled": false
37
}]
38
}
39
}

Object legend

Object

Description

name

The label for the configuration for your own reference.

endpoint

The log data endpoint that the agent should forward log data to. For example:
AP - ap.data.logs.insight.rapid7.com,
AU - au.data.logs.insight.rapid7.com,
CA - ca.data.logs.insight.rapid7.com,
EU - eu.data.logs.insight.rapid7.com,
US-1 - us.data.logs.insight.rapid7.com,
US-2 - us2.data.logs.insight.rapid7.com,
US-3 - us3.data.logs.insight.rapid7.com

region

The specific region where your deployment of InsightOps is based. For example:
ap, au, ca, eu, us, us2, us3

api-key

This is your Read Write API key that can be found in Settings > API Keys.

state-file

This is the path to a local file that tracks the last entry that was sent to InsightOps. When the agent restarts, it will send logs from that point to minimize missing log data while the agent is offline.

formatter

Setting formatter to "plain" will send the entries to InsightOps exactly as they were written to in the local log file.

Remove this field if you want the log lines to be sent in Syslog format RFC 5424 - adding a timestamp, host and other information.

metrics

This section allows you to send the host's resource utilization metrics to InsightOps.

system-stat-token

Enter a log token in this field to define which log will receive metrics data in InsightOps. You can generate a token using Add Data > Quick Add in the application.

windows-eventlog

This section defines if you want to automatically collect entries from the Windows Event log. These entries will be converted to JSON for easy search analysis.

This does not apply to Domain Controllers.

token

Enter a log token in this field to define which log will receive Windows events in InsightOps. You can generate a token using Add Data > Quick Add in the application.

logs

This section is a list of log files on the host that you want to follow. As new lines are written to these logs, updates will be sent to InsightOps in real time.

name

The label for the configuration for your own reference.

destination

When used, this field will automatically create a Log set and a log if a matching one does not exist.

Replace this field with a token field if you want to write an existing log in InsightOps or aggregate log data in one location in InsightOps.

path

This is the path to the file you wish to follow on the host running the agent. For files on Windows machines, you will need to escape the backslash. For example:
"path": "c:\\logs\\mylogfile.log".

InsightOps supports log file rotation for certain log rollover policies that do not allow you to specify an absolute destination file name. This is typical when log files are timestamped or assigned a sequential number. You can specify a wildcard in the filename so the contents of any active log file matching the pattern in the specified directory will be forwarded to the same log in InsightOps. Be aware that the Insight Agent is not capable of watching a directory itself. Rather, it's designed to watch logs that rotate on dates and sequential numbers, or a syslog rotation.

For example:

"path": "c:\\logs\\mylogfile-*.log"


When specifying a wildcard rule, make sure that your wildcard is anchored to a specific log filename base. Do not specify unanchored wildcards, such as "path": "c:\\logs\\*.log", because this will pull any and all log files being written to your specified directory and result in an inordinate amount of data transmission.


Note that wildcards are not necessary in typical syslog rotation schemes. The typical scheme will perform this rotation by renaming the original logname.log to logname.log.0 before creating a new logname.log to replace it. In this situation, you only need to specify logname.log by itself without any wildcard character.

enabled

This is a boolean value of true or false and is used to specify if the file should be followed or not.

proxy-type

Specifies the application protocol. At this time, only HTTP is supported.

proxy-url

Specifies the address of your proxy server. This can be a hostname or IP address.

proxy-port

Specifies the port that the Insight Agent will use to communicate with your proxy server. If you're specifying a Collector in your proxy definition, this must be set to 8037.