InsightOps - Configure Insight Agent Log Encryption Through a Collector

Although Insight Agent-based data sources in InsightOps are designed to communicate directly with the Insight platform, you can force the agent to use a collector to securely send this data instead. To implement this communication method, create a Custom Logs data source in InsightOps and configure your agents with a customized logging.json file.

To enable this collector encryption method:

• Create a Custom Logs Data Source and Download the SSL Certificate from InsightOps
• Append the SSL Certificate to cafile.pem
  • cafile.pem Windows Default Directory
  • cafile.pem Mac and Linux Default Directory
• Retrieve your InsightOps API Key
• Download, Configure, and Locate Your logging.json File
  • Windows logging.json replacement text
  • Linux logging.json replacement text
  • Windows logging.json placement directory
  • Linux logging.json placement directory
• Restart the Insight Agent Service and Verify Event Collection

Create a Custom Logs Data Source and Download the SSL Certificate from InsightOps

In order to ingest Insight Agent logs through a collector, you will need to configure a Custom Logs data source in InsightOps dedicated to agent traffic. Each agent that you intend for this solution must have the necessary Rapid7 SSL certificates in order for the collector to receive and encrypt the data. You can download the SSL certificate from the Custom Logs data source configuration panel in InsightOps.

To create your data source and download your certificate:

1. In InsightOps, go to the left menu and click the Data Collection tab.
2. On the “Add Data Source” screen, browse to the “Security Data” section and click Custom Logs.
3. In the “Collector” field, select the collector that you want the Insight Agent to target.
4. In the “Event Source” field, select Rapid7 Raw Data from the dropdown list.
5. Set the timezone. This must correspond to the timezone of the source data.
6. In the “Collection Method” section, click Listen For Syslog.
7. Enter a port number that your collector will listen on for agent traffic.

• Take note of the port number that you set here and set it aside for later. You will need this port number when you configure your logging.json file.
• This port number must be unique from all your other configured data sources.

8. Check the Encrypted box. This will automatically switch the selected protocol to TCP and produce a Download Certificate button.
9. Click Download Certificate. Your Rapid7CA.pem certificate file will download to your machine.
10. Click Save when finished.

Append the SSL Certificate to cafile.pem

Now that you have your certificate file, you can append its contents to the cafile.pem file in your Insight Agent installation directory. Default locations for this file by operating system are as follows:

cafile.pem Windows Default Directory

C:\Program Files\Rapid7\Insight Agent\components\bootstrap\common\ssl\cafile.pem

cafile.pem Mac and Linux Default Directory

/opt/rapid7/ir_agent/components/bootstrap/common/ssl/cafile.pem

To append your certificate file to cafile.pem:

1. Open your certificate file in a text editor. This file shows one or more certificates denoted by the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- lines. Copy the entire contents of your certificate file, including these opening and closing certificate lines.
2. In the installation directory of the Insight Agent that you want to configure, browse to and open the cafile.pem file with a text editor.
3. Paste the copied contents of your certificate file in cafile.pem on a new line below the previously existing text.
4. Save and close the file.

Retrieve your InsightOps API Key

Now that your SSL certificates have been appended to cafile.pem on each of your agents, you need to retrieve your InsightOps API key. The logging.json file on your Insight Agents needs this API key in order to send its logs to InsightOps.

To retrieve your API key:

1. In InsightOps, expand your left menu and click the Settings tab.
2. Click API Keys on the tab browser.
3. In the “Read/Write” section, copy the API key shown and set it aside for later use.

Download, Configure, and Locate Your logging.json File

At this stage, you should have the following configurations in place:

• A custom logs data source configured for Rapid7 Raw Data
• Rapid7 SSL certificates appended to cafile.pem on each of your intended agents

You should also have the following addresses and key values ready for reference:

• The Fully Qualified Domain Name (FQDN) and port number of the collector that you selected while configuring your custom logs data source
• An InsightOps API key

After verifying that you have all these items ready, you can download and configure the logging.json file for each of your intended agents. The logging.json file configured in this step instructs the agent to send operating system-specific logs through your chosen collector using your InsightOps API key.

The structure of the logging.json file varies between Windows and Linux operating systems. If you need to implement this communication method for both Windows and Linux-based agents, you must configure two logging.json files.

To configure logging.json:

1. In InsightOps, go to the left menu and click the Data Collection tab.
2. On the “Add Data Source” screen, browse to the “Insight Agent” section and click Windows or Linux depending on which operating system you want to configure for first.
3. On the configuration panel, skip ahead to step 4. Copy the values shown for "endpoint" and "region" from the provided logging.json template. These values are automatically populated according to your organization, and you will need them when you configure your replacement logging.json text in the next steps.
4. Still on the configuration panel, proceed to step 5 and click Download. The logging.json file will download to your machine.
5. Open logging.json with a text editor and delete its entire contents. Replace it with the provided text based on the operating system you are configuring for.

Windows logging.json replacement text

{
  "config": {
    "datahub":"<collector_FQDN>:<port>",
    "name": "Windows host",
    "endpoint": "<region_endpoint_url>",
    "region": "<region_code>",
    "api-key": "<API_key>",
    "state-file": "C:\\Program Files\\Rapid7\\Insight Agent\\components\\insight_agent\\common\\state.file",
    "windows-eventlog": {
      "enabled": true,
       "token": ""
    },
    "logs": [
    ]
  }
}

Linux logging.json replacement text

{
  "config": {
    "datahub":"<collector_FQDN>:<port>",
    "name": "Linux host",
    "endpoint": "<region_endpoint_url>",
    "region": "<region_code>",
    "api-key": "<API_key>",
    "state-file": "/opt/rapid7/ir_agent/components/insight_agent/common/config/logs.state",
    "logs": [
      {
        "name": "System log",
        "token": "",
        "path": "/var/log/syslog",
        "enabled": true
      },
      {
        "name": "Kernel",
        "token": "",
        "path": "/var/log/kern.log",
        "enabled": true
      },
      {
        "name": "AuditD",
        "token": "",
        "path": "/var/log/audit/audit.log",
        "enabled": true
      },
      {
        "name": "Authentication",
        "token": "",
        "path": "/var/log/auth.log",
        "enabled": true
      }
    ]
  }

6. After you paste this replacement text in your logging.json file, substitute all placeholder values with the proper information. The placeholder values you need to substitute are as follows:

• <collector_FQDN> - The hostname of your target collector. This must correspond to the collector that you selected while configuring your custom logs data source.
• <port> - The port number that the agent is sending traffic through. As with your collector hostname, this must match the port number that you specified while configuring your custom logs data source.
• <region_endpoint_url> and <region_code> - The endpoint URL and code that corresponds to your organizational data region, copied earlier in this procedure.
• <API-key> - Your InsightOps API key.

7. Save the file when finished.
8. With your logging.json file configured, place it in the following Insight Agent directory of your intended asset.

Windows logging.json placement directory

C:\Program Files\Rapid7\Insight Agent\components\insight_agent\common\config

Linux logging.json placement directory

/opt/rapid7/ir_agent/components/insight_agent/common/config

Restart the Insight Agent Service and Verify Event Collection

Now that everything is in place, you can restart the Insight Agent service and verify that InsightOps is receiving log events.

To restart the agent service and verify event collection:

1. Restart the agent service using the command that corresponds to your endpoint operating system. See Agent Controls for a list of restart commands.
2. After all your intended agents have restarted, return to InsightOps and click the Data Collection tab on your left menu.
3. On the “Add Data Source” screen, click Manage data in the upper right corner.
4. Click the Event Sources tab.
5. In the “Product Types” section, click the Rapid7 listing to display your custom logs data source.
6. Click View raw log to see events arriving on your collector.

You can also verify event collection through the Log Search interface:

1. Click the Log Search tab on your left menu.
2. In the log set category section, expand the Raw Log dropdown. Your “Log Entries” table will begin populating with new events.