Collector Troubleshooting

If you are having issues with your InsightIDR Collector, you can use some of the troubleshooting steps below to try and resolve the issues:

Linux Specific Errors include:

Collector Activation Key Does Not Work

First, make sure you have the correct activation key. It's located in the AgentKey.html file in the insightidr/agent_key subdirectory of the destination directory where you installed the Collector.

If the key is correct but still does not work, it may have been voided. This can occur if you do not activate the Collector immediately after installing it or if you have restarted the server where the Collector is installed.

If the activation key has been voided, you will need to delete and reinstall your Collector.

Collector Shows as Inactive

Try restarting the Collector service. Contact Rapid7 support if restarting does not fix your issue.

  • For Linux Collectors, run service collector restart from the command line. On some systems, this command may need superuser privileges (sudo).
  • For Windows collectors, open the Services app and restart the 'Collector' service.

Collector Failed to Activate

If activation fails, there is likely a network or routing configuration that is preventing your Collector host from communicating with the Insight platform. Remember that the following URLs should be allowlisted in firewalls/web proxies:

US Customers

US Region 1

US Region 2

US Region 3

European Customers

Canada Customers

Australian Customers

Japan Customers

Increase RAM to the Collector

You can increase the amount of RAM allocated to the Collector in environments that require a lot of RAM. To do so, place a file in the same directory where you installed the Collector with the name collector.vmoptions which contains the line: -Xmx#g.

where "#" is the number of GB of memory the Collector should use.

  • For a 4GB machine, you can tell the collector to use 3GB of memory by putting -Xmx3g in the file.
  • For an 8GB machine, you can tell the collector to take 6GB of memory by saving a collector.vmoptions file in the collector directory with the line -Xmx6g.

Collector Log Location

If you need to view the Collector logs for troubleshooting, you can find them in the following location: [installation_directory]/collector/logs

Syslog Data Not Appearing

If the Collector has a local firewall running, that firewall may be blocking the port you configured for the event source. Check your firewall settings to make sure the device can communicate with the InsightIDR Collector through the configured port. If firewall settings seem to be correct, try stopping the current event source and configuring a Rapid7 Generic Syslog event source to listen to the same port. If the generic syslog shows EPM, there is a problem with the log format. Contact support for further assistance.

Monitor an Unsupported Event Source

You can use the Rapid7 Universal Event Sources to monitor certain unsupported event sources. You can also attempt the same with Generic Syslog.

Error: Unqualified Hostname

Error: The hostname [hostname of machine running the Collector] is not fully qualified. Agents may not be able to connect to this collector

To resolve this error, complete one of these troubleshooting flows:

On a VM:

  1. Power off your virtual machine.
  2. Change the machine name to a FQDN.
  3. Power on your virtual machine.

For Windows:

  1. In the Control Panel, go to Network and Sharing Center, and select Change Settings in the Computer Name, domain, and workgroup settings section.
  2. Right-click the network adapter you are configuring and choose Properties.
  3. Select Internet Protocol 4 (TCP/IPv4) and then choose Properties.
  4. Select Advanced and then select DNS.
  5. Append the DNS suffix (or suffixes).
  6. In the Windows File Explorer, navigate to your local disk > Windows > System > drivers > etc and locate the hosts file.
  7. Ensure you are in administrator mode, then open the host file using a text editor.
  8. At the end of the text file, enter your local IP address, followed by a space and the domain name. For example, 127.0.0.1 collector1.rapid7poc.com.
  9. Save the file.
  10. To validate that you completed steps 1 - 8 correctly, open the Administrator: Command Prompt and ping the domain name you entered in the text editor.
  11. Use this flow to access the Registry Editor: Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters. Edit the registry value's Domain.
    • InsightIDR Unqualified Hostname Registry Editor Example

For Linux:

  1. Configure the /etc/hosts file so that the first entry is IP Hostname Alias. For example:
 
1
IPAddress     Hostname                  Alias
2
127.0.0.1     thishost.mydomain.org    thishost
  1. Restart your network to apply the changes.

Collector stuck writing to spillover directory

In situations where there are connectivity issues to the cloud, the InsightIDR Collector may start to continuously write data to the spillover directory. The spillover directory is meant to act as a temporary way to save data until the connection is reestablished to the Insight Cloud.

However, if the Collector reestablishes connection to the cloud but does not stop writing data to the spillover directory, it may impact the Collector performance. If you are seeing this behavior and it correlates with high CPU and prolonged delayed events, use the following instructions to stop it:

  1. Stop the collector by running the following command: sudo service collector stop
  2. If the command to stop the Collector service times out, use the following command to kill the process instead: kill $(ps aux | grep '[c]ollector | awk '{print $2}')
    • If you are using Windows, you can kill the process via the Task Manager
  3. You must then move or remove the contents of the Spillover Directory.
    • If you are a Mac user, the directory is located here: /rapid7//spillover-directory/
    • If you are a Windows user, the directory is located in the Collector installation folder.
  4. Start the Collector via the Server Manager in Windows or by running the following command: /etc/init.d/collector start
  5. Check InsightIDR to confirm the Collector is running normally and that both CPU and memory are in the green.
  6. Monitor the Collector logs over the next 24 hours to ensure the above steps corrected the issue.

No credential found with ID

Error: No credential found with ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

This error indicates that there is an issue with the wiring at the event source level on the collector. To resolve this error, delete the event source from InsightIDR, then recreate it.

Collector blocked by Antivirus

Endpoint security applications (such as McAfee Threat Intelligence Exchange, CylancePROTECT, Carbon Black, and others) may flag, block, or delete the Collector from your assets depending on your detection and response settings.

To prevent this from happening, we recommend that you configure an allow list rule for the directory of the collector so your endpoint security software does not accidentally target it.

Installation Error on Linux

When installing the Collector on a Linux host, the following error can occur:

java
1
Unpacking JRE ...
2
Starting Installer ...
3
java.lang.NoClassDefFoundError: java.awt.Container
4
        at com.install4j.runtime.installer.frontend.headless.AbstractHeadlessScreenExecutor.init(AbstractHeadlessScreenExecutor.java:67)
5
        at com.install4j.runtime.installer.frontend.headless.ConsoleScreenExecutor.init(ConsoleScreenExecutor.java:24)
6
        at com.install4j.runtime.installer.frontend.headless.InstallerConsoleScreenExecutor.init(InstallerConsoleScreenExecutor.java:6)
7
        at com.install4j.runtime.installer.Installer.getScreenExecutor(Installer.java:92)
8
        at com.install4j.runtime.installer.Installer.runInProcess(Installer.java:58)
9
        at com.install4j.runtime.installer.Installer.main(Installer.java:46)
10
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
12
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
13
        at java.lang.reflect.Method.invoke(Unknown Source)
14
        at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:65)
15
        at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:57)

To resolve this issue, run the installer in console mode by adding -csudo ./InsightSetup-Linux64.sh -c

Cannot Find Linux Activation Key

If you cannot find the activation key for Linux installations, you can find it here: /opt/rapid7/collector/agent-key/Agent_Key.html

Uninstall Error on RH Linux

When uninstalling the Collector on a RHEL machine, complete the following if you see this error:

 
1
Could not display the GUI. This application needs access to an X Server.
2
 *******************************************************************
3
You can also run this application in console mode without 
4
access to an X server by passing the argument -c
5
*******************************************************************
  1. Install the RPM package redhat-lsb.i686
  2. Run the uninstall script again.

Linux Collector Missing Collector Details

View your Linux Collector details on Data Collection > Data Collection Health > Collectors.

If your Linux Collectors are not showing details such as the hostname, IP address, OS version, or CPU and Memory usage, the Collector may be having trouble running code from the /tmp directory.

Some distributions of Linux prevent code from running in the /tmp directory for security reasons. To verify that this is the issue, open your Collector's log file and look for log entries similar to the following:

 
1
java.lang.UnsatisfiedLinkError: /tmp/jna-3506402/jna5825717272410834572.tmp: /tmp/jna-3506402/jna5825717272410834572.tmp: failed to map segment from shared object: Operation not permitted

To fix this issue, execute the following command in the /opt/rapid7/collector directory:

 
1
sudo bash -c 'mkdir tmp && echo "-Djava.io.tmpdir=/opt/rapid7/collector/tmp" > collector.vmoptions'

Then, restart the Collector service after executing the above command to resolve the issue.

Increase file descriptor limits for the collector on Linux

For certain OS's like CentOS & RHEL 6 the version of glibc used is incompatible with the version required by the Collector. Usually this can be identified by see the following log message in the collector logs:

 
1
WARN cnc-sync-worker-00 com.rapid7.razor.collector.endpoint.cloudproxy.impl.jna.ProcessLimits:101 - Failed to link to the C standard library - native methods will be disabled 
2
java.lang.UnsatisfiedLinkError: /tmp/jna-3506402/jna1776262045738086760.tmp: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /tmp/jna-3506402/jna1776262045738086760.tmp)

To resolve this issue, do the following:

  1. Edit /opt/rapid7/collector/collector and add the following lines after the initial comments:
Shell
1
# Uncomment the following line to add additional VM parameters
2
# INSTALL4J_ADD_VM_PARAMS=
3
 
4
ulimit -Hn 100000
5
ulimit -Sn 100000
  1. Restart the collector service by running.
 
1
service collector restart
  1. Verify the limit has been changed successful by looking for the following output in /opt/rapid7/collector/logs/collector.log
 
1
2017-11-20 06:21:01.039 INFO cnc-sync-worker-00 com.rapid7.razor.collector.endpoint.cloudproxy.impl.EndpointCloudProxy:119 - numCores: 1, maxFileDescriptors:100000
2
2017-11-20 06:21:01.044 INFO cnc-sync-worker-00 com.rapid7.razor.collector.endpoint.cloudproxy.impl.EndpointCloudProxy:120 - ThrottleMaxConnectionsPerSecond: 10, MaxPendingTask: 1024(128 * 8)

Collector Service Not Auto Starting With Host

The Collector installs and sets up the system and service manager so the Collector can automatically start with the host. If this fails, there is likely a host OS incompatibility or interference from other softwares, for example, endpoint protection software.

To reboot the Collector Service:

  1. Open the Collector terminal and log in as a root user. Enter sudo su -
  2. Open Crontab. Enter crontab -e
  3. Enter the following line: @reboot /opt/rapid7/collector/collector restart
  4. Save the file. The method to save depends on the editor used. For vi, enter :wq)
  5. Test the change by rebooting the machine. Enter shutdown -r now
  6. On the Collector host, check that the service is running. Enter service collector status
  7. Monitor the platform to make sure the Collector reports in after the system is rebooted.