Collector Troubleshooting
If you are having issues with your InsightIDR Collector, you can use some of the troubleshooting steps below to try and resolve the issues:
- Collector Activation Key Does Not Work
- Collector Shows as Inactive
- Increasing RAM to the Collector
- Collector Log Location
- Syslog Data Not Appearing
- Monitor an Unsupported Event Source
- Error: Unqualified Hostname
- Collector stuck writing to spillover directory
- No credential found with ID
- Tombstone Error
- Collector blocked by Antivirus
Linux Specific Errors include:
- Installation Error on Linux
- Cannot Find Linux Activation Key
- Uninstall Error on RH Linux
- Linux Collectors Missing Collector Details
- Increase file descriptor limits for the collector on Linux
Collector Activation Key Does Not Work
First, make sure you have the correct activation key. It's located in the AgentKey.html file in the insightidr/agent_key subdirectory of the destination directory where you installed the Collector.
If the key is correct but still does not work, it may have been voided. This can occur if you do not activate the Collector immediately after installing it or if you have restarted the server where the Collector is installed.
If the activation key has been voided, you will need to delete and reinstall your Collector.
Collector Shows as Inactive
Try restarting the Collector service. Contact Rapid7 support if restarting does not fix your issue.
- For Linux Collectors, run
service collector restart
from the command line. On some systems, this command may need superuser privileges (sudo
). - For Windows collectors, open the Services app and restart the 'Collector' service.
Collector Failed to Activate
If activation fails, there is likely a network or routing configuration that is preventing your Collector host from communicating with the Insight platform. Remember that the following URLs should be allowlisted in firewalls/web proxies:
US Customers
US Region 1
US Region 2
- https://us2.data.insight.rapid7.com
- https://us2.endpoint.ingress.rapid7.com
- https://s3.us-east-2.amazonaws.com
US Region 3
- https://us3.data.insight.rapid7.com
- https://us3.endpoint.ingress.rapid7.com
- https://s3.us-west-2.amazonaws.com
European Customers
- https://eu.data.insight.rapid7.com
- https://eu.endpoint.ingress.rapid7.com
- https://s3.eu-central-1.amazonaws.com
Canada Customers
- https://ca.data.insight.rapid7.com
- https://ca.endpoint.ingress.rapid7.com
- https://s3-ca-central-1.amazonaws.com
Australian Customers
- https://au.data.insight.rapid7.com
- https://au.endpoint.ingress.rapid7.com
- https://s3-ap-southeast-2.amazonaws.com
Japan Customers
- https://ap.data.insight.rapid7.com
- https://ap.endpoint.ingress.rapid7.com
- https://s3-ap-southeast-2.amazonaws.com
Increase RAM to the Collector
You can increase the amount of RAM allocated to the Collector in environments that require a lot of RAM. To do so, place a file in the same directory where you installed the Collector with the name collector.vmoptions
which contains the line: -Xmx#g
.
where "#" is the number of GB of memory the Collector should use.
- For a 4GB machine, you can tell the collector to use 3GB of memory by putting
-Xmx3g
in the file. - For an 8GB machine, you can tell the collector to take 6GB of memory by saving a collector.vmoptions file in the collector directory with the line
-Xmx6g
.
Collector Log Location
If you need to view the Collector logs for troubleshooting, you can find them in the following location: [installation_directory]/collector/logs
Syslog Data Not Appearing
If the Collector has a local firewall running, that firewall may be blocking the port you configured for the event source. Check your firewall settings to make sure the device can communicate with the InsightIDR Collector through the configured port. If firewall settings seem to be correct, try stopping the current event source and configuring a Rapid7 Generic Syslog event source to listen to the same port. If the generic syslog shows EPM, there is a problem with the log format. Contact support for further assistance.
Monitor an Unsupported Event Source
You can use the Rapid7 Universal Event Sources to monitor certain unsupported event sources. You can also attempt the same with Generic Syslog.
Error: Unqualified Hostname
Error: The hostname [hostname of machine running the Collector] is not fully qualified. Agents may not be able to connect to this collector
To resolve this error, complete one of these troubleshooting flows:
On a VM:
- Power off your virtual machine.
- Change the machine name to a FQDN.
- Power on your virtual machine.
For Windows:
- In the Control Panel, go to Network and Sharing Center, and select Change Settings in the Computer Name, domain, and workgroup settings section.
- Right-click the network adapter you are configuring and choose Properties.
- Select Internet Protocol 4 (TCP/IPv4) and then choose Properties.
- Select Advanced and then select DNS.
- Append the DNS suffix (or suffixes).
- In the Windows File Explorer, navigate to your local disk > Windows > System > drivers > etc and locate the hosts file.
- Ensure you are in administrator mode, then open the host file using a text editor.
- At the end of the text file, enter your local IP address, followed by a space and the domain name. For example,
127.0.0.1 collector1.rapid7poc.com
. - Save the file.
- To validate that you completed steps 1 - 8 correctly, open the Administrator: Command Prompt and ping the domain name you entered in the text editor.
- Use this flow to access the Registry Editor:
Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
. Edit the registry value's Domain.
For Linux:
- Configure the /etc/hosts file so that the first entry is
IP Hostname Alias
. For example:
1IPAddress Hostname Alias2127.0.0.1 thishost.mydomain.org thishost
- Restart your network to apply the changes.
Collector stuck writing to spillover directory
In situations where there are connectivity issues to the cloud, the InsightIDR Collector may start to continuously write data to the spillover directory. The spillover directory is meant to act as a temporary way to save data until the connection is reestablished to the Insight Cloud.
However, if the Collector reestablishes connection to the cloud but does not stop writing data to the spillover directory, it may impact the Collector performance. If you are seeing this behavior and it correlates with high CPU and prolonged delayed events, use the following instructions to stop it:
- Stop the collector by running the following command:
sudo service collector stop
- If the command to stop the Collector service times out, use the following command to kill the process instead:
kill $(ps aux | grep '[c]ollector | awk '{print $2}')
- If you are using Windows, you can kill the process via the Task Manager
- You must then move or remove the contents of the Spillover Directory.
- If you are a Mac user, the directory is located here: /rapid7//spillover-directory/
- If you are a Windows user, the directory is located in the Collector installation folder.
- Start the Collector via the Server Manager in Windows or by running the following command:
/etc/init.d/collector start
- Check InsightIDR to confirm the Collector is running normally and that both CPU and memory are in the green.
- Monitor the Collector logs over the next 24 hours to ensure the above steps corrected the issue.
No credential found with ID
Error: No credential found with ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
This error indicates that there is an issue with the wiring at the event source level on the collector. To resolve this error, delete the event source from InsightIDR, then recreate it.
Collector blocked by Antivirus
Endpoint security applications (such as McAfee Threat Intelligence Exchange, CylancePROTECT, Carbon Black, and others) may flag, block, or delete the Collector from your assets depending on your detection and response settings.
To prevent this from happening, we recommend that you configure an allow list rule for the directory of the collector so your endpoint security software does not accidentally target it.
Installation Error on Linux
When installing the Collector on a Linux host, the following error can occur:
java
1Unpacking JRE ...2Starting Installer ...3java.lang.NoClassDefFoundError: java.awt.Container4at com.install4j.runtime.installer.frontend.headless.AbstractHeadlessScreenExecutor.init(AbstractHeadlessScreenExecutor.java:67)5at com.install4j.runtime.installer.frontend.headless.ConsoleScreenExecutor.init(ConsoleScreenExecutor.java:24)6at com.install4j.runtime.installer.frontend.headless.InstallerConsoleScreenExecutor.init(InstallerConsoleScreenExecutor.java:6)7at com.install4j.runtime.installer.Installer.getScreenExecutor(Installer.java:92)8at com.install4j.runtime.installer.Installer.runInProcess(Installer.java:58)9at com.install4j.runtime.installer.Installer.main(Installer.java:46)10at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)11at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)12at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)13at java.lang.reflect.Method.invoke(Unknown Source)14at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:65)15at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:57)
To resolve this issue, run the installer in console mode by adding
-c
sudo ./InsightSetup-Linux64.sh -c
Cannot Find Linux Activation Key
If you cannot find the activation key for Linux installations, you can find it here: /opt/rapid7/collector/agent-key/Agent_Key.html
Uninstall Error on RH Linux
When uninstalling the Collector on a RHEL machine, complete the following if you see this error:
1Could not display the GUI. This application needs access to an X Server.2*******************************************************************3You can also run this application in console mode without4access to an X server by passing the argument -c5*******************************************************************
- Install the RPM package
redhat-lsb.i686
- Run the uninstall script again.
Linux Collector Missing Collector Details
View your Linux Collector details on Data Collection > Data Collection Health > Collectors.
If your Linux Collectors are not showing details such as the hostname, IP address, OS version, or CPU and Memory usage, the Collector may be having trouble running code from the /tmp
directory.
Some distributions of Linux prevent code from running in the /tmp
directory for security reasons. To verify that this is the issue, open your Collector's log file and look for log entries similar to the following:
1java.lang.UnsatisfiedLinkError: /tmp/jna-3506402/jna5825717272410834572.tmp: /tmp/jna-3506402/jna5825717272410834572.tmp: failed to map segment from shared object: Operation not permitted
To fix this issue, execute the following command in the /opt/rapid7/collector
directory:
1sudo bash -c 'mkdir tmp && echo "-Djava.io.tmpdir=/opt/rapid7/collector/tmp" > collector.vmoptions'
Then, restart the Collector service after executing the above command to resolve the issue.
Increase file descriptor limits for the collector on Linux
For certain OS's like CentOS & RHEL 6 the version of glibc used is incompatible with the version required by the Collector. Usually this can be identified by see the following log message in the collector logs:
1WARN cnc-sync-worker-00 com.rapid7.razor.collector.endpoint.cloudproxy.impl.jna.ProcessLimits:101 - Failed to link to the C standard library - native methods will be disabled2java.lang.UnsatisfiedLinkError: /tmp/jna-3506402/jna1776262045738086760.tmp: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /tmp/jna-3506402/jna1776262045738086760.tmp)
To resolve this issue, do the following:
- Edit
/opt/rapid7/collector/collector
and add the following lines after the initial comments:
Shell
1# Uncomment the following line to add additional VM parameters2# INSTALL4J_ADD_VM_PARAMS=34ulimit -Hn 1000005ulimit -Sn 100000
- Restart the collector service by running.
1service collector restart
- Verify the limit has been changed successful by looking for the following output in
/opt/rapid7/collector/logs/collector.log
12017-11-20 06:21:01.039 INFO cnc-sync-worker-00 com.rapid7.razor.collector.endpoint.cloudproxy.impl.EndpointCloudProxy:119 - numCores: 1, maxFileDescriptors:10000022017-11-20 06:21:01.044 INFO cnc-sync-worker-00 com.rapid7.razor.collector.endpoint.cloudproxy.impl.EndpointCloudProxy:120 - ThrottleMaxConnectionsPerSecond: 10, MaxPendingTask: 1024(128 * 8)
Collector Service Not Auto Starting With Host
The Collector installs and sets up the system and service manager so the Collector can automatically start with the host. If this fails, there is likely a host OS incompatibility or interference from other softwares, for example, endpoint protection software.
To reboot the Collector Service:
- Open the Collector terminal and log in as a root user. Enter
sudo su -
- Open Crontab. Enter
crontab -e
- Enter the following line:
@reboot /opt/rapid7/collector/collector restart
- Save the file. The method to save depends on the editor used. For vi, enter
:wq
) - Test the change by rebooting the machine. Enter
shutdown -r now
- On the Collector host, check that the service is running. Enter
service collector status
- Monitor the platform to make sure the Collector reports in after the system is rebooted.