Enhanced Endpoint Telemetry
When a process starts on an endpoint that is monitored by the Insight Agent, the agent collects this data and sends it to InsightIDR in the form of detailed logs. These logs include valuable information such as asset name, hostname, process start time, and more.
The Enhanced Endpoint Telemetry add-on module
Enhanced Endpoint Telemetry is available as an add-on to your InsightIDR license. When you purchase the EET add-on module, you receive full access to the archive of process start data that is captured by the Insight Agent. For pricing information, contact your Rapid7 Account Representative.
Benefits of Enhanced Endpoint Telemetry
While InsightIDR provides out-of-the-box detections for suspicious and malicious events, the information captured by the Insight Agent contains rich metadata that is useful for creating custom detections, accelerating alert investigations, and facilitating complete incident response.
You can use enhanced endpoint telemetry to improve the security of your organization in a variety of ways:
- Create custom alerts for tactics that uniquely target your organization, such as a powershell command executed by your custom web application.
- Investigate suspicious activity in context to understand the likelihood of a real threat. For example, you can query your process start data and display all commands that were executed before and after an alert was captured.
- Understand the true scope of an event and respond with greater precision. You can query all endpoint data to determine if a suspicious event was isolated to a specific user or asset, or if multiple users and assets were affected.
- Identify when users are playing games or running other unapproved software on an endpoint.
Requirements
To access EET data, you need:
- A license for the Enhanced Endpoint Telemetry add-on module. For pricing and purchasing options, contact your Rapid7 representative.
- The latest version of the Insight Agent installed on your endpoints.
Data retention
EET data is available for 30 days in hot storage and 90 days in cold storage. For details about log storage and retention in InsightIDR, view this solution brief.
View Endpoint Activity in Log Search
From the InsightIDR left menu, click Log Search. Select the Endpoint Activity log set. Adjust the date range on the date picker as needed.
Query your Process Start Data
In this section, we'll go over ways to unlock the value embedded in your Process Start data.
For sample queries, check out the following sections:
- Query by historical user and asset data
- Drill into a process that triggered an alert
- Group by process command line
- Find Unsigned Windows process
- Investigate suspicious login activity
- Find malicious hashes
Create custom queries
Custom queries allow you to capture and alert on activity that is uniquely relevant to your organization. To query by endpoint activity data, go to the InsightIDR left menu and click Log Search. Select the Endpoint Activity log set, and create a query using the query bar. To get started, you can check out these sample queries.
Tips for querying your data:
- Select the Case insensitive and partial mapping checkbox in Log Search
- Select the Endpoint Activity logset filter
- Replace values in bold from the queries below with real data from your logs
- Update the date range selected in the date picker as needed.
Enable case insensitive and partial mapping in log search
We strongly recommend that you select the Case insensitive and partial mapping checkbox before running a search to ensure the best possible results.
Save a query
After creating a query, click the Save button to the right of the query bar.
View saved queries
To view your saved queries, select the Queries dropdown to the left of the query bar.
Sample Queries
The queries contain placeholder variables (indicated by bold text) that should be replaced with real values from your logs.
Tips for querying your endpoint activity data:
- Select the Case insensitive and partial mapping checkbox in Log Search
- Select the Endpoint Activity logset filter
- Use Advanced mode
- Replace values in bold from the queries below with real data from your logs
- Update the date range selected in the date picker as needed
Query historical user and asset data
When a suspicious event is detected on the endpoint, you can create a query to view all activity that occurred on the host prior to the event. The following section provides 2 queries that you can use to display users and assets associated with a process start event.
Find unique assets associated with a process
where('hostname'="%HOSTNAME%") groupby(process.cmd_line, process.exe_path) calculate(count)
Find unique assets and users associated with a process
where('hostname'="%HOSTNAME%" and ‘process.username’=’'%USER_NAME%') groupby(process.cmd_line, process.exe_path) calculate(count)
Drill into a process that triggered an alert
When a suspicious process is identified, you can review additional information such as user, hash, and environment variables by including the Process ID and the hostname of the asset in your query.
where(process.pid="PID" and hostname="%HOSTNAME%", loose)
Group by process and host
The following query groups your data by process/command line to create patterns of text that make it easy to spot infrequently run commands. Anomalous command lines often indicate that suspicious activity has occurred.
Before running the sample query, be sure to select a date range and update the values in bold as outlined below:
- Replace %PROCESS_NAME% with the name of the process you want to group by
- Replace %HOSTNAME% with the hostname of the asset
- Replace %LIMIT% with desired number of returned results, e.g. 20
where("process.name" = "%PROCESS_NAME%" AND "hostname" = "%HOSTNAME%") groupby(data.cmdLine) calculate(count) limit(%LIMIT%)
Find unsigned Windows processes
The query below can help you identify potentially malicious unsigned Windows processes.
where("process.exe_file.signing_status.status"!="SIGNED_VALID") groupby(process.name, hostname) calculate(count)
Investigate suspicious login activity
If an Insight Agent is installed on a destination asset, you can identify suspicious login activity using the following query. Replace values in bold with real data from your logs.
where('hostname'="%HOSTNAME%" and ‘process.username’=’"%USER_NAME%")
Find malicious hashes
You can use the following query to identify all instances of a malicious file on a machine, regardless of whether or not the file is currently being accessed.
where( "process.exe_file.hashes.sha1" = "%HASH%") groupby(hostname) calculate(count)
Create alerts from custom queries
You can create an alert based on a custom query. To do so, enter your query in the Query Bar, and click the Search icon. Then, click Add Alert and select an alert type. For more information on creating alerts, see Create and Manage Custom Alerts.
Add process start event data to an investigation
All of your process start data is available for use in Investigations.
To add process start events to an investigation:
- Open the investigation you want to update.
- Select Explore Contextual Data > Search Logs. This will take you to Log Search.
- Select the Endpoint Activity log set.
- Define your query. Refer to the examples above, or check out the Log Search documentation for guidance on writing queries.
- Select the checkboxes beside the log lines you want to add to the investigation, and click Send to Investigation.
- Add the applicable context to the log lines.
- Click Save. The log line will then appear in the Investigation timeline.
Enhanced Endpoint Telemetry Metadata
This section provides a detailed look at the endpoint activity data the Insight Agent sends to Log Search. You can create any number of queries to group and alert on this data. Note that the fields that appear in Log Search vary on your operating system.
Process Start Event
The following table provides information about the process start event metadata collected by the Insight Agent. Some fields vary based on Operating System. If "All" is listed in the Operating System column, the field will be sent to Log Search regardless of operating system.
Field | Description | Operating System |
---|---|---|
hostname | Hostname of the endpoint running the process | All |
dns_domain | Domain of the endpoint running the process | All |
os_type | Endpoint operating system | All |
r7_hostid | Rapid7 Host ID | All |
process | All data related to the captured process | All |
parent_process | All data related to the process that spawned the started process | All |
env_vars | This object shows the environment variables when the process and its parent were launched. The parent value is listed only if it differs from the process value. The process value can be used to find processes that made changes to environment variables prior to launching a child process. If the env_vars of a process is null and the parent process includes environment variables, we will populate this with the parent’s environment variables. | Windows |
duplicated_events | This represents the count of identical events that occurred in a process. This is populated only if we see similar process events. | Linux |
Process Details
The Insight Agent collects and sends the following information about both the process triggering the event and the parent process. Some fields vary based on the Operating System the process or executible file is running on. If "All" is listed in the Operating System column, the field is sent to Log Search regardless of operating system.
Field | Description | Operating System |
---|---|---|
start_time | Time that this process started | All |
name | Name of process | All |
pid | System Process ID | All |
ppid | Parent system process ID | Mac/Linux, parent only |
r7_id | Agent-generated ID unique to a process start | All |
exe_path | Path to the executable | All |
img_path | Path to the executable. This may differ from exe_path if the executable is on a mounted remote file share. Only sent if it differs from exe_path. | Windows |
cmd_line | Command line invocation used to start process, including arguments | All |
username | Local user who started the process | All |
account_domain | AD domain of the user who started the process | Windows |
uid | User ID | Mac/Linux |
group | Group name | Mac/Linux |
gid | Group ID | Mac/Linux |
euid_name | Effective user name | Mac/Linux |
euid | Effective user ID | Mac/Linux |
egid_name | Effective group name | Mac/Linux |
egid | Effective group ID | Mac/Linux |
ruid_name | Real user name | Mac |
ruid | Real user ID | Mac |
rgid_name | Real group name | Mac |
rgid | Real group ID | Mac |
fsuid | File system user ID | Linux |
fsgid | File system group ID | Linux |
suid | Saved user ID | Linux |
sgid | Saved group ID | Linux |
session | Login session ID that launched the process | All |
addr | Remote address that the user is connecting from | Mac |
port | Port the process used | Mac |
exe_file | Information about the executable file | All |
Executable File
The following table outlines the metadata that the Insight Agent collects from the executable file.
Field | Description | Operating System |
---|---|---|
exe_file.owner | Owner of the executable file | All |
exe_file.uid | ID of the executable file owner | Mac/Linux |
exe_file.group | Group of the executable file | Mac/Linux |
exe_file.gid | Group ID of the executable file | Mac/Linux |
exe_file.permissions | Permissions string of the executable file | Mac/Linux |
exe_file.orig_filename | The original filename from the file metadata | Windows |
exe_file.description | The description from the file metadata | Windows |
exe_file.product_name | Product name of the executable as reported by the file metadata | Windows |
exe_file.author | Company who produced the executable as reported by the file metadata | Windows |
exe_file.version | Build version of the file from file metadata | Windows |
exe_file.created | Executable file creation date | All |
exe_file.last_modified | Executable file last modification date | All |
exe_file.last_accessed | Executable file last accessed date | Mac/Linux |
exe_file.size | Executable file size | All |
exe_file.internal_name | Internal name of the executable file from the metadata | Windows |
hashes | Collection of different hashes of the process | All |
signing_status | Signature status | Windows |
signing_chain | Signature chain | Windows |
countersigning_chain | Countersignature chain | Windows |
Hashes
The Insight Agent collects and sends the following process hash information:
Field | Description | Operating System |
---|---|---|
hashes.md5 | MD5 hash | All |
hashes.sha256 | SHA256 hash | All |
hashes.sha1 | SHA1 hash | All |