Enhanced Endpoint Telemetry
New Log Search is available for Open Preview
We are rolling out a new Log Search experience to customers with an open preview starting January 31st, 2023. You can still use original Log Search during this open preview. Both the original and New Log Search will exist in parallel until development is complete. For now, review the topic on new Log Search and navigate to the Log Search Open Preview page in InsightIDR to become familiar with the new layout. Check back soon for fully updated documentation.
While InsightIDR provides out-of-the-box detection rules for suspicious and malicious events, the information captured by the Insight Agent contains rich metadata that is useful for accelerating alert investigations and facilitating complete incident response.
Available with InsightIDR's Advanced and Ultimate Packages
Enhanced Endpoint Telemetry (EET) is an Ultimate package feature, now widely available to InsightIDR Advanced customers with a limited EET data retention period of seven days. When you purchase the Ultimate package, you receive full access to the archive of process start data captured by the Insight Agent. For pricing and packaging information, visit InsightIDR's Packages and Subscriptions page.
Requirements
To access EET data, you need:
- A license for the Advanced or Ultimate packages, or access to the previously available EET add-on module. For pricing and packaging information, visit InsightIDR's Packages and Subscriptions page.
- The latest version of the Insight Agent installed on your endpoints.
Data retention
By default EET data is available for 13 months. For details about log storage and retention in InsightIDR, view this solution brief.
For InsightIDR Advanced customers, EET data is only available for the previous 7 days.
Optimization and performance tuning
InsightIDR engineering teams utilize a variety of tuning measures to optimize for system performance and data storage limits. These measures may include removal of excessively noisy, irrelevant, or duplicated data that would otherwise clutter dashboards and log sets, as well as data compression to make the best use of your available storage space. When implementing these measures, InsightIDR engineering teams work closely with Rapid7 researchers and security experts to ensure we are collecting data that is the most effective for detecting and investigating malicious activity in your environment.
Query EET data with the Endpoint Activity log set
You can view all of your Endpoint Activity data in Log Search and run queries to analyze activity relevant to your organization.
- Navigate to Log Search.
- Select the Endpoint Activity log set to include all Endpoint Activity logs.
- Update the time range as needed.
- Create a query. To get started, you can check out these sample queries.
Enable case insensitive and partial mapping for inclusive results
Select the Case insensitive and partial mapping checkbox before running a search to ensure differences in capitalization don't lead to missing query results.
Logs available in the Endpoint Activity log set
The Endpoint Activity log set contains three log types: Process Start, Netbios Poisoning, and Local Service Creation.
These Process Start log contains log events where a process starts on an endpoint. The Netbios Poisoning log contains log events where protocol poisoning is seen on an asset. The Local Service Creation log contains log events where any new services are installed on an asset, for example, PowerShell.
Save a query
Access the data you need faster by saving queries for future use. To save a query:
- Create a query and click the Save button to the right of the query bar.
- Name your query.
- Optionally, choose to include your Current Log Selection and Current Time Range.
View saved queries
To view your saved queries, select the Queries dropdown to the left of the query bar.
Sample Queries
EET example queries can be used to search your Endpoint Activity log data.
Tips for querying your endpoint activity data:
- Select the Case insensitive and partial mapping checkbox in Log Search.
- Use Advanced mode.
- Replace values in bold from the example queries with values from your logs.
- Update the time range as needed.
Query process start data
Group by
There may be instances where you want to find data based on specific criteria. You can group your logs by domain, operating systems, file descriptions etc.
- groupby(dns_domain)
- groupby(os_type)
- groupby(process.exe_file.description)
- groupby(process.exe_file.product_name)
Find all unique assets with chosen software running
where(process.name="process.exe" OR parent_process.name="process.exe")groupby(hostname)calculate(unique:hostname)limit(1000)
Find software version
Parent Process
where(parent_process.name="process.exe")groupby(parent_process.exe_file.version)calculate(unique:hostname)
Child Process
where(process.name="process.exe")groupby(process.exe_file.version)calculate(unique:hostname)
Find hosts with psexec or psexecsvc running as either the parent or child process
- where(process.name="psexec.exe" OR parent_process.name="psexec.exe")groupby(hostname)calculate(unique:hostname)limit(1000)
- where(process.name="psexecsvc.exe" OR parent_process.name="psexecsvc.exe")groupby(hostname)calculate(unique:hostname)
Find ping process by hostname and command line
where(process.name="ping.exe")groupby(hostname, process.cmd_line)
Find Netstat processes by hostname and command line
where(process.name="netstat.exe")groupby(hostname, process.cmd_line)
Find unsigned Windows processes
where(process.exe_file.signing_status.status="UNISGNED") broupby(process.name, hostname) calculate(count)
Find processes that contain a specific word in their command line and/or by hostname
where(process.cmd_line icontains "your word")groupby(hostname, process.name)
Find RDP by hostname and command line
where(process.name="mstsc.exe")groupby(hostname, process.cmd_line)
Find Microsoft Management Console and its command line
where(process.name="mmc.exe")groupby(process.cmd_line)
Find MMC launching ADUC by endpoint and username
where(process.name="mmc.exe" AND process.cmd_line icontains "dsa.msc")groupby(hostname, parent_process.username)
Find Mimikatz by hostname and parent process command line
where("mimikatz", loose)groupby(hostname, parent_process.cmd_line)
Group by processes running on unique hosts
groupby(process.name)calculate(unique:hostname)limit(1000)
Find MSIExec installations
where(process.name="msiexec.exe" AND process.cmd_line icontains "/i")groupby(process.cmd_line)
Find MSIExec quiet installations
where(process.name="msiexec.exe" AND process.cmd_line icontains "/quiet")groupby(process.cmd_line)
Group by Linux process permissions and process name
where(os_type="LINUX")groupby(process.exe_file.permission, process.name)
Group by process reputation
groupby(process.hash_reputation.reputation)
Find all processes by name and by reputation
- where(process.hash_reputation.reputation='Known')groupby(process.name)
- where(process.hash_reputation.reputation='Unknown')groupby(process.name)
- where(process.hash_reputation.reputation=’Malicious’)groupby(process.name)
Find processes by reliability
where(process.hash_reputation.reputation='process_reputation')groupby(process.hash_reputation.reliability)
Group by overall process reliability
groupby(process.hash_reputation.reliability)
Find processes by reputational threat level
where(process.hash_reputation.reputation='process_reputation')groupby(process.hash_reputation.threat_level)
Find processes by name, hostname, and username
where(process.hash_reputation.reputation='process_reputation')groupby(process.name, hostname, process.username)
Find PUPs by process name and threat level
where(process.hash_reputation.classification.type='PUA')groupby(process.name, process.hash_reputation.threat_level)
Find PUPs by name, hostname, and username
where(process.hash_reputation.classification.type='PUA')groupby(process.name, hostname, process.username)
Find adware by process name
where(process.hash_repuation.classification.type='Adware')groupby(process.name)
Group by process classification types
groupby(process.hash_reputation.classification.type)
Find password documents
where(process.name=/(winword|excel|notepad|notepad++|textpad).exe/i AND process.cmd_line=/(.password.).(doc|txt|xls).*/i)groupby(process.cmd_line)
Group by all process threat levels
groupby(process.hash_reputation.threat_level)
Find low threat level processes by name
where(process.hash_reputation.threat_level='Low')groupby(process.name)
Find processes by threat level
where(process.hash_reputation.reputation='process_reputation')groupby(process.hash_reputation.threat_level)
Find processes with well-known reputation
where(process.hash_reputation.reliability IN ['Very high', 'High'])groupby(process.name)
Find command lines showing the Taskkill.exe process
where(process.name="taskkill.exe")groupby(process.cmd_line)
Find hostname and users using the PsLoggedon.exe utility
where(process.name="PsLoggedon.exe")groupby(hostname, process.username)
Note: This query shows connected users on the local machine and remote connections using local endpoint resources.
Find hostname and users running the native screencapture.exe software
where(process.name='ScreenCapture.exe')groupby(hostname, process.username)
Find processes, hostnames, and users running programs for SSH and/or Telnet
where(process.exe_file.description icontains-any ["ssh", "telnet"])groupby(process.name, hostname, process.username)
Find hostnames and users performing the <code>whoami</code> command line
where(process.name="whoami.exe")groupby(hostname, process.username)
Query process start data (Powershell)
These queries are designed to help you find useful information related to the PowerShell process.
Find Powershell processes
Parent Process
where(process.name="powershell.exe")groupby(parent_process.name)
Child Process
where(parent_process.name="powershell.exe")groupby(process.name)
Find Powershell process command lines that are not empty or are running a ps1 script
Parent Process
where(parent_process.name="powershell.exe" AND parent_process.cmd_line NOT IIN ["null", ".ps1"])groupby(parent_process.cmd_line)
Child Process
where(process.name="powershell.exe" AND process.cmd_line NOT IIN ["null", ".ps1"])groupby(process.cmd_line)
Find assets running powershell_ISE as either the parent or child process
where(process.name="powershell_ise.exe" OR parent_process.name="powershell_ise.exe")groupby(hostname)calculate(unique:hostname)
Query process start data (PSEXEC)
These queries are designed to help you find useful information related to the PSEXEC process.
Find all psexec command lines
where(process.name="psexec.exe")groupby(process.cmd_line)
Find psexec running the remote process in the system account
where(process.name="psexec.exe" AND process.cmd_line icontains ".s.")groupby(process.cmd_line)
Find remote assets that Psexec is running processes under the system account
where(process.name="psexec.exe" AND process.cmd_line icontains ".s." AND /psexec.exe (?P<remote_asset>[^ ]*)/)groupby(remote_asset)
Query historical user and asset data
Find unique assets associated with a process
where(hostname='hostname') groupby(process.cmd_line, process.exe_path) calculate(count)
Find unique assets and users associated with a process
where(hostname='hostname' and process.username=’username') groupby(process.cmd_line, process.exe_path) calculate(count)
Queries for Threat Hunting
These queries are designed to help you hunt for threats by analyzing endpoint user, process, and command line data.
Find commands being carried out by strings and stopping, all within the command line
where(process.name="cmd.exe" AND process.cmd_line ISTARTS-WITH "cmd.exe /C")groupby(process.cmd_line)
Find command line attempts to remove certain folder attributes
where(process.cmd_line icontains "attrib -h -s -r")
Find hosts and users that have launched a local endpoint's User Account Management
where(process.name IIN ["Netplwiz.exe", "lusmgr.msc"])groupby(hostname, process.username)
Find suspicious login activity
where(hostname="hostname" and process.username="username")
Find additional information about a process that triggered an alert
where(process.pid='process_id' and hostname='hostname', loose)
Find infrequently run commands
where(process.name='process_name' AND hostname='hostname') groupby(data.cmdLine) calculate(count) limit(limit)
Note: Before running the sample query, replace process_name and hostname with the name of the process and host you want to group by.
Find malicious hashes
where(process.exe_file.hashes.sha1="hash") groupby(hostname) calculate(count)
Find command lines where a setup.exe file is being launched from within the downloads folder
where(process.cmd_line icontains-all ["\downloads","setup.exe"])groupby(process.cmd_line)
Find commonly abused commands
Initial Investigation
where(process.cmd_line ISTARTS-WITH-ANY ["tasklist", "ver", "ipconfig", "systeminfo", "net time", "netstat", "whoami", "net start", "qprocess", "query"])groupby(hostname, process.cmd_line)
Reconnaissance
where(process.cmd_line ISTARTS-WITH-ANY ["dir", "net view", "ping", "net use", "type", "net user", "net localgroup", "net group", "net config", "net share"])groupby(hostname, process.cmd_line)
Spread of infection
where(process.cmd_line ISTARTS-WITH-ANY ["at", "reg", "wmic", "netsh advfirewall", "sc", "rundll32"])groupby(hostname, process.cmd_line)
Add process start event data to an investigation
All of your process start data is available for use in Investigations.
To add process start events to an investigation:
- Open the investigation you want to update.
- Select Explore Contextual Data > Search Logs. This will take you to Log Search.
- Select the Endpoint Activity log set.
- Define your query. Refer to the examples above, or check out the Log Search documentation for guidance on writing queries.
- Select the checkboxes beside the log lines you want to add to the investigation, and click Send to Investigation.
- Add the applicable context to the log lines.
- Click Save. The log line will then appear in the Investigation timeline.
Enhanced Endpoint Telemetry Metadata
This section provides a detailed look at the endpoint activity data the Insight Agent sends to Log Search. You can create any number of queries to group and alert on this data. Note that the fields that appear in Log Search vary on your operating system.
Process Start Event
The following table provides information about the process start event metadata collected by the Insight Agent. Some fields vary based on Operating System. If "All" is listed in the Operating System column, the field will be sent to Log Search regardless of operating system.
| Field | Description | Operating System |
|---|---|---|
| hostname | Hostname of the endpoint running the process | All |
| dns_domain | Domain of the endpoint running the process | All |
| os_type | Endpoint operating system | All |
| r7_hostid | Rapid7 Host ID | All |
| process | All data related to the captured process | All |
| parent_process | All data related to the process that spawned the started process | All |
| env_vars | This object shows the environment variables when the process and its parent were launched. The parent value is listed only if it differs from the process value. The process value can be used to find processes that made changes to environment variables prior to launching a child process. If the env_vars of a process is null and the parent process includes environment variables, we will populate this with the parent’s environment variables. | Windows |
| duplicated_events | This represents the count of identical events that occurred in a process. This is populated only if we see similar process events. | Linux |
Process Details
The Insight Agent collects and sends the following information about both the process triggering the event and the parent process. Some fields vary based on the Operating System the process or executible file is running on. If "All" is listed in the Operating System column, the field is sent to Log Search regardless of operating system.
| Field | Description | Operating System |
|---|---|---|
| start_time | Time that this process started | All |
| name | Name of process | All |
| pid | System Process ID | All |
| ppid | Parent system process ID | Mac/Linux, parent only |
| r7_id | Agent-generated ID unique to a process start | All |
| exe_path | Path to the executable | All |
| img_path | Path to the executable. This may differ from exe_path if the executable is on a mounted remote file share. Only sent if it differs from exe_path. | Windows |
| cmd_line | Command line invocation used to start process, including arguments | All |
| username | Local user who started the process | All |
| account_domain | AD domain of the user who started the process | Windows |
| uid | User ID | Mac/Linux |
| group | Group name | Mac/Linux |
| gid | Group ID | Mac/Linux |
| euid_name | Effective user name | Mac/Linux |
| euid | Effective user ID | Mac/Linux |
| egid_name | Effective group name | Mac/Linux |
| egid | Effective group ID | Mac/Linux |
| ruid_name | Real user name | Mac |
| ruid | Real user ID | Mac |
| rgid_name | Real group name | Mac |
| rgid | Real group ID | Mac |
| fsuid | File system user ID | Linux |
| fsgid | File system group ID | Linux |
| suid | Saved user ID | Linux |
| sgid | Saved group ID | Linux |
| session | Login session ID that launched the process | All |
| addr | Remote address that the user is connecting from | Mac |
| port | Port the process used | Mac |
| exe_file | Information about the executable file | All |
Executable File
The following table outlines the metadata that the Insight Agent collects from the executable file.
| Field | Description | Operating System |
|---|---|---|
| exe_file.owner | Owner of the executable file | All |
| exe_file.uid | ID of the executable file owner | Mac/Linux |
| exe_file.group | Group of the executable file | Mac/Linux |
| exe_file.gid | Group ID of the executable file | Mac/Linux |
| exe_file.permissions | Permissions string of the executable file | Mac/Linux |
| exe_file.orig_filename | The original filename from the file metadata | Windows |
| exe_file.description | The description from the file metadata | Windows |
| exe_file.product_name | Product name of the executable as reported by the file metadata | Windows |
| exe_file.author | Company who produced the executable as reported by the file metadata | Windows |
| exe_file.version | Build version of the file from file metadata | Windows |
| exe_file.created | Executable file creation date | All |
| exe_file.last_modified | Executable file last modification date | All |
| exe_file.last_accessed | Executable file last accessed date | Mac/Linux |
| exe_file.size | Executable file size | All |
| exe_file.internal_name | Internal name of the executable file from the metadata | Windows |
| hashes | Collection of different hashes of the process | All |
| signing_status | Signature status | Windows |
| signing_chain | Signature chain | Windows |
| countersigning_chain | Countersignature chain | Windows |
Hashes
The Insight Agent collects and sends the following process hash information:
| Field | Description | Operating System |
|---|---|---|
| hashes.md5 | MD5 hash | All |
| hashes.sha256 | SHA256 hash | All |
| hashes.sha1 | SHA1 hash | All |