Leverage Enhanced Endpoint Telemetry Data

While InsightIDR provides out-of-the-box detection rules for suspicious and malicious events, the information captured by the Insight Agent contains rich metadata that is useful for accelerating investigations and facilitating complete incident response. We refer to this data from the Insight Agent as Enhanced Endpoint Telemetry (EET) data, as it provides a more robust understanding of your endpoints’ activity.

Requirements

To access EET data, you need:

  • A license for the Advanced or Ultimate packages, or access to the previously available EET add-on module. For pricing and packaging information, visit InsightIDR’s Packages and Subscriptions page.
  • The Insight Agent installed on your endpoints.

Data retention

For InsightIDR Ultimate customers, EET data is available for 13 months. For InsightIDR Advanced customers, EET data is available for the previous 7 days only.

For details about log storage and retention in InsightIDR, view this solution brief.

Query EET data with the Endpoint Activity log set

You can view all of your EET data in Log Search using the Endpoint Activity log set. Run queries on this log set to analyze the activity relevant to your organization.

  1. In Log Search, select the Endpoint Activity log set.
  2. Update the time range as needed.
  3. Create a query:
    • Refer to the sample queries to get started.
    • Add the loose() clause to ensure differences in capitalization don’t lead to missing query results. Read more about loose search.
  4. Optionally save your query so that you can use it again later.

Logs available in the Endpoint Activity log set

The Endpoint Activity log set contains three logs:

  • Process Start - Contains events where a process starts on an endpoint.
  • Netbios Poisoning - Contains events where protocol poisoning is seen on an asset.
  • Local Service Creation - Contains events where any new services are installed on an asset, for example, PowerShell.

Sample Queries

Use these example queries to search your Endpoint Activity log data.

Replace values in bold from the example queries with values from your logs.

Query process start data

These queries are designed to help you find useful information in your Process Start data.

Groupby

There may be instances where you want to find data based on specific criteria. You can group your logs by domain, operating systems, file descriptions etc.

groupby(dns_domain) groupby(os_type) groupby(process.exe_file.description) groupby(process.exe_file.product_name)

Find all unique assets with chosen software running

where(process.name="**process.exe**" OR parent_process.name="**process.exe**")groupby(hostname)calculate(unique:hostname)limit(1000)

Find software version

Parent Process

where(parent_process.name="**process.exe**")groupby(**parent_process.exe_file.version**)calculate(unique:hostname)

Child Process

where(process.name="**process.exe**")groupby(**process.exe_file.version**)calculate(unique:hostname)

Find hosts with psexec or psexecsvc running as either the parent or child process

where(process.name="**psexec.exe**" OR parent_process.name="**psexec.exe**")groupby(hostname)calculate(unique:hostname)limit(1000) where(process.name="**psexecsvc.exe**" OR parent_process.name="**psexecsvc.exe**")groupby(hostname)calculate(unique:hostname)

Find ping process by hostname and command line

where(process.name="**ping.exe**")groupby(hostname, process.cmd_line)

Find Netstat processes by hostname and command line

where(process.name="**netstat.exe**")groupby(hostname, process.cmd_line)

Find unsigned Windows processes

where(process.exe_file.signing_status.status="UNISGNED") broupby(process.name, hostname) calculate(count)

Find processes that contain a specific word in their command line and/or by hostname

where(process.cmd_line icontains "**your word**")groupby(hostname, process.name)

Find RDP by hostname and command line

where(process.name="**mstsc.exe**")groupby(hostname, process.cmd_line)

Find Microsoft Management Console and its command line

where(process.name="**mmc.exe**")groupby(process.cmd_line)

Find MMC launching ADUC by endpoint and username

where(process.name="**mmc.exe**" AND process.cmd_line icontains "**dsa.msc**")groupby(hostname, parent_process.username)

Find Mimikatz by hostname and parent process command line

where("**mimikatz**", loose)groupby(hostname, parent_process.cmd_line)

Group by processes running on unique hosts

groupby(process.name)calculate(unique:hostname)limit(1000)

Find MSIExec installations

where(process.name="**msiexec.exe**" AND process.cmd_line icontains "**/i**")groupby(process.cmd_line)

Find MSIExec quiet installations

where(process.name="**msiexec.exe**" AND process.cmd_line icontains "**/quiet**")groupby(process.cmd_line)

Group by Linux process permissions and process name

where(os_type="**LINUX**")groupby(process.exe_file.permission, process.name)

Group by process reputation

groupby(process.hash_reputation.reputation)

Find all processes by name and by reputation

where(process.hash_reputation.reputation='Known')groupby(process.name) where(process.hash_reputation.reputation='Unknown')groupby(process.name) where(process.hash_reputation.reputation=’Malicious’)groupby(process.name)

Find processes by reliability

where(process.hash_reputation.reputation='process_reputation')groupby(process.hash_reputation.reliability)

Group by overall process reliability

groupby(process.hash_reputation.reliability)

Find processes by reputational threat level

where(process.hash_reputation.reputation='process_reputation')groupby(process.hash_reputation.threat_level)

Find processes by name, hostname, and username

where(process.hash_reputation.reputation='process_reputation')groupby(process.name, hostname, process.username)

Find PUPs by process name and threat level

where(process.hash_reputation.classification.type='PUA')groupby(process.name, process.hash_reputation.threat_level)

Find PUPs by name, hostname, and username

where(process.hash_reputation.classification.type='PUA')groupby(process.name, hostname, process.username)

Find adware by process name

where(process.hash_repuation.classification.type='Adware')groupby(process.name)

Group by process classification types

groupby(process.hash_reputation.classification.type)

Find password documents

where(process.name=/(winword|excel|notepad|notepad++|textpad).exe/i AND process.cmd_line=/(.password.).(doc|txt|xls).*/i)groupby(process.cmd_line)

Group by all process threat levels

groupby(process.hash_reputation.threat_level)

Find low threat level processes by name

where(process.hash_reputation.threat_level='Low')groupby(process.name)

Find processes by threat level

where(process.hash_reputation.reputation='process_reputation')groupby(process.hash_reputation.threat_level)

Find processes with well-known reputation

where(process.hash_reputation.reliability IN ['Very high', 'High'])groupby(process.name)

Find command lines showing the Taskkill.exe process

where(process.name="**taskkill.exe**")groupby(process.cmd_line)

Find hostname and users using the PsLoggedon.exe utility

where(process.name="**PsLoggedon.exe**")groupby(hostname, process.username)

Note: This query shows connected users on the local machine and remote connections using local endpoint resources.

Find hostname and users running the native screencapture.exe software

where(process.name='ScreenCapture.exe')groupby(hostname, process.username)

Find processes, hostnames, and users running programs for SSH and/or Telnet

where(process.exe_file.description icontains-any ["**ssh**", "**telnet**"])groupby(process.name, hostname, process.username)

Find hostnames and users performing the whoami command line

where(process.name="**whoami.exe**")groupby(hostname, process.username)

Query process start data (Powershell)

These queries are designed to help you find useful information related to the PowerShell process.

Find Powershell processes

Parent Process

where(process.name="**powershell.exe**")groupby(parent_process.name)

Child Process

where(parent_process.name="**powershell.exe**")groupby(process.name)

Find Powershell process command lines that are not empty or are running a ps1 script

Parent Process

where(parent_process.name="**powershell.exe**" AND parent_process.cmd_line NOT IIN ["**null**", "**.ps1**"])groupby(parent_process.cmd_line)

Child Process

where(process.name="**powershell.exe**" AND process.cmd_line NOT IIN ["**null**", "**.ps1**"])groupby(process.cmd_line)

Find assets running powershell_ISE as either the parent or child process

where(process.name="**powershell_ise.exe**" OR parent_process.name="**powershell_ise.exe**")groupby(hostname)calculate(unique:hostname)

Query process start data (PSEXEC)

These queries are designed to help you find useful information related to the PSEXEC process.

Find all psexec command lines

where(process.name="**psexec.exe**")groupby(process.cmd_line)

Find psexec running the remote process in the system account

where(process.name="**psexec.exe**" AND process.cmd_line icontains "**.s.**")groupby(process.cmd_line)

Find remote assets that Psexec is running processes under the system account

where(process.name="**psexec.exe**" AND process.cmd_line icontains "**.s.**" AND /psexec.exe \(?P<remote_asset>[^ ]*)/)groupby(remote_asset)

Query historical user and asset data

When a suspicious event is detected on an endpoint, you can create a query to view all activity that occurred on the host prior to the event. You can use the queries in this section to display users and assets associated with a process start event.

Find unique assets associated with a process

where(hostname='**hostname**') groupby(process.cmd_line, process.exe_path) calculate(count)

Find unique assets and users associated with a process

where(hostname='**hostname**' and process.username=’**username**') groupby(process.cmd_line, process.exe_path) calculate(count)

Queries for Threat Hunting

These queries are designed to help you hunt for threats by analyzing endpoint user, process, and command line data.

Find commands being carried out by strings and stopping, all within the command line

where(process.name="**cmd.exe**" AND process.cmd_line ISTARTS-WITH "**cmd.exe /C**")groupby(process.cmd_line)

Find command line attempts to remove certain folder attributes

where(process.cmd_line icontains "attrib -h -s -r")

Find hosts and users that have launched a local endpoint’s User Account Management

where(process.name IIN ["**Netplwiz.exe**", "lusmgr.msc"])groupby(hostname, process.username)

Find suspicious login activity

where(hostname="**hostname**" and process.username="**username**")

Find additional information about a process that triggered a detection

where(process.pid='**process_id**' and hostname='**hostname**', loose)

Find infrequently run commands

where(process.name='**process_name**' AND hostname='**hostname**') groupby(data.cmdLine) calculate(count) limit(**limit**)

Note: Before running the sample query, replace process_name and hostname with the name of the process and host you want to group by.

Find malicious hashes

where(process.exe_file.hashes.sha1="**hash**") groupby(hostname) calculate(count)

Find command lines where a setup.exe file is being launched from within the downloads folder

where(process.cmd_line icontains-all ["\downloads\","setup.exe"])groupby(process.cmd_line)

Find commonly abused commands

Initial Investigation

where(process.cmd_line ISTARTS-WITH-ANY ["tasklist", "ver", "ipconfig", "systeminfo", "net time", "netstat", "whoami", "net start", "qprocess", "query"])groupby(hostname, process.cmd_line)

Reconnaissance

where(process.cmd_line ISTARTS-WITH-ANY ["dir", "net view", "ping", "net use", "type", "net user", "net localgroup", "net group", "net config", "net share"])groupby(hostname, process.cmd_line)

Spread of infection

where(process.cmd_line ISTARTS-WITH-ANY ["at", "reg", "wmic", "netsh advfirewall", "sc", "rundll32"])groupby(hostname, process.cmd_line)

Understand the Enhanced Endpoint Telemetry metadata

This section provides the endpoint activity data that the Insight Agent sends to Log Search. You can create queries to group and detect on this data.

EET metadata varies by operating system

The keys that appear in Log Search vary based on the operating system of the device where the event occurred.

Process Start Event

The following table provides information about the process start event metadata collected by the Insight Agent. When you purchase the InsightIDR Ultimate package, you receive full access to the archive of process start data captured by the Insight Agent.

If All is listed in the Operating System column, the field is sent to Log Search regardless of the operating system of the device the event occurred on.

FieldDescriptionOperating System
hostnameThe hostname of the endpoint running the process.All
dns_domainThe domain of the endpoint running the process.All
os_typeThe endpoint’s operating system.All
r7_hostidThe Rapid7 Host ID.All
processAll data related to the captured process.All
parent_processAll data related to the process that spawned the started process.All
env_varsThis object shows the environment variables when the process and its parent were launched. The parent value is listed only if it differs from the process value. The process value can be used to find processes that made changes to environment variables prior to launching a child process. If the env_vars of a process is null and the parent process includes environment variables, InsightIDR populates env_vars with the parent’s environment variables.Windows
duplicated_eventsThe count of identical events that occurred in a process. This key is populated only if InsightIDR sees similar process events.Linux

Process Details

The Insight Agent collects and sends the following information about both the process triggering the event and the parent process.

Some fields vary based on the operating system that the process or executable file is running on. If All is listed in the Operating System column, the field is sent to Log Search regardless of operating system.

|Field |Description |Operating System | |---|---|---|---| |start_time | The time that this process started. |All | |name | The name of the process. |All | |pid | The system’s Process ID. |All | |ppid | The parent system’s process ID. |Mac/Linux, parent only | |r7_id | The Insight Agent-generated ID, unique to a process start. |All | |exe_path |The path to the executable. |All | |img_path | The path to the executable. This value might differ from exe_path if the executable is on a mounted remote file share. This key is sent to Log Search only if its value differs from exe_path. |Windows | |cmd_line |The command line invocation used to start the process, including arguments.|All | |username |The local user who started the process. |All | |account_domain |The AD domain of the user who started the process. |Windows | |uid |The user ID. |Mac/Linux | |group |The group name. |Mac/Linux | |gid |The group ID. |Mac/Linux | |euid_name |The effective user name. |Mac/Linux | |euid |The effective user ID. |Mac/Linux | |egid_name |The effective group name. |Mac/Linux | |egid |The effective group ID. |Mac/Linux | |ruid_name |The real user name. |Mac | |ruid |The real user ID. |Mac | |rgid_name |The real group name. |Mac | |rgid |The real group ID. |Mac | |fsuid |The file system user ID. |Linux | |fsgid |The file system group ID. |Linux | |suid |The saved user ID. | Linux | |sgid |The saved group ID. |Linux | |session |The login session ID that launched the process. |All | |addr |The remote address that the user is connecting from. |Mac | |port |The port the process used. |Mac | |exe_file |The information about the executable file. |All |

Executable File

The following table outlines the metadata that the Insight Agent collects from the executable file.

|Field |Description |Operating System | |---|---|---|---| |exe_file.owner |The owner of the executable file. |All | |exe_file.uid |The ID of the executable file owner. |Mac/Linux | |exe_file.group |The group of the executable file. |Mac/Linux | |exe_file.gid |The group ID of the executable file. |Mac/Linux | |exe_file.permissions |The permissions string of the executable file. |Mac/Linux | |exe_file.orig_filename |The original filename from the file metadata. | Windows | |exe_file.description |The description from the file metadata. |Windows | |exe_file.product_name |The product name of the executable, as reported by the file metadata. | Windows | |exe_file.author |The company who produced the executable, as reported by the file metadata. |Windows | |exe_file.version |The build version of the file, from the file metadata. |Windows | |exe_file.created |The executable file’s creation date. |All | |exe_file.last_modified |The executable file’s last modification date. |All | |exe_file.last_accessed |The executable file’s last accessed date. |Mac/Linux | |exe_file.size |The executable file’s size. |All | |exe_file.internal_name |The internal name of the executable file, from the metadata. | Windows | |hashes |The collection of different hashes of the process. |All | |signing_status |The signature status. |Windows | |signing_chain |The signature chain. |Windows | |countersigning_chain |The countersignature chain. |Windows |

Hashes

The Insight Agent collects and sends the following process hash information:

|Field |Description |Operating System | |---|---|---|---| |hashes.md5 |The MD5 hash. |All | |hashes.sha256 |The SHA256 hash. |All | |hashes.sha1 |The SHA1 hash. |All |