Leverage Enhanced Endpoint Telemetry Data

While InsightIDR provides out-of-the-box detection rules for suspicious and malicious events, the information captured by the Insight Agent contains rich metadata that is useful for accelerating investigations and facilitating complete incident response. We refer to this data from the Insight Agent as Enhanced Endpoint Telemetry (EET) data, as it provides a more robust understanding of your endpoints’ activity.

Requirements

To access EET data, you need:

  • A license for the Advanced or Ultimate packages, or access to the previously available EET add-on module. For pricing and packaging information, visit InsightIDR's Packages and Subscriptions page.
  • The Insight Agent installed on your endpoints.

Data retention

For InsightIDR Ultimate customers, EET data is available for 13 months. For InsightIDR Advanced customers, EET data is available for the previous 7 days only.

For details about log storage and retention in InsightIDR, view this solution brief.

Query EET data with the Endpoint Activity log set

You can view all of your EET data in Log Search using the Endpoint Activity log set. Run queries on this log set to analyze the activity relevant to your organization.

  1. In Log Search, select the Endpoint Activity log set.
  2. Update the time range as needed.
  3. Create a query:
    • Refer to the sample queries to get started.
    • Add the loose() clause to ensure differences in capitalization don't lead to missing query results. Read more about loose search.
  4. Optionally save your query so that you can use it again later.

Logs available in the Endpoint Activity log set

The Endpoint Activity log set contains three logs:

  • Process Start - Contains events where a process starts on an endpoint.
  • Netbios Poisoning - Contains events where protocol poisoning is seen on an asset.
  • Local Service Creation - Contains events where any new services are installed on an asset, for example, PowerShell.

Sample Queries

Use these example queries to search your Endpoint Activity log data.

Replace values in bold from the example queries with values from your logs.

Query process start data
These queries are designed to help you find useful information in your Process Start data.

Groupby

There may be instances where you want to find data based on specific criteria. You can group your logs by domain, operating systems, file descriptions etc.

1
groupby(dns_domain)
2
groupby(os_type)
3
groupby(process.exe_file.description)
4
groupby(process.exe_file.product_name)

Find all unique assets with chosen software running

1
where(process.name="**process.exe**" OR parent_process.name="**process.exe**")groupby(hostname)calculate(unique:hostname)limit(1000)

Find software version

Parent Process

1
where(parent_process.name="**process.exe**")groupby(**parent_process.exe_file.version**)calculate(unique:hostname)

Child Process

1
where(process.name="**process.exe**")groupby(**process.exe_file.version**)calculate(unique:hostname)

Find hosts with psexec or psexecsvc running as either the parent or child process

1
where(process.name="**psexec.exe**" OR parent_process.name="**psexec.exe**")groupby(hostname)calculate(unique:hostname)limit(1000)
2
where(process.name="**psexecsvc.exe**" OR parent_process.name="**psexecsvc.exe**")groupby(hostname)calculate(unique:hostname)

Find ping process by hostname and command line

1
where(process.name="**ping.exe**")groupby(hostname, process.cmd_line)

Find Netstat processes by hostname and command line

1
where(process.name="**netstat.exe**")groupby(hostname, process.cmd_line)

Find unsigned Windows processes

1
where(process.exe_file.signing_status.status="UNISGNED") broupby(process.name, hostname) calculate(count)

Find processes that contain a specific word in their command line and/or by hostname

1
where(process.cmd_line icontains "**your word**")groupby(hostname, process.name)

Find RDP by hostname and command line

1
where(process.name="**mstsc.exe**")groupby(hostname, process.cmd_line)

Find Microsoft Management Console and its command line

1
where(process.name="**mmc.exe**")groupby(process.cmd_line)

Find MMC launching ADUC by endpoint and username

1
where(process.name="**mmc.exe**" AND process.cmd_line icontains "**dsa.msc**")groupby(hostname, parent_process.username)

Find Mimikatz by hostname and parent process command line

1
where("**mimikatz**", loose)groupby(hostname, parent_process.cmd_line)

Group by processes running on unique hosts

1
groupby(process.name)calculate(unique:hostname)limit(1000)

Find MSIExec installations

1
where(process.name="**msiexec.exe**" AND process.cmd_line icontains "**/i**")groupby(process.cmd_line)

Find MSIExec quiet installations

1
where(process.name="**msiexec.exe**" AND process.cmd_line icontains "**/quiet**")groupby(process.cmd_line)

Group by Linux process permissions and process name

1
where(os_type="**LINUX**")groupby(process.exe_file.permission, process.name)

Group by process reputation

1
groupby(process.hash_reputation.reputation)

Find all processes by name and by reputation

1
where(process.hash_reputation.reputation='Known')groupby(process.name)
2
where(process.hash_reputation.reputation='Unknown')groupby(process.name)
3
where(process.hash_reputation.reputation=’Malicious’)groupby(process.name)

Find processes by reliability

1
where(process.hash_reputation.reputation='process_reputation')groupby(process.hash_reputation.reliability)

Group by overall process reliability

1
groupby(process.hash_reputation.reliability)

Find processes by reputational threat level

1
where(process.hash_reputation.reputation='process_reputation')groupby(process.hash_reputation.threat_level)

Find processes by name, hostname, and username

1
where(process.hash_reputation.reputation='process_reputation')groupby(process.name, hostname, process.username)

Find PUPs by process name and threat level

1
where(process.hash_reputation.classification.type='PUA')groupby(process.name, process.hash_reputation.threat_level)

Find PUPs by name, hostname, and username

1
where(process.hash_reputation.classification.type='PUA')groupby(process.name, hostname, process.username)

Find adware by process name

1
where(process.hash_repuation.classification.type='Adware')groupby(process.name)

Group by process classification types

1
groupby(process.hash_reputation.classification.type)

Find password documents

1
where(process.name=/(winword|excel|notepad|notepad++|textpad).exe/i AND process.cmd_line=/(.password.).(doc|txt|xls).*/i)groupby(process.cmd_line)

Group by all process threat levels

1
groupby(process.hash_reputation.threat_level)

Find low threat level processes by name

1
where(process.hash_reputation.threat_level='Low')groupby(process.name)

Find processes by threat level

1
where(process.hash_reputation.reputation='process_reputation')groupby(process.hash_reputation.threat_level)

Find processes with well-known reputation

1
where(process.hash_reputation.reliability IN ['Very high', 'High'])groupby(process.name)

Find command lines showing the Taskkill.exe process

1
where(process.name="**taskkill.exe**")groupby(process.cmd_line)

Find hostname and users using the PsLoggedon.exe utility

1
where(process.name="**PsLoggedon.exe**")groupby(hostname, process.username)

Note: This query shows connected users on the local machine and remote connections using local endpoint resources.

Find hostname and users running the native screencapture.exe software

1
where(process.name='ScreenCapture.exe')groupby(hostname, process.username)

Find processes, hostnames, and users running programs for SSH and/or Telnet

1
where(process.exe_file.description icontains-any ["**ssh**", "**telnet**"])groupby(process.name, hostname, process.username)

Find hostnames and users performing the <code>whoami</code> command line

1
where(process.name="**whoami.exe**")groupby(hostname, process.username)
Query process start data (Powershell)

These queries are designed to help you find useful information related to the PowerShell process.

Find Powershell processes

Parent Process

1
where(process.name="**powershell.exe**")groupby(parent_process.name)

Child Process

1
where(parent_process.name="**powershell.exe**")groupby(process.name)

Find Powershell process command lines that are not empty or are running a ps1 script

Parent Process

1
where(parent_process.name="**powershell.exe**" AND parent_process.cmd_line NOT IIN ["**null**", "**.ps1**"])groupby(parent_process.cmd_line)

Child Process

1
where(process.name="**powershell.exe**" AND process.cmd_line NOT IIN ["**null**", "**.ps1**"])groupby(process.cmd_line)

Find assets running powershell_ISE as either the parent or child process

1
where(process.name="**powershell_ise.exe**" OR parent_process.name="**powershell_ise.exe**")groupby(hostname)calculate(unique:hostname)
Query process start data (PSEXEC)

These queries are designed to help you find useful information related to the PSEXEC process.

Find all psexec command lines

1
where(process.name="**psexec.exe**")groupby(process.cmd_line)

Find psexec running the remote process in the system account

1
where(process.name="**psexec.exe**" AND process.cmd_line icontains "**.s.**")groupby(process.cmd_line)

Find remote assets that Psexec is running processes under the system account

1
where(process.name="**psexec.exe**" AND process.cmd_line icontains "**.s.**" AND /psexec.exe \(?P<remote_asset>[^ ]*)/)groupby(remote_asset)
Query historical user and asset data
When a suspicious event is detected on an endpoint, you can create a query to view all activity that occurred on the host prior to the event. You can use the queries in this section to display users and assets associated with a process start event.

Find unique assets associated with a process

1
where(hostname='**hostname**') groupby(process.cmd_line, process.exe_path) calculate(count)

Find unique assets and users associated with a process

1
where(hostname='**hostname**' and process.username=’**username**') groupby(process.cmd_line, process.exe_path) calculate(count)
Queries for Threat Hunting

These queries are designed to help you hunt for threats by analyzing endpoint user, process, and command line data.

Find commands being carried out by strings and stopping, all within the command line

1
where(process.name="**cmd.exe**" AND process.cmd_line ISTARTS-WITH "**cmd.exe /C**")groupby(process.cmd_line)

Find command line attempts to remove certain folder attributes

1
where(process.cmd_line icontains "attrib -h -s -r")

Find hosts and users that have launched a local endpoint's User Account Management

1
where(process.name IIN ["**Netplwiz.exe**", "lusmgr.msc"])groupby(hostname, process.username)

Find suspicious login activity

1
where(hostname="**hostname**" and process.username="**username**")

Find additional information about a process that triggered a detection

1
where(process.pid='**process_id**' and hostname='**hostname**', loose)

Find infrequently run commands

1
where(process.name='**process_name**' AND hostname='**hostname**') groupby(data.cmdLine) calculate(count) limit(**limit**)

Note: Before running the sample query, replace process_name and hostname with the name of the process and host you want to group by.

Find malicious hashes

1
where(process.exe_file.hashes.sha1="**hash**") groupby(hostname) calculate(count)

Find command lines where a setup.exe file is being launched from within the downloads folder

1
where(process.cmd_line icontains-all ["\downloads\","setup.exe"])groupby(process.cmd_line)

Find commonly abused commands

Initial Investigation

1
where(process.cmd_line ISTARTS-WITH-ANY ["tasklist", "ver", "ipconfig", "systeminfo", "net time", "netstat", "whoami", "net start", "qprocess", "query"])groupby(hostname, process.cmd_line)

Reconnaissance

1
where(process.cmd_line ISTARTS-WITH-ANY ["dir", "net view", "ping", "net use", "type", "net user", "net localgroup", "net group", "net config", "net share"])groupby(hostname, process.cmd_line)

Spread of infection

1
where(process.cmd_line ISTARTS-WITH-ANY ["at", "reg", "wmic", "netsh advfirewall", "sc", "rundll32"])groupby(hostname, process.cmd_line)

Understand the Enhanced Endpoint Telemetry metadata

This section provides the endpoint activity data that the Insight Agent sends to Log Search. You can create queries to group and detect on this data.

EET metadata varies by operating system

The keys that appear in Log Search vary based on the operating system of the device where the event occurred.

Process Start Event

The following table provides information about the process start event metadata collected by the Insight Agent. When you purchase the InsightIDR Ultimate package, you receive full access to the archive of process start data captured by the Insight Agent.

If All is listed in the Operating System column, the field is sent to Log Search regardless of the operating system of the device the event occurred on.

FieldDescriptionOperating System
hostnameThe hostname of the endpoint running the process.All
dns_domainThe domain of the endpoint running the process.All
os_typeThe endpoint's operating system.All
r7_hostidThe Rapid7 Host ID.All
processAll data related to the captured process.All
parent_processAll data related to the process that spawned the started process.All
env_varsThis object shows the environment variables when the process and its parent were launched. The parent value is listed only if it differs from the process value. The process value can be used to find processes that made changes to environment variables prior to launching a child process. If the env_vars of a process is null and the parent process includes environment variables, InsightIDR populates env_vars with the parent’s environment variables.Windows
duplicated_eventsThe count of identical events that occurred in a process. This key is populated only if InsightIDR sees similar process events.Linux

Process Details

The Insight Agent collects and sends the following information about both the process triggering the event and the parent process.

Some fields vary based on the operating system that the process or executable file is running on. If All is listed in the Operating System column, the field is sent to Log Search regardless of operating system.

FieldDescriptionOperating System
start_timeThe time that this process started.All
nameThe name of the process.All
pidThe system's Process ID.All
ppidThe parent system's process ID.Mac/Linux, parent only
r7_idThe Insight Agent-generated ID, unique to a process start.All
exe_pathThe path to the executable.All
img_pathThe path to the executable. This value might differ from exe_path if the executable is on a mounted remote file share. This key is sent to Log Search only if its value differs from exe_path.Windows
cmd_lineThe command line invocation used to start the process, including arguments.All
usernameThe local user who started the process.All
account_domainThe AD domain of the user who started the process.Windows
uidThe user ID.Mac/Linux
groupThe group name.Mac/Linux
gidThe group ID.Mac/Linux
euid_nameThe effective user name.Mac/Linux
euidThe effective user ID.Mac/Linux
egid_nameThe effective group name.Mac/Linux
egidThe effective group ID.Mac/Linux
ruid_nameThe real user name.Mac
ruidThe real user ID.Mac
rgid_nameThe real group name.Mac
rgidThe real group ID.Mac
fsuidThe file system user ID.Linux
fsgidThe file system group ID.Linux
suidThe saved user ID.Linux
sgidThe saved group ID.Linux
sessionThe login session ID that launched the process.All
addrThe remote address that the user is connecting from.Mac
portThe port the process used.Mac
exe_fileThe information about the executable file.All

Executable File

The following table outlines the metadata that the Insight Agent collects from the executable file.

FieldDescriptionOperating System
exe_file.ownerThe owner of the executable file.All
exe_file.uidThe ID of the executable file owner.Mac/Linux
exe_file.groupThe group of the executable file.Mac/Linux
exe_file.gidThe group ID of the executable file.Mac/Linux
exe_file.permissionsThe permissions string of the executable file.Mac/Linux
exe_file.orig_filenameThe original filename from the file metadata.Windows
exe_file.descriptionThe description from the file metadata.Windows
exe_file.product_nameThe product name of the executable, as reported by the file metadata.Windows
exe_file.authorThe company who produced the executable, as reported by the file metadata.Windows
exe_file.versionThe build version of the file, from the file metadata.Windows
exe_file.createdThe executable file's creation date.All
exe_file.last_modifiedThe executable file's last modification date.All
exe_file.last_accessedThe executable file's last accessed date.Mac/Linux
exe_file.sizeThe executable file's size.All
exe_file.internal_nameThe internal name of the executable file, from the metadata.Windows
hashesThe collection of different hashes of the process.All
signing_statusThe signature status.Windows
signing_chainThe signature chain.Windows
countersigning_chainThe countersignature chain.Windows

Hashes

The Insight Agent collects and sends the following process hash information:

FieldDescriptionOperating System
hashes.md5The MD5 hash.All
hashes.sha256The SHA256 hash.All
hashes.sha1The SHA1 hash.All