Log Set Guidance

One of the first challenges when building a working query in Log Search is knowing which logs to search. You must know which log sets contain the logs you need and which logs contain the keys and values you need.

This topic describes the common log sets in InsightIDR and the type of data they contain. With this information, you can make informed choices about which log sets to select when you are running a query in Log Search. Making the right selection saves you time and helps Log Search to perform faster.

In InsightIDR, log sets typically correspond to an event type. By default, a log set is defined by the event sources that generate the logs, such as a Firewall, DNS server, or Microsoft Active Directory™.

Tip for Getting Started with Selecting Logs

If you are unsure where to look to find the log data you need, check out the Event Types and Keys topic, which lists all of the keys (also known as fields) that are parsed from the logs. You can also select a short time range and run a search without a query to view the keys and values that are returned. Then, when you have chosen a key to look up, run a groupby query on that key to view all of the possible values it contains. For example, to see a list of all of the assets in your organization that are using the firewall, select the Firewall Activity log set and run the query groupby(asset).

Example: Investigating Suspicious External Authentication

Alex, a security analyst at an online retail company, wants to conduct an investigation into internal users who are logging into Office 365 from outside Canada, where the company's headquarters are located.

They review the available log sets and determine that the Ingress Authentication logs will show external authentications using Office 365.

Alex selects the Ingress Authentication log set and selects a time range of 1 month. They then examine the keys that are available within this log set.

They determine that they'll need to run a query where the service is o365 and the geoip_country_name is not Canada, so they run the query: where("geoip_country_name" != "Canada" AND "service" = o365)groupby(geoip_country_name).

Alex can now examine the results of the query and determine what countries other than Canada users are attempting to authenticate from.

Example: Investigating User Exfiltration

Sam, a Chief Information Security Officer (CISO) at a large Fintech company, has received a report of suspected exfiltration and wants to investigate the amount of data that the user in question is transferring to and from external applications.

Sam selects the Network Flow log set and queries the key that will find the user they want to investigate. They enter the query where(source_user="John Doe") and select a time range of 7 days. Sam runs the query and views the results in the table view.

The information that Sam wants to target is around the applications that John Doe is connecting and transferring data to. They target a key called app_protocol_description, which contains the names of the applications that are being used by users in the organization.

They can run a groupby query on this key to get a list of the applications that John Doe is using. To do this, they add this syntax to their query and click Run: groupby(app_protocol_description).

Log Search displays a bar chart and a list of the applications that John Doe is using. Sam can then analyze the frequency of usage and the amount of data being transferred and determine whether further investigation is needed.

Log Sets and Their Contents

This table contains a list of some of the log sets that are available in Log Search. You can use this list to decide which log set you want to search and then go to the Event Types and Keys reference topic to identify the exact keys to add to your query.

Log Set NameContentsEvent Type and KeysEvent Sources and Documentation
Active Directory Admin ActivityContains logs about administrator activities in Microsoft Active Directory™, such as user account management, and security group management.ad_adminView the related event sources.
Advanced Malware AlertContains logs from your organization's advanced firewall or a module on your firewall.advanced_malwareView the related event sources.
Asset AuthenticationContains logs about authentications to assets, based on observations from domain controllers and activity observed by the Insight Agent. Can include both domain authentications and authentications using local accounts.asset_authView the Insight Agent documentation.
Cloud Service ActivityContains logs that track end user activity on cloud services, such as Microsoft Azure™, Office 365™, AWS CloudTrail™, or Zoom™.cloud_service_activityView the related event sources.
Cloud Service Admin ActivityContains logs about the administrator activities on everyday cloud services, such as new account creation in Microsoft Azure™, Office 365™, AWS CloudTrail™, or Zoom™.cloud_service_adminView the related event sources.
DNS QueryContains the DNS queries that are logged on DNS servers. DNS queries resolve host or website names to their IP addresses. These logs are useful for finding what website addresses are being visited from particular assets.dnsView the related event sources.
Endpoint Activity*Contains logs about activity observed by the Insight Agent, including details about the execution of processes (such as process start events) on the asset.process_start_eventsView the Insight Agent documentation.
Endpoint Agent (MDR only)Contains logs from the Insight Agent that give details about beacon data, job requests, and process snapshots.n/aView the Insight Agent documentation.
File Access ActivityContains logs from the Insight Agent that result from file access activity monitoring (FAAM). This activity is captured from systems that are hosting Windows file shares and have activity auditing turned on.file_accessView the FAAM documentation.
File Modification ActivityContains logs that result from file integrity monitoring (FIM). When FIM is enabled in InsightIDR, this activity is captured from Windows and Linux systems that have file system monitoring enabled.file_modificationView the FIM documentation.
Firewall ActivityContains logs of network connections that are established across firewalls.firewallView the related event sources.
Host To IP ObservationsContains logs reporting relationships between hostnames and IP addresses, used to power the attribution engine of InsightIDR. The activity can come from DHCP, VPN or Active Directory event sources. It can also come from the Insight Agent when the agent is able to communicate with a Collector. In addition, Insight Network Sensors can gather this information from network flow logs.host_name_to_ipView the related event sources for DHCP, VPN, and Active Directory.
IDS AlertContains logs of alerts that come from intrusion detection systems (IDS) or intrusion prevention systems (IPS); typically, advanced firewall modules or sensors.idsView the related event sources.
Ingress AuthenticationContains logs that track user authentication attempts to corporate systems and cloud services from the public Internet.ingress_authView the related event sources.
Network Flow**Contains logs that come from Insight Network Sensors. This is only available to customers who have Enhanced Network Traffic Analysis (ENTA). Network flow tracks the entities that assets are connecting to and what network protocols they are using.flowView the Network Traffic Analysis documentation.
Raw LogContains logs that are collected from a 'raw data' event source, such as the Custom Logs or Generic Syslog event sources. This log set can also contain logs from named event sources that are not properly formatted for parsing. Tip: Raw logs can be a great datasource for building dashboards and visualizations.No schema is available, because the data varies based on the original event source.View the related event sources.
Third Party AlertContains logs that are generated by alerts from third-party services outside of InsightIDR, such as Amazon GuardDuty™.third_party_alertView the related event sources.
Unparsed DataContains logs that were collected from a named event source but that are not forensically relevant. These logs are of a type that are used for detection rules or they are missing key information that is required to make them forensically interesting, such as an external IP address, user name, or other important information. You can enable the collection of unparsed data by selecting the Send Unparsed Data option when you configure event sources.No schema is available, because the data varies based on the original event source.View the Unparsed Logs documentation.
Virus AlertContains logs that are generated when a virus is detected, cleaned, and quarantined by the antivirus system.virusView the related event sources.
Web Proxy ActivityContains logs that show requests to web URLs that are intercepted or monitored by a device like a web proxy.web_proxyView the related event sources.

* Available only to customers who have Enhanced Endpoint Telemetry (EET) as part of InsightIDR's Ultimate Package.

** Available only to customers who have Enhanced Network Traffic Analysis (ENTA) as part of InsightIDR's Ultimate Package.