Log Sets You Can Search
One of the first challenges when building a working query in Log Search is knowing which logs to search. You must know which log sets contain the logs you need and which logs contain the keys and values you need.
This topic describes the log sets available in InsightIDR and the type of data they contain. With this information, you can make informed choices about which log sets to select when you are running a query in Log Search. Making the right selection saves you time and helps Log Search to perform faster.
In InsightIDR, log sets typically correspond to an event type. By default, a log set is defined by the event sources that generate the logs, such as a Firewall Activity, DNS Query, or Microsoft Active Directory™Admin Activity. Expand the Log Sources panel in Log Search to explore your log sets.
Tip for getting started with selecting logs
If you are unsure where to find the log data you need, check out the Keys to Use in Your Queries topic, which lists all of the keys (also known as fields) that are parsed from the logs. You can also select a short time range and run a search without a query to view the keys and values that are returned. Then, when you have chosen a key to look up, run a query with a groupby clause on that key to view all of the possible values it contains. For example, to see a list of all of the assets in your organization that are using the firewall, select the Firewall Activity log set and run the query groupby(asset).
Example: Investigating suspicious external authentication
Alex, a security analyst at an online retail company, wants to conduct an investigation into internal users who are logging into Office 365 from outside Canada, where the company's headquarters are located.
They review the available log sets and determine that the Ingress Authentication logs will show external authentications using Office 365.
Alex selects the Ingress Authentication log set and selects a time range of 1 month. They then examine the keys that are available within this log set.
They determine that they'll need to run a query where the service is o365 and the geoip_country_name is not Canada, so they run the query: where(geoip_country_name != "Canada" AND service = o365)groupby(geoip_country_name).
Alex can now examine the results of the query and determine what countries other than Canada users are attempting to authenticate from.
Example: Investigating user exfiltration
Sam, a Chief Information Security Officer (CISO) at a large Fintech company, has received a report of suspected exfiltration and wants to investigate the amount of data that the user in question is transferring to and from external applications.
Sam selects the Network Flow log set and queries the key that will find the user they want to investigate. They enter the query where(source_user="John Doe") and select a time range of 7 days. Sam runs the query and views the results in the Table view.
The information that Sam wants to target is around the applications that John Doe is connecting and transferring data to. They target a key called app_protocol_description, which contains the names of the applications that are being used by users in the organization.
They can run a groupby() query on this key to get a list of the applications that John Doe is using. To do this, they add this syntax to their query and click Run: groupby(app_protocol_description).
Log Search displays a bar chart and a list of the applications that John Doe is using. Sam can then analyze the frequency of usage and the amount of data being transferred and determine whether further investigation is needed.
Understand log sets and their contents
This table contains a list of some of the log sets that are available in Log Search. You can use this list to decide which log set you want to search and then go to the Keys to Use in Your Queries reference topic to identify the exact keys to add to your query.
| Log Set Name | Contents | Event Type and Keys | Event Sources and Documentation |
|---|---|---|---|
| Active Directory Admin Activity | Contains logs about administrator activities in Microsoft Active Directory™, such as user account management, and security group management. | ad_admin | View the related event sources. |
| Advanced Malware Alert | Contains logs from your organization's advanced firewall or a module on your firewall. | advanced_malware | View the related event sources. |
| Asset Authentication | Contains logs about authentications to assets, based on observations from domain controllers and activity observed by the Insight Agent. Can include both domain authentications and authentications using local accounts. | asset_auth | View the Insight Agent documentation. |
| Audit Logs | Contains logs about the actions recorded in the InsightIDR audit log. This log set originates from actions taken in InsightIDR, rather than an event source or the Insight Agent. | investigations alerts | View the Investigations documentation. |
| Cloud Service Activity | Contains logs that track end user activity on cloud services, such as Microsoft Azure™, Office 365™, AWS CloudTrail™, or Zoom™. | cloud_service_activity | View the related event sources. |
| Cloud Service Admin Activity | Contains logs about the administrator activities on everyday cloud services, such as new account creation in Microsoft Azure™, Office 365™, AWS CloudTrail™, or Zoom™. | cloud_service_admin | View the related event sources. |
| DNS Query | Contains the DNS queries that are logged on DNS servers. DNS queries resolve host or website names to their IP addresses. These logs are useful for finding what website addresses are being visited from particular assets. This log set contains deduplicated data. | dns | View the related event sources. |
| Endpoint Activity* | Contains logs about activity observed by the Insight Agent, including details about the execution of processes (such as process start events) on the asset. | process_start_events | View the Insight Agent documentation. |
| Endpoint Agent (MDR only) | Contains logs from the Insight Agent that give details about beacon data, job requests, and process snapshots. | n/a | View the Insight Agent documentation. |
| Endpoint Health | Contains logs from the Insight Agent that provide details about health such as issues when collecting telemetry data from the endpoint. | endpoint_health | View the Insight Agent documentation. |
| File Access Activity | Contains logs from the Insight Agent that result from file access activity monitoring (FAAM). This activity is captured from systems that are hosting Windows file shares and have activity auditing turned on. | file_access | View the FAAM documentation. |
| File Modification Activity | Contains logs that result from file integrity monitoring (FIM). When FIM is enabled in InsightIDR, this activity is captured from Windows and Linux systems that have file system monitoring enabled. | file_modification | View the FIM documentation. |
| Firewall Activity | Contains logs of network connections that are established across firewalls. This log set contains deduplicated data. | firewall | View the related event sources. |
| Host To IP Observations | Contains logs reporting relationships between hostnames and IP addresses, used to power the attribution engine of InsightIDR. The activity can come from DHCP, VPN or Active Directory event sources. It can also come from the Insight Agent when the agent is able to communicate with a Collector. In addition, Insight Network Sensors can gather this information from network flow logs. | host_name_to_ip | View the related event sources for DHCP, VPN, and Active Directory. |
| IDS Alert | Contains logs of alerts that come from intrusion detection systems (IDS) or intrusion prevention systems (IPS); typically, advanced firewall modules or sensors. | ids | View the related event sources. |
| Ingress Authentication | Contains logs that track user authentication attempts to corporate systems and cloud services from the public Internet. | ingress_auth | View the related event sources. |
| Network Flow** | Contains logs that come from Insight Network Sensors. This is only available to customers who have Enhanced Network Traffic Analysis (ENTA). Network flow tracks the entities that assets are connecting to and what network protocols they are using. | flow | View the Network Traffic Analysis documentation. |
| Raw Log | Contains logs that are collected from a 'raw data' event source, such as the Custom Logs or Generic Syslog event sources. This log set can also contain logs from named event sources that are not properly formatted for parsing. Tip: Raw logs can be a great datasource for building dashboards and visualizations. | No schema is available, because the data varies based on the original event source. | View the related event sources. |
| Third Party Alert | Contains logs that are generated by alerts from third-party services outside of InsightIDR, such as Amazon GuardDuty™. | third_party_alert | View the related event sources. |
| Unparsed Data | Contains logs that were collected from a named event source but that are not forensically relevant. These logs are of a type that are used for detection rules or they are missing key information that is required to make them forensically interesting, such as an external IP address, user name, or other important information. You can enable the collection of unparsed data by selecting the Send Unparsed Data option when you configure event sources. | No schema is available, because the data varies based on the original event source. | View the Unparsed Logs documentation. |
| Virus Alert | Contains logs that are generated when a virus is detected, cleaned, and quarantined by the antivirus system. | virus | View the related event sources. |
| Web Proxy Activity | Contains logs that show requests to web URLs that are intercepted or monitored by a device like a web proxy. This log set contains deduplicated data. | web_proxy | View the related event sources. |
* Available only to customers who have Enhanced Endpoint Telemetry (EET) as part of InsightIDR's Advanced or Ultimate Packages. Learn more about the EET data retention periods.
** Available only to customers who have Enhanced Network Traffic Analysis (ENTA) as part of InsightIDR's Ultimate Package.