Log Search

Your connected event sources and environment systems produce data in the form of raw logs. Within InsightIDR, log data is categorized as log sets, logs, or log entries:

  • A log set is a collection of multiple log streams. In InsightIDR, a log set is defined by the type of event within in the log stream, such as Firewall, DNS, Active Directory, and other event types.
  • A log is a collection of log entries, or a single log stream. In InsightIDR, logs are typically named based on the source of the log stream. For example, Firewall: New York Office.
  • A log entry is an individual log event.

Log Search takes every log of raw, collected data and automatically sorts them into Log Sets for you. New log events are added to the existing log and grouped with other logs in the log set based on when the log event was created. Once you apply a search to a log, a log set, or multiple log sets, you can do multiple things including:

Set your Syntax Highlighting Preferences

Syntax Highlighting: Phased release begins February 2022

We will be rolling out new Syntax Highlighting functionality over the next few weeks. Syntax Highlighting will be enabled by default upon release. You can change this setting by going to User Profile > Profile Settings.

Syntax highlighting applies contrasting colors and text formatting to distinct components of a query – such as clauses, keys, values, and comparison operators – in Log Search. Syntax Highlighting is enabled by default whether you’re using Simple or Advanced Mode.

Query in log search with syntax highlighting applied

View Syntax and Color Key
FormatSyntax Details
LEQLOrange text is applied to:
- Keywords
- Operators (Logical and Comparison)
- Analytical Function
- Functions
- Parameters
Regular ExpressionsBold formatting is applied to all regular expressions.
ValuesThese values appear in plain text:
- Text and number
- Strings
- Lists
- IP Search
KeysKeys appear as blue text.
ErrorsSyntax errors are reverted to plain text and underlined in red.

Turn Syntax Highlighting On

  1. From the top menu bar, expand the User Profile menu, and click Profile Settings.
  2. Locate the Enhanced Log Search section.
  3. Select the Display Syntax Highlighting checkbox.

Turn Syntax Highlighting Off

  1. From the top menu bar, expand the User Profile menu, and click Profile Settings.
  2. Locate the Enhanced Log Search section.
  3. Clear the Display Syntax Highlighting checkbox.

Edit Log Streams and Log Events

Not seeing the log data you expected?

This could be tied to data retention or enrichment. Log data is stored in InsightIDR based on your retention policy. For more information, see Log Data Collection and Storage. InsightIDR may add enrichment data, such as geolocation or organizational data to events, however no original event data is ever removed or replaced. Events in InsightIDR reflect the events as they were originally generated by the source system.

To edit logs and log sets:

  1. In Log Search, from the list of logs or log sets, identify the one you want to edit and hover your cursor over it to click the ellipsis.
  2. From the dropdown menu, click Edit Settings.
  3. After you are finished editing, click Save.

User actions might be added to the audit log

If audit logging is active, the Platform Audit Log will contain a record of log creation, edits, or deletion. It is not currently updated for log sets.

Delete Logs and Log Sets

Depending on your user permissions, you can permanently delete logs and log sets. However, there may be some restrictions or prerequisites to consider.

If a log is associated with an active event source, collector, or network sensor, you must first delete that corresponding entity from either the Data Collection or Sensor Management screens.

If you are not an administrator, then you may need to ask your administrator to delete the event source, collector, or network sensor before you can delete the log or log set.

Restrictions on deleting log sets

If a log set is generated by a collector-based source, it is not possible to delete the log set. Only log sets that are directly sent to Log Search can be deleted.

To delete logs and log sets:

  1. In Log Search, from the list of logs or log sets, identify the one you want to delete, hover your cursor over it, and click the ellipsis that appears.
  2. From the dropdown menu, click Delete Log or Delete Log Set.

Data will be permanently deleted

This action permanently deletes the log or log set and its data. If you need to retain the log data for security investigation or compliance purposes, you should carefully consider whether it should be deleted.

Log View Options

When looking at the log entries, you can make reading logs easier by viewing log data in JSON format. Click Entries or Table to change views.

Viewing logs

All normalized log entries can be queried either by searching for a string or by searching for a keyword=value pair.

While in Table View, you can quickly create a query based on a selected key, filter by column, or click on source_user and enter a username to search by, and run calculations.

Searching Your Data

InsightIDR allows users different ways of searching their data, including Regex, String, KeyValue, or Keyword search. See Use a Search Language for more information.

Or, you can build queries off of the provided Example Queries.

You can also build a query in Table View:

  1. While in Table view, select the keyword that you want to query on.
  2. Select an operator and enter a key value.
  3. Click Search.
  4. You can now search within your query.

Export Data

You can export parseable logs to share with stakeholders at your convenience. When viewing the log entries table, select Export to CSV.

Export data

You will see a confirmation message appear. Your CSV file will be available in the Report Archive under the "Entries Export" tab.