Log Search

In InsightIDR, the connected event sources and environment systems produce data in the form of raw logs.

A log is a collection of hundreds or thousands of log entries, which is data that is streamed from an event source.

Logs are typically named based on the event source, for example, Firewall: New York Office. However, you can also name the logs yourself.

Log Search takes every log of raw, collected data, and automatically sorts the logs into log sets. A log set is, therefore, a collection of multiple log streams.

In InsightIDR, a log set is defined (by default) by the type of event within the log stream, such as Firewall, DNS, Active Directory, and other event types. You can also define your own custom log sets, for example, to organize raw data.

Log Search Glossary

For a full list of the terms that are used in Log Search, see the Glossary.

As your event sources continue to be active, more log entries are added to the existing logs. With this log structure in place, you can do multiple things including:

Set your Syntax Highlighting Preferences

Syntax highlighting applies contrasting colors to help you distinguish between the components of a query – such as clauses, keys, values, and comparison operators.

Syntax highlighting helps you to recognize when your query might be missing a component or if it's written incorrectly. If the system does not recognize a component by highlighting it with a specific color, you can find the mistake and make the correction more easily.

For example, this image shows a LEQL clause, operator, and function highlighted in orange, keys in blue, values in black, and regular expressions in purple. With this highlighting, you can validate your query as you are building it.

Query in log search with syntax highlighting applied

Try syntax highlighting for yourself by selecting some logs and entering one of our example queries.

Expand to view all syntax and color keys
FormatSyntax Details
LEQLOrange text is applied to:
- Keywords (including clauses)
- Operators (logical and comparison)
- Analytical functions
- Functions
- Parameters
Regular ExpressionsPurple text is applied to all regular expressions.
ValuesThese values appear as black text:
- Text and numbers
- Strings
- Lists
- IP Search
KeysKeys appear as blue text.
ErrorsSyntax errors are reverted to plain text and underlined in red.

Syntax Highlighting is active by default, but it can be turned off.

Turn Syntax Highlighting On or Off

  1. From the top menu bar, expand the User Profile menu, and click Profile Settings.
  2. Find the Enhanced Log Search section.
  3. Select or deselect Display Syntax Highlighting.

Edit Log Streams and Log Events

Not seeing the log data you expected?

This could be tied to data retention or enrichment. Log data is stored in InsightIDR based on your retention policy. For more information, see Log Data Collection and Storage. InsightIDR may add enrichment data, such as geolocation or organizational data to events, however no original event data is ever removed or replaced. Events in InsightIDR reflect the events as they were originally generated by the source system.

To edit logs and log sets:

  1. In Log Search, from the list of logs or log sets, identify the one you want to edit and hover your cursor over it to click the ellipsis.
  2. From the dropdown menu, click Edit Settings.
  3. After you are finished editing, click Save.

User actions might be added to the audit log

If audit logging is active, the Platform Audit Log will contain a record of log creation, edits, or deletion. It is not currently updated for log sets.

Delete Logs and Log Sets

Depending on your user permissions, you can permanently delete logs and log sets. However, there may be some restrictions or prerequisites to consider.

If a log is associated with an active event source, collector, or network sensor, you must first delete that corresponding entity from either the Data Collection or Sensor Management screens.

If you are not an administrator, then you may need to ask your administrator to delete the event source, collector, or network sensor before you can delete the log or log set.

Restrictions on deleting log sets

If a log set is generated by a collector-based source, it is not possible to delete the log set. Only log sets that are directly sent to Log Search can be deleted.

To delete logs and log sets:

  1. In Log Search, from the list of logs or log sets, identify the one you want to delete, hover your cursor over it, and click the ellipsis that appears.
  2. From the dropdown menu, click Delete Log or Delete Log Set.

Data will be permanently deleted

This action permanently deletes the log or log set and its data. If you need to retain the log data for security investigation or compliance purposes, you should carefully consider whether it should be deleted.

Log View Options

When looking at the log entries, you can make reading logs easier by viewing log data in JSON format. Click Entries or Table to change views.

Viewing logs

All normalized log entries can be queried either by searching for a string or by searching for a keyword=value pair.

While in Table View, you can quickly create a query based on a selected key, filter by column, or click on source_user and enter a username to search by, and run calculations.

Searching Your Data

InsightIDR allows users different ways of searching their data, including Regex, String, KeyValue, or Keyword search. See Use a Search Language for more information.

Or, you can build queries off of the provided Example Queries.

You can also build a query in Table View:

  1. While in Table view, select the keyword that you want to query on.
  2. Select an operator and enter a key value.
  3. Click Search.
  4. You can now search within your query.

Export Data

You can export parseable logs to share with stakeholders at your convenience. When viewing the log entries table, select Export to CSV.

Export data

You will see a confirmation message appear. Your CSV file will be available in the Report Archive under the "Entries Export" tab.

Want to archive your log data?

If you want to learn about archiving your log data, read our topic about Data Archiving.

Optimization and performance tuning

InsightIDR engineering teams utilize a variety of tuning measures to optimize for system performance and data storage limits. These measures may include removal of excessively noisy, irrelevant, or duplicated data that would otherwise clutter dashboards and log sets, as well as data compression to make the best use of your available storage space. When implementing these measures, InsightIDR engineering teams work closely with Rapid7 researchers and security experts to ensure we are collecting data that is the most effective for detecting and investigating malicious activity in your environment.