Log Search

In InsightIDR, the connected event sources and environment systems produce data in the form of raw logs.

A log is a collection of hundreds or thousands of log entries, which is data that is streamed from an event source.

Logs are typically named based on the event source, for example, Firewall: New York Office. However, you can also name the logs yourself.

Log Search takes every log of raw, collected data, and automatically sorts the logs into log sets. A log set is, therefore, a collection of multiple log streams.

In InsightIDR, a log set is defined (by default) by the type of event within the log stream, such as Firewall, DNS, Active Directory, and other event types. You can also define your own custom log sets, for example, to organize raw data.

As your event sources continue to be active, more log entries are added to the existing logs. With this log structure in place, you can do multiple things including:

• Search logs for specific terms with a Search Language.
• Build your own query to group by a field or calculate specific items.
• View logs in Visual Search.
• Create tags and Alerts on your log data.
• Export data to share with stakeholders.

Set your Syntax Highlighting Preferences

Syntax highlighting applies contrasting colors to help you distinguish between the components of a query – such as clauses, keys, values, and comparison operators.

Syntax highlighting helps you to recognize when your query might be missing a component or if it's written incorrectly. If the system does not recognize a component by highlighting it with a specific color, you can find the mistake and make the correction more easily.

For example, this image shows a LEQL clause, operator, and function highlighted in orange, keys in blue, values in black, and regular expressions in purple. With this highlighting, you can validate your query as you are building it.

Try syntax highlighting for yourself by selecting some logs and entering one of our example queries.

Expand to view all syntax and color keys

Format	Syntax Details
	Orange text is applied to: - Keywords (including clauses) - Operators (logical and comparison) - Analytical functions - Functions - Parameters
	Purple text is applied to all regular expressions.
	These values appear as black text: - Text and numbers - Strings - Lists - IP Search
	Keys appear as blue text.
	Syntax errors are reverted to plain text and underlined in red.

Syntax Highlighting is active by default, but it can be turned off.

Manage Syntax Highlighting

To manage syntax highlighting:

1. InsightIDR, go to the left menu and click Settings.
2. Select User Preferences and, under Profile Preferences, turn the LEQL Editor on or off.

Edit Log Streams and Log Events

To edit logs and log sets:

1. In Log Search, from the list of logs or log sets, identify the one you want to edit and hover your cursor over it to click the ellipsis.
2. From the dropdown menu, click Edit Settings.
3. After you are finished editing, click Save.

Delete Logs and Log Sets

Depending on your user permissions, you can permanently delete logs and log sets. However, there may be some restrictions or prerequisites to consider.

If a log is associated with an active event source, collector, or network sensor, you must first delete that corresponding entity from either the Data Collection or Sensor Management screens.

If you are not an administrator, then you may need to ask your administrator to delete the event source, collector, or network sensor before you can delete the log or log set.

To delete logs and log sets:

1. In Log Search, from the list of logs or log sets, identify the one you want to delete, hover your cursor over it, and click the ellipsis that appears.
2. From the dropdown menu, click Delete Log or Delete Log Set.

Log View Options

When viewing your log entries, you can change views based on your preferences. Click Entries, Table, or Visualizations to change views.

In Entries View, you can:

• Select a key to add it to the query bar and find all occurrences of the key in the selected logs and time range.
• Select the info icon to open the Entry Inspector for the selected log line.
• Select the search icon to view additional context for the selected log line.
• Manage Keys to include or exclude data from your view.

In Table View, you can:

• Add a search expression to the Query Builder by selecting a column header, selecting an operator, and entering a value or key.
• Manage Columns to include or exclude data from your view.

In Visualizations View, you can:

• View Rapid7-generated cards for your log data.
• Add a card.
• Save the visualizations to a dashboard.

Log Labels

Log labels are applied to provide visual cues that indicate potential issues with your logs. Within InsightIDR, there are currently two default labels, Exception and OutOfOrder. See Default Labels for more information. You can also Create Log Labels with an Alert.

Searching Your Data

InsightIDR provides different ways to search your data, including Regex, String, KeyValue, or Keyword search.

Read Use a Search Language for more information, or you can build queries off of the provided Example Queries.

You can also build a query in Table View:

1. While in Table view, select the keyword that you want to query on.
2. Select an operator and enter a key value.
3. Click Search.

Export Data

You can export parseable logs to share with stakeholders at your convenience.

When viewing your query results in the Entries or Table tabs, select the Download dropdown, then Export to CSV.

An export confirmation message will appear. You can access your CSV by navigating to Settings > Log Search > Exports.

Optimization and performance tuning

InsightIDR engineering teams utilize a variety of tuning measures to optimize for system performance and data storage limits. These measures may include removal of excessively noisy, irrelevant, or duplicated data that would otherwise clutter dashboards and log sets, as well as data compression to make the best use of your available storage space. When implementing these measures, InsightIDR engineering teams work closely with Rapid7 researchers and security experts to ensure we are collecting data that is the most effective for detecting and investigating malicious activity in your environment.