Log Search

In InsightIDR, the event sources and environment systems gather data in the form of raw logs. A log is a collection of log entries, which are pieces of data that are streamed from event sources. Logs are typically named based on the event source, for example, Firewall: New York Office. However, you can also name the logs yourself.

Log Search ingests every log of raw data and sorts the log into a log set based on the log’s event type. A single device, like a domain controller, can collect log entries that flow into multiple log sets, such as Active Directory Admin Activity and Asset Authentication. A log set is therefore, a collection of multiple logs. You can also define your own custom log sets, for example, to organize raw data.

As long as your event sources are active and working properly, log entries are added to the existing logs. With this log structure in place, you can do multiple things, including:

• Search logs for specific terms with a search language.
• Build your own query to group by a field or calculate specific items.
• Create and manage detection rules for your log data.
• Export data to share with stakeholders.

Explore the interface

Open Log Search and use the descriptions below for a quick overview of the Log Search experience.

• Search: Log Search offers the ability to query in what was previously known as "Advanced mode" only.
• Query Help: Quickly access tips and tricks for building a query.
• Collapse the search bar: You can collapse the search bar, leaving more space to view your results.
• Lock the search bar: Lock the search bar in a collapsed or visible position. Your selection persists as long as you remain in your current browser.
• Order: Log Search will order your query results by newest ingestion time first by default, and will persist your order selection for future logins. You can choose to order your query results by oldest ingestion time first by selecting the up arrow.
• Run: Log Search requires you to select Run or press the Return key to run a query.
• Home tab: Learn how to search your logs and leverage saved queries, Rapid7-provided example queries, and your recent queries from the Home tab.
• Data and Analysis tab navigation: You can now quickly switch between your Data and Analysis tabs to quickly understand your query's results.
• Hide the timeline: You can hide the timeline in the Data tab, leaving more space to view your results.
• Lock the timeline: Lock the timeline in the Data tab in a hidden or visible position. Your selection persists as long as you remain in your current browser.
• Settings: Use the Settings dropdown in the Data tab to select your data view and edit the keys shown in your results.
• Export to CSV: Click the arrow in the Data tab to export your log data to CSV.
• Context menu: Click on keys and values in your Results table and in the Bar chart view to add them to the query bar.
• Click-and-drag timeline: Click and drag along your query's timeline to magnify a specific period of your query's time range.
• Always-open feedback: Share feedback with the InsightIDR Log Search team anytime.

Set your Syntax Highlighting Preferences

Syntax highlighting applies contrasting colors to help you distinguish between the components of a query – such as clauses, keys, values, and comparison operators.

Syntax highlighting helps you to recognize when your query might be missing a component or if it's written incorrectly. If the system does not recognize a component by highlighting it with a specific color, you can find the mistake and make the correction more easily.

For example, this image shows a LEQL clause, operator, and function highlighted in orange, keys in blue, values in black, and regular expressions in purple. With this highlighting, you can validate your query as you are building it.

Try syntax highlighting for yourself by selecting some logs and entering one of our example queries.

Expand to view all syntax and color keys

Format	Syntax Details
	Orange text is applied to: - Keywords (including clauses) - Operators (logical and comparison) - Analytical functions - Functions - Parameters
	Purple text is applied to all regular expressions.
	These values appear as black text: - Text and numbers - Strings - Lists - IP Search
	Keys appear as blue text.
	Syntax errors are reverted to plain text and underlined in red.

Manage Syntax Highlighting

Syntax Highlighting is active by default, but it can be turned off.

To manage syntax highlighting:

1. In InsightIDR, go to the left menu and click Settings.
2. Select User Preferences > Profile Preferences > switch the LEQL Editor on or off.

Search your log data

You can search your log data using the Log Entry Query Language (LEQL) or regular expression (regex). Whichever approach you choose, you can search for key-value pairs, strings, and keywords.

Learn more about the components for building a query, how to search your logs, or start with the example queries provided by Rapid7.

You can also build a query using the context menu:

1. After you run a query, select any clickable key or value.
2. Choose a clause, function, key-value pair, or value to add it to the query bar.
3. Click Run.

Write a LEQL query

Our powerful search language, Log Entry Query Language (LEQL), allows you to construct queries and extract the valuable data within your logs. LEQL follows SQL-style syntax that makes constructing a query simple and intuitive. As you type in Log Search, the query bar automatically suggests the elements of LEQL that you can use in your query. These suggestions help you write queries more easily so you can reach the data you need faster. To open the suggestions dropdown, use the shortcut Ctrl+Space (Mac) or Ctrl+Shift+Space (Windows). If you'd like to learn more about the elements of LEQL, view the Components for Building a Query topic.

To specify, rename, or reorder the keys that are returned in your results, use the select() clause at the start of a query.

To search for specific values in your log data or exclude specific values from your results, use a where() clause. You can add any LEQL operators, clauses, and variables to a where() clause to narrow your search results. You can use only one where() clause in a query.

Use the groupby() clause to visualize and group your data by the keys you specified in your query.

Use the calculate() function to analyze and visualize your log data.

You can use a regular expression (regex) with LEQL to search for more advanced patterns. InsightIDR supports the re2 version of regex.

If you need help writing queries, you can start with the example queries provided by Rapid7.

LEQL Order of Execution

LEQL processes the different query clauses and functions in a specific order. If you were to use every type of clause and function—both filtering and statistical—that LEQL offers, the order of execution would be:

1. select()
2. where()
3. groupby()
4. calculate()
5. having()
6. sort()
7. limit(n)
8. timeslice(n)

View and analyze your log data

Log Search offers different methods to understand your log data depending on what you’re searching for. You can click through the Home, Data, and Analysis tabs to view your data in different ways. For example, the Expanded view in the Data tab is the best option if you need to view an individual log entry, whereas the Bar chart in the Analysis tab is the best option if you want to assess the quantitative results of a query that uses a calculation.

In the Home tab, you can:

• Use recent and saved queries to quickly search your log data
• Try an example query best suited for your common use case

In the Data tab, you can:

• View your search results in the format that suits your needs:
  • The Expanded view is best for viewing detailed log entries with hierarchy.
  • The Wrapped view is best for viewing a large amount of data without scrolling horizontally.
  • The Single Line view is best for scanning through a large amount of data and comparing the length of log entries.
  • The Table view is best for scanning through a large amount of data and comparing the values of individual keys across log entries.
• Edit keys to include or exclude data from your view.
• Use the context menu to build or add to your query directly from your results by clicking on keys or values.

In Analysis tab you can:

• View your statistical search results in the format that suits your needs:
  • When you use a groupby() clause, a bar chart displays the results.
  • When you use a calculate function, an area chart displays the results.
  • When you use a multi-groupby query, you can choose to view either a table of the results or a bar chart. In the multi-layer bar chart, you can click on a bar to view the next set of groups based on the order the keys are listed in the groupby() clause of your query.
  • The Table view provides greater detail and a time series chart to help you spot patterns.
• To add keys or values to your search query directly from a chart or visualization, click on the key or value to open the context menu.

Customize the view of your query data

In the Settings dropdown of the Data tab, you can choose to display the log data in Expanded view, Wrapped view, Single Line view, and Table view.

Note: Your selected keys will persist until you change your log set selections in the Log Sources panel.

In the Edit Keys modal, you can select and remove keys to best fit your needs. The list of Available keys includes all of the keys in the log's schema and any keys that are not listed in the log's schema but are referenced in the log data. The list of Selected keys includes all of the keys that appear in your original search results.

Here’s how to use the Edit Keys modal:

• To view a key in your results, click Select to move it from Available to Selected.
• To move all of the Available keys to your Selected keys, click the right arrow. To move all of your
• Selected keys back to the list of Available keys, click the left arrow.
• To apply your selections to your log data, click Apply. To discard your changes and display only the keys that were originally present in your results, click Restore to Default.

Save your queries for later use

You can save a frequently used query for quick access later. Saving a query can save you valuable time when you need it most.

To save a query:

1. Run a query.
2. Click the (•••) Query Actions button.
3. Select Save Query.
4. Fill in the Save Query modal and click Save. A confirmation banner will appear.

You can access your saved queries on the Home, Data, and Analysis tabs.

Save pre-computed queries

Queries can also be saved as a pre-computed query. Pre-computed queries compute the results for statistical searches in real time, as log entries are received by the Log Search system. Pre-computed queries return results faster than a conventional search and can be viewed in your Log Management settings or through a custom dashboard card. You should consider using a pre-computed query if:

• You regularly perform the same LEQL statistical search on the same logs
• Your statistical search typically takes a long time to return results

To save a pre-computed query:

1. Run a query.
2. Click the (•••) Query Actions button.
3. Select Save as Pre-Computed Query.
4. Fill in the necessary information and then click Save. You can access your saved pre-computed query in your Log Management settings or by adding your saved pre-computed query to an existing custom dashboard card.

Manage your pre-computed queries

View and manage your pre-computed queries in the Log Management settings.

1. Click Settings > Account Settings > Log Management > Pre-Computed Queries.
2. Choose a pre-computed query to manage > click the (•••) Query Actions button > select Edit.
3. In the Edit Pre-Computed Query modal, make edits to your chosen pre-computed query.
4. Click Save.

The edits you make will affect any dashboard cards that use the pre-computed query. You may notice a delay in your dashboard view.

Save query results to a dashboard card

You can save your query results as a dashboard card to view your log data with other related visualizations.

To save query results to a dashboard:

1. View your query results in the Data or Analysis tabs.
2. Click the arrow in the table or chart header and select Save as dashboard card.
3. Fill in the Save as Dashboard Card modal and click Save. A confirmation will banner appear.

You can access the card by navigating to Dashboards and Reports and searching for the dashboard. You can edit the dashboard card after creating it by clicking the Settings icon on the card and selecting Edit.

Export your data

You can export log entries to share with stakeholders at your convenience. When you export log entries as a CSV file, the parsed keys from the key-value pairs are used as the headers in the CSV file. Any log entries that contain values without keys will not be exported into the CSV file.

Similarly, you can download the results of a query that leverages a groupby() clause or calculate() function as a CSV file. This download contains the raw data for the bar and area charts on the Analysis tab. The download will resemble the table view on the Analysis tab, excluding the the Over time column.

To export your data from the Data tab:

1. View your query results in the Data tab.
2. Click Sharing Actions in the Data tab header and select Export to CSV.
   • If you've edited the keys in your results, optionally choose to include all of the keys available in your results in your export.
3. Review the export modal and ensure all of the information is correct, then click Export. An export confirmation message will appear.

You can access the CSV by navigating to Settings > Log Management > Exports. Select the export's date to download the file to your browser.

To export your data from the Analysis tab:

1. View your query results in the Analysis tab.
2. Click Sharing Actions in the Analysis tab header and select Download as CSV.
3. Optionally, rename the export.
4. Review the export modal and ensure all of the information is correct, then click Download. The CSV file will download to your browser.

Delete logs and log sets

Users with the Log Search Admin role can permanently delete logs and log sets.

Deletion restrictions:

• You must have the correct permissions to delete logs in the UI.
• Logs and log sets cannot be deleted until you have deleted all associated active event sources, collectors, and network sensors from the Data Collection or Sensor Management pages.
  • If you are unable to delete a log or log set, contact an administrator to delete the active event sources, collectors, or network sensors associated with the log or log set you want to delete.
• Logs generated by the Insight Agent cannot be deleted (for example, process_start log).
• The deletion icon does not appear for log sets created by InsightIDR.

To delete logs or log sets:

1. In the log selector, identify the item you want to delete.
2. Hover over your selected log or log set.
3. Click the delete icon.

To track and record deletion events, enable Audit Logging through the Insight Platform.

View your search statistics in Log Management

You can view statistical and event queries directly in Log Search or in your Log Management settings to evaluate Log Search’s performance over time. Any concerns you have after reviewing this data can be shared with the InsightIDR support team.

In Log Search, you can view a condensed version of the search statistics by clicking Search Statistics in the timeline after running a valid query. To view the full details of a query, navigate to your Log Management settings. There, you can review the last 100 queries from the past seven days for both statistics and events. To change views, set the toggle to the type of query you want to review. Statistical queries use a groupby() clause, calculate function, or both. Event queries use a where clause or are created through an empty search.

These statistics offer full transparency into the:

• Volume of data searched, explained by the Logs, Bytes Searched, and Events Searched columns
• Time range used to search the data
• Duration of the search
• Index factor achieved by the query
• Number of matched events

InsightIDR optimizes data collection and organization

InsightIDR engineering teams utilize a variety of tuning measures to optimize system performance and data storage limits when ingesting data into Log Search. These measures may include:

• The removal of excessively noisy, irrelevant, or duplicated data that would otherwise clutter dashboards and log sets.
• Data compression to make the best use of your available storage space.

InsightIDR engineering teams work closely with Rapid7 researchers and security experts to ensure we are collecting the data that is the most effective for detecting and investigating malicious activity in your environment.

InsightIDR will automatically parse log events that are in a key-value pair (KVP) or JSON format for easy use of advanced analytics. The KVP and JSON documentation details the specific formats the system will parse. If your logs are not in a standard KVP or JSON format, you can utilize regular expression field extraction to gain access to the same search capabilities.