Log Search

In InsightIDR, the event sources and environment systems gather data in the form of raw logs. A log is a collection of log entries, which are pieces of data that are streamed from event sources. Logs are typically named based on the event source, for example, Firewall: New York Office. However, you can also name the logs yourself.

Log Search ingests every log of raw data and sorts the log into a log set based on the log’s event type. A single device, like a domain controller, can collect log entries that flow into multiple log sets, such as Active Directory Admin Activity and Asset Authentication. A log set is therefore, a collection of multiple logs. You can also define your own custom log sets, for example, to organize raw data.

As long as your event sources are active and working properly, log entries are added to the existing logs. With this log structure in place, you can do multiple things, including:

• Search logs for specific terms with a search language.
• Build your own query to group by a field or calculate specific items.
• Create alerts for your log data.
• Export data to share with stakeholders.

Explore the interface

Open Log Search and use the descriptions below for a quick overview of the Log Search experience.

• Search: Log Search offers the ability to query in what was previously known as "Advanced mode" only.
• Query Help: Quickly access tips and tricks for building a query.
• Collapsible log selections: Click the arrow to hide your log selections, leaving more space for your query and results.
• Order: You can now order your query results by newest ingestion time first. Log Search will order your query results by newest ingestion time first by default, and will persist your order selection for future logins.
• Run: Log Search requires you to select Run or press the Return key to run a query.
• Home tab: Learn how to search your logs and leverage saved queries, Rapid7-provided example queries, and your recent queries from the Home tab.
• Data and Analysis tab navigation: You can now quickly switch between your Data and Analysis tabs to quickly understand your query's results.
• Settings: Use the Settings dropdown in the Data tab to select your data view and edit the keys shown in your results.
• Export to CSV: Click the arrow in the Data tab to export your log data to CSV.
• Context menu: Click on keys and values in your Results table and in the Bar chart view to add them to the query bar.
• Click-and-drag timeline: Click and drag along your query's timeline to magnify a specific period of your query's time range.
• Always-open feedback: Share feedback with the InsightIDR Log Search team anytime.

Set your Syntax Highlighting Preferences

Syntax highlighting applies contrasting colors to help you distinguish between the components of a query – such as clauses, keys, values, and comparison operators.

Syntax highlighting helps you to recognize when your query might be missing a component or if it's written incorrectly. If the system does not recognize a component by highlighting it with a specific color, you can find the mistake and make the correction more easily.

For example, this image shows a LEQL clause, operator, and function highlighted in orange, keys in blue, values in black, and regular expressions in purple. With this highlighting, you can validate your query as you are building it.

Try syntax highlighting for yourself by selecting some logs and entering one of our example queries.

Expand to view all syntax and color keys

Format	Syntax Details
	Orange text is applied to: - Keywords (including clauses) - Operators (logical and comparison) - Analytical functions - Functions - Parameters
	Purple text is applied to all regular expressions.
	These values appear as black text: - Text and numbers - Strings - Lists - IP Search
	Keys appear as blue text.
	Syntax errors are reverted to plain text and underlined in red.

Manage Syntax Highlighting

Syntax Highlighting is active by default, but it can be turned off.

To manage syntax highlighting:

1. In InsightIDR, go to the left menu and click Settings.
2. Select User Preferences > Profile Preferences > switch the LEQL Editor on or off.

Search your log data

You can search your log data using the Log Entry Query Language (LEQL) or regular expression (regex). Whichever approach you choose, you can search for key-value pairs, strings, and keywords.

Learn more about the components for building a query, how to search your logs, or start with the example queries provided by Rapid7.

You can also build a query using the context menu:

1. After you run a query, select any clickable key or value.
2. Choose a clause, function, key-value pair, or value to add it to the query bar.
3. Click Run.

Write a LEQL query

Our powerful search language, Log Entry Query Language (LEQL), allows you to construct queries and extract the valuable data within your logs. LEQL follows SQL-style syntax that makes constructing a query simple and intuitive.

Use a where() clause at the start of a query to search for specific values in your log data or exclude specific values from your results. You can add any LEQL operators, clauses, and variables to a where() clause to narrow your search results. You can use only one where() clause in a query.

Use the groupby() clause to visualize and group your data by the key(s) you specified in your query.

Use the calculate() function to analyze and visualize your log data.

You can use a regular expression (regex) with LEQL to search for more advanced patterns. InsightIDR supports the re2 version of regex.

If you need help writing queries, you can start with the example queries provided by Rapid7.

LEQL Order of Execution

LEQL processes the different query clauses and functions in a specific order. If you were to use every type of clause and function—both filtering and statistical—that LEQL offers, the order of execution would be:

1. where()
2. groupby()
3. calculate()
4. having()
5. sort()
6. limit(n)
7. timeslice(n)

View and analyze your log data

Log Search offers different methods to understand your log data depending on what you’re searching for. You can click through the Home, Data, and Analysis tabs to view your data in different ways. For example, the Expanded view in the Data tab is the best option if you need to view an individual log line, whereas the Bar chart in the Analysis tab is the best option if you want to assess the quantitative results of a query that uses a calculation.

In the Home tab, you can:

• Use recent and saved queries to quickly search your log data
• Try an example query best suited for your common use case

In the Data tab, you can:

• View your search results in the format that suits your needs:
  • The Expanded view is best for viewing detailed log entries with hierarchy.
  • The Wrapped view is best for viewing a large amount of data without scrolling horizontally.
  • The Single Line view is best for scanning through a large amount of data and comparing the length of log entries.
  • The Table view is best for scanning through a large amount of data and comparing the values of individual keys across log entries.
• Edit keys to include or exclude data from your view.
• Use the context menu to build or add to your query directly from your results by clicking on keys or values.

In Analysis tab you can:

• View your statistical search results in the format that suits your needs:
  • When you use a groupby() clause, a bar chart displays the results.
  • When you use a calculate function, an area chart displays the results.
  • When you use a multi-groupby query, you can choose to view either a table of the results or a bar chart. In the multi-layer bar chart, you can click on a bar to view the next set of groups based on the order the keys are listed in the groupby() clause of your query.
  • The Table view provides greater detail and a time series chart to help you spot patterns.
• To add keys or values to your search query directly from a chart or visualization, click on the key or value to open the context menu.

Customize the view of your query data

In the Settings dropdown of the Data tab, you can choose to display the log data in Expanded view, Wrapped view, Single Line view, and Table view.

Note: Your selected keys will persist until you change your log set selections in the Log Sources panel.

In the Edit Keys modal, you can select and remove keys to best fit your needs. The list of Available keys includes all of the keys in the log's schema and any keys that are not listed in the log's schema but are referenced in the log data. The list of Selected keys includes all of the keys that appear in your original search results.

Here’s how to use the Edit Keys modal:

• To view a key in your results, click Select to move it from Available to Selected.
• To move all of the Available keys to your Selected keys, click the right arrow. To move all of your
• Selected keys back to the list of Available keys, click the left arrow.
• To apply your selections to your log data, click Apply. To discard your changes and display only the keys that were originally present in your results, click Restore to Default.

Save your queries for later use

You can save a frequently used query for quick access later. Saving a query can save you valuable time when you need it most.

To save a query

1. Run a query.
2. Click the ellipsis next to the Run button.
3. Select Save Query.
4. Fill in the Save Query modal and click Save. A confirmation banner appears.

You can access your saved queries on the Home tab.

Save query results to a dashboard card

You can save your query results as a dashboard card to view your log data with other related visualizations.

To save query results to a dashboard:

1. View your query results in the Data or Analysis tabs.
2. Click the arrow in the table or chart header and select Save as dashboard card.
3. Fill in the Save as Dashboard Card modal and click Save. A confirmation banner appears.

You can access the card by navigating to Dashboards and Reports and searching for the dashboard. You can edit the dashboard card after creating it by clicking the Settings icon on the card and selecting Edit.

Export your data

You can export log entries to share with stakeholders at your convenience.

When you export log entries as a CSV file, the parsed keys from the key-value pairs are used as the headers in the CSV file. Any log entries that contain values without keys will not be exported into the CSV file.

To export your data:

1. View your query results in the Data or Analysis tabs.
2. Click the arrow in the Results table header and select Export to CSV.
   • If you're exporting results from the Data tab and you've edited the keys in your results, optionally choose to include all of the keys available in your results in your export.
   • If you're exporting results from the Analysis tab, name your export.
3. Review the export modal and ensure all of the information is correct, then click Export. An export confirmation message will appear.

You can access the CSV by navigating to Settings > Log Search > Exports.

View your Log Search Query Statistics

You can review the last 100 statistical queries that have been run over the last seven days by navigating to Settings > Log Search > Log Search Statistics. Statistical queries use a groupby() clause, calculate function, or both.

These statistics offer full transparency into the:

• Volume of data searched, explained by the Logs, Bytes Searched, and Events Searched columns
• Time range used to search the data
• Duration of the search
• Index Factor achieved by the query
• Number of matched events

With this view, you can benchmark Log Search’s performance over time and easily determine any concerns to share with the InsightIDR support team.

InsightIDR optimizes data collection and organization

InsightIDR engineering teams utilize a variety of tuning measures to optimize system performance and data storage limits when ingesting data into Log Search. These measures may include:

• The removal of excessively noisy, irrelevant, or duplicated data that would otherwise clutter dashboards and log sets.
• Data compression to make the best use of your available storage space.

InsightIDR engineering teams work closely with Rapid7 researchers and security experts to ensure we are collecting the data that is the most effective for detecting and investigating malicious activity in your environment.

InsightIDR will automatically parse log events that are in a key-value pair (KVP)or JSON format for easy use of advanced analytics. The KVP and JSON documentation details the specific formats the system will parse. If your logs are not in a standard KVP or JSON format, you can utilize regular expression field extraction to gain access to the same search capabilities.