Log Search
In InsightIDR, the event sources and environment systems gather data in the form of raw logs. A log is a collection of log entries, which are pieces of data that are streamed from event sources. Logs are typically named based on the event source, for example, Firewall: New York Office. However, you can also name the logs yourself.
Log Search ingests every log of raw data and sorts the log into a log set based on the log’s event type. A single device, like a domain controller, can collect log entries that flow into multiple log sets, such as Active Directory Admin Activity and Asset Authentication. A log set is therefore, a collection of multiple logs. You can also define your own custom log sets, for example, to organize raw data.
Log Search Glossary
For a full list of the terms that are used in Log Search, view the Glossary.
As long as your event sources are active and working properly, log entries are added to the existing logs. With this log structure in place, you can do multiple things, including:
- Search logs for specific terms with a search language.
- Build your own query to group by a field or calculate specific items.
- Create and manage detection rules for your log data.
- Export data to share with stakeholders.
Explore the interface
Open Log Search and use the descriptions below for a quick overview of the Log Search experience.
- Search: Log Search offers the ability to query in what was previously known as "Advanced mode" only.
- Query Help: Quickly access tips and tricks for building a query.
- Collapse the search bar: You can collapse the search bar, leaving more space to view your results.
- Lock the search bar: Lock the search bar in a collapsed or visible position. Your selection persists as long as you remain in your current browser.
- Order: Log Search will order your query results by newest ingestion time first by default, and will persist your order selection for future logins. You can choose to order your query results by oldest ingestion time first by selecting the up arrow.
- Run: Log Search requires you to select Run or press the Return key to run a query.
- Home tab: Learn how to search your logs and leverage saved queries, Rapid7-provided example queries, and your recent queries from the Home tab.
- Data and Analysis tab navigation: You can now quickly switch between your Data and Analysis tabs to quickly understand your query's results.
- Hide the timeline: You can hide the timeline in the Data tab, leaving more space to view your results.
- Lock the timeline: Lock the timeline in the Data tab in a hidden or visible position. Your selection persists as long as you remain in your current browser.
- Settings: Use the Settings dropdown in the Data tab to select your data view and edit the keys shown in your results.
- Export to CSV: Click the arrow in the Data tab to export your log data to CSV.
- Context menu: Click on keys and values in your Results table and in the Bar chart view to add them to the query bar.
- Click-and-drag timeline: Click and drag along your query's timeline to magnify a specific period of your query's time range.
- Always-open feedback: Share feedback with the InsightIDR Log Search team anytime.
Set your Syntax Highlighting Preferences
Syntax highlighting applies contrasting colors to help you distinguish between the components of a query – such as clauses, keys, values, and comparison operators.
Syntax highlighting helps you to recognize when your query might be missing a component or if it's written incorrectly. If the system does not recognize a component by highlighting it with a specific color, you can find the mistake and make the correction more easily.
For example, this image shows a LEQL clause, operator, and function highlighted in orange, keys in blue, values in black, and regular expressions in purple. With this highlighting, you can validate your query as you are building it.
Try syntax highlighting for yourself by selecting some logs and entering one of our example queries.
Expand to view all syntax and color keys
Manage Syntax Highlighting
Syntax Highlighting is active by default, but it can be turned off.
To manage syntax highlighting:
- In InsightIDR, go to the left menu and click Settings.
- Select User Preferences > Profile Preferences > switch the LEQL Editor on or off.
Search your log data
You can search your log data using the Log Entry Query Language (LEQL) or regular expression (regex). Whichever approach you choose, you can search for key-value pairs, strings, and keywords.
Learn more about the components for building a query, how to search your logs, or start with the example queries provided by Rapid7.
You can also build a query using the context menu:
- After you run a query, select any clickable key or value.
- Choose a clause, function, key-value pair, or value to add it to the query bar.
- Click Run.
Write a LEQL query
Our powerful search language, Log Entry Query Language (LEQL), allows you to construct queries and extract the valuable data within your logs. LEQL follows SQL-style syntax that makes constructing a query simple and intuitive. As you type in Log Search, the query bar automatically suggests the elements of LEQL that you can use in your query. These suggestions help you write queries more easily so you can reach the data you need faster. To open the suggestions dropdown, use the shortcut Ctrl+Space (Mac) or Ctrl+Shift+Space (Windows). If you'd like to learn more about the elements of LEQL, view the Components for Building a Query topic.
Limitations and availability of automatic LEQL suggestions
This capability will gradually roll out to all customers. Only the Log Search query bar automatically suggests elements of LEQL. Key and value suggestions are not currently supported. Use the up and down arrows on your keyboard to navigate through the suggested options. Use the left and right arrows on your keyboard to move to to different lines in your query. The query bar in Legacy Log Search does not offer this capability.
To specify, rename, or reorder the keys that are returned in your results, use the select()
clause at the start of a query.
To search for specific values in your log data or exclude specific values from your results, use a where()
clause. You can add any LEQL operators, clauses, and variables to a where()
clause to narrow your search results. You can use only one where()
clause in a query.
Use the groupby()
clause to visualize and group your data by the keys you specified in your query.
Use the calculate()
function to analyze and visualize your log data.
You can use a regular expression (regex) with LEQL to search for more advanced patterns. InsightIDR supports the re2 version of regex.
If you need help writing queries, you can start with the example queries provided by Rapid7.
LEQL Order of Execution
LEQL processes the different query clauses and functions in a specific order. If you were to use every type of clause and function—both filtering and statistical—that LEQL offers, the order of execution would be:
select()
where()
groupby()
calculate()
having()
sort()
limit(n)
timeslice(n)
View and analyze your log data
Log Search offers different methods to understand your log data depending on what you’re searching for. You can click through the Home, Data, and Analysis tabs to view your data in different ways. For example, the Expanded view in the Data tab is the best option if you need to view an individual log entry, whereas the Bar chart in the Analysis tab is the best option if you want to assess the quantitative results of a query that uses a calculation.
Read a log entry with persistent highlight
Click a log entry on the Data tab to scroll through your log data horizontally without losing track of the log entry you're examining.
In the Home tab, you can:
- Use recent and saved queries to quickly search your log data
- Try an example query best suited for your common use case
In the Data tab, you can:
- View your search results in the format that suits your needs:
- The Expanded view is best for viewing detailed log entries with hierarchy.
- The Wrapped view is best for viewing a large amount of data without scrolling horizontally.
- The Single Line view is best for scanning through a large amount of data and comparing the length of log entries.
- The Table view is best for scanning through a large amount of data and comparing the values of individual keys across log entries.
- Edit keys to include or exclude data from your view.
- Use the context menu to build or add to your query directly from your results by clicking on keys or values.
In Analysis tab you can:
- View your statistical search results in the format that suits your needs:
- When you use a
groupby()
clause, a bar chart displays the results. - When you use a
calculate
function, an area chart displays the results. - When you use a multi-groupby query, you can choose to view either a table of the results or a bar chart. In the multi-layer bar chart, you can click on a bar to view the next set of groups based on the order the keys are listed in the
groupby()
clause of your query. - The Table view provides greater detail and a time series chart to help you spot patterns.
- When you use a
- To add keys or values to your search query directly from a chart or visualization, click on the key or value to open the context menu.
Customize the view of your query data
In the Settings dropdown of the Data tab, you can choose to display the log data in Expanded view, Wrapped view, Single Line view, and Table view.
Note: Your selected keys will persist until you change your log set selections in the Log Sources panel.
In the Edit Keys modal, you can select and remove keys to best fit your needs. The list of Available keys includes all of the keys in the log's schema and any keys that are not listed in the log's schema but are referenced in the log data. The list of Selected keys includes all of the keys that appear in your original search results.
Here’s how to use the Edit Keys modal:
- To view a key in your results, click Select to move it from Available to Selected.
- To move all of the Available keys to your Selected keys, click the right arrow. To move all of your
- Selected keys back to the list of Available keys, click the left arrow.
- To apply your selections to your log data, click Apply. To discard your changes and display only the keys that were originally present in your results, click Restore to Default.
Save your queries for later use
You can save a frequently used query for quick access later. Saving a query can save you valuable time when you need it most.
To save a query:
- Run a query.
- Click the (•••) Query Actions button.
- Select Save Query.
- Fill in the Save Query modal and click Save. A confirmation banner will appear.
You can access your saved queries on the Home, Data, and Analysis tabs.
Save pre-computed queries
Queries can also be saved as a pre-computed query. Pre-computed queries compute the results for statistical searches in real time, as log entries are received by the Log Search system. Pre-computed queries return results faster than a conventional search and can be viewed in your Log Management settings or through a custom dashboard card. You should consider using a pre-computed query if:
- You regularly perform the same LEQL statistical search on the same logs
- Your statistical search typically takes a long time to return results
You must have the correct permissions to create pre-computed queries
Only InsightIDR administrators or analysts can create pre-computed queries.
To save a pre-computed query:
- Run a query.
- Click the (•••) Query Actions button.
- Select Save as Pre-Computed Query.
- Fill in the necessary information and then click Save. You can access your saved pre-computed query in your Log Management settings or by adding your saved pre-computed query to an existing custom dashboard card.
Only 100 pre-computed queries can be saved
A 100 limit of pre-computed queries can be saved for each organization.
Manage your pre-computed queries
View and manage your pre-computed queries in the Log Management settings.
- Click Settings > Account Settings > Log Management > Pre-Computed Queries.
- Choose a pre-computed query to manage > click the (•••) Query Actions button > select Edit.
- In the Edit Pre-Computed Query modal, make edits to your chosen pre-computed query.
- Click Save.
The edits you make will affect any dashboard cards that use the pre-computed query. You may notice a delay in your dashboard view.
Save query results to a dashboard card
You can save your query results as a dashboard card to view your log data with other related visualizations.
To save query results to a dashboard:
- View your query results in the Data or Analysis tabs.
- Click the arrow in the table or chart header and select Save as dashboard card.
- Fill in the Save as Dashboard Card modal and click Save. A confirmation will banner appear.
You can access the card by navigating to Dashboards and Reports and searching for the dashboard. You can edit the dashboard card after creating it by clicking the Settings icon on the card and selecting Edit.
Export your data
You can export log entries to share with stakeholders at your convenience. When you export log entries as a CSV file, the parsed keys from the key-value pairs are used as the headers in the CSV file. Any log entries that contain values without keys will not be exported into the CSV file.
Similarly, you can download the results of a query that leverages a groupby()
clause or calculate()
function as a CSV file. This download contains the raw data for the bar and area charts on the Analysis tab. The download will resemble the table view on the Analysis tab, excluding the the Over time
column.
Some log sets contain deduplicated data
When you export log data from the DNS Query, Firewall Activity, or Web Proxy Activity log sets, the export can contain deduplicated log entries. Because exports are limited to 1 million log entries, data deduplication offers a clearer view of your log activity without the obstruction of repetitive activity cluttering the results. Learn more about data deduplication.
To export your data from the Data tab:
- View your query results in the Data tab.
- Click Sharing Actions in the Data tab header and select Export to CSV.
- If you've edited the keys in your results, optionally choose to include all of the keys available in your results in your export.
- Review the export modal and ensure all of the information is correct, then click Export. An export confirmation message will appear.
You can access the CSV by navigating to Settings > Log Management > Exports. Select the export's date to download the file to your browser.
To export your data from the Analysis tab:
- View your query results in the Analysis tab.
- Click Sharing Actions in the Analysis tab header and select Download as CSV.
- Optionally, rename the export.
- Review the export modal and ensure all of the information is correct, then click Download. The CSV file will download to your browser.
Want to archive your log data?
If you want to learn how to archive your log data, read our Data Archiving topic.
Delete logs and log sets
Users with the Log Search Admin role can permanently delete logs and log sets.
Deletion restrictions:
- You must have the correct permissions to delete logs in the UI.
- Logs and log sets cannot be deleted until you have deleted all associated active event sources, collectors, and network sensors from the Data Collection or Sensor Management pages.
- If you are unable to delete a log or log set, contact an administrator to delete the active event sources, collectors, or network sensors associated with the log or log set you want to delete.
- Logs generated by the Insight Agent cannot be deleted (for example,
process_start log
). - The deletion icon does not appear for log sets created by InsightIDR.
To delete logs or log sets:
- In the log selector, identify the item you want to delete.
- Hover over your selected log or log set.
- Click the delete icon.
Data will be permanently deleted
This action permanently deletes the log or log set and its data. If you need to retain the log data for security, investigation, or compliance purposes, carefully consider whether it should be deleted.
To track and record deletion events, enable Audit Logging through the Insight Platform.
View your Search Statistics in Log Management
You can view statistical and event queries in your Log Management settings to evaluate Log Search’s performance over time. Any concerns you have after reviewing this data can be shared with the InsightIDR support team. You can review the last 100 queries from the past seven days for both statistics and events. To change views, set the toggle to the type of query you want to review. Statistical queries use a groupby()
clause, calculate
function, or both. Event queries use a where
clause or are created through an empty search.
These statistics offer full transparency into the:
- Volume of data searched, explained by the
Logs
,Bytes Searched
, andEvents Searched
columns - Time range used to search the data
- Duration of the search
- Index Factor achieved by the query
- Number of matched events
Administrator role required
To view Log Search Statistics you must be an administrator.
InsightIDR optimizes data collection and organization
InsightIDR engineering teams utilize a variety of tuning measures to optimize system performance and data storage limits when ingesting data into Log Search. These measures may include:
- The removal of excessively noisy, irrelevant, or duplicated data that would otherwise clutter dashboards and log sets.
- Data compression to make the best use of your available storage space.
InsightIDR engineering teams work closely with Rapid7 researchers and security experts to ensure we are collecting the data that is the most effective for detecting and investigating malicious activity in your environment.
InsightIDR will automatically parse log events that are in a key-value pair (KVP) or JSON format for easy use of advanced analytics. The KVP and JSON documentation details the specific formats the system will parse. If your logs are not in a standard KVP or JSON format, you can utilize regular expression field extraction to gain access to the same search capabilities.
Rapid7 stores and retains logs centrally
Rapid7 stores and retains logs centrally for security, compliance, and operational needs. These centralized logs cannot be altered after they have been submitted to our logging system. You can still delete logs but only in certain circumstances, for example, if there is no active collector associated with the log or event source. In such cases, an immutable audit log is recorded and will capture what action took place, the time the action occurred, and who completed the action.