New Log Search is available for Open Preview
We are rolling out a new Log Search experience to customers with an open preview starting January 31st, 2023. You can still use original Log Search during this open preview. Both the original and New Log Search will exist in parallel until development is complete. For now, review the topic on new Log Search and navigate to the Log Search Open Preview page in InsightIDR to become familiar with the new layout. Check back soon for fully updated documentation.
In InsightIDR, the connected event sources and environment systems produce data in the form of raw logs.
A log is a collection of hundreds or thousands of log entries, which is data that is streamed from an event source.
Logs are typically named based on the event source, for example, Firewall: New York Office. However, you can also name the logs yourself.
Log Search takes every log of raw, collected data, and automatically sorts the logs into log sets. A log set is, therefore, a collection of multiple log streams.
In InsightIDR, a log set is defined (by default) by the type of event within the log stream, such as Firewall, DNS, Active Directory, and other event types. You can also define your own custom log sets, for example, to organize raw data.
Log Search Glossary
For a full list of the terms that are used in Log Search, see the Glossary.
As your event sources continue to be active, more log entries are added to the existing logs. With this log structure in place, you can do multiple things including:
- Search logs for specific terms with a Search Language.
- Build your own query to group by a field or calculate specific items.
- View logs in Visual Search.
- Create tags and Alerts on your log data.
- Export data to share with stakeholders.
Set your Syntax Highlighting Preferences
Syntax highlighting applies contrasting colors to help you distinguish between the components of a query – such as clauses, keys, values, and comparison operators.
Syntax highlighting helps you to recognize when your query might be missing a component or if it's written incorrectly. If the system does not recognize a component by highlighting it with a specific color, you can find the mistake and make the correction more easily.
For example, this image shows a LEQL clause, operator, and function highlighted in orange, keys in blue, values in black, and regular expressions in purple. With this highlighting, you can validate your query as you are building it.
Try syntax highlighting for yourself by selecting some logs and entering one of our example queries.
Expand to view all syntax and color keys
Syntax Highlighting is active by default, but it can be turned off.
Manage Syntax Highlighting
To manage syntax highlighting:
- InsightIDR, go to the left menu and click Settings.
- Select User Preferences and, under Profile Preferences, turn the LEQL Editor on or off.
Edit Log Streams and Log Events
Not seeing the log data you expected?
This could be tied to data retention or enrichment. Log data is stored in InsightIDR based on your retention policy. For more information, see Log Data Collection and Storage. InsightIDR may add enrichment data, such as geolocation or organizational data to events, however no original event data is ever removed or replaced. Events in InsightIDR reflect the events as they were originally generated by the source system.
To edit logs and log sets:
- In Log Search, from the list of logs or log sets, identify the one you want to edit and hover your cursor over it to click the ellipsis.
- From the dropdown menu, click Edit Settings.
- After you are finished editing, click Save.
User actions might be added to the audit log
If audit logging is active, the Platform Audit Log will contain a record of log creation, edits, or deletion. It is not currently updated for log sets.
Delete Logs and Log Sets
Depending on your user permissions, you can permanently delete logs and log sets. However, there may be some restrictions or prerequisites to consider.
If a log is associated with an active event source, collector, or network sensor, you must first delete that corresponding entity from either the Data Collection or Sensor Management screens.
If you are not an administrator, then you may need to ask your administrator to delete the event source, collector, or network sensor before you can delete the log or log set.
Restrictions on deleting log sets
If a log set is generated by a collector-based source, it is not possible to delete the log set. Only log sets that are directly sent to Log Search can be deleted.
To delete logs and log sets:
- In Log Search, from the list of logs or log sets, identify the one you want to delete, hover your cursor over it, and click the ellipsis that appears.
- From the dropdown menu, click Delete Log or Delete Log Set.
Data will be permanently deleted
This action permanently deletes the log or log set and its data. If you need to retain the log data for security investigation or compliance purposes, you should carefully consider whether it should be deleted.
Log View Options
When viewing your log entries, you can change views based on your preferences. Click Entries, Table, or Visualizations to change views.
In Entries View, you can:
- Select a key to add it to the query bar and find all occurrences of the key in the selected logs and time range.
- Select the info icon to open the Entry Inspector for the selected log line.
- Select the search icon to view additional context for the selected log line.
- Manage Keys to include or exclude data from your view.
In Table View, you can:
- Add a search expression to the Query Builder by selecting a column header, selecting an operator, and entering a value or key.
- Manage Columns to include or exclude data from your view.
In Visualizations View, you can:
- View Rapid7-generated cards for your log data.
- Add a card.
- Save the visualizations to a dashboard.
Log labels are applied to provide visual cues that indicate potential issues with your logs. Within InsightIDR, there are currently two default labels,
OutOfOrder. See Default Labels for more information. You can also Create Log Labels with an Alert.
Searching Your Data
InsightIDR provides different ways to search your data, including Regex, String, KeyValue, or Keyword search.
Read Use a Search Language for more information, or you can build queries off of the provided Example Queries.
You can also build a query in Table View:
- While in Table view, select the keyword that you want to query on.
- Select an operator and enter a key value.
- Click Search.
You can export parseable logs to share with stakeholders at your convenience.
When viewing your query results in the Entries or Table tabs, select the Download dropdown, then Export to CSV.
An export confirmation message will appear. You can access your CSV by navigating to Settings > Log Search > Exports.
Want to archive your log data?
If you want to learn how to archive your log data, read our Data Archiving topic.
Optimization and performance tuning
InsightIDR engineering teams utilize a variety of tuning measures to optimize for system performance and data storage limits. These measures may include removal of excessively noisy, irrelevant, or duplicated data that would otherwise clutter dashboards and log sets, as well as data compression to make the best use of your available storage space. When implementing these measures, InsightIDR engineering teams work closely with Rapid7 researchers and security experts to ensure we are collecting data that is the most effective for detecting and investigating malicious activity in your environment.