In InsightIDR, the connected event sources and environment systems produce data in the form of raw logs.
A log is a collection of hundreds or thousands of log entries, which is data that is streamed from an event source.
Logs are typically named based on the event source, for example, Firewall: New York Office. However, you can also name the logs yourself.
Log Search takes every log of raw, collected data, and automatically sorts the logs into log sets. A log set is, therefore, a collection of multiple log streams.
In InsightIDR, a log set is defined (by default) by the type of event within the log stream, such as Firewall, DNS, Active Directory, and other event types. You can also define your own custom log sets, for example, to organize raw data.
Log Search Glossary
For a full list of the terms that are used in Log Search, see the Glossary.
As your event sources continue to be active, more log entries are added to the existing logs. With this log structure in place, you can do multiple things including:
- Search logs for specific terms with a Search Language.
- Build your own query to group by a field or calculate specific items.
- View logs in Visual Search.
- Create tags and Alerts on your log data.
- Export data to share with stakeholders.
Set your Syntax Highlighting Preferences
Syntax highlighting applies contrasting colors to help you distinguish between the components of a query – such as clauses, keys, values, and comparison operators.
Syntax highlighting helps you to recognize when your query might be missing a component or if it's written incorrectly. If the system does not recognize a component by highlighting it with a specific color, you can find the mistake and make the correction more easily.
For example, this image shows a LEQL clause, operator, and function highlighted in orange, keys in blue, values in black, and regular expressions in purple. With this highlighting, you can validate your query as you are building it.
Try syntax highlighting for yourself by selecting some logs and entering one of our example queries.
Expand to view all syntax and color keys
Syntax Highlighting is active by default, but it can be turned off.
Turn Syntax Highlighting On or Off
Syntax highlighting settings are moving to a new location
As of January 2023, syntax highlighting settings are moving from the Profile Settings menu at the top right of the screen to a new section of the left navigation, which you can find by clicking Settings -> User Preferences -> LEQL Editor. This navigation update is taking place as part of a phased release, so if your environment is not yet displaying the new settings menu, you can expect to see the updates shortly.
- From the top menu bar, expand the User Profile menu, and click Profile Settings.
- Find the Enhanced Log Search section.
- Select or deselect Display Syntax Highlighting.
Edit Log Streams and Log Events
Not seeing the log data you expected?
This could be tied to data retention or enrichment. Log data is stored in InsightIDR based on your retention policy. For more information, see Log Data Collection and Storage. InsightIDR may add enrichment data, such as geolocation or organizational data to events, however no original event data is ever removed or replaced. Events in InsightIDR reflect the events as they were originally generated by the source system.
To edit logs and log sets:
- In Log Search, from the list of logs or log sets, identify the one you want to edit and hover your cursor over it to click the ellipsis.
- From the dropdown menu, click Edit Settings.
- After you are finished editing, click Save.
User actions might be added to the audit log
If audit logging is active, the Platform Audit Log will contain a record of log creation, edits, or deletion. It is not currently updated for log sets.
Delete Logs and Log Sets
Depending on your user permissions, you can permanently delete logs and log sets. However, there may be some restrictions or prerequisites to consider.
If a log is associated with an active event source, collector, or network sensor, you must first delete that corresponding entity from either the Data Collection or Sensor Management screens.
If you are not an administrator, then you may need to ask your administrator to delete the event source, collector, or network sensor before you can delete the log or log set.
Restrictions on deleting log sets
If a log set is generated by a collector-based source, it is not possible to delete the log set. Only log sets that are directly sent to Log Search can be deleted.
To delete logs and log sets:
- In Log Search, from the list of logs or log sets, identify the one you want to delete, hover your cursor over it, and click the ellipsis that appears.
- From the dropdown menu, click Delete Log or Delete Log Set.
Data will be permanently deleted
This action permanently deletes the log or log set and its data. If you need to retain the log data for security investigation or compliance purposes, you should carefully consider whether it should be deleted.
Log View Options
When looking at the log entries, you can make reading logs easier by viewing log data in JSON format. Click Entries or Table to change views.
All normalized log entries can be queried either by searching for a string or by searching for a keyword=value pair.
While in Table View, you can quickly create a query based on a selected key, filter by column, or click on source_user and enter a username to search by, and run calculations.
Log labels are applied to provide visual cues that indicate potential issues with your logs. Within InsightIDR, there are currently two default labels,
OutOfOrder. See Default Labels for more information. You can also Create Log Labels with an Alert.
Searching Your Data
InsightIDR allows users different ways of searching their data, including Regex, String, KeyValue, or Keyword search. See Use a Search Language for more information.
Or, you can build queries off of the provided Example Queries.
You can also build a query in Table View:
- While in Table view, select the keyword that you want to query on.
- Select an operator and enter a key value.
- Click Search.
- You can now search within your query.
You can export parseable logs to share with stakeholders at your convenience. When viewing the log entries table, select Export to CSV.
You will see a confirmation message appear. Your CSV file will be available in the Reports tab of the Dashboards and Reports page.
Want to archive your log data?
If you want to learn about archiving your log data, read our topic about Data Archiving.
Optimization and performance tuning
InsightIDR engineering teams utilize a variety of tuning measures to optimize for system performance and data storage limits. These measures may include removal of excessively noisy, irrelevant, or duplicated data that would otherwise clutter dashboards and log sets, as well as data compression to make the best use of your available storage space. When implementing these measures, InsightIDR engineering teams work closely with Rapid7 researchers and security experts to ensure we are collecting data that is the most effective for detecting and investigating malicious activity in your environment.