Custom Parsing Tool

The Custom Parsing Tool gives you the ability to create custom parsing rules for log events to extract the data you need. You can either parse logs in a format that is unknown to InsightIDR, which allows you to pull and monitor data that is not automatically extracted by InsightIDR, or further parse log entries in common formats. Using a custom parser, you can extract log data that is most relevant to your organizational needs.

For example, if you are using an electronic health record (EHR) tool, you may want to parse out patient IDs, login successes and failures, and EHR event types. With this data, you can create operational dashboards that track the data that is uniquely important to your business needs.

Building custom parsing rules is simple. The Custom Parser provides an interface you can use to show us exactly what you want to extract from the logs. Based on your input, it auto-generates the patterns needed to extract the data.

Things to Know about Custom Parsing Rules

As you are building a custom parser, here are some things to keep in mind:

  • Once your parsing rule is created, you can expect a 5-10 minute delay before your parsed data shows up in log search.
  • Previously collected data will not be parsed with your new parsing rules. Only data collected after the parsing rules are implemented will be parsed with them.
  • Parsing rules can impact current dashboards. You will need to check your filters to assess how the parsing rules will impact your dashboards and alerts.
  • You will not be able to edit the parser after you save it. You will need to delete it and start over.
  • Data that is parsed with a custom parsing rule appears in log search with a “custom_data” tag in the log line. If your data does not appear in log search, it could indicate an issue with your parsing rule. For more details, see the Troubleshooting section.

How Custom Parsers Work

Custom parsers apply to raw events, which are lines of text collected by an event source. Your custom parsing rule will parse raw events when it is uploaded to the Insight cloud. To build your rule, you define the values extracted from a log line and the values you want to map them to. With the custom parsing tool, you can also normalize the structure of your logs, making it easier to find the fields you want to extract.

** Steps to build a custom parsing rule: **

  1. Launch the custom parsing tool
  2. Select the log you want to sample and use to extract fields
  3. Filter on a subset of your data
  4. Extract fields
  5. Bulk apply rules

Build a Custom Parsing Rule

To build your rule, define the values extracted from a log line and the values you want to map them to.

To get started

  1. Launch the custom parsing tool. From the left menu, go to Log Search, and select Custom Data Parsing > Create Custom Parsing Rule.
  2. Once you are in the Custom Parsing Tool, complete tasks 1-5 to build your custom rules.

Task 1: Name your rule.

You should use a name that is unique and also descriptive so that you can easily find the rule later. For example, if you want to create parsing rules for your firewall logs, you could enter, “Firewall Log 1".

To add a name

  1. Enter a unique name.
  2. Click Next: Select Logs.

Task 2: Select a log

This task involves two steps: selecting the log and setting the time range. The time range allows you to preview a specific range of logs that have the fields you want to parse. Log lines are displayed from oldest to newest, so it’s important to select a time range that will generate the most relevant sampling of data. In Task 4, you will extract fields from these log lines.

To select a log:

  1. Under Step 1: Log select the log that you want to extract fields from.
  2. Under Step 2: Sample Log Lines select a time range. If you don’t see any sample data, you may need to increase the time range you’ve selected.
  3. Click Next: Create a Filter.

Task 3: Create a filter

Next, let’s determine whether you need to create a filter. To do this, review the structure of your log lines: if your log streams have multiple formats for the log events, complete this step to ensure that your custom parsing rule applies to only the relevant logs. However, if your log lines are uniformly formatted, you can skip this step and proceed to Task 4: Extract Fields.

Filters added to your custom parsing rule are applied to incoming logs before they are parsed. Say for example, you have an event source that sends DNS, VPN, and firewall events in a single log line. You can create a filter that focuses on firewall events, giving you increased visibility into malicious activity happening along that layer.

To create a filter:

  1. Enter the values that you want to include as part of your filter.
  2. Click Apply. This filter will be applied to any data before it is parsed by this rule
  3. Click Next: Extract Fields.

Task 4: Extract Fields from the log

  1. Highlight the data you want to extract from the sample log. The Custom Parsing tool will automatically highlight the matching data in the other log lines. When extracting data, we recommend that you not include brackets or quotation marks, as this may make it more difficult for you to search for your fields later.
  2. When you are satisfied with the selected data, click Validate.
  3. Name your field, and click Add Field.
  4. Repeat steps 1-3 until you have extracted all the fields you want to include in your rule.
  5. Click Next: Bulk Apply Rules

Task 5: Bulk Apply Rules

You can apply your new custom parsing rules to other logs. To do so, select your logs from the list of suggested logs.

You must choose logs that are identical to the format of the parsing rule that you created. For example, if your sample logs are firewall logs, you can select matching firewall logs from this list.

Manage Custom Parsing Rules

Edit a Parsing Rule

You cannot edit a parser after you create it. If you need to make a change, you will need to delete the rule and recreate it.

Previously collected data will not be parsed with your new parsing rules. Only data collected after the parsing rules are implemented will be parsed with them.

View All Parsing Rules

From the left menu, go to your log search and choose Custom Data Parsing > Manage Parsing Rules. The Manage Parsing Rules table displays all of the parsing rules.

View Extracted Fields for a Parsing Rule

From the left menu, go to your log search and choose Custom Data Parsing > Manage Parsing Rules. The Manage Parsing Rules table displays all of the parsing rules.

The Extracted Fields column shows the extracted fields for each parsing rule.

View Logs for a Parsing Rule

From the left menu, go to your log search and choose Custom Data Parsing > Manage Parsing Rules. The Manage Parsing Rules table displays all of the parsing rules.

The Logs column shows the logs that the parsing rule is applied to.

Delete a Parsing Rule

Deleting a parsing rule may affect the dashboards, queries, and custom alerts that use its data. Please review your queries. After you delete a parsing rule, you cannot revert the changes.

Note that once you delete your parsing rule, new incoming data will be unparsed. Previously parsed data will remain in a parsed state.

To delete a parsing rule:

  1. From the left menu, go to your log search and choose Custom Data Parsing > Manage Parsing Rules. The Manage Parsing Rules table displays all of the parsing rules.
  2. Find the parsing rule you want to delete and click the Delete icon.

Troubleshoot your Custom Parsing Rules

My parser stopped parsing my logs

If the Custom Parsing Tool is no longer parsing your logs, the most likely reason is that the incoming logs no longer match the parsing rule (s) that you created, usually because a change made to the sending device has changed the format of the logs. Although you may not be aware of the device being changed, the vendor may have changed the format during a product upgrade or an admin might have changed the logging configuration. To fix this, you should create new parsing rule (s) that match the current logs. Depending on the log data, you may also wish to delete the old parsing rules.