Use a Search Language

InsightIDR allows you to use several different languages while searching through your logs:

• Quotes
• Numbers
• Operators
  • Logical Operators
  • Comparison Operators
    • Using NOT with Comparison Operators
  • Order of Operation
• Keyword Search
• Regular Expression Keyword Search
  • Partial Matching
  • Case Insensitive Search and Comparison
  • Regular Expression Operators
• Regular Expression Field Extraction
  • Benefits
  • Advanced Capabilities
• Key Value Pair and JSON Search
  • Key Expressions
• IP Search
  • Things to know about CIDR notation in InsightIDR
• Analytic Functions and Visualizations
  • Count
  • Sum
  • Average
  • Count Unique
  • Min
  • Max
  • Groupby
  • Timeslice
  • Percentile
  • Bytes
  • Standard Deviation

Quotes

You can use quotes to match an exact string with Log Search. Place single quotes ' ' or double quotes “ ” around the string.

In addition, you can use both types of triple quotes ''' ''' and """ """ in all possible combinations and scenarios. Triple quotes will use everything in between them unchanged, including complicated cases of searching by strings containing mixed quotes.

Examples:

 
1
"""abc"""
2
'''abc'''
3
key = """abc"""
4
key = '''abc'''
5
key CONTAINS-ANY ["""aaa""", '''bbb''', "ccc", 'ddd']
6
data.cmd_line ICONTAINS """WHERE "(name='java.exe' OR name='launch.exe')"""

Numbers

Numerical values in search queries must be formatted as signed or unsigned integer or floating-point values, in simple or scientific notation.

Examples:

 
1
answer = 42
2
time_dif < -1e-03
3
PI >= 3.1415
4
A = 6.0221409e+23

For the best search results, numeric values should fit in ranges of Java Double or Long data types. Otherwise, the value will be treated as a string.

If a log event contains my_key: 1000, you can use any of the following searches to return it:

 
1
my_key = 1000
2
my_key = 1e3
3
my_key = 1.0e+03
4
my_key = 1000.0

In comparison, operations such as <, >, <=, >= can use numbers with units. However, units are not calculated as part of the comparison. For example, searching for a value<100bytes would not return a result with value=200bits.

Operators

InsightIDR supports both logical and comparison operators, which allow you to create more complex searches. The guide below will introduce both sets of operators available to use while constructing a query.

Logical Operators

InsightIDR supports the following logical operators to create comprehensive search criteria. Logical operators are treated as case-insensitive by LEQL.

Comparison Operators

Comparison operators can be used for Key Value Pairs (KVP) search and Regular Expression search.

Using NOT with Comparison Operators

You can use NOT in front of these operators:

• CONTAINS
• ICONTAINS
• STARTS-WITH
• ISTARTS-WITH
• IN
• IIN
• CONTAINS-ANY
• ICONTAINS-ANY
• CONTAINS-ALL
• ICONTAINS-ALL
• STARTS-WITH-ANY
• ISTARTS-WITH-ANY

Examples:

key NOT IN [aaa, bbb]

key1, key2 NOT STARTS-WITH aaa

Order of Operation

As with other query languages, LEQL follows boolean logic rules and respects brackets. This should be considered if you want to control the operator precedence that the query follows.

To understand the operator precedence of AND and OR operators in LEQL, it helps to consider a simple math equation.

In basic mathematics, multiplication, division, addition, and subtraction are the operations. We know that multiplication and division have higher precedence than addition and subtraction. And so, if we want to modify that behavior, we use brackets.

For example, in this equation, multiplication is done before addition:

 
1
1 + 2 * 3 = 7      

In this equation that contains brackets, multiplication is after addition:

 
1
(1 + 2) * 3 = 9

The same concepts apply to LEQL:

• AND operators are evaluated first
• OR operators are evaluated second

And, if you want to modify that behavior, you use brackets.

Example

You have a log entry that looks like this:

 
1
Given logline:
2
{"city": "London", "action": "logout" }

These two queries are functionally the same. They search for logins in both London and Dublin:

 
1
where(city = London OR city = Dublin AND action = login)   == True
2
where(city = London OR (city = Dublin AND action = login)) == True

But, if you're trying to query for logins in either London or Dublin, you would type:

 
1
where((city = London OR city = Dublin) AND action = login) == False

By adding a set of brackets around the key-value pairs of both cities, you can override the operator precedence of the AND and OR operators.

Keyword Search

Keyword search will work on all logs regardless of their format. Keyword searches are case sensitive by default and will match a full string until it is delimited by a non-letter character. For example:

 
1
Apr 13 20:01:01 hostname run-parts(/etc/cron.hourly)[26263]: starting 0anacron
2
Apr 13 20:01:01 hostname run-parts(/etc/cron.hourly)[26272]: finished 0anacron

InsightIDR will match the events by searching for “etc” or “run” because the text is delimited by whitespace and non-letter characters. InsightIDR would not match “hour” but will match “hourly."

Keyword search can be combined with logical operators. For example, “starting AND finished” would return both log events.

Note: When you list a series of keywords, InsightIDR automatically assumes an "AND" between each keyword. If you want to match an exact string, place “Double Quotes” around the search.

Regular Expression Keyword Search

Regular expressions can greatly enhance the power of your keyword searching. Regular expressions must be wrapped with two forward slashes (“/”). The two most common cases for regex search in InsightIDR are below.

Partial Matching

By default, regex search will return partial matches. The search below will match “complete,” “completely,” and “completed.”

 
1
where(/complete/)

Case Insensitive Search and Comparison

To perform a case-insensitive keyword search, you can use this query format.

 
1
where(/error/i)

This query will match instances of the word 'error', such as, Error, ERROR, error, and any other variation of case.

You can query keys that contain a specific value, regardless of case, by using the NOCASE keyword:

 
1
where(key = NOCASE(value))

You can also query a key that contains a value anywhere in the string, regardless of case, by using ICONTAINS:

 
1
where(key ICONTAINS value)

To perform a case-insensitive key comparison, you can use this query format:

 
1
where(key1==key2, loose)

For example, you might have a log line that contains irregular capitalization of values like this:

 
1
key1=VALUE0 key2=vaLue

The case-insensitive key comparison query finds results where the value of key1 fully contains the value of key2 at any position, regardless of the case of the characters.

Regular Expression Operators

Regular expressions use special characters to enable searching for more advanced patterns. These characters are *, +, .,\, [,], (, ), {, }.If you need to use special characters as ordinary characters, you will need to escape them with a backward slash (\).

Regular Expression Field Extraction

Regex grouping and naming allows you to identify values in your log events and give these values a name, similar to having a key value pair in your log events. You can then use this named capture group to perform more complex search functions.

For example, what if you wanted to extract the IP address from the following raw log data?

 
1
<11>Mar 14 09:24:58 _hostname_ SSH: No User. Possible reasons: Invalid username, invalid license, error while accessing user database <SessionID=33711845, Listener=10.224.9.243:22, Client=13.91.103.73:1984, User=elasticsearch>

You can use this query to find the IP as a source_address: where(/Client=(?P<source_address>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/) groupby(source_address) calculate(count).

Benefits

Regular Expression Field Extraction gives you the ability to identify key pieces of information in your logs which are not in a Key Value Format such that search functions can be applied to the values in your logs. By assigning a name to the identified value(s), these values can be used with our advanced search functions such as GroupBy() or for calculating values such as counts, sums, averages or unique instance counts. They can also be used for comparisons when creating alerts. This means you can create a type of Key Value Pairing out of non-Key Value Pair log formats.

• Uses standard RE2 regex syntax for named capture groups.
• Is not dependent on any log type or structure.
• Removes requirement for data to be in KVP formats.
• Can be used in queries and saved for creating dashboard items.

A regex named capture group is declared by using the following syntax in your expression:

 
1
(?P<name>regExp)

The result returned from the query to the right of the ‘>’ will be assigned the name enclosed in the ‘< >’.

Consider the sample log events below that contain a specific value such as ‘total sale’:

 
1
12:12:14 new sales event – customer Tim – total sale 24.45 – item blanket
2
12:12:15 new sales event – customer Tim – total sale 100.45 – item jacket
3
12:12:16 new sales event – customer Tim – total sale 1000.33 – item computer

The Regular expression to find the value following ‘total sale’ and assign this value to a named variable called ‘saleValue’ is:

 
1
where(/total sale (?P<saleValue>\d*)/)

Once you have captured the key you can then perform the full range of LEQL functions against it. For example:

 
1
where(/total sale (?P<saleValue>\d*)/) calculate(average:saleValue)

In the example, ‘saleValue’ is the name that the digits that follow ‘total sale’ will be assigned because of the regular expression. Once the digits are assigned to saleValue, then an Average calculation can be applied to these numbers.

Regular expression field extraction is extremely useful in this scenario because the value is not in a Key Value Format (KVP), making it hard to tell most systems what value to use. By using regex named capture group syntax, it is now easy to identify the value and assign it a name. This name is then used as part of the search query. It is also possible to save the query and use it for creating a dashboard item.

Advanced Capabilities

To learn more about our advanced regular expression search capabilities, please read the Regular Expression Log Search Documentation.

Key Value Pair and JSON Search

InsightIDR will automatically parse log events that are in a KVP or JSON format for easy use of advanced analytics. The KVP and JSON parsing documentation details the specific formats the system will parse. If your logs are not in a standard KVP or JSON format you can utilize the regular expression field extraction to gain access to the same search capabilities.

To understand how to write KVP and JSON searches, let's look at the example. Given the log events below

Search for all log events with a response time over 25 to return the first two log events, such as:

 
1
where(response_time>25)

You can then add the logical operator OR to include events from containerID 14 to return all three log events:

 
1
where(response_time>25 OR containerID=14)

NOTE: See more about numerical values and notations in the Numbers chapter.

Regular Expression Field Extraction

If your logs do not contain any KVPs, you can designate a KVP relationship for a given string using Regular Expression Field Extraction. This will give you access to all of the advanced search, analytic, and visualization capabilities available for KVP and JSON log format.

Key Expressions

The left-hand side of the key-value predicate, e.g. the key, can be specified in various ways.

Literal key, key = value, returns log events where the field with name key matches the search value.

Nested Key, obj.field1 = value will return events with a nested structure from the example log event {“obj":{"fld1":{"fld2":"val"}}. To check if a field in an event contains a specified nested key, for example, field1, use query obj.field1/.

Wildcard key, key.* = value, returns log events where a nested field of the object or an item of nested array matches the search value. This search retrieves the log events like {"key": {"field1": "value", "field2": "garbage"}} and {"key": ["value", "garbage"]}. Same results can be achieved using, correspondingly, key.field1 = value OR key.field2 = value and key.0 = value OR key.1 = value queries. Wildcard keys not only simplify search queries, but also allow finding log events with unknown field names and arbitrary order of array items. To search for log events with more complicated structure it's possible to specify more than one wildcard symbol, for example, key.*.field1.* = value.

Note, this type of keys only work with JSON log events for now.

Any key, key1, key2 = value, returns log events where any of the keys match search value. Two or more keys can be specified, each key can be a literal or wildcard key. Using this key expression is a shortcut for equivalent query key1 = value OR key2 = value.

All keys, ALL(key1, key2) = value, returns log events where all keys match search value. Two or more keys can be specified, each key can be a literal or wildcard key. Using this key expression is a shortcut for equivalent query key1 = value AND key2 = value.

Note, that key-value searches support various operators, such as =, !=, IN, CONTAINS etc.

IP Search

InsightIDR supports classless inter-domain routing (CIDR) notation, which allows you to search for a range of IP addresses on your network without using complicated regular expressions. This means you can easily view the most active servers, users, and applications on your network.

Things to know about CIDR notation in InsightIDR

• You can use this capability to search flow data generated by the Insight Network Sensor and any log data that contains IPv4 Addresses.
• This requires a key=value search. IP() on its own does not work.
• Allowed subnet values are /1 to /32.

In Log Search, enter a query with the following format:

• Simple Search destination_address = IP(192.168.0.0/24)
• Advanced Search where(destination_address = IP(192.168.0.0/24))

where

• destination_address is the field in the log data you want to filter by
• 192.168.0.0 is the IP address to use as the comparison
• /24 is the block of addresses you want to search

The previous query would return any addresses in the range 192.168.0.1 to 192.168.0.254.

You can adjust the network range of your query by updating the subnet value. For example, replacing /24 with /16 would return any addresses in the range 192.168.0.1 to 192.168.255.254.

Analytic Functions and Visualizations

With our powerful LEQL functions, you are able to produce queries that will easily visualize your data without any preprocessing required.

Count

Log search also supports returning a count of matched search results. Append calculate(COUNT) to your search query or press the calculate to get the number of search results. An example can be seen below.

 
1
where(status=500) calculate(COUNT)

Sum

You can use the Sum function to total the values of your name value pairs. If you had a KVP for sale_value and wanted to know the total sales for a specified time period you would use the following query:

 
1
where(sale_total>0) calculate(SUM:sale_total).

Average

The Average modifier is similar to the sum modifier, but it computes the mean of the values matching the search criteria. For instance, to get the average value of your sales, you might invoke a search like:

 
1
 where(sale_total>0) calculate(AVERAGE:sale_total)

Count Unique

The Count Unique keyword returns an approximation of the number of unique values for a given key. It takes one parameter: the name of the key. For example, if you have the KVP userID in your log file and want to find the number of unique users, you should use the following query:

 
1
 where(userID) calculate(UNIQUE:userID)

Min

The Min function will return the minimum value of the key for each time period. For example, the query below will return the shortest response time for each time period:

 
1
where(status=200) calculate(MIN:responseTime)

Max

The Max function will return the maximum value of the key for each time period. For example, the query below will return the longest response time for each time period:

 
1
where(status=200) calculate(Max:responseTime)

Groupby

Read more about the groupby function and how to use it.

Timeslice

By default, InsightIDR calculates 10 equal time intervals when performing a count, min, max or average query. However, you can use the timeslice function to manually set the number of time intervals.

The valid input for timeslice is a number between 1 and 200 (inclusive). For example, the query below could be used against a 1-hour search period to return the number of 404 errors that occurred per minute.

 
1
where(status=404) calculate(count) timeslice(60)

You can also specify units of time: seconds (s), minutes (m), hours (h) and days (d). For example, the query below will return the number of 404 errors in a 30-minute period.

 
1
where(status=404) calculate(count) timeslice(30m)

It is important to note that data points on the result graph are evenly distributed with the best attempt to satisfy the requirements of time slice values. InsightIDR uses the following methods of calculation:

• The number of data points is calculated as ceil(timerange / time slice), where time range = to - from.
• The granularity of the dots is calculated as timerange / number of data points.
• The value for the first data point is calculated on the time range [from ... from + (granularity * 1)].
• The value for the second data point is calculated on the time range [ from + (granularity * 1) ... from + (granularity * 2)].

Example Let's imagine that we are searching for logs in a time frame of 20 minutes from 13:00 to 13:20 and want to apply a timeslice interval of 5 minutes - timeslice(5m). The resulting graph will contain 4 points with values shown at 13:00, 13:05, 13:10 and 13:15.

If we want to change the interval to timeslice(9m), the resulting graph will have 3 data points, because ceil(20 minutes / 9 minutes) = 3 data points. The granularity works out as 20 minutes / 3 data points = 6 minutes 40 seconds. The 3 data points will appear at positions around 13:00, 13:06 and 13:13. And the values will be calculated on the time frames of 13:00 - 13:06:40, 13:06:40 - 13:13:20, and 13:13:20 - 13:20:00.

Percentile

The Percentile option now allows you to exclude outliers from your search functions. In Simple mode, you can select either a 95th or 99th percentile search function based on a key that contains a number.

If you are using Advanced mode, you can specify your own percentile value by using percentile(N):key or pctl(N):key in your calculate function. For example, calculate(pctl(95):total).

Bytes

The Bytes option lets you calculate the size of your logs in byte form. This is useful for users who wish to verify the size of the logs that they have sent to their account. A simple query that would calculate the size of the given log would be where(/.*/) calculate(bytes)

Standard Deviation

The Standard Deviation option lets you calculate the standard deviation of a given series values. This is useful when trying to establish what values would be considered within normal variance to a given mean. An example use case for Standard Deviation is response times.

 
1
calculate(standarddeviation:service)

You can also use the keyword sd as a shortcut. For example: calculate(sd:service)