JSON & KVP

What are JSON and KVP?

JSON structured logs are regularly produced by software components and are promoted as a logging best practice. JSON formatted logs are easily produced and consumed by a variety of software components and are human readable. If you are not ready to make the move to JSON, using key-value pairs (KVPs) into your logs provides similar benefits. Both formats will allow you to take advantage of our advanced search and analytic capabilities.

Why use Key Value Pair and JSON Search?

InsightOps will automatically parse log events that are in a Key Value Pair (KVP) or JSON format for easy use of advanced analytics. The KVP and JSON parsing document details the specific formats the system will parse. If your logs are not in a standard KVP or JSON format you can utilize the regular expression field extraction to gain access to the same advanced analytics.

To understand which KVP and JSON formats, examine the log events below:

Searches can be easily written to return important log events

Search for all log events with a response time over 25 to return the first two log events, such as:

 
1
where(repsonse_time>25)

You can then add the logical operator OR to include events from containerID 14 to return all three log events:

 
1
where(repsonse_time>25 OR containerID=14)

InsightOps will automatically identify and index fields within your JSON and KVP log events. Keys are automatically highlighted and clickable. Clicking on a field will populate the search bar to allow for quick searching of across your data (Key Highlighting is temporarily disabled). The specification for which log formats will be parsed automatically is detailed in the JSON and KVP Specification Section of this document. 

JSON Parsing Specification

The system will automatically parse JSON formatted log events formatted in compliance with the JSON Specification (IETF RFC 4627). To take advantage of the prettification option, the entry must be a single object ({ … }) or array of JSON objects. JSON objects that appear embedded in unstructured data will still have their keys recognized but you will not be able to expand the JSON hierarchy.

KVP Parsing Specification

InsightOps is able to automatically parse a wide variety of KVP formats in compliance with the following specification:

• A key-value pair consists of an identifier called a KEY, a SEPARATOR, a VALUE, and a prefix and suffix termed the KVDELIMITER.

KVDELIMITER KEY SEPARATOR VALUE KVDELIMITER

Term Definitions

KVDELIMITER

• can be space (” “), comma (“,”), tab “\t” or “\r” or “\n” or “\r\n”

KEY

• the allowed list of characters is any Alphanumeric plus “_” “.” “$” “@”.

• must have at least one non-numeric character from the allowed list of characters
• a ” or ‘ either side of a key will result in anything between the quotations marks being interpreted as the KEY, e.g. “key[1]” will give a KEY of key[1]
• can be 0 or 1 spaces (Ascii 0x20 or ” “)

SEPARATOR

• is “=” or “:”
• can be preceded by and/or followed by a space
• in order to appear in a key or value, the separator must be between a pair of quotes or escaped

VALUE

• The value can consist of any alphanumeric character and “-” “_” “.” “$” “@”.
• The value can be empty.

• a ” or ‘ either side of a value will result in anything between the quotations marks being interpreted as the VALUE, e.g. key1=”value description” will give a KEY of key1 and a VALUE of “value description”
• Placing quotes around the VALUE is required for parsing URLs or other items that contain slashes (/)
• a SEPARATOR in a value will be treated such that this value (and the possible key preceding it) will not be treated as a key value pair, e.g. so timestamps not broken up

UNIT

• This is the alphabetical text or “%” which follows the last number in a Numeric only VALUE up to the next space or first non alphabetical character or up to 2 characters (whichever occurs first) – those 2 characters will not be considered as part of value for querying, i.e. 100ms is treated as 100.

CHARACTER ESCAPING

• this is used to allow a KEY or VALUE to contain a character that would not otherwise be allowed, e.g. a space or quote or comma or bracket is not allowed in a KEY or VALUE unless escaped.
• this is supported in both a KEY and a VALUE by adding a backslash (“\”) in front of the character to be escaped or by including the entire KEY or VALUE in quotes (pairs of ” or pairs of ‘). As per the definition of KEY and VALUE, the \ is not an allowed character and only appears to support escaping, e.g a space can be included by either using quotes as in name=”David Tracey” or escaping as in name=David\ Tracey. Regarding quotes being in pairs of ” or pairs of ‘, this means that a ‘ does not finish an escaped sequence started with ” and it is possible for one type of quote to be appear within quotes of the other type, e.g. somekey = “David’s One” will include the ‘ in the value.
• note that the \ is only treated as escaping a character in the process of determining a name or a value

Notes

• "key=value" "key :value" "key = value" "key: value" are all valid formats for key-value pairs. Based on these formats, this definition means a key-value pair must have both a key and a value.
  • The format "key:value" is only supported when you parse or extract log data. When you search your log data, the KVP must be formatted like "key=value".
  • The characters - and + are not allowed in a key unless they are escaped with quotation marks " ".
  • The characters . and _ are allowed in keys and values.
• A hyphen - can be used before any LEQL key or value and before a key-value pair. If the hyphen occurs immediately before a number, without a space between, it indicates a negative value. For example, -5. In all other cases when a hyphen appears before a string, key-value pair, regex, or IP search, it is treated as a a shortcut for LEQL negation. A hyphen can be a part of any value, and does not require escaping.
• The only supported encoding is UTF-8 Encoding.
• JSON data requires keys to contain only numeric characters. To correctly query JSON data, the key must be enclosed in quotation marks. For example, groupby("42").