Universal Event Sources
InsightIDR can universally support selected data types from any product’s logs, so long as you convert the log output from your product to JSON that matches the Universal Event Format (UEF) contract.
This allows you to send data from otherwise non-InsightIDR event sources into InsightIDR and still experience the same functionality as InsightIDR event sources, such as legacy detection rules. All logs that meet the UEF contract will also appear in log search.
Need help transforming your logs?
Read instructions on transforming your logs in this Rapid7 blog post or on the Transform Logs to UEF help page.
Required Format
Field | Required? | Validation | Description |
---|---|---|---|
version | Yes | Must be a documented version string for the Universal Event event_type. | Each event_type is versioned. This version reflects the version of the specific event_type for a Universal Event. |
event_type | Yes | Any documented supported value. | The event type of this Universal Event. |
time | Yes | Must be a valid ISO 8601 extended timestamp with millisecond precision, such as the following: yyyy-MM-ddTHH:mm:SS.SSSZ | The ISO 8601 extended timestamp. See Time Validation for more information. |
custom_data | No | Must be a JSON object. | Use this field to send any additional information. This data will be available for log search and LEQL queries. |
Please refer to the documentation for each event_type to see additional required and optional fields that apply.
Timestamps
Timestamps with fewer than 3 significant digits in the sub-second portion will be augmented to 3 significant digits. For example, 2018-06-07T12:34:56.1Z
will be interpreted as 2018-06-07T12:34:56.100Z
.
Additionally, 2018-06-07T12:34:56Z
(seconds only precision) will be interpreted as 2018-06-07T12:34:56.000Z
.
Timestamps with greater than 3 significant digits in the sub-second portion will be truncated to 3 significant digits. For example, 2018-06-07T12:34:56.123456Z
(sub-millisecond precision) will be interpreted as 2018-06-07T12:34:56.123Z
.
Expected Format
You must send events to the InsightIDR collector in UTF-8 format, with each log entry representing a single event and a newline delimiting each event.
For example, a DHCP event would be represented as: {"event_type":"DHCP_LEASE","version": “v1”,"time": "2018-06-08T18:18:18.123Z","client_hostname":"pc.acme.com","client_ip":"10.6.102.53","operation": "OBTAIN"}
The same event in expanded JSON format would be:
1{2"version": “v1”,3"event_type": "DHCP_LEASE",4"time": "2018-06-07T18:18:31.000123+0100",56// event type-specific fields here7“client_hostname”: ”pc.acme.com”,8“client_ip”: ”10.6.102.53”,9“operation”: “OBTAIN”10}
Log Formatting Requirements
InsightIDR will only process UEF events that meet the exact format stipulated for a given event_type
and version
. If the logs do not contain the required information or violate the contract of the UEF, InsightIDR will not parse the logs and they will not be processed by the InsightIDR analytics engine or appear in log search.
Possible Violations of UEF
Possible violations of the UEF contract include:
- Malformed JSON objects
- Log entries that contain more than one or only part of a UEF event
- UEF events with unrecognized child fields on the root member object
- UEF events without all of the required fields
- UEF events containing an invalid value for an optional or required field
- For example: If a field is child of the root member object and the contract mandates the value be an IPv4 address, but the value for that field is not a valid IPv4 address
Supported Event Types
InsightIDR supports the following universal event sources: