Transform Logs to Universal Event Format

If you want to use InsightIDR's Rapid7 Universal Event Sources, you'll need to convert your logs into the exact UEF contract.

To convert your logs into the accepted format:

1. Prepare for log collection using a log manipulation tool.
2. Pull in raw logs by editing conf files.
3. Transform your logs into UEF.
4. Send to InsightIDR as a Universal Event Source.
5. Verify Log Parsing.

Prepare for Log Collection

This guide will be using NXLog as the log manipulation tool. Read detailed information and documentation about NXLog here: https://nxlog.co/documentation

To prepare for log collection, take the following steps:

1. Open services.msc. Stop the service before testing. Restart the service for each test.
2. Install NXLog in the default location on the same system from which you want to collect logs.
   • For NXLog, this location is C:\Program Files (x86)\nxlog\
3. This folder contains a subfolder called conf. Make a copy of the original nxlog.conf file for editing purposes and keep the original as a backup.
4. Create a diagnostic log for testing. Call it nxlog.log and write to a folder, or its typical location: C:\Program Files (x86)\nxlog\data
5. Periodically check the nxlog.log file for errors that may potentially accumulate while you are testing.
6. Open nxlog.conf file in your preferred text editor.

Edit the NXLog Conf File

Specific that logs will come in, convert to the Rapid7 Universal Event Format, and send to InsightIDR Collector.

The NXLog configuration file has three main parts: global directives, Input and Output blocks, and a Route block.

• The global directives section defines what NXLog can do.
• The Input block is used to read in the source logs and the Output block is used to define how the logs will be forwarded.
• The Route block tells NXLog the order in which to process the Input and Output blocks.

The default shell for NXLog looks like this:

Default Shell for NXLog
1
## This is a sample configuration file. See the nxlog reference manual about the
2
## configuration options. It should be installed locally and is also available
3
## online at http://nxlog.org/docs/
4

5
## Please set the ROOT to the folder your nxlog was installed into,
6
## otherwise it will not start.
7

8
#define ROOT C:\Program Files\nxlog
9
define ROOT C:\Program Files (x86)\nxlog
10

11
Moduledir %ROOT%\modules
12
CacheDir %ROOT%\data
13
Pidfile %ROOT%\data\nxlog.pid
14
SpoolDir %ROOT%\data
15
LogFile %ROOT%\data\nxlog.log
16

17
<Extension _json>
18
    Module  xm_json
19
</Extension>
20

21
#Add in required extensions here
22

23
<Input in>
24
#Add in input method here
25
 </Input>
26
 
27
 <Output out>
28
#Add in output method here
29
</Output>
30

31
 <Route 1>
32
	 Path		 in => out
33
</Route>

To edit the conf file to produce the desired UEF:

1. If you haven't already, stop the service.
2. Decide on an input method. You can read about input methods in the [NXlog Guides].
3. Decide on an output method, which are the three ways that InsightIDR can collect data from the Universal Event Source.
   • In this example, NXLog is converting the logs to Universal Ingress Authentication, so the output methods are: syslog, tail file, and watch directory.
4. Define extensions used in the NXLog file, such as xm_syslog and xm_json.
5. Restart the service and test the configuration file.
6. Watch for errors in the diagnostic conf file using the Log Tail tool.
7. Fix any issues before continuing.
8. Stop the service again.
9. If necessary, add an extension into the nxlog.conf to read and reformat the logs. Add this modification to the input method of step 2 and output method of step 3.
10. Remove all the extra fields from the logs that the UEF does not need.
11. Rename fields in the logs to match what the UEF expects.
12. Add in additional fields that the UEF expects.
13. If applicable, convert timestamps to ISO 8601 extended format.

Transform to UEF

Once NXLog has properly converted your logs, convert them to JSON.

1. Add the to_json procedure to the bottom of the input block.
2. Save the file and restart the service.
3. Test the nxlog.conf file to make sure it is the proper JSON object according to Rapid7 Universal Ingress Authentication.

It should look like this:

json
1
{"time":"2018-09-29T01:25:31Z","source_ip":"36.7.19.12","authentication_target":"CracklingBean","account":"jfrost","authentication_result":"SUCCESS","version":"v1","event_type":"INGRESS_AUTHENTICATION"}
2
{"time":"2018-09-29T01:26:31Z","source_ip":"71.64.8.111","authentication_target":"CracklingBean","account":"mbolo","authentication_result":"FAILURE","version":"v1","event_type":"INGRESS_AUTHENTICATION"}

How to Configure This Event Source

Take the following steps to configure the event source:

1. From your dashboard, select Data Collection on the left hand menu.
2. When the "Data Collection" page appears, click the Setup Event Source dropdown and choose Add Event Source.
3. From the “Security Data” section, click the icon for the UES you are converting. The “Add Event Source” panel appears.
4. Choose your collector and event source. You can also name your event source if you want.
5. Choose the collection method you specified in your output method.
6. Enter the required information for that collection methods.
   • When using syslog, optionally choose to encrypt by choosing TCP and downloading the Rapid7 Certificate.
7. Click Save.

It may take a few minutes for logs to begin appearing. To check, go to Data Collection > Event Sources tab. Find the Universal Event Source and click View Raw Log in the bottom right.

Verify Parsing

After InsightIDR successfully ingests the transformed logs, you must verify that they are parsing. Parsed logs will appear in Log search.

For Universal Ingress Authentication, logs will also appear in the ingress map.