NXLog

NXLog is a tool that converts files to syslog, which can be useful when an application produces log output that is not accepted by InsightIDR.

Install and Configure NXLog

To start using NXLog in your environment:

  1. Download the latest version of NXLog here: http://nxlog.co/products/nxlog-community-edition/download
  2. Install NXLog locally and set the ROOT to the folder in which your NXLog was installed, otherwise NXLog will not start. The path should be similar to this: C:\Program Files (x86)\nxlog\conf\nxlog.conf
  3. Once installed, stop the NXLog service.
  4. Open the NXLog configuration file and paste the following into to the file, adjusting for your account as necessary:
Sample Configuration File
1
## This is a sample configuration file. See the nxlog reference manual about the
2
## configuration options. It should be installed locally and is also available
3
## online at http://nxlog.org/docs/
4
5
## Please set the ROOT to the folder your nxlog was installed into,
6
## otherwise it will not start.
7
8
#define ROOT C:\Program Files\nxlog
9
define ROOT C:\Program Files (x86)\nxlog
10
11
Moduledir %ROOT%\modules
12
CacheDir %ROOT%\data
13
Pidfile %ROOT%\data\nxlog.pid
14
SpoolDir %ROOT%\data
15
LogFile %ROOT%\data\nxlog.log
16
17
<Extension _syslog>
18
Module xm_syslog
19
</Extension>
20
21
<Input in>
22
Module im_msvistalog
23
ReadFromLast True
24
# For windows 2003 and earlier use the following:
25
# Module im_mseventlog
26
27
Query <QueryList>\
28
<Query Id="0" Path="Security">\
29
<Select Path="Security">*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0) and (EventID=4769 or EventID=4728 or EventID=4732 or EventID=4756 or EventID=4720 or EventID=4722 or EventID=4725 or EventID=4740 or EventID=4767 or EventID=4624 or EventID=4625 or EventID=4704 or EventID=4648 or EventID=4724 or EventID=4741 or EventID=4738 or EventID=4768 or EventID=4723)]]</Select>\
30
</Query>\
31
</QueryList>
32
33
</Input>
34
35
<Output out>
36
Module om_tcp
37
Host COLLECTOR-HOST-NAME
38
Port COLLECTOR-PORT
39
Exec to_syslog_snare();
40
</Output>
41
42
<Route 1>
43
Path in => out
44
</Route>

For additional configuration options, see the NXLog reference manual here: http://nxlog.org/docs/

  1. Restart the NXLog service.

You can now use the new NXLog configuration.

Collect Windows Server Logs

If you need to collect security log events from your environment, but do not want to use a Domain Admin account, you can configure NXLog to collect events for you.

Use one of the following configuration files to collect security logs:

You can look at other examples and also configure the NXLog files to send encrypted messages.

Active Directory

You can use NXLog to collect domain controller security logs for Active Directory.

To use NXLog to collect the security log:

  1. Install NXLog on the domain controller.
  2. Create a new nxlog.conf file to replace the default file. The default location of this file is C:\Program Files (x86)\nxlog\conf.
  3. To collect and forward domain controller security logs to your InsightIDR Collector, use the following configuration file as an example:
text
1
#This sample nxlog.conf file can be used to read the Microsoft Security Log and send it to InsightIDR using syslog.
2
#Send the logs to InsightIDR in syslog_snare format.
3
4
#define ROOT C:\Program Files\nxlog
5
define ROOT C:\Program Files (x86)\nxlog
6
7
Moduledir %ROOT%\modules
8
CacheDir %ROOT%\data
9
Pidfile %ROOT%\data\nxlog.pid
10
SpoolDir %ROOT%\data
11
LogFile %ROOT%\data\nxlog.log
12
13
<Extension _syslog>
14
Module xm_syslog
15
</Extension>
16
17
<Input in_securitylog>
18
Module im_msvistalog
19
Query <QueryList> \
20
<Query Id="0"> \
21
<Select Path="Security">*</Select> \
22
</Query> \
23
</QueryList>
24
<Exec>
25
$Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");
26
$raw_event = $Message;
27
to_syslog_snare();
28
</Exec>
29
</Input>
30
31
<Output out_securitylog>
32
Module om_udp
33
#This is the IP address of the InsightIDR Collector
34
Host 192.168.0.46
35
#This is the port configured on the Event Source
36
Port 10341
37
</Output>
38
39
<Route 1>
40
Path in_securitylog=> out_securitylog
41
</Route>

Microsoft DHCP

You can use NXLog to collect DHCP logs for Microsoft DHCP.

To use NXLog to collect the Microsoft DHCP log:

  1. Install NXLog on the DHCP server.
  2. Create a new nxlog.conf file to replace the default file.The default location of this file is C:\Program Files (x86)\nxlog\conf.
  3. To collect and forward DHCP logs to your InsightIDR collector, use the following configuration file as an example:
text
1
#This sample nxlog.conf file can be used to read Microsoft DHCP log files and send them to InsightIDR using syslog.
2
#Send the logs to InsightIDR in syslog_bsd format.
3
4
#define ROOT C:\Program Files\nxlog
5
define ROOT C:\Program Files (x86)\nxlog
6
7
Moduledir %ROOT%\modules
8
CacheDir %ROOT%\data
9
Pidfile %ROOT%\data\nxlog.pid
10
SpoolDir %ROOT%\data
11
LogFile %ROOT%\data\nxlog.log
12
13
<Extension _syslog>
14
Module xm_syslog
15
</Extension>
16
17
<Input in_dhcp>
18
Module im_file
19
#Change the location below to where the DHCP logs are.
20
File "C:\\Windows\\Sysnative\\dhcp\\DhcpSrvLog-*.log"
21
InputType LineBased
22
SavePos TRUE
23
<Exec>
24
if $raw_event =~ /^#/ drop();
25
else
26
{
27
to_syslog_bsd();
28
}
29
</Exec>
30
</Input>
31
32
<Output out>
33
#If you wish to send the logs using tcp instead of udp, change the module below to om_tcp.
34
Module om_udp
35
#This is the IP address of the InsightIDR Collector
36
Host 192.168.0.156
37
#This is the port configured on the Event Source
38
Port 10400
39
</Output>
40
41
<Route 1>
42
Path in_dhcp=> out
43
</Route>

Microsoft DNS

You can use NXLog to collect DNS logs for Microsoft DNS.

To use NXLog to collect the DNS log:

  1. Install NXLog on the DNS server.
  2. Create a new nxlog.conf file to replace the default file.The default location of this file is C:\Program Files (x86)\nxlog\conf.
  3. Enable the DNS file rollover. See the DNS page for instructions on how to enable rollover.
  4. To collect and forward DNS logs to your InsightIDR collector, use the following configuration file as an example:
text
1
#This sample nxlog.conf file can be used to read Microsoft DNS log files and send them to InsightIDR using encrypted syslog.
2
#Send the logs to InsightIDR in syslog_bsd format.#define ROOT C:\Program Files\nxlog
3
4
define ROOT C:\Program Files (x86)\nxlog
5
6
Moduledir %ROOT%\modules
7
CacheDir %ROOT%\data
8
Pidfile %ROOT%\data\nxlog.pid
9
SpoolDir %ROOT%\data
10
LogFile %ROOT%\data\nxlog.log
11
12
<Extension _syslog>
13
Module xm_syslog
14
</Extension>
15
16
<Input in_dns>
17
Module im_file
18
File 'c:\dnslogs\dns*.log'
19
InputType LineBased
20
SavePos TRUE
21
<Exec>
22
if $raw_event =~ /^#/ drop();
23
else
24
{
25
to_syslog_bsd();
26
}
27
</Exec>
28
</Input>
29
30
<Output out_dns>
31
Module om_udp
32
#This is the IP address of the InsightIDR Collector
33
Host 192.168.0.46
34
#This is the port configured on the Event Source
35
Port 10350
36
</Output>
37
38
<Route 1>
39
Path in_dns=>out_dns
40
</Route>

Microsoft Exchange Message Tracking Logs

You can use NXLog to collect Microsoft Exchange Message Tracking logs.

To use NXLog to collect the Microsoft Exchange Message Tracking logs:

  1. Install NXLog on the Exchange server.
  2. Create a new nxlog.conf file to replace the default file.The default location of this file is C:\Program Files (x86)\nxlog\conf.
  3. Use the following configuration file as an example to collect and forward the Message Tracking logs to your InsightIDR collector:
text
1
#This sample nxlog.conf file can be used to collect the Exchange Message Tracking Log and send it to InsightIDR.
2
#define ROOT C:\Program Files\nxlog
3
define ROOT C:\Program Files (x86)\nxlog
4
5
Moduledir %ROOT%\modules
6
CacheDir %ROOT%\data
7
Pidfile %ROOT%\data\nxlog.pid
8
SpoolDir %ROOT%\data
9
LogFile %ROOT%\data\nxlog.log
10
define CertDir C:\Program Files (x86)\nxlog\cert
11
12
#Define the modules that will be used by nxlog.
13
14
<Extension json>
15
Module xm_json
16
</Extension>
17
18
<Extension _syslog>
19
Module xm_syslog
20
</Extension>
21
22
<Extension MessageTrackingLog>
23
Module xm_csv
24
Fields $date-time,$client-ip,$client-hostname,$server-ip,$server-hostname,$source-context,$connector-id,$source,$event-id,$internal-message-id,$message-id,$network-message-id,$recipient-address,$recipient-status,$total-bytes,$recipient-count,$related-recipient-address,$reference,$message-subject,$sender-address,$return-path,$message-info,$directionality,$tenant-id,$original-client-ip,$original-server-ip,$custom-data,$transport-traffic-type,$log-id,$schema-version
25
#FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string
26
Delimiter ,
27
QuoteMethod None
28
</Extension>
29
30
<Input in_MessageTrackingLog>
31
Module im_file
32
File 'c:\logs\logfile*.log'
33
InputType LineBased
34
SavePos TRUE
35
PollInterval 1
36
<Exec>
37
if $raw_event =~ /^#/ drop();
38
else
39
{
40
MessageTrackingLog->parse_csv();
41
$EventTime = parsedate($date + " " + $time);
42
$SourceName = "MessageTrackingLog";
43
$raw_event = to_json();
44
}
45
</Exec>
46
</Input>
47
48
<Output out_MesssageTrackingLog>
49
Module om_udp
50
#This is the IP address of the InsightIDR Collector
51
Host 192.168.0.156
52
#This is the port configured on the Universal Event Source
53
Port 10060
54
</Output>
55
56
<Route 1>
57
Path in_MessageTrackingLog=>out_MesssageTrackingLog
58
</Route>

After you create this replacement file, configure a Custom Log event source in InsightIDR.

Microsoft IIS

You can use NXLog to collect Microsoft IIS logs.

To use NXLog to collect IIS logs:

  1. Install NXLog on the IIS server.
  2. Create a new nxlog.conf file to replace the default file.The default location of this file is C:\Program Files (x86)\nxlog\conf.
  3. Use the following configuration file to collect and forward the IIS logs to your InsightIDR collector:
text
1
## This is a sample configuration file. See the nxlog reference manual about the
2
## configuration options. It should be installed locally and is also available
3
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html
4
## Please set the ROOT to the folder your nxlog was installed into,
5
## otherwise it will not start.
6
#define ROOT C:\Program Files\nxlog
7
define ROOT C:\Program Files (x86)\nxlog
8
Moduledir %ROOT%\modules
9
CacheDir %ROOT%\data
10
Pidfile %ROOT%\data\nxlog.pid
11
SpoolDir %ROOT%\data
12
LogFile %ROOT%\data\nxlog.log
13
# Include fileop when rotating logs or while debugging
14
#<Extension fileop>
15
#Module xm_fileop
16
#</Extension>
17
<Extension json>
18
Module xm_json
19
</Extension>
20
# Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
21
<Extension w3c>
22
Module xm_csv
23
Fields $date, $time, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, #$c_ip, $cs_User_Agent, $cs_Referer, $sc_status, $sc_substatus, $sc_win32_status, $time_taken
24
FieldTypes string, string, string, string, string, string, integer, string, string, string, #string, integer, integer, integer, integer
25
Delimiter ' '
26
</Extension>
27
# Convert the IIS logs to JSON and use the original event time
28
<Input IIS_Site1>
29
Module im_file
30
File "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex*"
31
SavePos TRUE
32
33
Exec if $raw_event =~ /^#/ drop(); \
34
else \
35
{ \
36
w3c->parse_csv(); \
37
$EventTime = parsedate($date + " " + $time); \
38
$SourceName = "IIS"; \
39
$raw_event = to_json(); \
40
}
41
</Input>
42
<Output IIS_Site1_out>
43
Module om_udp
44
Host <IDR_COLLECTOR_IP/DNS>
45
Port <IDR_EVENT_PORT>
46
#Please note that if you do not see the IIS logs in Log Search, you will need to uncomment the line below.
47
#The Generic Syslog event source requires that the logs have a syslog header, as specified by RFC3164. This will add the required RFC3164 syslog header to the logs.
48
#Exec to_syslog_bsd();
49
</Output>
50
<Route IIS_Site1>
51
Path IIS_Site1 => IIS_Site1_out
52
</Route>

After you create this replacement file, configure a Generic Syslog event source in InsightIDR, which parses the IIS logs into a searchable format for Log Search.

Send Encrypted Logs

If you prefer to use encrypted syslog to transport logs to InsightIDR, you can use om_ssl in your output module. For example:

text
1
#Download the certificate from InsightIDR and place it in the folder you specify below.
2
<Output out_dns>
3
Module om_ssl
4
#This is the IP address of the InsightIDR Collector
5
Host 10.10.10.10
6
#This is the port configured on the Event Source
7
Port 10001
8
#specify cert location, the default location is nxlog\cert.
9
CAFile %CertDir%\Rapid7CA.pem
10
</Output>

Other Examples

It is possible to have multiple “in” and “out” sections in the nxlog.conf file. For example, if you want to collect the security log, DHCP logs, and DNS logs from a single server, use the following nxlog.conf file:

text
1
#This sample nxlog.conf file can be used to read the Windows Security log, DHCP, and DNS logs from a domain controller.
2
#define ROOT C:\Program Files\nxlog
3
define ROOT C:\Program Files (x86)\nxlog
4
5
Moduledir %ROOT%\modules
6
CacheDir %ROOT%\data
7
Pidfile %ROOT%\data\nxlog.pid
8
SpoolDir %ROOT%\data
9
LogFile %ROOT%\data\nxlog.log
10
11
<Extension _syslog>
12
Module xm_syslog
13
</Extension>
14
15
#Collect the Security log.
16
<Input in_securitylog>
17
Module im_msvistalog
18
Query <QueryList> \
19
<Query Id="0"> \
20
<Select Path="Security">*</Select> \
21
</Query> \
22
</QueryList>
23
<Exec>
24
$Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");
25
$raw_event = $Message;
26
to_syslog_snare();
27
</Exec>
28
</Input>
29
30
#Collect the Microsoft DHCP logs.
31
<Input in_dhcp>
32
Module im_file
33
#Change the location below to where the DHCP logs are.
34
File "C:\\Windows\\Sysnative\\dhcp\\DhcpSrvLog-*.log"
35
InputType LineBased
36
SavePos TRUE
37
<Exec>
38
if $raw_event =~ /^#/ drop();
39
else
40
{
41
to_syslog_bsd();
42
}
43
</Exec>
44
</Input>
45
46
#Collect the Microsoft DNS log (s).
47
<Input in_dns>
48
Module im_file
49
File 'c:\dnslogs\dns*.log'
50
InputType LineBased
51
SavePos TRUE
52
<Exec>
53
if $raw_event =~ /^#/ drop();
54
else
55
{
56
to_syslog_bsd();
57
}
58
</Exec>
59
</Input>
60
61
<Output out_securitylog>
62
Module om_udp
63
#This is the IP address of the InsightIDR Collector
64
Host 192.168.0.46
65
#This is the port configured on the Event Source
66
Port 10001
67
# Exec to_json(); $message = $raw_event;
68
</Output>
69
70
<Output out_dhcp>
71
#If you wish to send the logs using tcp instead of udp, change the module below to om_tcp.
72
Module om_udp
73
#This is the IP address of the InsightIDR Collector
74
Host 192.168.0.46
75
#This is the port configured on the Event Source
76
Port 10101
77
</Output>
78
79
<Output out_dns>
80
Module om_udp
81
#This is the IP address of the InsightIDR Collector
82
Host 192.168.0.46
83
#This is the port configured on the Event Source
84
Port 10201
85
</Output>
86
87
<Route 1>
88
Path in_securitylog=> out_securitylog
89
</Route>
90
91
<Route 2>
92
Path in_dhcp=> out_dhcp
93
</Route>
94
95
<Route 3>
96
Path in_dns=> out_dns
97
</Route>