NXLog

NXLog is a tool that converts files to syslog, which can be useful when an application produces log output that is not accepted by InsightIDR.

Install and Configure NXLog

To start using NXLog in your environment:

  1. Download the latest version of NXLog here: http://nxlog.co/products/nxlog-community-edition/download
  2. Install NXLog locally and set the ROOT to the folder in which your NXLog was installed, otherwise NXLog will not start. The path should be similar to this: C:\Program Files (x86)\nxlog\conf\nxlog.conf
  3. Once installed, stop the NXLog service.
  4. Open the NXLog configuration file and paste the following into to the file, adjusting for your account as necessary:
Sample Configuration File
1
## This is a sample configuration file. See the nxlog reference manual about the
2
## configuration options. It should be installed locally and is also available
3
## online at http://nxlog.org/docs/
4
 
5
## Please set the ROOT to the folder your nxlog was installed into,
6
## otherwise it will not start.
7
 
8
#define ROOT C:\Program Files\nxlog
9
define ROOT C:\Program Files (x86)\nxlog
10
 
11
Moduledir %ROOT%\modules
12
CacheDir %ROOT%\data
13
Pidfile %ROOT%\data\nxlog.pid
14
SpoolDir %ROOT%\data
15
LogFile %ROOT%\data\nxlog.log
16
 
17
<Extension _syslog>
18
    Module      xm_syslog
19
</Extension>
20
 
21
<Input in>
22
    Module      im_msvistalog
23
    ReadFromLast  True
24
# For windows 2003 and earlier use the following:
25
#   Module      im_mseventlog
26
 
27
Query <QueryList>\
28
  <Query Id="0" Path="Security">\
29
     <Select Path="Security">*[System[(Level=1  or Level=2 or Level=3 or Level=4 or Level=0) and (EventID=4769 or EventID=4728 or EventID=4732 or EventID=4756 or EventID=4720 or EventID=4722 or EventID=4725 or EventID=4740 or EventID=4767 or EventID=4624 or EventID=4625 or EventID=4704 or EventID=4648 or EventID=4724 or EventID=4741 or EventID=4738 or EventID=4768 or EventID=4723)]]</Select>\
30
  </Query>\
31
</QueryList>
32
 
33
</Input>
34
 
35
<Output out>
36
    Module      om_tcp
37
    Host        COLLECTOR-HOST-NAME
38
    Port        COLLECTOR-PORT
39
    Exec        to_syslog_snare();
40
</Output>
41
 
42
<Route 1>
43
    Path        in => out
44
</Route>

For additional configuration options, see the NXLog reference manual here: http://nxlog.org/docs/

  1. Restart the NXLog service.

You can now use the new NXLog configuration.

Collect Windows Server Logs

If you need to collect security log events from your environment, but do not want to use a Domain Admin account, you can configure NXLog to collect events for you.

Use one of the following configuration files to collect security logs:

You can look at other examples and also configure the NXLog files to send encrypted messages.

Active Directory

You can use NXLog to collect domain controller security logs for Active Directory.

To use NXLog to collect the security log:

  1. Install NXLog on the domain controller.
  2. Create a new nxlog.conf file to replace the default file. The default location of this file is C:\Program Files (x86)\nxlog\conf.
  3. To collect and forward domain controller security logs to your InsightIDR Collector, use the following configuration file as an example:
text
1
#This sample nxlog.conf file can be used to read the Microsoft Security Log and send it to InsightIDR using syslog.
2
#Send the logs to InsightIDR in syslog_snare format.
3


4
#define ROOT C:\Program Files\nxlog
5
define ROOT C:\Program Files (x86)\nxlog
6


7
Moduledir %ROOT%\modules
8
CacheDir %ROOT%\data
9
Pidfile %ROOT%\data\nxlog.pid
10
SpoolDir %ROOT%\data
11
LogFile %ROOT%\data\nxlog.log
12


13
<Extension _syslog>
14
    Module  xm_syslog
15
</Extension>
16


17
<Input in_securitylog>
18
   Module im_msvistalog
19
   Query <QueryList> \
20
          <Query Id="0"> \
21
           <Select Path="Security">*</Select> \
22
          </Query> \
23
         </QueryList>
24
	<Exec>	 
25
		$Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");
26
		$raw_event = $Message;
27
		to_syslog_snare();
28
	</Exec>
29
</Input>
30


31
<Output out_securitylog>
32
   Module      om_udp
33
   #This is the IP address of the InsightIDR Collector
34
   Host			192.168.0.46
35
   #This is the port configured on the Event Source
36
   Port        10341
37
</Output>
38


39
<Route 1>
40
   Path        in_securitylog=> out_securitylog
41
</Route>

Microsoft DHCP

You can use NXLog to collect DHCP logs for Microsoft DHCP.

To use NXLog to collect the Microsoft DHCP log:

  1. Install NXLog on the DHCP server.
  2. Create a new nxlog.conf file to replace the default file.The default location of this file is C:\Program Files (x86)\nxlog\conf.
  3. To collect and forward DHCP logs to your InsightIDR collector, use the following configuration file as an example:
text
1
#This sample nxlog.conf file can be used to read Microsoft DHCP log files and send them to InsightIDR using syslog.
2
#Send the logs to InsightIDR in syslog_bsd format.
3


4
#define ROOT C:\Program Files\nxlog
5
define ROOT C:\Program Files (x86)\nxlog
6


7
Moduledir %ROOT%\modules
8
CacheDir %ROOT%\data
9
Pidfile %ROOT%\data\nxlog.pid
10
SpoolDir %ROOT%\data
11
LogFile %ROOT%\data\nxlog.log
12


13
<Extension _syslog>
14
    Module  xm_syslog
15
</Extension>
16


17
<Input in_dhcp>
18
 Module  im_file
19
 #Change the location below to where the DHCP logs are.
20
 File "C:\\Windows\\Sysnative\\dhcp\\DhcpSrvLog-*.log"
21
 InputType LineBased
22
 SavePos TRUE
23
 <Exec>
24
	if $raw_event =~ /^#/ drop();
25
	else
26
	{
27
		to_syslog_bsd();
28
	}
29
 </Exec>
30
 </Input>
31
 
32
<Output out>
33
   #If you wish to send the logs using tcp instead of udp, change the module below to om_tcp.
34
   Module      om_udp
35
   #This is the IP address of the InsightIDR Collector
36
   Host			192.168.0.156
37
   #This is the port configured on the Event Source
38
   Port        10400
39
</Output>
40


41
<Route 1>
42
   Path        in_dhcp=> out
43
</Route>

Microsoft DNS

You can use NXLog to collect DNS logs for Microsoft DNS.

To use NXLog to collect the DNS log:

  1. Install NXLog on the DNS server.
  2. Create a new nxlog.conf file to replace the default file.The default location of this file is C:\Program Files (x86)\nxlog\conf.
  3. Enable the DNS file rollover. See the DNS page for instructions on how to enable rollover.
  4. To collect and forward DNS logs to your InsightIDR collector, use the following configuration file as an example:
text
1
#This sample nxlog.conf file can be used to read Microsoft DNS log files and send them to InsightIDR using encrypted syslog.
2
#Send the logs to InsightIDR in syslog_bsd format.#define ROOT C:\Program Files\nxlog
3


4
define ROOT C:\Program Files (x86)\nxlog
5


6
Moduledir %ROOT%\modules
7
CacheDir %ROOT%\data
8
Pidfile %ROOT%\data\nxlog.pid
9
SpoolDir %ROOT%\data
10
LogFile %ROOT%\data\nxlog.log
11


12
<Extension _syslog>
13
    Module  xm_syslog
14
</Extension>
15


16
<Input in_dns>
17
 Module  im_file
18
 File 'c:\dnslogs\dns*.log'
19
 InputType LineBased
20
 SavePos TRUE
21
 <Exec>
22
	if $raw_event =~ /^#/ drop();
23
	else
24
	{
25
		to_syslog_bsd();
26
	}
27
 </Exec>
28
 </Input>
29


30
<Output out_dns>
31
   Module      om_udp
32
   #This is the IP address of the InsightIDR Collector
33
   Host			192.168.0.46
34
   #This is the port configured on the Event Source
35
   Port        10350
36
</Output>
37


38
<Route 1>
39
   Path        in_dns=>out_dns
40
</Route>

Microsoft Exchange Message Tracking Logs

You can use NXLog to collect Microsoft Exchange Message Tracking logs.

To use NXLog to collect the Microsoft Exchange Message Tracking logs:

  1. Install NXLog on the Exchange server.
  2. Create a new nxlog.conf file to replace the default file.The default location of this file is C:\Program Files (x86)\nxlog\conf.
  3. Use the following configuration file as an example to collect and forward the Message Tracking logs to your InsightIDR collector:
text
1
#This sample nxlog.conf file can be used to collect the Exchange Message Tracking Log and send it to InsightIDR.
2
#define ROOT C:\Program Files\nxlog
3
define ROOT C:\Program Files (x86)\nxlog
4


5
Moduledir %ROOT%\modules
6
CacheDir %ROOT%\data
7
Pidfile %ROOT%\data\nxlog.pid
8
SpoolDir %ROOT%\data
9
LogFile %ROOT%\data\nxlog.log
10
define CertDir C:\Program Files (x86)\nxlog\cert
11


12
#Define the modules that will be used by nxlog.
13


14
<Extension json>
15
    Module xm_json
16
</Extension>
17


18
<Extension _syslog>
19
    Module  xm_syslog
20
</Extension>
21


22
<Extension MessageTrackingLog>
23
    Module      xm_csv
24
	Fields 		$date-time,$client-ip,$client-hostname,$server-ip,$server-hostname,$source-context,$connector-id,$source,$event-id,$internal-message-id,$message-id,$network-message-id,$recipient-address,$recipient-status,$total-bytes,$recipient-count,$related-recipient-address,$reference,$message-subject,$sender-address,$return-path,$message-info,$directionality,$tenant-id,$original-client-ip,$original-server-ip,$custom-data,$transport-traffic-type,$log-id,$schema-version
25
    	#FieldTypes  string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string
26
    	Delimiter   ,
27
	QuoteMethod	None
28
</Extension>
29


30
<Input in_MessageTrackingLog>
31
	Module  im_file
32
	File 'c:\logs\logfile*.log'
33
	InputType LineBased
34
	SavePos TRUE
35
	PollInterval 1
36
	<Exec>
37
		if $raw_event =~ /^#/ drop();
38
		else
39
		{
40
			MessageTrackingLog->parse_csv();
41
			$EventTime = parsedate($date + " " + $time);
42
            $SourceName = "MessageTrackingLog";				
43
            $raw_event = to_json();	
44
		}
45
	</Exec>
46
 </Input>
47


48
<Output out_MesssageTrackingLog>
49
    Module      om_udp
50
	#This is the IP address of the InsightIDR Collector
51
    Host        192.168.0.156
52
	#This is the port configured on the Universal Event Source
53
    Port        10060
54
</Output>
55


56
<Route 1>
57
   Path        in_MessageTrackingLog=>out_MesssageTrackingLog
58
</Route>

After you create this replacement file, configure a Custom Log event source in InsightIDR.

Microsoft IIS

You can use NXLog to collect Microsoft IIS logs.

To use NXLog to collect IIS logs:

  1. Install NXLog on the IIS server.
  2. Create a new nxlog.conf file to replace the default file.The default location of this file is C:\Program Files (x86)\nxlog\conf.
  3. Use the following configuration file to collect and forward the IIS logs to your InsightIDR collector:
text
1
## This is a sample configuration file. See the nxlog reference manual about the
2
## configuration options. It should be installed locally and is also available
3
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html
4
## Please set the ROOT to the folder your nxlog was installed into,
5
## otherwise it will not start.
6
#define ROOT C:\Program Files\nxlog
7
define ROOT C:\Program Files (x86)\nxlog
8
Moduledir %ROOT%\modules
9
CacheDir %ROOT%\data
10
Pidfile %ROOT%\data\nxlog.pid
11
SpoolDir %ROOT%\data
12
LogFile %ROOT%\data\nxlog.log
13
# Include fileop when rotating logs or while debugging
14
#<Extension fileop>
15
#Module xm_fileop
16
#</Extension>
17
<Extension json>
18
    Module      xm_json
19
</Extension>
20
# Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
21
<Extension w3c>
22
    Module xm_csv
23
    Fields $date, $time, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, #$c_ip, $cs_User_Agent, $cs_Referer, $sc_status, $sc_substatus, $sc_win32_status, $time_taken
24
    FieldTypes string, string, string, string, string, string, integer, string, string, string, #string, integer, integer, integer, integer
25
    Delimiter ' '
26
</Extension>
27
# Convert the IIS logs to JSON and use the original event time
28
<Input IIS_Site1>
29
    Module    im_file
30
    File    "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex*"
31
    SavePos  TRUE
32
 
33
    Exec if $raw_event =~ /^#/ drop();              \
34
       else                         \
35
       {                            \
36
            w3c->parse_csv();                    \
37
            $EventTime = parsedate($date + " " + $time);  \
38
            $SourceName = "IIS";              \
39
            $raw_event = to_json();             \
40
       }
41
</Input>
42
<Output IIS_Site1_out>
43
   Module      om_udp
44
   Host        <IDR_COLLECTOR_IP/DNS>
45
   Port        <IDR_EVENT_PORT>
46
   #Please note that if you do not see the IIS logs in Log Search, you will need to uncomment the line below.  
47
   #The Generic Syslog event source requires that the logs have a syslog header, as specified by RFC3164.  This will add the required RFC3164 syslog header to the logs.
48
  #Exec         to_syslog_bsd();
49
</Output>
50
<Route IIS_Site1>
51
    Path        IIS_Site1 => IIS_Site1_out
52
</Route>

After you create this replacement file, configure a Generic Syslog event source in InsightIDR, which parses the IIS logs into a searchable format for Log Search.

Send Encrypted Logs

If you prefer to use encrypted syslog to transport logs to InsightIDR, you can use om_ssl in your output module. For example:

text
1
#Download the certificate from InsightIDR and place it in the folder you specify below.
2
<Output out_dns>
3
   Module      om_ssl
4
   #This is the IP address of the InsightIDR Collector
5
   Host	       10.10.10.10
6
   #This is the port configured on the Event Source
7
   Port        10001
8
   #specify cert location, the default location is nxlog\cert.
9
   CAFile      %CertDir%\Rapid7CA.pem
10
</Output>

Other Examples

It is possible to have multiple “in” and “out” sections in the nxlog.conf file. For example, if you want to collect the security log, DHCP logs, and DNS logs from a single server, use the following nxlog.conf file:

text
1
#This sample nxlog.conf file can be used to read the Windows Security log, DHCP, and DNS logs from a domain controller.
2
#define ROOT C:\Program Files\nxlog
3
define ROOT C:\Program Files (x86)\nxlog
4


5
Moduledir %ROOT%\modules
6
CacheDir %ROOT%\data
7
Pidfile %ROOT%\data\nxlog.pid
8
SpoolDir %ROOT%\data
9
LogFile %ROOT%\data\nxlog.log
10


11
<Extension _syslog>
12
    Module  xm_syslog
13
</Extension>
14


15
#Collect the Security log.
16
<Input in_securitylog>
17
   Module im_msvistalog
18
   Query <QueryList> \
19
          <Query Id="0"> \
20
           <Select Path="Security">*</Select> \
21
          </Query> \
22
         </QueryList>
23
	<Exec>	 
24
		$Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");
25
		$raw_event = $Message;
26
		to_syslog_snare();
27
	</Exec>
28
</Input>
29


30
#Collect the Microsoft DHCP logs.
31
<Input in_dhcp>
32
 Module  im_file
33
 #Change the location below to where the DHCP logs are.
34
 File "C:\\Windows\\Sysnative\\dhcp\\DhcpSrvLog-*.log"
35
 InputType LineBased
36
 SavePos TRUE
37
 <Exec>
38
	if $raw_event =~ /^#/ drop();
39
	else
40
	{
41
		to_syslog_bsd();
42
	}
43
 </Exec>
44
 </Input>
45


46
#Collect the Microsoft DNS log (s).
47
<Input in_dns>
48
 Module  im_file
49
 File 'c:\dnslogs\dns*.log'
50
 InputType LineBased
51
 SavePos TRUE
52
 <Exec>
53
	if $raw_event =~ /^#/ drop();
54
	else
55
	{
56
		to_syslog_bsd();
57
	}
58
 </Exec>
59
 </Input>
60


61
<Output out_securitylog>
62
   Module      om_udp
63
   #This is the IP address of the InsightIDR Collector
64
   Host			192.168.0.46
65
   #This is the port configured on the Event Source
66
   Port        10001
67
 #  Exec        to_json(); $message = $raw_event;
68
</Output>
69


70
<Output out_dhcp>
71
   #If you wish to send the logs using tcp instead of udp, change the module below to om_tcp.
72
   Module      om_udp
73
   #This is the IP address of the InsightIDR Collector
74
   Host			192.168.0.46
75
   #This is the port configured on the Event Source
76
   Port        10101
77
</Output>
78


79
<Output out_dns>
80
   Module      om_udp
81
   #This is the IP address of the InsightIDR Collector
82
   Host			192.168.0.46
83
   #This is the port configured on the Event Source
84
   Port        10201
85
</Output>
86


87
<Route 1>
88
   Path        in_securitylog=> out_securitylog
89
</Route>
90


91
<Route 2>
92
   Path        in_dhcp=> out_dhcp
93
</Route>
94


95
<Route 3>
96
   Path        in_dns=> out_dns
97
</Route>