NXLog
NXLog is a tool that converts files to syslog, which can be useful when an application produces log output that is not accepted by InsightIDR.
Install and Configure NXLog
To start using NXLog in your environment:
- Download the latest version of NXLog here: http://nxlog.co/products/nxlog-community-edition/download
- Install NXLog locally and set the ROOT to the folder in which your NXLog was installed, otherwise NXLog will not start. The path should be similar to this:
C:\Program Files (x86)\nxlog\conf\nxlog.conf
- Once installed, stop the NXLog service.
- Open the NXLog configuration file and paste the following into to the file, adjusting for your account as necessary:
Sample Configuration File
1## This is a sample configuration file. See the nxlog reference manual about the2## configuration options. It should be installed locally and is also available3## online at http://nxlog.org/docs/45## Please set the ROOT to the folder your nxlog was installed into,6## otherwise it will not start.78#define ROOT C:\Program Files\nxlog9define ROOT C:\Program Files (x86)\nxlog1011Moduledir %ROOT%\modules12CacheDir %ROOT%\data13Pidfile %ROOT%\data\nxlog.pid14SpoolDir %ROOT%\data15LogFile %ROOT%\data\nxlog.log1617<Extension _syslog>18Module xm_syslog19</Extension>2021<Input in>22Module im_msvistalog23ReadFromLast True24# For windows 2003 and earlier use the following:25# Module im_mseventlog2627Query <QueryList>\28<Query Id="0" Path="Security">\29<Select Path="Security">*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0) and (EventID=4769 or EventID=4728 or EventID=4732 or EventID=4756 or EventID=4720 or EventID=4722 or EventID=4725 or EventID=4740 or EventID=4767 or EventID=4624 or EventID=4625 or EventID=4704 or EventID=4648 or EventID=4724 or EventID=4741 or EventID=4738 or EventID=4768 or EventID=4723)]]</Select>\30</Query>\31</QueryList>3233</Input>3435<Output out>36Module om_tcp37Host COLLECTOR-HOST-NAME38Port COLLECTOR-PORT39Exec to_syslog_snare();40</Output>4142<Route 1>43Path in => out44</Route>
For additional configuration options, see the NXLog reference manual here: http://nxlog.org/docs/
- Restart the NXLog service.
You can now use the new NXLog configuration.
Collect Windows Server Logs
If you need to collect security log events from your environment, but do not want to use a Domain Admin account, you can configure NXLog to collect events for you.
Use one of the following configuration files to collect security logs:
You can look at other examples and also configure the NXLog files to send encrypted messages.
Active Directory
You can use NXLog to collect domain controller security logs for Active Directory.
To use NXLog to collect the security log:
- Install NXLog on the domain controller.
- Create a
new nxlog.conf
file to replace the default file. The default location of this file isC:\Program Files (x86)\nxlog\conf
. - To collect and forward domain controller security logs to your InsightIDR Collector, use the following configuration file as an example:
text
1#This sample nxlog.conf file can be used to read the Microsoft Security Log and send it to InsightIDR using syslog.2#Send the logs to InsightIDR in syslog_snare format.34#define ROOT C:\Program Files\nxlog5define ROOT C:\Program Files (x86)\nxlog67Moduledir %ROOT%\modules8CacheDir %ROOT%\data9Pidfile %ROOT%\data\nxlog.pid10SpoolDir %ROOT%\data11LogFile %ROOT%\data\nxlog.log1213<Extension _syslog>14Module xm_syslog15</Extension>1617<Input in_securitylog>18Module im_msvistalog19Query <QueryList> \20<Query Id="0"> \21<Select Path="Security">*</Select> \22</Query> \23</QueryList>24<Exec>25$Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");26$raw_event = $Message;27to_syslog_snare();28</Exec>29</Input>3031<Output out_securitylog>32Module om_udp33#This is the IP address of the InsightIDR Collector34Host 192.168.0.4635#This is the port configured on the Event Source36Port 1034137</Output>3839<Route 1>40Path in_securitylog=> out_securitylog41</Route>
Microsoft DHCP
You can use NXLog to collect DHCP logs for Microsoft DHCP.
To use NXLog to collect the Microsoft DHCP log:
- Install NXLog on the DHCP server.
- Create a new
nxlog.conf
file to replace the default file.The default location of this file isC:\Program Files (x86)\nxlog\conf
. - To collect and forward DHCP logs to your InsightIDR collector, use the following configuration file as an example:
text
1#This sample nxlog.conf file can be used to read Microsoft DHCP log files and send them to InsightIDR using syslog.2#Send the logs to InsightIDR in syslog_bsd format.34#define ROOT C:\Program Files\nxlog5define ROOT C:\Program Files (x86)\nxlog67Moduledir %ROOT%\modules8CacheDir %ROOT%\data9Pidfile %ROOT%\data\nxlog.pid10SpoolDir %ROOT%\data11LogFile %ROOT%\data\nxlog.log1213<Extension _syslog>14Module xm_syslog15</Extension>1617<Input in_dhcp>18Module im_file19#Change the location below to where the DHCP logs are.20File "C:\\Windows\\Sysnative\\dhcp\\DhcpSrvLog-*.log"21InputType LineBased22SavePos TRUE23<Exec>24if $raw_event =~ /^#/ drop();25else26{27to_syslog_bsd();28}29</Exec>30</Input>3132<Output out>33#If you wish to send the logs using tcp instead of udp, change the module below to om_tcp.34Module om_udp35#This is the IP address of the InsightIDR Collector36Host 192.168.0.15637#This is the port configured on the Event Source38Port 1040039</Output>4041<Route 1>42Path in_dhcp=> out43</Route>
Microsoft DNS
You can use NXLog to collect DNS logs for Microsoft DNS.
To use NXLog to collect the DNS log:
- Install NXLog on the DNS server.
- Create a new
nxlog.conf
file to replace the default file.The default location of this file isC:\Program Files (x86)\nxlog\conf
. - Enable the DNS file rollover. See the DNS page for instructions on how to enable rollover.
- To collect and forward DNS logs to your InsightIDR collector, use the following configuration file as an example:
text
1#This sample nxlog.conf file can be used to read Microsoft DNS log files and send them to InsightIDR using encrypted syslog.2#Send the logs to InsightIDR in syslog_bsd format.#define ROOT C:\Program Files\nxlog34define ROOT C:\Program Files (x86)\nxlog56Moduledir %ROOT%\modules7CacheDir %ROOT%\data8Pidfile %ROOT%\data\nxlog.pid9SpoolDir %ROOT%\data10LogFile %ROOT%\data\nxlog.log1112<Extension _syslog>13Module xm_syslog14</Extension>1516<Input in_dns>17Module im_file18File 'c:\dnslogs\dns*.log'19InputType LineBased20SavePos TRUE21<Exec>22if $raw_event =~ /^#/ drop();23else24{25to_syslog_bsd();26}27</Exec>28</Input>2930<Output out_dns>31Module om_udp32#This is the IP address of the InsightIDR Collector33Host 192.168.0.4634#This is the port configured on the Event Source35Port 1035036</Output>3738<Route 1>39Path in_dns=>out_dns40</Route>
Microsoft Exchange Message Tracking Logs
You can use NXLog to collect Microsoft Exchange Message Tracking logs.
To use NXLog to collect the Microsoft Exchange Message Tracking logs:
- Install NXLog on the Exchange server.
- Create a new
nxlog.conf
file to replace the default file.The default location of this file isC:\Program Files (x86)\nxlog\conf
. - Use the following configuration file as an example to collect and forward the Message Tracking logs to your InsightIDR collector:
text
1#This sample nxlog.conf file can be used to collect the Exchange Message Tracking Log and send it to InsightIDR.2#define ROOT C:\Program Files\nxlog3define ROOT C:\Program Files (x86)\nxlog45Moduledir %ROOT%\modules6CacheDir %ROOT%\data7Pidfile %ROOT%\data\nxlog.pid8SpoolDir %ROOT%\data9LogFile %ROOT%\data\nxlog.log10define CertDir C:\Program Files (x86)\nxlog\cert1112#Define the modules that will be used by nxlog.1314<Extension json>15Module xm_json16</Extension>1718<Extension _syslog>19Module xm_syslog20</Extension>2122<Extension MessageTrackingLog>23Module xm_csv24Fields $date-time,$client-ip,$client-hostname,$server-ip,$server-hostname,$source-context,$connector-id,$source,$event-id,$internal-message-id,$message-id,$network-message-id,$recipient-address,$recipient-status,$total-bytes,$recipient-count,$related-recipient-address,$reference,$message-subject,$sender-address,$return-path,$message-info,$directionality,$tenant-id,$original-client-ip,$original-server-ip,$custom-data,$transport-traffic-type,$log-id,$schema-version25#FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string26Delimiter ,27QuoteMethod None28</Extension>2930<Input in_MessageTrackingLog>31Module im_file32File 'c:\logs\logfile*.log'33InputType LineBased34SavePos TRUE35PollInterval 136<Exec>37if $raw_event =~ /^#/ drop();38else39{40MessageTrackingLog->parse_csv();41$EventTime = parsedate($date + " " + $time);42$SourceName = "MessageTrackingLog";43$raw_event = to_json();44}45</Exec>46</Input>4748<Output out_MesssageTrackingLog>49Module om_udp50#This is the IP address of the InsightIDR Collector51Host 192.168.0.15652#This is the port configured on the Universal Event Source53Port 1006054</Output>5556<Route 1>57Path in_MessageTrackingLog=>out_MesssageTrackingLog58</Route>
After you create this replacement file, configure a Custom Log event source in InsightIDR.
Microsoft IIS
You can use NXLog to collect Microsoft IIS logs.
To use NXLog to collect IIS logs:
- Install NXLog on the IIS server.
- Create a new
nxlog.conf
file to replace the default file.The default location of this file isC:\Program Files (x86)\nxlog\conf
. - Use the following configuration file to collect and forward the IIS logs to your InsightIDR collector:
text
1## This is a sample configuration file. See the nxlog reference manual about the2## configuration options. It should be installed locally and is also available3## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html4## Please set the ROOT to the folder your nxlog was installed into,5## otherwise it will not start.6#define ROOT C:\Program Files\nxlog7define ROOT C:\Program Files (x86)\nxlog8Moduledir %ROOT%\modules9CacheDir %ROOT%\data10Pidfile %ROOT%\data\nxlog.pid11SpoolDir %ROOT%\data12LogFile %ROOT%\data\nxlog.log13# Include fileop when rotating logs or while debugging14#<Extension fileop>15#Module xm_fileop16#</Extension>17<Extension json>18Module xm_json19</Extension>20# Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.21<Extension w3c>22Module xm_csv23Fields $date, $time, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, #$c_ip, $cs_User_Agent, $cs_Referer, $sc_status, $sc_substatus, $sc_win32_status, $time_taken24FieldTypes string, string, string, string, string, string, integer, string, string, string, #string, integer, integer, integer, integer25Delimiter ' '26</Extension>27# Convert the IIS logs to JSON and use the original event time28<Input IIS_Site1>29Module im_file30File "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex*"31SavePos TRUE3233Exec if $raw_event =~ /^#/ drop(); \34else \35{ \36w3c->parse_csv(); \37$EventTime = parsedate($date + " " + $time); \38$SourceName = "IIS"; \39$raw_event = to_json(); \40}41</Input>42<Output IIS_Site1_out>43Module om_udp44Host <IDR_COLLECTOR_IP/DNS>45Port <IDR_EVENT_PORT>46#Please note that if you do not see the IIS logs in Log Search, you will need to uncomment the line below.47#The Generic Syslog event source requires that the logs have a syslog header, as specified by RFC3164. This will add the required RFC3164 syslog header to the logs.48#Exec to_syslog_bsd();49</Output>50<Route IIS_Site1>51Path IIS_Site1 => IIS_Site1_out52</Route>
After you create this replacement file, configure a Generic Syslog event source in InsightIDR, which parses the IIS logs into a searchable format for Log Search.
Send Encrypted Logs
If you prefer to use encrypted syslog to transport logs to InsightIDR, you can use om_ssl
in your output module. For example:
text
1#Download the certificate from InsightIDR and place it in the folder you specify below.2<Output out_dns>3Module om_ssl4#This is the IP address of the InsightIDR Collector5Host 10.10.10.106#This is the port configured on the Event Source7Port 100018#specify cert location, the default location is nxlog\cert.9CAFile %CertDir%\Rapid7CA.pem10</Output>
Other Examples
It is possible to have multiple “in” and “out” sections in the nxlog.conf
file. For example, if you want to collect the security log, DHCP logs, and DNS logs from a single server, use the following nxlog.conf
file:
text
1#This sample nxlog.conf file can be used to read the Windows Security log, DHCP, and DNS logs from a domain controller.2#define ROOT C:\Program Files\nxlog3define ROOT C:\Program Files (x86)\nxlog45Moduledir %ROOT%\modules6CacheDir %ROOT%\data7Pidfile %ROOT%\data\nxlog.pid8SpoolDir %ROOT%\data9LogFile %ROOT%\data\nxlog.log1011<Extension _syslog>12Module xm_syslog13</Extension>1415#Collect the Security log.16<Input in_securitylog>17Module im_msvistalog18Query <QueryList> \19<Query Id="0"> \20<Select Path="Security">*</Select> \21</Query> \22</QueryList>23<Exec>24$Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");25$raw_event = $Message;26to_syslog_snare();27</Exec>28</Input>2930#Collect the Microsoft DHCP logs.31<Input in_dhcp>32Module im_file33#Change the location below to where the DHCP logs are.34File "C:\\Windows\\Sysnative\\dhcp\\DhcpSrvLog-*.log"35InputType LineBased36SavePos TRUE37<Exec>38if $raw_event =~ /^#/ drop();39else40{41to_syslog_bsd();42}43</Exec>44</Input>4546#Collect the Microsoft DNS log (s).47<Input in_dns>48Module im_file49File 'c:\dnslogs\dns*.log'50InputType LineBased51SavePos TRUE52<Exec>53if $raw_event =~ /^#/ drop();54else55{56to_syslog_bsd();57}58</Exec>59</Input>6061<Output out_securitylog>62Module om_udp63#This is the IP address of the InsightIDR Collector64Host 192.168.0.4665#This is the port configured on the Event Source66Port 1000167# Exec to_json(); $message = $raw_event;68</Output>6970<Output out_dhcp>71#If you wish to send the logs using tcp instead of udp, change the module below to om_tcp.72Module om_udp73#This is the IP address of the InsightIDR Collector74Host 192.168.0.4675#This is the port configured on the Event Source76Port 1010177</Output>7879<Output out_dns>80Module om_udp81#This is the IP address of the InsightIDR Collector82Host 192.168.0.4683#This is the port configured on the Event Source84Port 1020185</Output>8687<Route 1>88Path in_securitylog=> out_securitylog89</Route>9091<Route 2>92Path in_dhcp=> out_dhcp93</Route>9495<Route 3>96Path in_dns=> out_dns97</Route>