NXLog

NXLog is a tool that converts files to syslog, which can be useful when an application produces log output that is not accepted by InsightIDR.

Install and Configure NXLog

To start using NXLog in your environment:

Download the latest version of NXLog here: http://nxlog.co/products/nxlog-community-edition/download
Install NXLog locally and set the ROOT to the folder in which your NXLog was installed, otherwise NXLog will not start. The path should be similar to this: C:\Program Files (x86)\nxlog\conf\nxlog.conf
Once installed, stop the NXLog service.
Open the NXLog configuration file and paste the following into to the file, adjusting for your account as necessary:


## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/docs/
 
## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
 
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
 
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
 
<Extension _syslog>
    Module      xm_syslog
</Extension>
 
<Input in>
    Module      im_msvistalog
    ReadFromLast  True
# For windows 2003 and earlier use the following:
#   Module      im_mseventlog
 
Query <QueryList>\
  <Query Id="0" Path="Security">\
     <Select Path="Security">*[System[(Level=1  or Level=2 or Level=3 or Level=4 or Level=0) and (EventID=4769 or EventID=4728 or EventID=4732 or EventID=4756 or EventID=4720 or EventID=4722 or EventID=4725 or EventID=4740 or EventID=4767 or EventID=4624 or EventID=4625 or EventID=4704 or EventID=4648 or EventID=4724 or EventID=4741 or EventID=4738 or EventID=4768 or EventID=4723)]]</Select>\
  </Query>\
</QueryList>
 
</Input>
 
<Output out>
    Module      om_tcp
    Host        COLLECTOR-HOST-NAME
    Port        COLLECTOR-PORT
    Exec        to_syslog_snare();
</Output>
 
<Route 1>
    Path        in => out
</Route>

For additional configuration options, see the NXLog reference manual here: http://nxlog.org/docs/

Restart the NXLog service.

You can now use the new NXLog configuration.

Collect Windows Server Logs

If you need to collect security log events from your environment, but do not want to use a Domain Admin account, you can configure NXLog to collect events for you.

Use one of the following configuration files to collect security logs:

You can look at other examples and also configure the NXLog files to send encrypted messages.

Active Directory

You can use NXLog to collect domain controller security logs for Active Directory.

To use NXLog to collect the security log:

Install NXLog on the domain controller.
Create a new nxlog.conf file to replace the default file. The default location of this file is C:\Program Files (x86)\nxlog\conf.
To collect and forward domain controller security logs to your InsightIDR Collector, use the following configuration file as an example:


#This sample nxlog.conf file can be used to read the Microsoft Security Log and send it to InsightIDR using syslog.
#Send the logs to InsightIDR in syslog_snare format.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input in_securitylog>
   Module im_msvistalog
   Query <QueryList> \
          <Query Id="0"> \
           <Select Path="Security">*</Select> \
          </Query> \
         </QueryList>
	<Exec>	 
		$Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");
		$raw_event = $Message;
		to_syslog_snare();
	</Exec>
</Input>

<Output out_securitylog>
   Module      om_udp
   #This is the IP address of the InsightIDR Collector
   Host			192.168.0.46
   #This is the port configured on the Event Source
   Port        10341
</Output>

<Route 1>
   Path        in_securitylog=> out_securitylog
</Route>

Microsoft DHCP

You can use NXLog to collect DHCP logs for Microsoft DHCP.

To use NXLog to collect the Microsoft DHCP log:

Install NXLog on the DHCP server.
Create a new nxlog.conf file to replace the default file.The default location of this file is C:\Program Files (x86)\nxlog\conf.
To collect and forward DHCP logs to your InsightIDR collector, use the following configuration file as an example:


#This sample nxlog.conf file can be used to read Microsoft DHCP log files and send them to InsightIDR using syslog.
#Send the logs to InsightIDR in syslog_bsd format.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input in_dhcp>
 Module  im_file
 #Change the location below to where the DHCP logs are.
 File "C:\\Windows\\Sysnative\\dhcp\\DhcpSrvLog-*.log"
 InputType LineBased
 SavePos TRUE
 <Exec>
	if $raw_event =~ /^#/ drop();
	else
	{
		to_syslog_bsd();
	}
 </Exec>
 </Input>
 
<Output out>
   #If you wish to send the logs using tcp instead of udp, change the module below to om_tcp.
   Module      om_udp
   #This is the IP address of the InsightIDR Collector
   Host			192.168.0.156
   #This is the port configured on the Event Source
   Port        10400
</Output>

<Route 1>
   Path        in_dhcp=> out
</Route>

Microsoft DNS

You can use NXLog to collect DNS logs for Microsoft DNS.

To use NXLog to collect the DNS log:

Install NXLog on the DNS server.
Create a new nxlog.conf file to replace the default file.The default location of this file is C:\Program Files (x86)\nxlog\conf.
Enable the DNS file rollover. See the DNS page for instructions on how to enable rollover.
To collect and forward DNS logs to your InsightIDR collector, use the following configuration file as an example:


#This sample nxlog.conf file can be used to read Microsoft DNS log files and send them to InsightIDR using encrypted syslog.
#Send the logs to InsightIDR in syslog_bsd format.#define ROOT C:\Program Files\nxlog

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input in_dns>
 Module  im_file
 File 'c:\dnslogs\dns*.log'
 InputType LineBased
 SavePos TRUE
 <Exec>
	if $raw_event =~ /^#/ drop();
	else
	{
		to_syslog_bsd();
	}
 </Exec>
 </Input>

<Output out_dns>
   Module      om_udp
   #This is the IP address of the InsightIDR Collector
   Host			192.168.0.46
   #This is the port configured on the Event Source
   Port        10350
</Output>

<Route 1>
   Path        in_dns=>out_dns
</Route>

Microsoft Exchange Message Tracking Logs

You can use NXLog to collect Microsoft Exchange Message Tracking logs.

To use NXLog to collect the Microsoft Exchange Message Tracking logs:

Install NXLog on the Exchange server.
Create a new nxlog.conf file to replace the default file.The default location of this file is C:\Program Files (x86)\nxlog\conf.
Use the following configuration file as an example to collect and forward the Message Tracking logs to your InsightIDR collector:


#This sample nxlog.conf file can be used to collect the Exchange Message Tracking Log and send it to InsightIDR.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
define CertDir C:\Program Files (x86)\nxlog\cert

#Define the modules that will be used by nxlog.

<Extension json>
    Module xm_json
</Extension>

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Extension MessageTrackingLog>
    Module      xm_csv
	Fields 		$date-time,$client-ip,$client-hostname,$server-ip,$server-hostname,$source-context,$connector-id,$source,$event-id,$internal-message-id,$message-id,$network-message-id,$recipient-address,$recipient-status,$total-bytes,$recipient-count,$related-recipient-address,$reference,$message-subject,$sender-address,$return-path,$message-info,$directionality,$tenant-id,$original-client-ip,$original-server-ip,$custom-data,$transport-traffic-type,$log-id,$schema-version
    	#FieldTypes  string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string
    	Delimiter   ,
	QuoteMethod	None
</Extension>

<Input in_MessageTrackingLog>
	Module  im_file
	File 'c:\logs\logfile*.log'
	InputType LineBased
	SavePos TRUE
	PollInterval 1
	<Exec>
		if $raw_event =~ /^#/ drop();
		else
		{
			MessageTrackingLog->parse_csv();
			$EventTime = parsedate($date + " " + $time);
            $SourceName = "MessageTrackingLog";				
            $raw_event = to_json();	
		}
	</Exec>
 </Input>

<Output out_MesssageTrackingLog>
    Module      om_udp
	#This is the IP address of the InsightIDR Collector
    Host        192.168.0.156
	#This is the port configured on the Universal Event Source
    Port        10060
</Output>

<Route 1>
   Path        in_MessageTrackingLog=>out_MesssageTrackingLog
</Route>

After you create this replacement file, configure a Custom Log event source in InsightIDR.

Microsoft IIS

You can use NXLog to collect Microsoft IIS logs.

To use NXLog to collect IIS logs:

Install NXLog on the IIS server.
Create a new nxlog.conf file to replace the default file.The default location of this file is C:\Program Files (x86)\nxlog\conf.
Use the following configuration file to collect and forward the IIS logs to your InsightIDR collector:


## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html
## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
# Include fileop when rotating logs or while debugging
#<Extension fileop>
#Module xm_fileop
#</Extension>
<Extension json>
    Module      xm_json
</Extension>
# Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c>
    Module xm_csv
    Fields $date, $time, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, #$c_ip, $cs_User_Agent, $cs_Referer, $sc_status, $sc_substatus, $sc_win32_status, $time_taken
    FieldTypes string, string, string, string, string, string, integer, string, string, string, #string, integer, integer, integer, integer
    Delimiter ' '
</Extension>
# Convert the IIS logs to JSON and use the original event time
<Input IIS_Site1>
    Module    im_file
    File    "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex*"
    SavePos  TRUE
 
    Exec if $raw_event =~ /^#/ drop();              \
       else                         \
       {                            \
            w3c->parse_csv();                    \
            $EventTime = parsedate($date + " " + $time);  \
            $SourceName = "IIS";              \
            $raw_event = to_json();             \
       }
</Input>
<Output IIS_Site1_out>
   Module      om_udp
   Host        <IDR_COLLECTOR_IP/DNS>
   Port        <IDR_EVENT_PORT>
   #Please note that if you do not see the IIS logs in Log Search, you will need to uncomment the line below.  
   #The Generic Syslog event source requires that the logs have a syslog header, as specified by RFC3164.  This will add the required RFC3164 syslog header to the logs.
  #Exec         to_syslog_bsd();
</Output>
<Route IIS_Site1>
    Path        IIS_Site1 => IIS_Site1_out
</Route>

After you create this replacement file, configure a Generic Syslog event source in InsightIDR, which parses the IIS logs into a searchable format for Log Search.

Send Encrypted Logs

If you prefer to use encrypted syslog to transport logs to InsightIDR, you can use om_ssl in your output module. For example:


#Download the certificate from InsightIDR and place it in the folder you specify below.
<Output out_dns>
   Module      om_ssl
   #This is the IP address of the InsightIDR Collector
   Host	       10.10.10.10
   #This is the port configured on the Event Source
   Port        10001
   #specify cert location, the default location is nxlog\cert.
   CAFile      %CertDir%\Rapid7CA.pem
</Output>

Other Examples

It is possible to have multiple “in” and “out” sections in the nxlog.conf file. For example, if you want to collect the security log, DHCP logs, and DNS logs from a single server, use the following nxlog.conf file:


#This sample nxlog.conf file can be used to read the Windows Security log, DHCP, and DNS logs from a domain controller.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
    Module  xm_syslog
</Extension>

#Collect the Security log.
<Input in_securitylog>
   Module im_msvistalog
   Query <QueryList> \
          <Query Id="0"> \
           <Select Path="Security">*</Select> \
          </Query> \
         </QueryList>
	<Exec>	 
		$Message = replace($Message, "\t", " "); $Message = replace($Message, "\n", " "); $Message = replace($Message, "\r", " ");
		$raw_event = $Message;
		to_syslog_snare();
	</Exec>
</Input>

#Collect the Microsoft DHCP logs.
<Input in_dhcp>
 Module  im_file
 #Change the location below to where the DHCP logs are.
 File "C:\\Windows\\Sysnative\\dhcp\\DhcpSrvLog-*.log"
 InputType LineBased
 SavePos TRUE
 <Exec>
	if $raw_event =~ /^#/ drop();
	else
	{
		to_syslog_bsd();
	}
 </Exec>
 </Input>

#Collect the Microsoft DNS log (s).
<Input in_dns>
 Module  im_file
 File 'c:\dnslogs\dns*.log'
 InputType LineBased
 SavePos TRUE
 <Exec>
	if $raw_event =~ /^#/ drop();
	else
	{
		to_syslog_bsd();
	}
 </Exec>
 </Input>

<Output out_securitylog>
   Module      om_udp
   #This is the IP address of the InsightIDR Collector
   Host			192.168.0.46
   #This is the port configured on the Event Source
   Port        10001
 #  Exec        to_json(); $message = $raw_event;
</Output>

<Output out_dhcp>
   #If you wish to send the logs using tcp instead of udp, change the module below to om_tcp.
   Module      om_udp
   #This is the IP address of the InsightIDR Collector
   Host			192.168.0.46
   #This is the port configured on the Event Source
   Port        10101
</Output>

<Output out_dns>
   Module      om_udp
   #This is the IP address of the InsightIDR Collector
   Host			192.168.0.46
   #This is the port configured on the Event Source
   Port        10201
</Output>

<Route 1>
   Path        in_securitylog=> out_securitylog
</Route>

<Route 2>
   Path        in_dhcp=> out_dhcp
</Route>

<Route 3>
   Path        in_dns=> out_dns
</Route>