Rsyslog

Rsyslog is a syslog daemon commonly deployed in Debian and Ubuntu systems. It typically uses a simple TCP connection to send logs line-by-line. We support two methods of forwarding rsyslog events to InsightOps, which are explained below. We recommend using our Token-based input method which brings additional security and is independent of the actual source IP address.

Create a new Host in the UI. Inside this host, create a new Log and select Token TCP and Register the log, you will receive a unique Token UUID which will be printed under the form and then beside the log name in the list of logs. Enter this token where it says TOKEN HERE below and place the two lines at the bottom of your rsyslog configuration file found at /etc/rsyslog.conf. Replace REGION_HERE with the region where your InsightOps account is located (e.g. eu, us, ca, au etc)

 
1
$template InsightOpsFormat,"TOKEN_HERE %HOSTNAME% %syslogtag%%msg%\n"
2

3
*.* @@REGION_HERE.data.logs.insight.rapid7.com:80;InsightOpsFormat

If you would rather use a more basic Input, then create a host in the InsightOps UI. Inside that, create a log and select Plain TCP/UDP and Register the log, you will receive a PORT number to use which will be printed under the form and beside the log name in the list of logs. Simply add the PORT number to the following line and place it at the bottom of your rsyslog configuration file found at /etc/rsyslog.conf. Replace REGION_HERE with the region where your InsightOps account is located (e.g. eu, us, ca, au etc)

 
1
*.* @@REGION_HERE.data.logs.insight.rapid7.com:PORT

Note: For UDP forwarding use a single @. For encrypted TLS connections use @@REGION_HERE.data.logs.insight.rapid7.com.

In order to accept any changes to its configuration, restart your syslog server with the following command and you should start to see your events displayed in the log file you just set up in your account:

 
1
sudo service rsyslog restart

Rsyslog can be configured to follow any files on your system. Full documentation on this can be found here. If you wanted to follow a file called /var/log/myapp/errors.log, you would enter the following lines in your /etc/rsyslog.conf configuration file, above the previous snippet we added.

 
1
$Modload imfile
2

3
$InputFileName /var/log/myapp/errors.log
4
$InputFileTag myapp
5
$InputFileStateFile myapp-file1
6
$InputFileSeverity info
7
$InputFileFacility local7
8
$InputRunFileMonitor
9

10
# Only entered once in case of following multiple files
11
# This will poll the file every 10 seconds
12
$InputFilePollInterval 10

InsightOps also provides a Python setup tool to setup your Rsyslog configuration. This tool can be downloaded via Github.

The setup tool allows you to automatically create a new Host and then for each file you wish to follow it will create a new log. The register documentation can be found here. Following a file is done by simply running sudo python le follow myfile where myfile is the path of the file you wish to follow.

More details can be found here. The setup tool can also read from a JSON configuration file which allows you to setup what files you wish to follow and what tokens to use. This is very useful is you are logging in an elastic enviroment. More details can be found here.

Rsyslog allows you to filter logs and assign different destinations to different events. Full documentation on this can be found here. If you wanted to send your nginx access log to a single log on InsightOps, you would first follow the above section so that the nginx access log is monitored, taking note of the value you gave for $InputFileTag. Then you would enter the following in your /etc/rsyslog.conf. Replace REGION_HERE with the region where your InsightOps account is located (e.g. eu, us, ca, au etc)

 
1
$template NginxTemplate,"TOKEN_HERE %HOSTNAME% %syslogtag%%msg%\n"
2

3
if $programname == 'mytag' then @@REGION_HERE.data.logs.insight.rapid7.com:80;NginxTemplate
4
& ~

You need to create a log in InsightOps and choose Token TCP and then the token thats printed in green is inserted above where it says TOKEN_HERE. You also need to make sure that the value you used for $InputFileTag matches the value that $programname is being checked against. The third line simply discards the event after its sent to InsightOps, you can omit this if you would like to also be sent somehwere else in your /etc/rsyslog.conf.

In untrusted networks, you can set up encryption with certificate validation. Make sure you have installed support for TLS. Usually this is achieved installing the rsyslog-gnutls package. Sample configuration to enable encryption may look like this:

 
1
$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt
2

3
$ActionSendStreamDriver gtls 
4
$ActionSendStreamDriverMode 1 
5
$ActionSendStreamDriverAuthMode x509/name 
6
$ActionSendStreamDriverPermittedPeer *.data.logs.insight.rapid7.com
7

8
$template InsightOpsFormat,"LOG-TOKEN %HOSTNAME% %syslogtag%%msg%\n" 
9
*.* @@REGION_HERE.data.logs.insight.rapid7.com:443;InsightOpsFormat

Note: you have to specify the right port number. For token based input, the port number is 443.

Replace REGION_HERE with the region where your InsightOps account is located (e.g. eu, us, ca, au etc)

In case of unreliable network connection you can instruct rsyslog to buffer log entries during network outage. For more details please refer to rsyslog reliable forwarding documentation.

The following is a quick guide to the parameters we have set in our Rsyslog.conf. Fpr further information make sure to check our Rsyslog’s Official Documentation $InputFileName: The location of the file that we wish to follow e.g /var/log/dmesg

$InputFileTag: The tag to be used for messages that originate from this file. If you would like to see the colon after the tag, you need to specify it here (like ‘tag=”myTagValue:”’).

$InputFileStateFile: This is the name of this file’s state file.

$InputFileFacility: The syslog facility to be assigned to lines read. Can be specified in textual form (e.g. “local0”, “local1”, …) or as numbers (e.g. 128 for “local0”). Textual form is suggested. Default is “local0”.

$InputFileSeverity: The syslog severity to be assigned to lines read. Can be specified in textual form (e.g. “info”, “warning”, …) or as numbers (e.g. 4 for “info”). Textual form is suggested. Default is “notice”.

$InputRunFileMonitor: This activates the current monitor. It has no parameters. If you forget this directive, no file monitoring will take place.

$InputFilePollInterval: This setting specifies how often files are to be polled for new data. For obvious reasons, it has effect only if imfile is running in polling mode. The time specified is in seconds. During each polling interval, all files are processed in a round-robin fashion. A short poll interval provides more rapid message forwarding, but requires more system resources. While it is possible, we stongly recommend not to set the polling interval to 0 seconds. That will make rsyslogd become a CPU hog, taking up considerable resources. It is supported, however, for the few very unusual situations where this level may be needed. Even if you need quick response, 1 seconds should be well enough. Please note that imfile keeps reading files as long as there is any data in them. So a “polling sleep” will only happen when nothing is left to be processed.

My logs stop forwarding at certain times, e.g. Midnight UTC? There is a known bug with the Rsyslog version currently available through Ubuntu repositories, when used in conjunction with logrotate. Rsyslog does not pick up the new files after they are rotated. To fix this, you can install a newer version of Rsyslog with the instructions found here.