Data Collection Methods

When you configure event sources with the Collector, you can use one of the following data collection methods:

Listen on Network Port

You can configure your application to forward log events to a syslog server, and then configure the InsightIDR Collector to "listen" on network port for syslog data on a unique port in order to receive it.

See Syslog Logging for more information.

Log Aggregator

If you want to collect logs that have already been collected by a SIEM or a Log Aggregator, you can send raw logs to the Collector using a unique port. See SIEMs/Log Aggregators for more information.

SQS

AWS SQS, or Amazon Simple Queue Services, is a managed queuing service that works with InsightIDR when sending messages as events. See AWS SQS for more information,

WMI

WMI (Windows Management Instrumentation) allows your Collector to retrieve your event source applications for events that are related to User Attribution. WMI is available for all Windows-based event sources, and it is recommended for data collection whenever possible.

See Ports Used by InsightIDR for port recommendations and other requirements.

Watch Directory

You can monitor a network location that hosts log files copied from a specified directory on a local or remote host.

Use this collection method for log files that "roll over" into new files, such as Microsoft DHCP or IIS log files used in OWA/ActiveSync.

The default option for the scan interval is 30 seconds, but that can be adjusted during configuration.

Interaction between Watch Directory and the Collector

The Collector will scan the watched directory after it is configured. All files in the directory will be scanned to their EOF position. Then, the Collector will re-scan at the specified scan interval. Once the first scan interval elapses, the Collector will run another check and consume any net new lines written past the previous EOF position.

Shared Remote Directory and Local Folder options

During configuration, you must specify a path for the directory. The options are Shared Remote Directory (the default option) or Local Folder.

You can select the Shared Remote Directory option to enhance security. Use this method for log files that are written continuously to a single file, such as Microsoft DNS.

This option requires the Collector to authenticate to the directory as it would for any file share.

Shared Remote Directory option for Watch Directory in Add Event Source

With the Local Folder option, you can specify a local folder path or a Windows UNC (Universal Naming Convention) path to a hosted network drive. If the directory contains other files, enter a file pattern to specify which files InsightIDR should collect from the Directory.

Local Folder option for Watch Directory in Add Event Source

Watch Directory specifications

There are specifications on how Watch Directory works:

  • Watch Directory supports SMB v1 (CIFS) and SMB v2. InsightIDR requires packet signing for SMB2 connections.
  • Each new line must be delimited by an \n newline character to be considered as a log entry. The limit for each line is 8192 characters.
  • The Watch Directory collection method requires write permissions to be granted to the service account for the target directory.

Watch Directory or Tail File?

You can use Watch Directory for monitoring multiple files. This collection method is best for log files that roll over into new files. Tail File is a method to monitor log files that are written continuously to a single file.

Tail File

You can configure InsightIDR to watch the network location where a host stores log data, and ingest any new data added to the log file on a local or remote host. Using the equivalent of the Unix tail command, InsightIDR will collect data written to the host disk every 20 seconds.

There are specifications on how Tail File works:

  • Tail File will read a file and set the last read position. It will only upload net new log entries written to it since the event source started.
  • Each new line must be delimited by a \n newline character to be considered as a "line". The limit for this is 2048 characters.
  • During configuration, you must specify a local file path or a Windows UNC (Universal Naming Convention) path to a hosted network drive.
  • Tail File supports SMB v1 (CIFS) and SMB v2. InsightIDR requires packet signing for SMB2 connections.

Shared Remote Directory and Local File Path options

During configuration, you must specify a path for the directory. The options are Shared Remote Directory (the default option) or Local File Path.

You can select the Shared Remote Directory option if the log files reside on a remote Windows file share.

This option requires the Collector to authenticate to the directory as it would for any file share.

Shared Remote Directory option for Tail File in Add Event Source

If the log files you want InsightIDR to collect are located on the Insight Collector, you can use the Local File Path option to collect them. In this case, specify the path to the local folder where the log files reside. If the folder contains other files that you do not want InsightIDR to collect, enter a file pattern to specify the files InsightIDR should collect from the folder.

Local File Path option for Tail File in Add Event Source

Amazon S3

You can configure InsightIDR to read logs that are stored in an Amazon S3 bucket. Information about setting up an Amazon S3 bucket can be found at: https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html

Amazon S3 Log Formatting

The Amazon S3 option supports reading files stored in an S3 bucket that contain either plain text or plain text files that have been compressed using gzip. In either case, the files should have new line characters separating each log entry or event. If the files contain log events spread across multiple lines, each line will be interpreted as a separate event. Amazon S3 does not support reading files with encrypted contents, binary data, or files that have been compressed using a method other than gzip.

Configure the Refresh Rate to your desired collection interval based on the type of logs being written to the S3 bucket. If you are not sure what to use, start with a setting of 5 minutes and adjust as needed.

Amazon S3 in Add Event Source

Amazon S3 Supported Regions

S3 RegionURL
US_STANDARDhttps://s3.amazonaws.com
US_EAST_OHIOhttps://s3.us-east-2.amazonaws.com
US_EAST_GOVCLOUDhttps://s3.us-gov-east-1.amazonaws.com
US_EAST_FIPS_GOVCLOUDhttps://s3-fips.us-gov-east-1.amazonaws.com
US_WEST_N_CALIFORNIAhttps://s3.us-west-1.amazonaws.com
US_WEST_OREGONhttps://s3.us-west-2.amazonaws.com
US_WEST_GOVCLOUDhttps://s3.us-gov-west-1.amazonaws.com
US_WEST_FIPS_GOVCLOUDhttps://s3-fips.us-gov-west-1.amazonaws.com
CA_CENTRALhttps://s3.ca-central-1.amazonaws.com
EU_IRELANDhttps://s3.eu-west-1.amazonaws.com
EU_LONDONhttps://s3.eu-west-2.amazonaws.com
EU_PARIShttps://s3.eu-west-3.amazonaws.com
EU_FRANKFURThttps://s3.eu-central-1.amazonaws.com
EU_MILANhttps://s3.eu-south-1.amazonaws.com
EU_STOCKHOLMhttps://s3.eu-north-1.amazonaws.com
AF_CAPE_TOWNhttps://s3.af-south-1.amazonaws.com
AP_MUMBAIhttps://s3.ap-south-1.amazonaws.com
AP_HONG_KONGhttps://s3.ap-east-1.amazonaws.com
AP_TOKYOhttps://s3.ap-northeast-1.amazonaws.com
AP_SEOULhttps://s3.ap-northeast-2.amazonaws.com
AP_OSAKAhttps://s3.ap-northeast-3.amazonaws.com
AP_SINGAPOREhttps://s3.ap-southeast-1.amazonaws.com
AP_SYDNEYhttps://s3.ap-southeast-2.amazonaws.com
AP_JAKARTAhttps://s3.ap-southeast-3.amazonaws.com
ME_BAHRAINhttps://s3.me-south-1.amazonaws.com
ME_UAEhttps://s3.me-central-1.amazonaws.com
SA_SAO_PAULOhttps://s3.sa-east-1.amazonaws.com