InsightIDR Essential | Quick Start Guide

As you get started, refer to this Quick Start Guide for guidance about how to approach InsightIDR's multi-step deployment process.

Not sure if you're in the right place?

If you purchased InsightIDR (not designated as Essential, Advanced, or Ultimate), please follow InsightIDR Quick Start Guide | Advanced for tasks and materials suited to your product.

InsightIDR Essential Flywheel

We’ve outlined the important deployment milestones in this recommended order:

  • Days 1-15: You'll install the Collector, deploy the Insight Agent, and set up five core event sources to begin aggregating your environment's data into InsightIDR to gain visibility and collect the right data to monitor key security controls.
  • Days 16-45: You'll explore Log Search to understand network activity and identify anomalous events within your environment.
  • Days 46-90: You'll expand your monitoring footprint with FIM, network traffic, and customizable capabilities such as basic detection rules (formerly known as custom alerts) and custom dashboards.

Product Overview and Core Concepts

InsightIDR contains many features, but your first 90 days will center around the key parts of the product outlined in this section. For an overview of the areas you'll explore, watch our InsightIDR Demo video!

Here's a preview of the topics we cover in this demonstration:

  • 0:00 - Overview
  • 0:31 - Data Collection
  • 2:04 - Log Search
  • 3:09 - Dashboards and Reports
  • 4:35 - Custom Alerts (now known as basic detection rules) and Alert Settings (now known as Detection Rules)
  • 5:27 - Settings
  • 6:21 - Conclusion
Core Components

On-premise components

Review InsightIDR's on-premises components to understand their purpose and value.

ComponentDescription
CollectorRequired for:
- Event source log collection

To retrieve data, the Collector either polls event sources for the data, or has data pushed to it from the event sources. By default, the Collector filters logs to cut down on duplicate or unnecessary data. The Collector sends the log data to the Insight Platform for analysis.

Each Collector must be installed on a dedicated host and only one Collector instance can be on each machine.
Insight AgentRequired for:
- Endpoint Agent log set
- File Integrity Monitoring (FIM) Log Search and Dashboard capabilities

An Insight Agent is lightweight software installed on an endpoint to report security-relevant events. To have the most cohesive understanding of your environment, it is recommended that you install an Insight Agent on all endpoints and servers you wish to monitor for File Integrity Monitoring purposes. The agent only collects data from the asset on which it is installed.
Insight Network SensorRequired for:
- IDS Alerts log set
- DNS Query log set
- Host to IP Observations log set
- DNS Query Out-of-the-Box Dashboard cards
- IDS Alert Out-of-the-Box Dashboard cards

The Insight Network Sensor allows you to capture and analyze the network traffic moving throughout your physical, virtual and cloud environments. InsightIDR leverages DNS and DHCP metadata that the sensor extracts from network packets to provide valuable network traffic log sets and Suricata based IDS alerts.
Service AccountRequired for:
- Event Source log collection

Service Accounts represent machine accounts that are typically designed for machine processing within your network. InsightIDR does not expect these accounts to be used outside your network.
Event SourceRequired for:
- Centralized log management
- Out-of-the-Box compliance dashboards

An event source is an application, appliance, server, service, or other IT asset that generates log events.

Basic Detection Rules and Basic Detection Rules Settings

Review this section to understand InsightIDR Essential's detection features, their purposes, values, and how they contribute to your environment's security.

ComponentDescription
Basic Detection RulesIf you need to detect on certain, custom events, create a basic detection rule. You can also use basic detection rules to create File Integrity Monitoring (FIM) and Intrusion Detection System (IDS) alerts and to configure your desired notification channels. There are three types of basic detection rules: Log Inactivity Detection Rules, Log Pattern Detection Rules, and Log Change Detection Rules.
Basic Detection Rules SettingsYou can create basic detection rules from Log Search or from the Basic Detection Rules tab of the Detection Rules page. All of your basic detection rules can be managed via the Basic Detection Rules tab.

Prepare for Deployment

To ensure you get the most out of your first 90 days with InsightIDR, it’s important to understand your deployment tasks and create a plan for deployment.

Supported Browsers

Rapid7 supports InsightIDR in Google Chrome (latest stable release) and Mozilla Firefox (latest stable release).

Set up a service account

Before provisioning resources and deploying InsightIDR, you must set up a Service Account to collect log data for InsightIDR. You can either designate an existing user account, or create a Service Account.

Review the Service Account requirements and set up an account.

Review system requirements

Each component of InsightIDR requires specific resources. Navigate to each component's designated page for a full list of requirements.

ComponentRequired Resources
Collector- 4 CPU cores with 2GHz+ on each core
- 8 GB RAM recommended
- 60 GB+ available disk space
- Configured with a Fully Qualified Domain Name (FQDN) such as idrcollector23.myorg.com

See Collector Requirements requirements for more information.
Insight AgentRequired ports for Collector communication through TCP:

- 5508
- 6608
- 8037

If you are using the Collector for endpoint monitoring, ensure the following ports are open:

- 5508
- 6608
- 20000
– 30000

See Insight Agent Requirements requirements for more information.
Service AccountYou can either designate an existing user account as a service account, or create a new account account as your service account, that meets all of the following requirements:

- Active Directory Permissions
- Microsoft DNS Permissions
- Microsoft DHCP Account Permissions

See Setting Up a Service Account for more information.
Event SourceDesignate a Service Account with the correct permissions

See Core Event Sources for more individual event source requirements.
Insight Network SensorAlthough the network sensor software itself runs in the form of a container, all physical or virtual network sensor hosts must run one of the following supported Linux operating systems. The version number shown for each one indicates the minimum supported version:

- Ubuntu Server 20.04 and later
- RHEL 7.2 and later
- CentOS 8 and later
- Fedora 30 and later
- SUSE 15.0 and later
- Debian 8.11 and later

See Network Sensor Host Requirements for more information.
Provision resources in your environment

We recommend provisioning specific resources as soon as possible to ensure a quick and easy deployment experience.

These core components require provisioning hardware:

Log in to the Insight Platform

Already have an Insight Platform account?

If you already have a platform account from a trial or existing subscription to another Rapid7 solution, you’re all set! Use your existing email address to log in to https://insight.rapid7.com/login.

The Rapid7 Insight Platform is your base within the ecosystem of Rapid7 cloud products and services. It provides a centralized location for administrative functions and makes navigating the Insight product suite simple and easy.

To log in to the platform, you need a Rapid7 Insight Platform account.

To create an account:

  1. Check your corporate email inbox for an email from the Rapid7 Insight Platform team.
  2. Visit insight.rapid7.com/login.
  3. Select Haven’t activated your account?.
  4. Enter your corporate email address to receive an activation email with next steps. If you do not receive an activation email, reach out to your Customer Adoption Manager (CAM) or Customer Success Manager (CSM).
  5. Refer to the activation email and follow the instructions to create and activate your Insight Platform account
Configure daily data archiving

InsightIDR stores your log data for a retention period of 13 months. If you need to retain data for longer than that period, such as for security investigation or compliance purposes, we recommend that you set up daily archiving. Archiving allows you to retain a copy of your log data using the storage capabilities of Amazon S3.

To set up data archiving, see Data Archiving.

Daily Archiving versus Historical Data Archiving

If you do not configure daily archiving, you can download a backup of your data up to 2 times a year using InsightIDR's Historical Data Archiving feature. This process can take several days to complete.

Days 1 - 15: Get Up and Running

Install Collectors, deploy Insight Agents, and set up core event sources so InsightIDR can begin collecting the right data to detect and investigate anomalous activity in your environment.

Install the Collector

The Collector is the on-premises component of InsightIDR that either polls data or receives data from Event Sources and makes it available for InsightIDR to analyze.

The Collector captures the data generated by your event sources, compresses the data, encrypts it, and pushes it up to the Insight Platform. The Insight Platform will then normalize, attribute, analyze, and present that data for search.

How to install

Identify all the servers where event source data originates from. Many networks consolidate security and network administration tools and services in a data center or corporate office. This central location is an ideal place to deploy a Collector on a dedicated host.

  1. Review the Collector Requirements.
  2. Install collectors on network servers or virtual machines that meet the following requirements:
    • 4 CPU cores with 2GHz+ on each core
    • 8 GB RAM recommended
    • 60 GB+ available disk space
    • Configured with a Fully Qualified Domain Name (FQDN) such as idrcollector23.myorg.com

For detailed setup instructions, see Collector Installation and Deployment.

Deploy the Insight Agent

The Insight Agent enables InsightIDR to provide continuous endpoint security monitoring, which is necessary for identifying the early signs of an attack, and powers File Integrity Monitoring (FIM).

The Insight Agent collects live system information – including basic asset identification information, running processes, and logs – from your assets and sends it to the Insight Platform for analysis. Combined with network and user data, this helps you identify stealthy attacks for both your critical and remote assets.

How to deploy the Insight Agent

Each Insight Agent only collects data from the endpoint on which it is installed. You must install the Agent on all assets you want InsightIDR to monitor. The Insight Agent can be installed directly on Windows, Linux, or Mac assets.

You can deploy the Insight Agent on your target assets using either a token-based installer or certificate package installer. The token-based installer uses a token to download configuration files, whereas the certificate package installer provides a .zip file with the configuration files.

  1. Decide which installer is best suited for your environment.
  2. Read the steps to download the installer and follow the instructions.
Set up core event sources

To take advantage of InsightIDR's out-of-the-box compliance dashboards and reports, you must set up these event sources.

Setup instructions for each event source can vary depending upon the technologies you use.

Active Directory

Active Directory provides security logs from your domain controllers and authentication and administrative events for your domain users. Make sure that you add one Active Directory event source for each domain controller.

To add the Active Directory event source:

  • Open ports 135, 139, and 445 between the Collector and Active Directory.
  • Designate a Service Account with the correct permissions.

For detailed setup instructions, see Active Directory.

DHCP

Dynamic Host Configuration Protocol (DHCP) event logs provide IP lease information to correlate each IP address with its assigned host at the time of the event.

To add the DHCP event source:

Choose one of the following methods:

  • Domain Admin Account: Use this method if you want to collect logs using a Domain Admin account.
  • NXLog: Use this method if you don’t want to use a Domain Admin account to collect logs.
DNS

Connecting DNS as an event source allows InsightIDR to gather DNS logs which provides more information about web traffic. DNS also provides greater visibility into destination URLs.

For instructions on how to configure DNS appliances with InsightIDR, see DNS.

VPN
VPN logs provide visibility into users' remote network ingress activity and allow you to collect and verify information about user activity. For setup instructions, see VPN.
Firewall

By introducing Firewall data, you allow InsightIDR to track traffic moving in and out of your network. More than simply collecting configuration logs and change logs, InsightIDR can automatically attribute connection events to the users and endpoints.

We recommend you set up separate Firewall event sources for each Firewall type.

To add a Firewall event source:

Determine which Firewall event source(s) best suits your needs and follow the related configuration steps.

For setup instructions for each type of Firewall event source, see Firewall.

Days 16 - 45: Assess and Detect

Explore Log Search and Dashboards and Reports to see these components work together. Start using InsightIDR's powerful query capabilities, and visualize your log data using dashboards.

Explore Log Search and Queries

Your connected event sources and environment systems produce data in the form of raw logs. Log Search takes every log of raw, collected data and automatically sorts them into log sets for you. New log events are added to the existing log and grouped with other logs in the log set, based on when the log event was created.

When data is ingested, repetitive activity is processed and combined into a single entry. The count of the repeated events are included in the combined entry, and other fields such as bytes transferred are summarized. To learn more about data deduplication, read Collector Overview.

Use queries to search for anomalous events in your environment

Log Entry Query Language (LEQL) is used when building queries in InsightIDR. LEQL supports Regular Expressions (RegEx) and can be used when creating basic detection rule queries and dashboards.

Log Search supports many InsightIDR capabilities:

  • Data ingestion
  • Detection
  • Basic detection rules
  • Dashboards and reporting
  • Compliance standards

Build and save a query

InsightIDR provides different ways to search your data, including RegEx, String, KeyValue, or Keyword searches. With LEQL, you can construct queries to extract the hidden data within your logs.

LEQL follows SQL-style syntax and constructing a query is simple and intuitive. InsightIDR requires you to input a where() clause first (if your query requires one) and add subsequent clause(s) and function(s), such as groupby() or calculate, respectively.

  • where () = search
  • groupby () = field
  • calculate (:) = function:field

Try it out

Test these queries in Log Search and see the results for yourself.

  • where(result ISTARTS-WITH "failed" AND logon_type!=NETWORK)groupby(destination_user, destination_asset)limit(5, 1)
  • where(action=MEMBER_ADDED_TO_SECURITY_GROUP AND group IIN [admin, desktop, network, support, "service accounts"])groupby(source_user, target_account, group)
  • where(source_account==target_account)groupby(action, source_user))

To save a query for future use, click the ellipsis next to the Run button and select Save Query.

See Log Search for more information.

Visualize your data with dashboards

Dashboards allow you to build custom views of the data you want to monitor. As a starting point, either create a new dashboard or use an existing dashboard. Add, edit, resize, and rearrange the data visualization cards to tailor the data view to your needs.

Explore the IT Operations Dashboard

The IT Operations Dashboard offers a high level view of Weekly Account Lockouts and Unlocks, Most Active admin accounts on user accounts, Most Active admin accounts on their own accounts, Top 5 Directory groups by most membership adds, and more.

The IT Operations Dashboard is populated by data sent from Active Directory, Firewall, VPN, and Cloud Service event sources. To take advantage of this dashboard, be sure those event sources have been set up.

  • Active Directory event sources send Active Directory Admin Activity, Asset Authentication and Raw log data.
  • Firewall and VPN event sources send Firewall Activity and Ingress Authentication logs.
  • Cloud Service event source send Ingress Authentication logs.

To set up the IT Operations Dashboard

  1. From the InsightIDR left menu, click Dashboards and Reports.
  2. From the dashboard list, click the Dashboard Library button. The Dashboard Library modal appears.
  3. Search for the IT Operations Dashboard and click Add.

Days 46 - 90: Optimize for Success

Configure basic detection rules (formerly known as custom alerts) and create custom dashboards to tailor InsightIDR to your environment's unique needs.

Configure Basic Detection Rules

Basic detection rules allow you to create alerts directly from Log Search data. Basic detection rules can be created manually, or configured to auto-populate. You can always switch to a different rule type during configuration. There are three kinds of basic detection rules.

Log Inactivity Detection Rules

Also known as “Up Down Monitoring”, inactivity rules can be used to notify you when an entire log, log group, or particular pattern becomes inactive for a given time period.

For setup instructions, see Auto-populate a Log Inactivity Detection Rule or Manually configure a Log Inactivity Detection Rule.

Log Pattern Detection Rules

In order for a detection to trigger, a log must match the exact pattern you enter as a search term. Detecting on patterns can be useful in situations such as monitoring server errors, critical exceptions, and general performance, and allows you to only monitor events that are important to you.

For setup instructions, see Auto-populate a Log Pattern Detection Rule or Manually create a Log Pattern Detection Rule.

Log Change Detection Rules

Log change detection rules will notify you when a condition changes, such as HTTP 500 errors in your web access logs. They are based on calculations that you apply to log(s) or logset(s).

For setup instructions, see Auto-populate a Log Change Detection Rule or Manually configure a Log Change Detection Rule.

Customize your dashboards

InsightIDR's dashboards are customizable to ensure your views of your environment meet your needs. While it's easier to customize a dashboard template, you can also create a new one from scratch. Although dashboard customization is up to you, we'd like to share some key elements to explore while building your dashboards.

Edit the cards

After adding a card to your dashboard, click the gear icon in the top right corner of a card and select Edit.

  1. Name Card : Change the card's name and description to provide more clarity from the dashboard view.
  2. Select Data Source :
    • Use a previously saved query and customize the Full Query to show the data you want to highlight.
    • Change the time range for the data shown in the card.
    • Change the log set the card's data is referencing.
  3. Configure Chart : Add a caption, add data labels, change the scale, and change the colors of your card
    • Captions are a free text form to provide additional context of the card's data from the dashboard view.
    • Colors are best for differentiating between multiple groups in one card. Try editing the colors of a Stacked Bar chart!
  4. Switch Charts : You can choose the best way to visualize your card's data from the charts listed in the Switch Charts section. InsightIDR will only make the charts available that can correctly display your data.

Apply a LEQL query to an entire dashboard

By clicking the filter icon in the top menu of your dashboard, you can add a query to filter all of your dashboard's data. Check out our documentation for help building a query.

Choosing the right chart for your data

To determine the best chart to display your data, here are some example use cases for each format.

  • For examining trends or activity over time, use an Area, Line, Stacked Area, or Stacked Bar chart.
  • For comparisons across different categories, use a Bar or Bar Horizontal chart.
  • For understanding top trends and groups, use a Packed Bubble, Word Cloud, or Pie chart.
  • For a quick glance at a single data point, use a Number chart.
  • For visualizing cyclical data in relation to time, use a Radial chart.
  • For an easy-to-scan view of individual data points across categories, use a Table chart.

Size your dashboard cards

Changing the size of your cards on your dashboard helps highlight certain data views you will use often, or grabs other users' attention when viewing your dashboard for the first time.

To adjust a card's size, click and drag the bottom-right corner.

How to create and save a new custom dashboard

  1. From the left menu, click Dashboards and Reports.
  2. From the dashboard list, click the New Dashboard button. The New Dashboard modal appears.
  3. Enter a Dashboard Name and Description. The Description is optional, but it helps differentiate all of the dashboards at your organization.
  4. Click the Create Dashboard button.

Now that you’ve saved your new dashboard, it’s available in the dashboard list. To add data visualization cards to your dashboard, you’ve been directed to the Card Library. You can find, add, and customize cards for your dashboard. If you can’t find a card you want in the Card Library, build your own in the Card Builder.

See Dashboard Cards to learn how to customize your cards.

Enable File Integrity Monitoring (FIM)

File Integrity Monitoring (FIM) allows you to audit changes to critical files and folders for compliance reasons on Windows systems running agent version 2.5.3.8 or later.

When you turn on FIM, the Insight Agent starts collecting FIM events. InsightIDR can then attribute users to file modification activity. You can create alerts based on certain file log events to notify you when one of your users modifies a critical file or folder.

How to configure FIM

Deploy the Network Sensor

During your first 45 days, you deployed Insight Agents and Collectors to monitor security controls. While these components are responsible for collecting data on your assets, they do not account for network traffic, which is the data moving between your assets. To provide the network traffic visibility that you need for network traffic analysis (NTA), Rapid7 offers the Insight Network Sensor with multiple deployment options.

The Insight Network Sensor allows you to monitor, capture, and assess the end-to-end network traffic moving throughout your physical and virtual environment. InsightIDR can then leverage this network sensor data for network traffic analysis and IDS (Intrusion Detection System) alerts.

Network Sensor deployment options

There are 3 options for deploying the network sensor. All options offer network traffic visibility, but are deployed and configured differently.

  • The Insight Network Sensor deployed on a physical server
  • The Insight Network Sensor deployed on a virtual machine
  • The Insight Network Sensor for AWS, which is deployed on an EC2 instance

To deploy an Insight Network Sensor, see the Insight Network Sensor Deployment Guide.

Day 90+: Continue your InsightIDR Exploration and Education

Discovering new aspects of InsightIDR doesn't stop after your first 90 days! Rapid7 has workshops, monthly product updates, and in-product messages to keep you informed about new features and the value you can gain from them.

The Rapid7 Academy

The Rapid7 Academy holds trainings, webcasts, workshops, and more, all led by our Rapid7 experts.

  • On-demand training helps you get started with Rapid7 products, answer frequently asked questions, and recommend best practices.
  • Rapid7 Webcasts are hosted by Rapid7's product teams and provide a forum where customers can learn about best practices as well as what’s new in their Rapid7 products.
  • Virtual Instuctor-Led Training Courses are live training sessions broken down by product and available for enrollment.
  • Certification Exams are product-specific exams to help you demonstrate your knowledge of using Rapid7's solutions as a cybersecurity professional.
  • Product Workshops are Rapid7's free trainings on all things, all products, and average about an hour long.
Communications

To make sure you receive the Rapid7 communications that best suit your needs, set your communication preferences.

  • Whether it's an emergent cybersecurity threat, a product update, or a notice of service degradation for maintenance, we'll alert you with an in-product message to ensure you're aware of all that affects your environment.
  • Rapid7's research provides information on a variety of topics, such as, cloud misconfigurations, vulnerability management, detection and response, application security, and more.
  • Rapid7's blog offers conversational guidance and information from our security experts.
  • InsightIDR's Release Notes and in-product What's New guides are published at the end of each month to notify you of the product updates, improvements, and bug fixes that have occurred during the month.
Our Communities

Rapid7 supports a range of open-source projects. Consider joining one of our Open-Source communities!

  • AttackerKB captures, highlights, and expands on security researcher knowledge to shed light on the specific conditions and characteristics that make a vulnerability exploitable and useful to attackers.
  • Velociraptor provides you with the ability to more effectively respond to a wide range of digital forensic and cyber incident response investigations and data breaches.
  • Metasploit empowers and arms defenders to stay one step ahead of the game by verifying vulnerabilities, managing security assessments, and improving security awareness.
  • Recog is a framework for identifying products, services, operating systems, and hardware by matching fingerprints against data returned from various network probes.

Our customer advocacy program, Rapid7 Voice, provides you with a network of customers, offers the chance to deepen your security expertise, and provides the opportunity to share input on future product developments.

Rapid7 Support

When you run into any problems with InsightIDR, first search our help site for potential solutions. If you need additional assistance, please contact Rapid7 Support through the customer portal.