Distributed Scan Engines

Distributed Scan Engines are separate from the Security Console and are strategically provisioned and located in a way that makes your scanning environment as efficient as possible. If you intend to maintain a production deployment of the Security Console, distributed Scan Engines are an absolute necessity.

This article guides you through the deployment and configuration process for distributed Scan Engines.

Deployment Process at a Glance

Deploying a distributed Scan Engine involves the following procedures:

  • Installing a new Scan Engine on a separate host machine
  • Pairing the Scan Engine to the Security Console according to your communication method of choice
  • Refreshing the Scan Engine in the Security Console to verify that the pairing was successful

Methods of Communication

Scan Engines and Security Consoles must be able to communicate with each other in order to initiate scans and integrate scan data. Distributed Scan Engines can communicate with a Security Console in two ways:

  • Console-to-Engine - This is the standard method of communication. In this case, the Security Console directs the Scan Engine to perform a task by initiating a TCP connection over port 40814.
  • Engine-to-Console - Also known as the Reverse communication method, engine-to-console pairings rely on the Scan Engine initiating the connection to the Security Console when the engine starts.

TIP

You can read more about these configurations on the Scan Engine Communication Methods Help page.

Requirements

To ensure that you’re set up for success with a distributed Scan Engine, make sure you complete the following preliminary steps:

  • Verify that your intended Scan Engine host meets the system requirements.
  • If firewalls are present on your network, make sure you whitelist the necessary ports for your Security Console and Scan Engine host according to the communication method of your choice. Consult the following table for port whitelist requirements.

Source

Destination

Port

Protocol

Console-to-Engine

Console

Scan Engine

40814

TCP

Engine-to-Console

Engine

Console

40815

TCP

NOTE

The ports shown in this table are the default ports used by the Security Console and Scan Engine. If you modify these default ports during the deployment procedure, make sure your firewall rules match your port modifications.

Download and Install the Scan Engine

After verifying that you meet the hardware and networking requirements, you’re ready to download and install the Scan Engine on your intended host machine.

REMINDER

Do not install a distributed Scan Engine on a host that already has a Security Console!

“Distributed” Scan Engines are so named because they use hardware entirely reserved for their use. All Security Console installations already include a local Scan Engine.

NOTE - Distributed Scan Engines on virtual machines

If you intend to deploy a distributed Scan Engine on a virtual machine, ensure that you provision the virtual machine with sufficient reserved memory according to the system requirements.

Configuring a virtual machine with shared memory may cause the Scan Engine to run out of memory during a scan.

Download the Scan Engine Installer

The Nexpose product installer is also responsible for installing distributed Scan Engines. Visit the download section of the quick start guide to download the Linux or Windows installer according to the operating system of your intended host machine.

Install the Scan Engine

TIP

The following procedure applies to both the Linux command line and the Windows installation wizard.

Launch the product installer to get started. Follow the initial prompts until you reach the component selection and communication direction step.

Select Components

After going through the necessary acknowledgements, you’ll be prompted to select which components you want to install.

Select Scan Engine only. This tells the installer that you intend to deploy a distributed Scan Engine.

Select a Communication Direction

After selecting your components, you’ll be prompted to select a communication direction.

Which communication method should I use?

Deciding how your Scan Engine communicates with the Security Console ultimately depends on the configuration and topology of your network. Production deployments commonly have both Scan Engine types in place in order to accommodate scanning conditions like asset location and the presence of firewalls.

See the Scan Engine Communication Methods Help page for best practices and use case information.

If you select the Engine-to-Console method, you will have the opportunity to configure a reverse pair with your Security Console during the Scan Engine installation. Although you can skip this pairing step if you want to, Rapid7 recommends that you take advantage of this pairing opportunity since the post-install reverse pairing procedure involves more complicated steps.

To configure a reverse pair during a Scan Engine installation:

  1. When prompted by the install wizard, enter the IP address of your Security Console.

  2. Provide the installer with the Security Console shared secret.

    • You can generate a shared secret in the Security Console by navigating to Administration > Scans > Manage scan engines > Generate Scan Engine Shared Secret > Generate.

    TIP

    Multiple Scan Engines can use the same console-generated shared secret for each of their reverse pairing procedures. However, shared secrets are only valid for 60 minutes. If your shared secret expires, you must generate a new one to complete any further reverse pairing procedures.

  3. Test your connection to ensure that your Security Console and Scan Engine can communicate properly.

  4. Continue with the rest of the Scan Engine installation.

  5. After your Scan Engine finishes installing, proceed directly to the Refresh Your New Scan Engine section of this guide.

If you select the Console-to-Engine method, you’ll need to configure a standard pair with your Security Console after the Scan Engine installation completes. Continue with the rest of the installation at this time. After your Scan Engine finishes installing, proceed to the Pair Your Scan Engine to the Security Console section of this guide.

Pair Your Scan Engine to the Security Console

NOTE

Make sure your new Scan Engine is running and reachable before proceeding with a post-installation pairing procedure. You must also have admin-level access to your Scan Engine host to complete these pairing procedures.

All new Scan Engines must be paired to the Security Console in order to be usable for scanning. These engine pairing procedures differ based on the method of communication you want to implement.

Consult one of the following pairing procedures for your communication method of choice:

Standard Pair (Console-to-Engine)

In order to configure a console-to-engine pairing, the Security Console must be made aware that a new Scan Engine is available for use and must be provided with instructions on how to reach it. Consequently, the first step of all standard pairing procedures is to add your new Scan Engine to the Security Console.

You can add a new Scan Engine within a site configuration or through the Administration page.

Add an Engine in a Site Configuration

To add a Scan Engine in a site configuration:

  1. In a new or existing site configuration, click the Engines tab.
    • You can create a new site by expanding the Create dropdown next to the left navigation menu, or access an existing site by clicking its edit icon in the “Sites” table of the Home page.
  2. Click the Add Scan Engine subtab.
  3. Name your Scan Engine.
  4. Enter the IP address of your Scan Engine in the Address field.
  5. If you need to adjust the default port number, modify the value shown in the Port field.
  6. Assuming your site has the minimum set of required information, click Save when finished.

Add an Engine through Administration

To add a Scan Engine through the Administration tab:

  1. On the Administration page, in the Scans > Scan Engines section, click Manage scan engines.
  2. Click New Engine.
  3. On the General tab, name your Scan Engine.
  4. Enter the IP address of your Scan Engine in the “Address” field.
  5. Click Save when finished.

Properly added Scan Engines generate a consoles.xml file on the Scan Engine host. You will modify this file in the next step.

Modify consoles.xml

The consoles.xml file generated on your Scan Engine host in the previous step contains an entry for the Security Console that added the Scan Engine. You must enable the console to complete the pairing.

To modify the consoles.xml file for a Linux or Windows host:

  1. Browse to the consoles.xml file in your Scan Engine directory. By default, this file is located in the following places according to the operating system of your Scan Engine host:
    • Linux - /opt/rapid7/nexpose/nse/conf/consoles.xml
    • Windows - Files\Rapid7\NeXpose\nse\conf\consoles.xml
  2. Open consoles.xml with a text editor of your choice. Navigate to the <consoles> element. The Security Console that added the Scan Engine appears as a <console> element with several attributes.
    • You can identify the correct Security Console by checking that the lastAddress attribute matches the IP address of the Security Console you want to pair with.
  3. Change the value for the enabled attribute from 0 to 1.
  4. Save and close the consoles.xml file.
  5. Restart the Scan Engine host so your changes can take effect.

Pairing procedure complete!

Proceed to the verification step to finish your Scan Engine deployment.

Reverse Pair (Engine-to-Console)

If you took advantage of the reverse pairing configuration opportunity during your Scan Engine installation, then you’ve already completed this step! Proceed directly to the Refresh Your New Scan Engine section of this guide to verify that your Scan Engine is ready for use.

However, if you installed a Scan Engine with the Engine-to-Console method selected without completing the reverse pairing step, you must complete the pairing with a separate procedure. See the Post-Installation Engine-to-Console Pairing page for instructions on how to do this.

Refresh Your New Scan Engine

After completing a standard or reverse pair for your Scan Engine, you must refresh its status to verify that the Security Console can communicate with it properly.

To refresh your new Scan Engine in the Security Console:

  1. On the Administration page, click Scans > Engines.
  2. In the Scan Engines section, click Refresh Displayed Engines. This ensures that your Scan Engine table is up-to-date.
  3. Locate the distributed Scan Engine that you paired to the Security Console. Click the icon in the “Refresh” column to complete the verification process.

If you have properly configured and paired your Scan Engine, it now displays up-to-date version and communication status information. The “Communication Status” column itself indicates both the current communication method by arrow and connection state by color. Arrows pointing to “Engine” indicate a standard pairing, while arrows pointing to “Console” indicate reverse pairing. Additionally, arrow icons can have the following color codes:

  • Green - Scan Engine is active
  • Orange - Scan Engine status is unknown
    • An “unknown” status indicates that the Security Console and the Scan Engine could not communicate even though no error was recorded. This is often the result of a significant lapse between pings. Refresh the Scan Engine status to attempt communication again.
  • Red - There are three possibilities associated with this color code:
    • Pending Authorization - This state indicates that the Security Console has not yet been enabled on the Scan Engine. Consoles must be enabled on the Scan Engine during the pairing procedure.
    • Incompatible - This state indicates that the Security Console and Scan Engine are on different versions. Consoles and engines must be on the same version in order to communicate properly.
    • Down - This state indicates that the Scan Engine is offline.

Congratulations!

You should now have successfully deployed a distributed Scan Engine.