Skip to Content
NexposeHIDDEN

Headless Scan Engine

The Headless Scan Engine is a solution that can be leveraged by those with scale requirements that extend beyond the capabilities of a singular security console. The Headless Scan Engine is also useful for those interested solely in scan data.

The Headless Scan Engine is a toolkit consisting of our scan engine remote management client and Nexpose’s Content Service . The Headless Scan Engine can be used in three ways: in conjunction with the Containerized Scan Engine, through our managed services, or by manual updates and management.

It is important to note that the Headless Scan Engine is strictly a tool for commands. Updates must be made and maintained externally.

Containerized Scan Engine

Though it is not required, it is recommended to use the Containerized Scan Engine to maintain updates when working with the Headless Scan Engine.

Headless Scan Engine Architecture Overview

Prerequisites

For full use of the Headless Scan Engine’s capabilities, adhere to the following:

$ python3 -m pip install pyopenssl
Requirement already satisfied: pyopenssl in /home/user/.local/lib/python3.8/site-packages (19.1.0)
Requirement already satisfied: cryptography>=2.8 in /usr/lib/python3/dist-packages (from pyopenssl) (2.8)
Requirement already satisfied: six>=1.5.2 in /usr/lib/python3/dist-packages (from pyopenssl) (1.14.0)

For more information on the Content Service refer to the following DockerHub article .

Establish trust between the Headless Scan Engine and a scan engine

The Headless Scan Engine can be used to establish trust with a scan engine but can also maintain a list of scan engines it trusts.

With the introduction of credential support, if a malicious actor impersonated a scan engine, it’s possible that they could obtain credentials if the user of the Headless Scan Engine is unaware of the malicious actor and tries to start a scan on the impersonated scan engine that includes credentials. To solve this, the Headless Scan Engine uses a known_hosts file, a concept borrowed from OpenSSH. The known_hosts file stores the base64 encoded public certificate of trusted scan engines and will verify the public certificate before commands are sent to the scan engine. This feature can be disabled using --always-trust-host.

The Headless Scan Engine supports mutual trust establishment using a SHA256 HMAC and a shared secret. A file called shared_secret.key is created if it does not exist. If the contents of this file is shared with a scan engine, the Headless Scan Engine can establish a mutual trust and authorize itself with the scan engine. The algorithm used to establish mutual trust requires both the Headless Scan Engine and the scan engine to prove knowledge of the shared secret. This protects from malicious actors that impersonate a scan engine. The shared secret is a hex encoded 128-bit random secret.

To use the shared secret, create a consoles.xml file before starting the scan engine:

<Consoles>
  <console id=\"1\" enabled=\"1\" lastAddress=\"${CLIENT_ADDRESS}\" plaintext_sharedSecret=\"${SHARED_SECRET}\" />
</Consoles>

If the consoles.xml file already exists, add the <console ... /> tag as a child of the <Consoles> tag. The CLIENT_ADDRESS and SHARED_SECRET properties should be populated and reference the IP address of the client and the shared secret found in the shared_secret.key file.

The client will not attempt mutual trust establishment if --always-trust-host is declared.

The Headless Scan Engine can also pair with a scan engine by implied trust with just a client IP address restriction. If the plaintext_sharedSecret attribute is not included in the <console ... /> tag, the scan engine will trust the first client connecting from the CLIENT_ADDRESS IP address stored in the lastAddress attribute. For this to work, the client must also have --always-trust-host declared. This is because the scan engine is not added to the known_hosts file if no challenge was done to establish mutual trust.

Supported credential services

  • cvs
  • ftp
  • http
  • as400
  • notes
  • htmlform
  • httpheaders
  • tds
  • sybase
  • cifs
  • cifshash
  • oracle
  • mysql
  • db2
  • pop
  • postgresql
  • remote execution
  • snmp
  • snmpv3
  • ssh
  • ssh-key
  • telnet
  • kerberos

Supported credential impersonation types

  • PBRUN
  • SESU
  • DZDO
  • SUDO
  • DZDOSU
  • SUDOSU
  • SU

Credential format

The credential format uses JSON to represent multiple attributes and their associated values. The attributes supported by the credential format include:

  • service
  • username
  • password
  • domain
  • private-key
  • impersonation_type
  • impersonation_username
  • impersonation_password

The service field is required no matter the service the credential is for.

Credential format for SSH key authentication

The SSH key must be a private key in PEM format.

If the private key is in RFC4716 it can be converted with the following OpenSSH keygen command: ssh-keygen -f private_key_file -p -m PEM. After entering the command, you will be required to enter a new passphrase. The private key will be saved alongside the new passphrase in PEM format.

The private key will have multiple lines but the credential format is a single line JSON value.

The private key can be converted into a single line with the following awk command: awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' private_key_file. If the private key has a passphrase, the password attribute of the credential will be used to decrypt the private key.

Credential format examples

  • {"service": "cifs", "username": "example", "password": "example"}
  • {"service": "cifs", "username": "example", "password": "example", "domain": "example"}
  • {"service": "ssh", "username": "example", "password": "example"}
  • {"service": "ssh", "username": "example", "password": "example", "impersonation\_type": "SU"}
  • {"service": "ssh-key", "username": "example", "private\_key": "example"}
  • {"service": "ssh-key", "username": "example", "password": "example", "private\_key": "example"}

Authenticate and authorize Headless Scan Engine and scan engine communication

Example scripts are written in Python

The following scripts were tested with version 3.8.2 of Python.

Generate authentication and authorization keys

$ ./main.py --regenerate-keys
2020-06-02 13:33:40,799 [INFO] Private key not provided, using default: private.key
2020-06-02 13:33:40,799 [INFO] Public key not provided, using default: public.key
2020-06-02 13:33:40,799 [INFO] Authorization key not provided, using default: authorization.key
2020-06-02 13:33:40,984 [ERROR] Scan engine host must be provided

Connect to a scan engine with the new authentication and authorization keys

2020-06-02 13:37:53,096 [INFO] Private key not provided, using default: private.key
2020-06-02 13:37:53,096 [INFO] Public key not provided, using default: public.key
2020-06-02 13:37:53,096 [INFO] Authorization key not provided, using default: authorization.key
2020-06-02 13:37:53,797 [ERROR] Uncaught exception: xx.x.xx.xxx:xxxxx
Traceback (most recent call last):
  File "./main.py", line 144, in connect
    client = ScanEngineClient(ssl_client_socket, authorization_key_data)
  File "/home/user/eclipse-workspace/platform-tools/products/ivm/nse/api/src/ScanEngineClient.py", line 42, in __init__
    self.__negotiate_protocol_version()
  File "/home/user/eclipse-workspace/platform-tools/products/ivm/nse/api/src/ScanEngineClient.py", line 62, in __negotiate_protocol_version
    raise Exception(error_message)
Exception: Unauthorized console connection from xx.x.xx.x:xxxxx (CN=user/user-laptop, OU=InsightVM, O=Rapid7, C=US, ST=MA)

Screen into the scan engine and enable the Headless Scan Engine as if it was a console

2020-06-02T13:38:17 [INFO] > show consoles
2020-06-02T13:38:17 [INFO] <Consoles>
<console id="8" enabled="0" connectTo="0" name="CN=user/user-laptop, OU=InsightVM, O=Rapid7, ST=MA, C=US" address="xx.x.xx.x" port="40815"/>
</Consoles>
> enable console 8
2020-06-02T13:38:45 [INFO] > enable console 8
2020-06-02T13:38:45 [INFO] Console successfully enabled

Headless Scan Engine commands

Start a scan

$ ./main.py --scan-engine-host xx.x.xx.xxx --scan-start --scan-template scanTemplates/full-audit-without-web-spider.xml --scan-target-networks xx.x.xx.x/20 --scan-target-domain-names openntpd.vuln.lax.rapid7.com
2020-06-02 14:04:12,259 [INFO] Private key not provided, using default: private.key
2020-06-02 14:04:12,259 [INFO] Public key not provided, using default: public.key
2020-06-02 14:04:12,259 [INFO] Authorization key not provided, using default: authorization.key
2020-06-02 14:04:13,563 [INFO] Scan name not provided, using default: user's scan
2020-06-02 14:04:13,563 [INFO] Scan started by not provided, using default: user
2020-06-02 14:04:13,563 [INFO] Scan target networks: xx.x.xx.x/20
2020-06-02 14:04:13,563 [INFO] Scan target domain names: openntpd.vuln.lax.rapid7.com
2020-06-02 14:04:13,804 [INFO] Scan started with ID: 40

Start a scan with credentials

$ ./main.py --scan-engine-host xx.x.xx.xxx --scan-start --scan-template scanTemplates/full-audit-without-web-spider.xml --scan-target-networks xx.x.xx.x/20 --scan-credentials "{'service':'cifs', 'username':'<username>', 'password':'<password>'}" --scan-credentials "{'service':'ssh', 'username':'<username>', 'password':'<password>'}"
2020-06-08 14:02:39,745 [INFO] Private key not provided, using default: private.key
2020-06-08 14:02:39,745 [INFO] Public key not provided, using default: public.key
2020-06-08 14:02:39,745 [INFO] Authorization key not provided, using default: authorization.key
2020-06-08 14:02:41,149 [INFO] Scan name not provided, using default: user's scan
2020-06-08 14:02:41,150 [INFO] Scan started by not provided, using default: user
2020-06-02 14:04:13,563 [INFO] Scan target networks: xx.x.xx.x/20
2020-06-08 14:02:41,497 [INFO] Scan started with ID: 40

Query the status of a scan

$ ./main.py --scan-engine-host xx.x.xx.xxx --scan-status --scan-id 40
2020-06-02 14:05:53,500 [INFO] Private key not provided, using default: private.key
2020-06-02 14:05:53,500 [INFO] Public key not provided, using default: public.key
2020-06-02 14:05:53,500 [INFO] Authorization key not provided, using default: authorization.key
2020-06-02 14:05:54,844 [INFO] Scan ID 40 will be used to query scan status
2020-06-02 14:05:55,144 [INFO] Scan ID 40 status: RUNNING

Query the statistics of a scan

$ ./main.py --scan-engine-host xx.x.xx.xxx --scan-statistics --scan-id 40
2020-06-05 13:37:25,972 [INFO] Private key not provided, using default: private.key
2020-06-05 13:37:25,972 [INFO] Public key not provided, using default: public.key
2020-06-05 13:37:25,972 [INFO] Authorization key not provided, using default: authorization.key
2020-06-05 13:37:27,329 [INFO] Scan ID 40 will be used to query scan statistics
2020-06-05 13:37:27,600 [INFO] Scan ID 40 statistics:
2020-06-05 13:37:27,600 [INFO] Scan status: RUNNING
2020-06-05 13:37:27,600 [INFO] Scan start time: Wed Jun 24 15:34:04 52398
2020-06-05 13:37:27,600 [INFO] Scan threads pending: 151
2020-06-05 13:37:27,600 [INFO] Scan threads active: 29
2020-06-05 13:37:27,600 [INFO] Scan threads done: 612421
2020-06-05 13:37:27,600 [INFO] Scan live node count: 2411
2020-06-05 13:37:27,601 [INFO] Scan dead node count: 1594
2020-06-05 13:37:27,601 [INFO] Scan started node count: 2076
2020-06-05 13:37:27,601 [INFO] Scan completed node count: 2051
2020-06-05 13:37:27,601 [INFO] Scan pending node count: 0

Download scan results

$ ./main.py --scan-engine-host xx.x.xx.xxx --scan-results-download --scan-results-directory ~/tmp --scan-id 40
2020-06-02 14:11:44,101 [INFO] Private key not provided, using default: private.key
2020-06-02 14:11:44,101 [INFO] Public key not provided, using default: public.key
2020-06-02 14:11:44,102 [INFO] Authorization key not provided, using default: authorization.key
2020-06-02 14:11:45,560 [INFO] Scan ID 40 will be used to download scan results
2020-06-02 14:11:45,560 [INFO] Scan results will be downloaded into directory: /home/user/tmp
2020-06-02 14:11:45,847 [INFO] Creating file: /home/user/tmp/scan.xlog.ctl
2020-06-02 14:11:45,995 [INFO] Creating file: /home/user/tmp/scan.xlog
2020-06-02 14:11:46,388 [INFO] Creating file: /home/user/tmp/scan.log

Stop a scan

$ ./main.py --scan-engine-host xx.x.xx.xxx --scan-stop --scan-id 40
2020-06-02 14:00:54,285 [INFO] Private key not provided, using default: private.key
2020-06-02 14:00:54,285 [INFO] Public key not provided, using default: public.key
2020-06-02 14:00:54,285 [INFO] Authorization key not provided, using default: authorization.key
2020-06-02 14:00:55,620 [INFO] Scan stopped by not provided, using default: user
2020-06-02 14:00:55,620 [INFO] Scan ID 40 will be used to stop scan
2020-06-02 14:01:11,488 [INFO] Scan ID 40 stopped

Delete a scan

$ ./main.py --scan-engine-host xx.x.xx.xxx --scan-delete --scan-id 40
2020-06-02 14:49:51,541 [INFO] Private key not provided, using default: private.key
2020-06-02 14:49:51,541 [INFO] Public key not provided, using default: public.key
2020-06-02 14:49:51,541 [INFO] Authorization key not provided, using default: authorization.key
2020-06-02 14:49:52,867 [INFO] Scan ID 40 will be used to delete scan
2020-06-02 14:49:53,178 [INFO] Scan ID 40 deleted

Additional scripts

Scan management start to finish

# !/bin/sh
 
SCAN_ID=$1
 
while [ 1 ];
do
  SCAN_STATUS=$(./main.py --scan-engine-host xx.x.xx.xxx --scan-status --scan-id ${SCAN_ID} 2>&1 | grep "RUNNING")
  if [ -z "${SCAN_STATUS}" ];
  then
    ./main.py --scan-engine-host xx.x.xx.xxx --scan-results-download --scan-results-directory ~/tmp --scan-id ${SCAN_ID}
    ./main.py --scan-engine-host xx.x.xx.xxx --scan-delete --scan-id ${SCAN_ID}
    break
  fi
  ./main.py --scan-engine-host xx.x.xx.xxx --scan-results-download --scan-results-directory ~/tmp --scan-id ${SCAN_ID}
  sleep 5
done