Build custom connectors with the SDK

The Surface Command Software Development Kit (SDK), surcom-sdk, provides tools to develop Surface Command data connectors for the Rapid7 Command Platform. To learn more about connectors, see Connectors. To learn more about Surface Command and Attack Surface Management, see Get started with Attack Surface Management.

Prerequisites

• Python version 3.11 or greater
  • Using a virtual environment is recommended
• Docker fully configured and running
• Permissions  to access Attack Surface Management capabilities and create an API key in the Command Platform
• Familiarity with the Attack Surface Management type system
• Optional:
  • Integrated Developer Environment (IDE)
    • Example: Microsoft Visual Studio Code
  • 1Password

Install and configure the SDK

Complete the following tasks to set up the SDK properly:

Task 1: Create an API Key

Generate a new user API key  in the Command Platform and copy the key before closing the window. You can save the key as an API Credential in 1Password (recommended) so it can be accessed using a 1Password Secret Reference . Alternatively, you can save it as plain text (not recommended).

Task 2: Configure your local environment

1. Open a terminal.

2. Create a development directory to house your SDK development files:

   cd ~
   mkdir development
   cd development

3. Clone the Surface Command Connectors repository :

   git clone https://github.com/rapid7/r7-surcom-connectors

4. Create a Python virtual environment:

   python -m venv sdk-dev-env
   source sdk-dev-env/bin/activate

5. Install the SDK:

   python -m pip install r7-surcom-sdk

Task 3: Configure the SDK

1. Optionally, turn on tab auto complete:

   surcom config enable-tabs

2. Initialize the SDK configuration:

   surcom config init

3. Enter the path to your Connectors Workspace:

> ~/development/r7-surcom-connectors/connectors/partners

1. Enter the URL for the Rapid7 Surface Command Platform (default: us.surface.insight.rapid7.com):

   > us2.surface.insight.rapid7.com

2. Enter a name for the connection (normally the name of your Organization). Connection name should follow the format: <region>-<staging|prod|poc>-<companyname> (for example, eu-prod-examplecorp):

   > us-staging-rapid7

3. Enter the API Key. Rapid7 recommends using a 1Password Secret Reference. See the 1Password documentation  for details. Plain text API keys are not recommended as they are saved directly to the configuration file.

4. Validate the configuration:

   surcom config test

This creates a surcom_config file at ~/.r7-surcom-sdk/surcom_config with your initial connection configuration.

SDK command-line interface (CLI) reference

This section provides reference information for the surcom-sdk command-line interface (CLI). It includes usage syntax, available commands, options, and examples.

Command syntax and tree

usage: 
    $ surcom <command> ...
    $ surcom -v <command> ...
    $ surcom --version
    
 
options:
  -h, --help     show this help message and exit
  -v, --verbose  Set the log level to DEBUG
  --version      Print the version of the surcom SDK and exit
 
commands:
  
    config       Configure the surcom-sdk
    connector    Develop Connectors for the Rapid7 Surface Command Platform
    type         Manage Surface Command Types
    data         Interact with Surface Command Data

config

Configures the SDK.

Usage:

surcom config <command> [options]

Commands:

connector

Builds Connectors.

Usage:

surcom connector <command> [options]

Commands:

type

Manages Surface Command types. To learn more about the Surface Command type system, see Explore the Attack Surface Management type system.

Usage:

surcom type <command> [options]

Commands:

data

Enables interaction with Attack Surface Management data.

Usage:

surcom data <command> [options]

Commands:

Build an example connector

The following sections build an example connector that fetches data from the MockServer  that is loaded with the SDK.

Task 1: Initialize the connector

1. Open a terminal.

2. Go to your connectors directory:

   cd ~/development/r7-surcom-connectors/connectors/partners

3. Initialize the connector:

   surcom connector init

4. Enter a display name (for example, Demo Connector):

   > Demo Connector

5. Enter a connector ID (for example, demo.connector):

   > demo.connector

6. Enter the author (for example, username):

   > username

A template connector specification file (connector.spec.yaml) is generated in your workspace (defined by the configuration file).

Task 2: Modify the connector specification file

1. Open the connector specification file (connector.spec.yaml) in the IDE of your choice. Rapid7 recommends using Microsoft Visual Studio Code.

2. Replace the template types:

   types:
       - DemoConnectorDevice
       - DemoConnectorUser

3. Update the import_all function:

   - id: import_all
       title: Import All
       description: Import all devices and users from Demo Connector
       return_types:
           - DemoConnectorDevice
           - DemoConnectorUser

4. Replace the settings block:

   settings:
       url:
           type: string
           title: URL
           description: >-
               Base URL of where the MockServer is hosted when you run it using the surcom-sdk.
           example: http://host.docker.internal:1080
           default: http://host.docker.internal:1080
    
       api_key:
           type: string
           title: API Key
           description: >-
               API key for the MockServer that is provided in the instructions.
               for this tutorial.
           format: password
    
       verify_tls:
           type: boolean
           title: Verify TLS?
           description: If enabled, verify the server's identity.
           default: true
    
       page_limit:
           type: integer
           title: Page Limit
           description: Maximum number of items to return per page.
           enum:
               - 2
               - 4
               - 6
           default: 6
    
       # This is an optional setting that can be used to control pagination.
       total_pages:
           type: integer
           title: Total Pages
           description: >-
               Total number of pages to retrieve. Set it to 0 to retrieve all pages.
           default: 2
           nullable: true

Task 3: Generate a basic connector structure

1. Open a terminal.

2. Go to the new connector directory:

   cd ~/development/r7-surcom-connectors/connectors/partners/demo_connector

3. Generate basic template code for your connector:

   surcom connector codegen

4. Confirm file structure for the connector has been created.

Task 4: Add connector functions

Functions are located in demo_connector/functions. For this example, you need to complete the following updates:

Update functions/helpers.py with new get_devices, get_users, and get_permissions methods and add a REQUIRED_PERMISSIONS list:

REQUIRED_PERMISSIONS = [
    "readUsers",
    "readDevices"
]
 
 
# Here is an example of a simple client that interacts with a third-party API.
class DemoConnectorClient():
 
    def __init__(
        self,
        user_log: Logger,
        settings: Settings
    ):
        # Expose the logger to the client
        self.logger = user_log
 
        # Expose the Connector Settings to the client
        self.settings = settings
 
        # Get the URL from the settings and ensure it is properly formatted
        self.base_url = settings.get("url").strip().rstrip("/")
 
        # Setup a Session using the Surcom HttpSession class
        self.session = HttpSession()
 
        # Use the value of our `verify_tls` setting to determine if we should verify TLS
        self.session.verify = settings.get("verify_tls")
 
        # Here we update the session header with the API Key from the Connector Settings.
        # Authentication methods will vary based on the third-party API. Refer to the specific
        # API documentation for details.
        self.session.headers.update({
            "x-api-key": settings.get("api_key")
        })
 
    def get_devices(
        self,
        page_number: int = -1,
    ):
 
        url = f"{self.base_url}/api/devices"
 
        params = {
            "page": page_number,
            "items_per_page": self.settings.get("page_limit", -1)
        }
 
        self.logger.debug("Requesting devices from '%s' with params: %s", url, params)
 
        r = self.session.get(url, params=params)
        r.raise_for_status()
 
        return r.json()
 
    def get_permissions(self):
 
        url = f"{self.base_url}/api/permissions"
 
        self.logger.debug("Calling permissions endpoint '%s'", url)
 
        r = self.session.get(url)
        r.raise_for_status()
 
        return r.json()
 
    def get_users(
        self,
        page_number: int = -1,
    ):
 
        url = f"{self.base_url}/api/users"
 
        params = {
            "page": page_number,
            "items_per_page": self.settings.get("page_limit", -1)
        }
 
        self.logger.debug("Requesting users from '%s' with params: %s", url, params)
 
        r = self.session.get(url, params=params)
        r.raise_for_status()
 
        return r.json()
 

Update functions/fn_import_all.py with:

from logging import Logger
from typing import Callable
 
 
from . import helpers
from .sc_settings import Settings
from .sc_types import DemoConnectorDevice, DemoConnectorUser
 
 
def _get_asset(
    user_log: Logger,
    settings: Settings,
    api_method: Callable,
    surcom_type: dict
):
    user_log.info("Getting '%s' from '%s'", surcom_type.__name__, settings.get("url"))
 
    # We set and keep track of the current page number
    current_page = 1
 
    while True:
 
        # Get some assets from the client
        r = api_method(current_page)
 
        # If we don't have a response or the response doesn't contain data, we log a warning and break
        if not r or not r.get("data") or not r.get("page"):
            user_log.warning("No data found in response for page %d. Response: %s", current_page, r)
            break
 
        data = r.get("data", [])
 
        if data:
 
            user_log.info("Processing %d items from page %d", len(data), current_page)
 
            # For each asset in the response, yield a Surcom type to ingest
            for asset in data:
 
                # Ensure the ID is a string
                if asset.get("id"):
                    asset["id"] = str(asset["id"])
 
                yield surcom_type(asset)
 
        # We check if we have reached the last page
        if r.get("page") == r.get("total_pages"):
            user_log.debug("Reached the last page: %d", current_page)
            break
 
        # If the total pages setting is > 0, we check if we have reached it
        elif settings.get("total_pages") and r.get("page") >= settings.get("total_pages"):
            user_log.info("Not getting any more pages")
            break
 
        # Else, we increment the page number and continue
        next_page = current_page + 1
        user_log.debug("Moving to the next page: %d", next_page)
        current_page = next_page
 
 
def import_all(
    user_log: Logger,
    settings: Settings
):
    # Instantiate the DemoConnectorClient
    client = helpers.DemoConnectorClient(user_log, settings)
 
    # Import all DemoConnectorDevice
    yield from _get_asset(
        user_log=user_log,
        settings=settings,
        api_method=client.get_devices,
        surcom_type=DemoConnectorDevice
    )
 
    # Import all DemoConnectorUser
    yield from _get_asset(
        user_log=user_log,
        settings=settings,
        api_method=client.get_users,
        surcom_type=DemoConnectorUser
    )
 

Task 5: Update your configuration, create a type, and import data

1. Use the SDK to start MockServer and load the tutorial:

   surcom mockserver load-tutorial

2. Invoke your connector function:

   surcom connector invoke -f import_all

3. Using the invoke command for the first time prompts you for the following settings:

   • url: Accept the default (http://host.docker.internal:1080).
   • api_key: Enter the 1Password Secret Reference for your API key (recommended) or enter surcom-demo-connector-api-key as a plain text string (not recommended).
   • verify_tls: Accept the default (True).
   • page_limit: Accept the default (6).
   • total_pages: Set to 2.

   This creates a new section ([connector.demo.connector]) in the configuration file (default: ~/.r7-surcom-sdk/surcom_config). You can update this file or use the CLI to overwrite the settings.

4. If you used a 1Password Secret Reference, authorize 1Password. After successful authorization, the output for the command is written to the build/output directory and a terminal message confirms this.

5. Create a custom type:

   surcom type generate build/output/DemoConnectorDevice.json 

6. Import connector data into Surface Command:

   surcom connector invoke -f import_all --import-data

7. Confirm the operation.

8. Verify the type definition is correctly imported into Surface Command by opening the Workspace and running the following query:

   MATCH (d:DemoConnectorDevice) RETURN d

9. Now implement the test function in functions/fn_test.py by following the source code in our repository . This function is invoked when you click the Test Connection button in the Surface Command UI.

10. Validate the connector and fix any critical issues.

    surcom connector validate

Troubleshoot your connector

surcom --help           # view high-level help
surcom connector --help # view help for a sub-command

[connector.demo.connector]
url = http://host.docker.internal:1080
api_key = "op://Employee/DEMO/SURCOM DEMO API KEY"
verify_tls = True
page_limit = 6
-total_pages = 2

surcom connector invoke -f test

surcom connector invoke -f import_all -s page_limit=4,api_key=xxx