Build custom connectors with the SDK

The Surface Command Software Development Kit (SDK), surcom-sdk, provides tools to develop Surface Command data connectors for the Rapid7 Command Platform. To learn more about connectors, see Connectors. To learn more about Surface Command and Attack Surface Management, see Get started with Attack Surface Management.

Prerequisites

• Python version 3.11 or greater
  • Using a virtual environment is recommended
• Docker fully configured and running
• Permissions  to access Attack Surface Management capabilities and create an API key in the Command Platform
• Familiarity with the Attack Surface Management type system
• Optional:
  • Integrated Developer Environment (IDE)
    • Example: Microsoft Visual Studio Code
  • 1Password

Install and configure the SDK

Complete the following tasks to set up the SDK properly:

Task 1: Create an API Key

Generate a new user API key  in the Command Platform and copy the key before closing the window. You can save the key as an API Credential in 1Password (recommended) so it can be accessed using a 1Password Secret Reference . Alternatively, you can save it as plain text (not recommended).

Task 2: Configure your local environment

1. Open a terminal.

2. Create a development directory to house your SDK development files:

   cd ~
   mkdir development
   cd development

3. Clone the Surface Command Connectors repository :

   git clone https://github.com/rapid7/r7-surcom-connectors

4. Create a Python virtual environment:

   python -m venv sdk-dev-env
   source sdk-dev-env/bin/activate

5. Install the SDK:

   python -m pip install r7-surcom-sdk

Task 3: Configure the SDK

1. Optionally, turn on tab auto complete:

   surcom config enable-tabs

2. Initialize the SDK configuration:

   surcom config init

3. Enter the path to your Connectors Workspace:

> ~/development/r7-surcom-connectors/connectors/partners

1. Enter the URL for the Rapid7 Surface Command Platform (default: us.surface.insight.rapid7.com):

   > us2.surface.insight.rapid7.com

2. Enter a name for the connection (normally the name of your Organization). Connection name should follow the format: <region>-<staging|prod|poc>-<companyname> (for example, eu-prod-examplecorp):

   > us-staging-rapid7

3. Enter the API Key. Rapid7 recommends using a 1Password Secret Reference. See the 1Password documentation  for details. Plain text API keys are not recommended as they are saved directly to the configuration file.

4. Validate the configuration:

   surcom config test

This creates a surcom_config file at ~/.r7-surcom-sdk/surcom_config with your initial connection configuration.

SDK command-line interface (CLI) reference

This section provides reference information for the surcom-sdk command-line interface (CLI). It includes usage syntax, available commands, options, and examples.

Command syntax and tree

usage: 
    $ surcom <command> ...
    $ surcom -v <command> ...
    $ surcom --version
    
 
options:
  -h, --help     show this help message and exit
  -v, --verbose  Set the log level to DEBUG
  --version      Print the version of the surcom SDK and exit
 
commands:
  
    config       Configure the surcom-sdk
    connector    Develop Connectors for the Rapid7 Surface Command Platform
    type         Manage Surface Command Types
    data         Interact with Surface Command Data

config

Configures the SDK.

Usage:

surcom config <command> [options]

Commands:

connector

Builds Connectors.

Usage:

surcom connector <command> [options]

Commands:

type

Manages Surface Command types. To learn more about the Surface Command type system, see Explore the Attack Surface Management type system.

Usage:

surcom type <command> [options]

Commands:

data

Enables interaction with Attack Surface Management data.

Usage:

surcom data <command> [options]

Commands:

Build an example connector

The following sections build an example connector that fetches data from the Reqres  service.

Task 1: Initialize the connector

1. Open a terminal.

2. Go to your connectors directory:

   cd ~/development/r7-surcom-connectors/connectors/partners

3. Initialize the connector:

   surcom connector init

4. Enter a display name (for example, My Req Res Demo):

   > My Req Res Demo

5. Enter a connector ID (for example, my.req.res.demo):

   > my.req.res.demo

6. Enter the author (for example, username):

   > username

A template connector specification file (connector.spec.yaml) is generated in your workspace (defined by the configuration file).

Task 2: Modify the connector specification file

1. Open the connector specification file (connector.spec.yaml) in the IDE of your choice. Rapid7 recommends using Microsoft Visual Studio Code.

2. Replace the template types:

   types:
       - MyReqResDemoUser
       - MyReqResDemoDevice

3. Update the import_all function:

   - id: import_all
       return_types:
           - MyReqResDemoUser
           - MyReqResDemoDevice

4. Replace the settings block:

   settings:
       url:
           type: string
           title: URL
           example: https://example.com
           default: https://reqres.in
    
       api_key:
           type: string
           title: API Key
           format: password
    
       verify_tls:
           type: boolean
           title: Verify TLS?
           description: If enabled, verify the server's identity
           default: true
    
       page_limit:
           type: integer
           title: Page Limit
           description: Maximum number of items to return per page
           enum:
               - 2
               - 4
               - 6
           default: 2
    
       # This is an optional setting that can be used to control pagination.
       total_pages:
           type: integer
           title: Total Pages
           description: >-
               Total number of pages to retrieve. Set it to 0 to retrieve all pages.
           default: 1
           nullable: true

Task 3: Generate a basic connector structure

1. Open a terminal.

2. Go to the new connector directory:

   cd ~/development/r7-surcom-connectors/connectors/partners/my_reqres_demo

3. Generate basic template code for your connector:

   surcom connector codegen

4. Confirm file structure for the connector has been created.

Task 4: Add connector functions

Functions are located in my_reqres_demo/functions. For this example, you need to complete the following updates:

Update functions/helpers.py with a new get_devices method:

# Here is an example of a simple client that interacts with a third-party API.
class MyReqResDemoClient():
 
    def __init__(
        self,
        user_log: Logger,
        settings: Settings
    ):
        # Expose the logger to the client
        self.logger = user_log
 
        # Expose the Connector Settings to the client
        self.settings = settings
 
        # Get the URL from the settings and ensure it is properly formatted
        self.base_url = settings.get("url").strip().rstrip("/")
 
        # Setup a Session using the Surcom HttpSession class
        self.session = HttpSession()
 
        # Use the value of our `verify_tls` setting to determine if we should verify TLS
        self.session.verify = settings.get("verify_tls")
 
        # Here we update the session header with the API Key from the Connector Settings.
        # Authentication methods will vary based on the third-party API. Refer to the specific
        # API documentation for details.
        self.session.headers.update({
            "x-api-key": settings.get("api_key")
        })
 
    def get_devices(self, page_number):
 
        url = f"{self.base_url}/api/devices"
 
        params = {
            "per_page": self.settings.get("page_limit"),
            "page": page_number
        }
 
        self.logger.debug("Requesting devices from '%s' with params: %s", url, params)
 
        r = self.session.get(url, params=params)
        r.raise_for_status()
 
        return r.json()

Update functions/fn_import_all.py with:

from logging import Logger
 
from . import helpers
from .sc_settings import Settings
from .sc_types import RequestResponseDemoDevice, RequestResponseDemoUser
 
 
def import_all(
    user_log: Logger,
    settings: Settings
):
    user_log.info("Getting '%s' from '%s'", RequestResponseDemoDevice.__name__, settings.get("url"))
 
    # Instantiate the RequestResponseDemoClient
    client = helpers.RequestResponseDemoClient(user_log, settings)
 
    # We set and keep track of the current page number
    current_page = 1
 
    while True:
 
        # Get some assets from the client
        r = client.get_devices(current_page)
 
        data = r.get("data", [])
 
        if data:
 
            user_log.info("Processing %d items from page %d", len(data), current_page)
 
            # For each asset in the response, yield a RequestResponseDemoDevice type to ingest
            for asset in data:
 
                # Ensure the ID is a string
                if asset.get("id"):
                    asset["id"] = str(asset["id"])
 
                yield RequestResponseDemoDevice(asset)
 
        # If this is the last page, break out
        # Otherwise, set the next page number to fetch
        if r.get("page") == r.get("total_pages"):
            user_log.debug("This is a debug log. We have reached the last page: %d", current_page)
            break
        else:
            next_page = current_page + 1
            user_log.debug("This is a debug log. Moving to the next page: %d", next_page)
            current_page = next_page

Task 5: Update your configuration, create a type, and import data

1. Go to the Reqres  website and click Get a Free API Key. Copy the key before closing the window.

   • You can save the key as an API Credential in 1Password (recommended) so it can be accessed using a 1Password Secret Reference . Alternatively, you can save it as plain text (not recommended).

2. Invoke your connector function:

   surcom connector invoke -f import_all

3. Using the invoke command for the first time prompts you for the following settings:

   • url: Accept the default (https://reqres.in).
   • api_key: Enter the 1Password Secret Reference for your API key (recommended) or a plain text string (not recommended).
   • verify_tls: Accept the default (True).
   • page_limit: Accept the default (2).
   • total_pages: Set to 0.

   This creates a new section ([connector.my.req.res.demo]) in the configuration file (default: ~/.r7-surcom-sdk/surcom_config). You can update this file or use the CLI to overwrite the settings.

4. If you used a 1Password Secret Reference, authorize 1Password. After successful authorization, the output for the command is written to the build/output directory and a terminal message confirms this.

5. Create a custom type:

   surcom type generate build/output/MyReqResDemoDevice.json 

6. Import connector data into Surface Command:

   surcom connector invoke -f import_all --import-data

7. Confirm the operation.

8. Verify the type definition is correct imported into Surface Command by opening the Workspace and running the following query:

   MATCH (r:RequestResponseDemoDevice) RETURN r

9. Validate the connector and fix any critical issues.

   surcom connector validate

Troubleshoot your connector

surcom --help           # view high-level help
surcom connector --help # view help for a sub-command

[connector.my.req.res.demo]
url = https://reqres.in
api_key = "op://Employee/REQRES/FREE API KEY"
verify_tls = True
page_limit = 2
-total_pages = 0

surcom connector invoke -f test

surcom connector invoke -f import_all -s page_limit=4,api_key=xxx