Plugin Tooling

The tooling is intended to be used to develop new or existing plugins.

Tooling

You can find a list of open-source plugins to reference, add a new feature, or fix a bug. We welcome and encourage pull requests!

Insight Plugin

This is the main tool for developing plugins. Installation instructions for insight-plugin is available in the Getting Started document.

Make

The make command constructs artifacts for plugin development.

Arguments

make image is used to package the plugin code into a Docker image which makes it available to run. This is typically the first thing you want to do when you’re working on an existing plugin to ensure it builds successfully.


❯ make image
[*] Use ``make menu`` for available targets
[*] Including available Makefiles: ../../tools/Makefiles/Colors.mk ../../tools/Makefiles/Helpers.mk
--
[*] Building plugin image
docker build --pull -t rapid7/base64:1.1.6 .

make tarball is used to build a tarball of the plugin. This is used for backward-compatible testing in Automation (InsightConnect).


❯ make tarball
INFO[0000] Building tarball rapid7-base64-1.1.1.tar.gz

Create

The subcommand create creates all the necessary plugin files and directories.

It can be used to generate a new plugin from a plugin spec (Plugin Spec) with the path of the spec as an argument.

insight-plugin create plugin.spec.yaml

Create examples

Command	Description
`insight-plugin create plugin.spec.yaml`	Create plugin files


❯ insight-plugin create plugin.spec.yaml
Create process complete!

Refresh

Anytime you modify the plugin spec, use the subcommand refresh to rebuild the skeleton with the new schema without compromising previously written code. This makes adding actions and updating the plugin spec simple and fast.

insight-plugin refresh

Refresh Examples

Command	Description
`insight-plugin refresh`	Refresh plugin files


❯ insight-plugin refresh
Refresh process complete!

Samples

The samples command is used to create the json test files that we use to test actions and triggers against the plugin docker image. Once you build out the plugin using make or make image, create the json test files using: insight-plugin samples <path/to/specfile>

Sample Examples

Command	Description
`insight-plugin samples`	Generate all samples
`insight-plugin samples {target_component}`	Generate a sample for a specific component


❯ insight-plugin samples
Samples process complete!

❯ insight-plugin samples encode
WARNING: SamplesUtil: tests/encode.json already exists, overwriting..
Samples process complete!

Run

The subcommand run for insight-plugin allows a user to quickly test and execute plugins from the command-line.

See the Running Plugins page for more detail. Below are a few examples.

Run Examples

Command	Description
`insight-plugin run tests/{test}.json`	Runs the test file test.json
`insight-plugin run -T`	Runs the connection test. Uses the first JSON file passed or first in the tests directory.
`insight-plugin run tests/{test}.json -A`	Execute the run method on the submit JSON test file for a printed assessment to post in a PR on our open-source Github repository
`insight-plugin run`	Execute the test and run methods for all JSON tests
`insight-plugin run tests/{test}.json --rebuild`	Rebuild the docker container before running the sample file


❯ insight-plugin run tests/encode.json
2023-08-22 14:41:40 [info     ] rapid7/Base64:1.1.6. Step name: encode

{"body": {"log": "", "status": "ok", "meta": {}, "output": {"data": "UmFwaWQ3"}}, "version": "v1", "type": "action_event"}

Server

You can also interact with the plugin over HTTP when it’s in REST mode:

insight-plugin server

Server Examples

Command	Description
`insight-plugin server`	Run plugin docker container in http mode
`insight-plugin server --rebuild`	Rebuild before running plugin docker container in http mode
`insight-plugin server -p 80:8080`	List of ports to expose to other containers and scripts
`insight-plugin server -V /Absolute/src:/Absolute/dest`	Volume to mount from local machine to container


❯ insight-plugin server
[2023-08-22 14:39:58 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2023-08-22 14:39:58 +0000] [1] [INFO] Listening at: http://0.0.0.0:10001 (1)
[2023-08-22 14:39:58 +0000] [1] [INFO] Using worker: threads
[2023-08-22 14:39:58 +0000] [8] [INFO] Booting worker with pid: 8

Then you can hit its run or test endpoint with a web request containing the JSON input as payload:


curl -d @tests/test.json http://127.0.0.1:10001/actions/test

Shell

You can also interact with the plugin via command line by running the plugin and Docker and entering the shell:

insight-plugin shell

Shell Examples

Command	Description
`insight-plugin shell`	Runs the plugin in Docker so that it may be accessed locally.
`insight-plugin shell -rebuild`	Rebuilds the Docker image before running it in a container and entering the shell.
`insight-plugin shell -V /Absolute/src:/Absolute/dest`	Volume to mount from local machine to container


❯ insight-plugin shell
$ pwd
/python/src
$ ls
Dockerfile  cisco_umbrella_destinations_rapid7_plugin.egg-info icon.png     setup.py
Makefile    dist      icon_cisco_umbrella_destinations  venv
bin     extension.png     plugin.spec.yaml
build     help.md      requirements.txt
$

Validate

The subcommand validate will run validation against the plugin, ensuring that all files including schema, plugin.spec.yaml and help.md are valid to the constraints applied by Automation (InsightConnect).

insight-plugin validate

Validate Examples

Command	Description
`insight-plugin validate`	Validate / Run checks against the plugin


❯ insight-plugin validate
[*] Running Integration Validators...
[*] Executing validator HelpValidator
[*] Executing validator ChangelogValidator
[*] Executing validator CloudReadyConnectionCredentialTokenValidator
[*] Executing validator RequiredKeysValidator
[*] Executing validator UseCaseValidator
[*] Executing validator SpecPropertiesValidator
[*] Executing validator SpecVersionValidator
[*] Executing validator FilesValidator
[*] Executing validator TagValidator
[*] Executing validator DescriptionValidator
[*] Executing validator TitleValidator
[*] Executing validator VendorValidator
[*] Executing validator DefaultValueValidator
[*] Executing validator IconValidator
[*] Executing validator RequiredValidator
[*] Executing validator VersionValidator
[*] Executing validator DockerfileParentValidator
[*] Executing validator ProfanityValidator
[*] Executing validator AcronymValidator
[*] Executing validator JSONValidator
[*] Executing validator OutputValidator
[*] Executing validator RegenerationValidator
[*] Executing validator HelpInputOutputValidator
[*] Executing validator SupportValidator
[*] Executing validator RuntimeValidator
[*] Executing validator VersionPinValidator
[*] Executing validator EncodingValidator
[*] Executing validator ExampleInputValidator
[*] Executing validator CloudReadyValidator
[*] Executing validator SupportedVersionValidator
[*] Executing validator UnapprovedKeywordsValidator
WARNING: Unsupported keywords found: vmware, cloud. The following keywords will not be searchable by the Extension Library. Please remove or update the invalid keywords from the keywords array in the plugin.spec.yaml file.
[*] Executing validator HelpExampleValidator
[*] Executing validator Version Increment Validator
[*] Executing validator NameValidator
[*] Executing validator ExceptionValidator
[*] Executing validator CredentialsValidator
[*] Executing validator PasswordValidator
[*] Executing validator PrintValidator
[*] Executing validator ConfidentialValidator
[*] Executing validator DockerValidator
[*] Executing validator URLValidator

Export

The subcommand export will export the Docker image as a .tar file which you can import into Automation (InsightConnect). The output of this command is what you will import into Automation (InsightConnect) as a Custom Plugin to begin using it. Note that this command will build the plugin first before exporting it.

insight-plugin export

Export Examples

Command	Description
`insight-plugin export`	Export a plugin Docker image to a tarball
`insight-plugin export --no-pull`	Flag to skip pulling the latest SDK base image during plugin export


❯ insight-plugin export
Beginning export..
Export process complete!

Analysis

This subcommand runs the static code analysis check on the plugin to analyse the source code for quality, reliability and security without executing the code. The plugin must succeed with no errors before it can be contributed to the public repo.


❯ insight-plugin analysis
Messages
========
 
komand_base64/actions/decode/action.py
  Line: 26
    pylint: logging-too-many-args / Too many arguments for logging format string (col 12)
 
komand_base64/connection/connection.py
  Line: 9
    pylint: signature-differs / Signature differs from overridden 'connect' method (col 4)
 
 
 
Check Information
=================
         Started: 2024-05-13 10:19:20.916095
        Finished: 2024-05-13 10:19:23.286805
      Time Taken: 2.37 seconds
       Formatter: grouped
        Profiles: /Users/comcnally/dev/insightconnect-plugins/prospector.yaml, no_doc_warnings, no_member_warnings, no_test_warnings, strictness_medium, strictness_high, strictness_veryhigh
      Strictness: from profile
  Libraries Used:
       Tools Run: bandit, mccabe, pyflakes, pylint
  Messages Found: 2

Checks

This subcommand runs the analysis and validate commands against the plugin alongside running the black package to refactor the codebase if needed. This is useful for mocking the same checks that are usually run against the plugins via GitHub actions. Especially useful for 3rd party contributors.


❯ insight-plugin checks
 
Linter
 
All done! ✨ 🍰 ✨
4 files left unchanged.
 
Analysis
 
Messages
========
 
komand_base64/actions/decode/action.py
  Line: 26
    pylint: logging-too-many-args / Too many arguments for logging format string (col 12)
 
komand_base64/connection/connection.py
  Line: 9
    pylint: signature-differs / Signature differs from overridden 'connect' method (col 4)
 
 
 
Check Information
=================
         Started: 2024-05-13 10:40:06.302222
        Finished: 2024-05-13 10:40:08.493606
      Time Taken: 2.19 seconds
       Formatter: grouped
        Profiles: /Users/comcnally/dev/insightconnect-plugins/prospector.yaml, no_doc_warnings, no_member_warnings, no_test_warnings, strictness_medium, strictness_high, strictness_veryhigh
      Strictness: from profile
  Libraries Used:
       Tools Run: bandit, mccabe, pyflakes, pylint
  Messages Found: 2
 
 
Validate
 
[*] Running Integration Validators...
[*] Executing validator HelpValidator
[*] Executing validator ChangelogValidator
[*] Executing validator CloudReadyConnectionCredentialTokenValidator
[*] Executing validator RequiredKeysValidator
[*] Executing validator UseCaseValidator
[*] Executing validator SpecPropertiesValidator
[*] Executing validator SpecVersionValidator
[*] Executing validator FilesValidator
[*] Executing validator TagValidator
[*] Executing validator DescriptionValidator
[*] Executing validator TitleValidator
[*] Executing validator VendorValidator
[*] Executing validator DefaultValueValidator
[*] Executing validator IconValidator
[*] Executing validator RequiredValidator
[*] Executing validator VersionValidator
[*] Executing validator DockerfileParentValidator
[*] Executing validator ProfanityValidator
[*] Executing validator AcronymValidator
[*] Executing validator JSONValidator
[*] Executing validator OutputValidator
[*] Executing validator RegenerationValidator
[*] Executing validator HelpInputOutputValidator
[*] Executing validator SupportValidator
[*] Executing validator RuntimeValidator
[*] Executing validator VersionPinValidator
[*] Executing validator EncodingValidator
[*] Executing validator ExampleInputValidator
[*] Executing validator CloudReadyValidator
[*] Executing validator SupportedVersionValidator
[*] Executing validator UnapprovedKeywordsValidator
WARNING: Unsupported keywords found: utilities. The following keywords will not be searchable by the Extension Library. Please remove or update the invalid keywords from the keywords array in the plugin.spec.yaml file.
[*] Executing validator HelpExampleValidator
[*] Executing validator Version Increment Validator
[*] Executing validator NameValidator
[*] Executing validator ExceptionValidator
[*] Executing validator CredentialsValidator
[*] Executing validator PasswordValidator
[*] Executing validator PrintValidator
[*] Executing validator ConfidentialValidator
[*] Executing validator DockerValidator

JSON to Spec

A script called json_to_spec.py converts a JSON object to Automation (InsightConnect) plugin spec data. It takes a path to a file containing JSON as an argument. It’s located in the open-source plugins repo .


$ curl -s https://ifconfig.co/json > ifconfig.json
$ tools/json_to_spec.py ifconfig.json
types: {}
 
output:
  city:
    description: ENTER DESCRIPTION
    title: ENTER TITLE
    type: string
  hostname:
    description: ENTER DESCRIPTION
    title: ENTER TITLE
    type: string
  ip:
    description: ENTER DESCRIPTION
    title: ENTER TITLE
    type: string
  country_iso:
    description: ENTER DESCRIPTION
    title: ENTER TITLE
    type: string
  ip_decimal:
    description: ENTER DESCRIPTION
    title: ENTER TITLE
    type: integer
  country:
    description: ENTER DESCRIPTION
    title: ENTER TITLE
    type: string

You can then copy and paste the output into your plugin spec rather than having to type it out manually.