Plugin Tooling

The tooling is intended to be used to develop new or existing plugins.

Tooling

You can find a list of open-source plugins to reference, add a new feature, or fix a bug. We welcome and encourage pull requests!

Insight Plugin

This is the main tool for developing plugins. Installation instructions for insight-plugin is available in the Getting Started document.

Make

The make command constructs artifacts for plugin development.

Arguments

make image is used to package the plugin code into a Docker image which makes it available to run. This is typically the first thing you want to do when you’re working on an existing plugin to ensure it builds successfully.

❯ make image [*] Use ``make menu`` for available targets [*] Including available Makefiles: ../../tools/Makefiles/Colors.mk ../../tools/Makefiles/Helpers.mk -- [*] Building plugin image docker build --pull -t rapid7/base64:1.1.6 .

make tarball is used to build a tarball of the plugin. This is used for backward-compatible testing in InsightConnect.

❯ make tarball INFO[0000] Building tarball rapid7-base64-1.1.1.tar.gz

Create

The subcommand create creates all the necessary plugin files and directories.

It can be used to generate a new plugin from a plugin spec (Plugin Spec) with the path of the spec as an argument.

insight-plugin create plugin.spec.yaml

Create examples

CommandDescription
insight-plugin create plugin.spec.yamlCreate plugin files
insight-plugin create plugin.spec.yaml Create process complete!

Refresh

Anytime you modify the plugin spec, use the subcommand refresh to rebuild the skeleton with the new schema without compromising previously written code. This makes adding actions and updating the plugin spec simple and fast.

insight-plugin refresh

Refresh Examples

CommandDescription
insight-plugin refreshRefresh plugin files
insight-plugin refresh Refresh process complete!

Samples

The samples command is used to create the json test files that we use to test actions and triggers against the plugin docker image. Once you build out the plugin using make or make image, create the json test files using: insight-plugin samples <path/to/specfile>

Sample Examples

CommandDescription
insight-plugin samplesGenerate all samples
insight-plugin samples {target_component}Generate a sample for a specific component
❯ insight-plugin samples Samples process complete! ❯ insight-plugin samples encode WARNING: SamplesUtil: tests/encode.json already exists, overwriting.. Samples process complete!

Run

The subcommand run for insight-plugin allows a user to quickly test and execute plugins from the command-line.

See the Running Plugins page for more detail. Below are a few examples.

Run Examples

CommandDescription
insight-plugin run tests/{test}.jsonRuns the test file test.json
insight-plugin run -TRuns the connection test. Uses the first JSON file passed or first in the tests directory.
insight-plugin run tests/{test}.json -AExecute the run method on the submit JSON test file for a printed assessment to post in a PR on our open-source Github repository
insight-plugin runExecute the test and run methods for all JSON tests
insight-plugin run tests/{test}.json --rebuildRebuild the docker container before running the sample file
❯ insight-plugin run tests/encode.json 2023-08-22 14:41:40 [info ] rapid7/Base64:1.1.6. Step name: encode {"body": {"log": "", "status": "ok", "meta": {}, "output": {"data": "UmFwaWQ3"}}, "version": "v1", "type": "action_event"}

Server

You can also interact with the plugin over HTTP when it’s in REST mode:

insight-plugin server

Server Examples

CommandDescription
insight-plugin serverRun plugin docker container in http mode
insight-plugin server --rebuildRebuild before running plugin docker container in http mode
insight-plugin server -p 80:8080List of ports to expose to other containers and scripts
insight-plugin server -V /Absolute/src:/Absolute/destVolume to mount from local machine to container
❯ insight-plugin server [2023-08-22 14:39:58 +0000] [1] [INFO] Starting gunicorn 20.0.4 [2023-08-22 14:39:58 +0000] [1] [INFO] Listening at: http://0.0.0.0:10001 (1) [2023-08-22 14:39:58 +0000] [1] [INFO] Using worker: threads [2023-08-22 14:39:58 +0000] [8] [INFO] Booting worker with pid: 8

Then you can hit its run or test endpoint with a web request containing the JSON input as payload:

curl -d @tests/test.json http://127.0.0.1:10001/actions/test

Shell

You can also interact with the plugin via command line by running the plugin and Docker and entering the shell:

insight-plugin shell

Shell Examples

CommandDescription
insight-plugin shellRuns the plugin in Docker so that it may be accessed locally.
insight-plugin shell -rebuildRebuilds the Docker image before running it in a container and entering the shell.
insight-plugin shell -V /Absolute/src:/Absolute/destVolume to mount from local machine to container
insight-plugin shell $ pwd /python/src $ ls Dockerfile cisco_umbrella_destinations_rapid7_plugin.egg-info icon.png setup.py Makefile dist icon_cisco_umbrella_destinations venv bin extension.png plugin.spec.yaml build help.md requirements.txt $

Validate

The subcommand validate will run validation against the plugin, ensuring that all files including schema, plugin.spec.yaml and help.md are valid to the constraints applied by InsightConnect.

insight-plugin validate

Validate Examples

CommandDescription
insight-plugin validateValidate / Run checks against the plugin
insight-plugin validate [*] Running Integration Validators... [*] Executing validator HelpValidator [*] Executing validator ChangelogValidator [*] Executing validator CloudReadyConnectionCredentialTokenValidator [*] Executing validator RequiredKeysValidator [*] Executing validator UseCaseValidator [*] Executing validator SpecPropertiesValidator [*] Executing validator SpecVersionValidator [*] Executing validator FilesValidator [*] Executing validator TagValidator [*] Executing validator DescriptionValidator [*] Executing validator TitleValidator [*] Executing validator VendorValidator [*] Executing validator DefaultValueValidator [*] Executing validator IconValidator [*] Executing validator RequiredValidator [*] Executing validator VersionValidator [*] Executing validator DockerfileParentValidator [*] Executing validator ProfanityValidator [*] Executing validator AcronymValidator [*] Executing validator JSONValidator [*] Executing validator OutputValidator [*] Executing validator RegenerationValidator [*] Executing validator HelpInputOutputValidator [*] Executing validator SupportValidator [*] Executing validator RuntimeValidator [*] Executing validator VersionPinValidator [*] Executing validator EncodingValidator [*] Executing validator ExampleInputValidator [*] Executing validator CloudReadyValidator [*] Executing validator SupportedVersionValidator [*] Executing validator UnapprovedKeywordsValidator WARNING: Unsupported keywords found: vmware, cloud. The following keywords will not be searchable by the Extension Library. Please remove or update the invalid keywords from the keywords array in the plugin.spec.yaml file. [*] Executing validator HelpExampleValidator [*] Executing validator Version Increment Validator [*] Executing validator NameValidator [*] Executing validator ExceptionValidator [*] Executing validator CredentialsValidator [*] Executing validator PasswordValidator [*] Executing validator PrintValidator [*] Executing validator ConfidentialValidator [*] Executing validator DockerValidator [*] Executing validator URLValidator

Export

The subcommand export will export the Docker image as a .tar file which you can import into InsightConnect. The output of this command is what you will import into InsightConnect as a Custom Plugin to begin using it. Note that this command will build the plugin first before exporting it.

insight-plugin export

Export Examples

CommandDescription
insight-plugin exportExport a plugin Docker image to a tarball
insight-plugin export --no-pullFlag to skip pulling the latest SDK base image during plugin export
❯ insight-plugin export Beginning export.. Export process complete!

Analysis

This subcommand runs the static code analysis check on the plugin to analyse the source code for quality, reliability and security without executing the code. The plugin must succeed with no errors before it can be contributed to the public repo.

❯ insight-plugin analysis Messages ======== komand_base64/actions/decode/action.py Line: 26 pylint: logging-too-many-args / Too many arguments for logging format string (col 12) komand_base64/connection/connection.py Line: 9 pylint: signature-differs / Signature differs from overridden 'connect' method (col 4) Check Information ================= Started: 2024-05-13 10:19:20.916095 Finished: 2024-05-13 10:19:23.286805 Time Taken: 2.37 seconds Formatter: grouped Profiles: /Users/comcnally/dev/insightconnect-plugins/prospector.yaml, no_doc_warnings, no_member_warnings, no_test_warnings, strictness_medium, strictness_high, strictness_veryhigh Strictness: from profile Libraries Used: Tools Run: bandit, mccabe, pyflakes, pylint Messages Found: 2

Checks

This subcommand runs the analysis and validate commands against the plugin alongside running the black package to refactor the codebase if needed. This is useful for mocking the same checks that are usually run against the plugins via GitHub actions. Especially useful for 3rd party contributors.

insight-plugin checks Linter All done! 🍰 4 files left unchanged. Analysis Messages ======== komand_base64/actions/decode/action.py Line: 26 pylint: logging-too-many-args / Too many arguments for logging format string (col 12) komand_base64/connection/connection.py Line: 9 pylint: signature-differs / Signature differs from overridden 'connect' method (col 4) Check Information ================= Started: 2024-05-13 10:40:06.302222 Finished: 2024-05-13 10:40:08.493606 Time Taken: 2.19 seconds Formatter: grouped Profiles: /Users/comcnally/dev/insightconnect-plugins/prospector.yaml, no_doc_warnings, no_member_warnings, no_test_warnings, strictness_medium, strictness_high, strictness_veryhigh Strictness: from profile Libraries Used: Tools Run: bandit, mccabe, pyflakes, pylint Messages Found: 2 Validate [*] Running Integration Validators... [*] Executing validator HelpValidator [*] Executing validator ChangelogValidator [*] Executing validator CloudReadyConnectionCredentialTokenValidator [*] Executing validator RequiredKeysValidator [*] Executing validator UseCaseValidator [*] Executing validator SpecPropertiesValidator [*] Executing validator SpecVersionValidator [*] Executing validator FilesValidator [*] Executing validator TagValidator [*] Executing validator DescriptionValidator [*] Executing validator TitleValidator [*] Executing validator VendorValidator [*] Executing validator DefaultValueValidator [*] Executing validator IconValidator [*] Executing validator RequiredValidator [*] Executing validator VersionValidator [*] Executing validator DockerfileParentValidator [*] Executing validator ProfanityValidator [*] Executing validator AcronymValidator [*] Executing validator JSONValidator [*] Executing validator OutputValidator [*] Executing validator RegenerationValidator [*] Executing validator HelpInputOutputValidator [*] Executing validator SupportValidator [*] Executing validator RuntimeValidator [*] Executing validator VersionPinValidator [*] Executing validator EncodingValidator [*] Executing validator ExampleInputValidator [*] Executing validator CloudReadyValidator [*] Executing validator SupportedVersionValidator [*] Executing validator UnapprovedKeywordsValidator WARNING: Unsupported keywords found: utilities. The following keywords will not be searchable by the Extension Library. Please remove or update the invalid keywords from the keywords array in the plugin.spec.yaml file. [*] Executing validator HelpExampleValidator [*] Executing validator Version Increment Validator [*] Executing validator NameValidator [*] Executing validator ExceptionValidator [*] Executing validator CredentialsValidator [*] Executing validator PasswordValidator [*] Executing validator PrintValidator [*] Executing validator ConfidentialValidator [*] Executing validator DockerValidator

JSON to Spec

A script called json_to_spec.py converts a JSON object to InsightConnect plugin spec data. It takes a path to a file containing JSON as an argument. It’s located in the open-source plugins repo.

$ curl -s https://ifconfig.co/json > ifconfig.json $ tools/json_to_spec.py ifconfig.json types: {} output: city: description: ENTER DESCRIPTION title: ENTER TITLE type: string hostname: description: ENTER DESCRIPTION title: ENTER TITLE type: string ip: description: ENTER DESCRIPTION title: ENTER TITLE type: string country_iso: description: ENTER DESCRIPTION title: ENTER TITLE type: string ip_decimal: description: ENTER DESCRIPTION title: ENTER TITLE type: integer country: description: ENTER DESCRIPTION title: ENTER TITLE type: string

You can then copy and paste the output into your plugin spec rather than having to type it out manually.