Plugin Tooling
The tooling is intended to be used to develop new or existing plugins.
Tooling
You can find a list of open-source plugins to reference, add a new feature, or fix a bug. We welcome and encourage pull requests!
Insight Plugin
This is the main tool for developing plugins. Installation instructions for insight-plugin
is available in the Getting Started document.
Make
The make
command constructs artifacts for plugin development.
Arguments
make image
is used to package the plugin code into a Docker image which makes it available to run. This is typically the first thing you want to do when you’re working on an existing plugin to ensure it builds successfully.
❯ make image
[*] Use ``make menu`` for available targets
[*] Including available Makefiles: ../../tools/Makefiles/Colors.mk ../../tools/Makefiles/Helpers.mk
--
[*] Building plugin image
docker build --pull -t rapid7/base64:1.1.6 .
make tarball
is used to build a tarball of the plugin. This is used for backward-compatible testing in InsightConnect.
❯ make tarball
INFO[0000] Building tarball rapid7-base64-1.1.1.tar.gz
Create
The subcommand create
creates all the necessary plugin files and directories.
It can be used to generate a new plugin from a plugin spec (Plugin Spec) with the path of the spec as an argument.
insight-plugin create plugin.spec.yaml
Create examples
Command | Description |
---|---|
insight-plugin create plugin.spec.yaml | Create plugin files |
❯ insight-plugin create plugin.spec.yaml
Create process complete!
Refresh
Anytime you modify the plugin spec, use the subcommand refresh
to rebuild the skeleton with the new schema without compromising previously written code. This makes adding actions and updating the plugin spec simple and fast.
insight-plugin refresh
Refresh Examples
Command | Description |
---|---|
insight-plugin refresh | Refresh plugin files |
❯ insight-plugin refresh
Refresh process complete!
Samples
The samples command is used to create the json test files that we use to test actions and triggers against the plugin docker image. Once you build out the plugin using make
or make image
, create the json test files using:
insight-plugin samples <path/to/specfile>
Sample Examples
Command | Description |
---|---|
insight-plugin samples | Generate all samples |
insight-plugin samples {target_component} | Generate a sample for a specific component |
❯ insight-plugin samples
Samples process complete!
❯ insight-plugin samples encode
WARNING: SamplesUtil: tests/encode.json already exists, overwriting..
Samples process complete!
Run
The subcommand run
for insight-plugin
allows a user to quickly test and execute plugins from the command-line.
See the Running Plugins page for more detail. Below are a few examples.
Run Examples
Command | Description |
---|---|
insight-plugin run tests/{test}.json | Runs the test file test.json |
insight-plugin run -T | Runs the connection test. Uses the first JSON file passed or first in the tests directory. |
insight-plugin run tests/{test}.json -A | Execute the run method on the submit JSON test file for a printed assessment to post in a PR on our open-source Github repository |
insight-plugin run | Execute the test and run methods for all JSON tests |
insight-plugin run tests/{test}.json --rebuild | Rebuild the docker container before running the sample file |
❯ insight-plugin run tests/encode.json
2023-08-22 14:41:40 [info ] rapid7/Base64:1.1.6. Step name: encode
{"body": {"log": "", "status": "ok", "meta": {}, "output": {"data": "UmFwaWQ3"}}, "version": "v1", "type": "action_event"}
Server
You can also interact with the plugin over HTTP when it’s in REST mode:
insight-plugin server
Server Examples
Command | Description |
---|---|
insight-plugin server | Run plugin docker container in http mode |
insight-plugin server --rebuild | Rebuild before running plugin docker container in http mode |
insight-plugin server -p 80:8080 | List of ports to expose to other containers and scripts |
insight-plugin server -V /Absolute/src:/Absolute/dest | Volume to mount from local machine to container |
❯ insight-plugin server
[2023-08-22 14:39:58 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2023-08-22 14:39:58 +0000] [1] [INFO] Listening at: http://0.0.0.0:10001 (1)
[2023-08-22 14:39:58 +0000] [1] [INFO] Using worker: threads
[2023-08-22 14:39:58 +0000] [8] [INFO] Booting worker with pid: 8
Then you can hit its run or test endpoint with a web request containing the JSON input as payload:
curl -d @tests/test.json http://127.0.0.1:10001/actions/test
Shell
You can also interact with the plugin via command line by running the plugin and Docker and entering the shell:
insight-plugin shell
Shell Examples
Command | Description |
---|---|
insight-plugin shell | Runs the plugin in Docker so that it may be accessed locally. |
insight-plugin shell -rebuild | Rebuilds the Docker image before running it in a container and entering the shell. |
insight-plugin shell -V /Absolute/src:/Absolute/dest | Volume to mount from local machine to container |
❯ insight-plugin shell
$ pwd
/python/src
$ ls
Dockerfile cisco_umbrella_destinations_rapid7_plugin.egg-info icon.png setup.py
Makefile dist icon_cisco_umbrella_destinations venv
bin extension.png plugin.spec.yaml
build help.md requirements.txt
$
Validate
The subcommand validate
will run validation against the plugin, ensuring that all files including schema, plugin.spec.yaml
and help.md
are valid to the constraints applied by InsightConnect.
insight-plugin validate
Validate Examples
Command | Description |
---|---|
insight-plugin validate | Validate / Run checks against the plugin |
❯ insight-plugin validate
[*] Running Integration Validators...
[*] Executing validator HelpValidator
[*] Executing validator ChangelogValidator
[*] Executing validator CloudReadyConnectionCredentialTokenValidator
[*] Executing validator RequiredKeysValidator
[*] Executing validator UseCaseValidator
[*] Executing validator SpecPropertiesValidator
[*] Executing validator SpecVersionValidator
[*] Executing validator FilesValidator
[*] Executing validator TagValidator
[*] Executing validator DescriptionValidator
[*] Executing validator TitleValidator
[*] Executing validator VendorValidator
[*] Executing validator DefaultValueValidator
[*] Executing validator IconValidator
[*] Executing validator RequiredValidator
[*] Executing validator VersionValidator
[*] Executing validator DockerfileParentValidator
[*] Executing validator ProfanityValidator
[*] Executing validator AcronymValidator
[*] Executing validator JSONValidator
[*] Executing validator OutputValidator
[*] Executing validator RegenerationValidator
[*] Executing validator HelpInputOutputValidator
[*] Executing validator SupportValidator
[*] Executing validator RuntimeValidator
[*] Executing validator VersionPinValidator
[*] Executing validator EncodingValidator
[*] Executing validator ExampleInputValidator
[*] Executing validator CloudReadyValidator
[*] Executing validator SupportedVersionValidator
[*] Executing validator UnapprovedKeywordsValidator
WARNING: Unsupported keywords found: vmware, cloud. The following keywords will not be searchable by the Extension Library. Please remove or update the invalid keywords from the keywords array in the plugin.spec.yaml file.
[*] Executing validator HelpExampleValidator
[*] Executing validator Version Increment Validator
[*] Executing validator NameValidator
[*] Executing validator ExceptionValidator
[*] Executing validator CredentialsValidator
[*] Executing validator PasswordValidator
[*] Executing validator PrintValidator
[*] Executing validator ConfidentialValidator
[*] Executing validator DockerValidator
[*] Executing validator URLValidator
Export
The subcommand export
will export the Docker image as a .tar
file which you can import into InsightConnect. The output of this command is what you will import into InsightConnect as a Custom Plugin to begin using it. Note that this command will build the plugin first before exporting it.
insight-plugin export
Export Examples
Command | Description |
---|---|
insight-plugin export | Export a plugin Docker image to a tarball |
insight-plugin export --no-pull | Flag to skip pulling the latest SDK base image during plugin export |
❯ insight-plugin export
Beginning export..
Export process complete!
Analysis
This subcommand runs the static code analysis check on the plugin to analyse the source code for quality, reliability and security without executing the code. The plugin must succeed with no errors before it can be contributed to the public repo.
❯ insight-plugin analysis
Messages
========
komand_base64/actions/decode/action.py
Line: 26
pylint: logging-too-many-args / Too many arguments for logging format string (col 12)
komand_base64/connection/connection.py
Line: 9
pylint: signature-differs / Signature differs from overridden 'connect' method (col 4)
Check Information
=================
Started: 2024-05-13 10:19:20.916095
Finished: 2024-05-13 10:19:23.286805
Time Taken: 2.37 seconds
Formatter: grouped
Profiles: /Users/comcnally/dev/insightconnect-plugins/prospector.yaml, no_doc_warnings, no_member_warnings, no_test_warnings, strictness_medium, strictness_high, strictness_veryhigh
Strictness: from profile
Libraries Used:
Tools Run: bandit, mccabe, pyflakes, pylint
Messages Found: 2
Checks
This subcommand runs the analysis
and validate
commands against the plugin alongside running the black
package to refactor the codebase if needed. This
is useful for mocking the same checks that are usually run against the plugins via GitHub actions.
Especially useful for 3rd party contributors.
❯ insight-plugin checks
Linter
All done! ✨ 🍰 ✨
4 files left unchanged.
Analysis
Messages
========
komand_base64/actions/decode/action.py
Line: 26
pylint: logging-too-many-args / Too many arguments for logging format string (col 12)
komand_base64/connection/connection.py
Line: 9
pylint: signature-differs / Signature differs from overridden 'connect' method (col 4)
Check Information
=================
Started: 2024-05-13 10:40:06.302222
Finished: 2024-05-13 10:40:08.493606
Time Taken: 2.19 seconds
Formatter: grouped
Profiles: /Users/comcnally/dev/insightconnect-plugins/prospector.yaml, no_doc_warnings, no_member_warnings, no_test_warnings, strictness_medium, strictness_high, strictness_veryhigh
Strictness: from profile
Libraries Used:
Tools Run: bandit, mccabe, pyflakes, pylint
Messages Found: 2
Validate
[*] Running Integration Validators...
[*] Executing validator HelpValidator
[*] Executing validator ChangelogValidator
[*] Executing validator CloudReadyConnectionCredentialTokenValidator
[*] Executing validator RequiredKeysValidator
[*] Executing validator UseCaseValidator
[*] Executing validator SpecPropertiesValidator
[*] Executing validator SpecVersionValidator
[*] Executing validator FilesValidator
[*] Executing validator TagValidator
[*] Executing validator DescriptionValidator
[*] Executing validator TitleValidator
[*] Executing validator VendorValidator
[*] Executing validator DefaultValueValidator
[*] Executing validator IconValidator
[*] Executing validator RequiredValidator
[*] Executing validator VersionValidator
[*] Executing validator DockerfileParentValidator
[*] Executing validator ProfanityValidator
[*] Executing validator AcronymValidator
[*] Executing validator JSONValidator
[*] Executing validator OutputValidator
[*] Executing validator RegenerationValidator
[*] Executing validator HelpInputOutputValidator
[*] Executing validator SupportValidator
[*] Executing validator RuntimeValidator
[*] Executing validator VersionPinValidator
[*] Executing validator EncodingValidator
[*] Executing validator ExampleInputValidator
[*] Executing validator CloudReadyValidator
[*] Executing validator SupportedVersionValidator
[*] Executing validator UnapprovedKeywordsValidator
WARNING: Unsupported keywords found: utilities. The following keywords will not be searchable by the Extension Library. Please remove or update the invalid keywords from the keywords array in the plugin.spec.yaml file.
[*] Executing validator HelpExampleValidator
[*] Executing validator Version Increment Validator
[*] Executing validator NameValidator
[*] Executing validator ExceptionValidator
[*] Executing validator CredentialsValidator
[*] Executing validator PasswordValidator
[*] Executing validator PrintValidator
[*] Executing validator ConfidentialValidator
[*] Executing validator DockerValidator
JSON to Spec
A script called json_to_spec.py
converts a JSON object to InsightConnect plugin spec data. It takes a path to a file containing JSON as an argument. It’s located in the open-source plugins repo .
$ curl -s https://ifconfig.co/json > ifconfig.json
$ tools/json_to_spec.py ifconfig.json
types: {}
output:
city:
description: ENTER DESCRIPTION
title: ENTER TITLE
type: string
hostname:
description: ENTER DESCRIPTION
title: ENTER TITLE
type: string
ip:
description: ENTER DESCRIPTION
title: ENTER TITLE
type: string
country_iso:
description: ENTER DESCRIPTION
title: ENTER TITLE
type: string
ip_decimal:
description: ENTER DESCRIPTION
title: ENTER TITLE
type: integer
country:
description: ENTER DESCRIPTION
title: ENTER TITLE
type: string
You can then copy and paste the output into your plugin spec rather than having to type it out manually.