Plugin Tooling

The tooling is intended to be used to develop new or existing plugins.

Tooling

You can find a list of open-source plugins to reference, add a new feature, or fix a bug. We welcome and encourage pull requests!

Insight Plugin

This is the main tool for developing plugins. Installation instructions for insight-plugin is available in the Getting Started document.

Make

The make command constructs artifacts for plugin development.

Arguments

make image is used to package the plugin code into a Docker image which makes it available to run. This is typically the first thing you want to do when you’re working on an existing plugin to ensure it builds successfully.

console
1
❯ make image
2
[*] Use ``make menu`` for available targets
3
[*] Including available Makefiles: ../../tools/Makefiles/Colors.mk ../../tools/Makefiles/Helpers.mk
4
--
5
[*] Building plugin image
6
docker build --pull -t rapid7/base64:1.1.6 .

make tarball is used to build a tarball of the plugin. This is used for backward-compatible testing in InsightConnect.

 
1
❯ make tarball
2
INFO[0000] Building tarball rapid7-base64-1.1.1.tar.gz

Create

The subcommand create creates all the necessary plugin files and directories.

It can be used to generate a new plugin from a plugin spec (Plugin Spec) with the path of the spec as an argument.

insight-plugin create plugin.spec.yaml

Create examples

shell
1
❯ insight-plugin create plugin.spec.yaml
2
Create process complete!

Refresh

Anytime you modify the plugin spec, use the subcommand refresh to rebuild the skeleton with the new schema without compromising previously written code. This makes adding actions and updating the plugin spec simple and fast.

insight-plugin refresh

Refresh Examples

shell
1
❯ insight-plugin refresh
2
Refresh process complete!

Samples

The samples command is used to create the json test files that we use to test actions and triggers against the plugin docker image. Once you build out the plugin using make or make image, create the json test files using: insight-plugin samples <path/to/specfile>

Sample Examples

 
1
❯ insight-plugin samples
2
Samples process complete!
3

4
❯ insight-plugin samples encode
5
WARNING: SamplesUtil: tests/encode.json already exists, overwriting..
6
Samples process complete!

Run

The subcommand run for insight-plugin allows a user to quickly test and execute plugins from the command-line.

See the Running Plugins page for more detail. Below are a few examples.

Run Examples

 
1
❯ insight-plugin run tests/encode.json
2
2023-08-22 14:41:40 [info     ] rapid7/Base64:1.1.6. Step name: encode
3

4
{"body": {"log": "", "status": "ok", "meta": {}, "output": {"data": "UmFwaWQ3"}}, "version": "v1", "type": "action_event"}

Server

You can also interact with the plugin over HTTP when it's in REST mode:

insight-plugin server

Server Examples

 
1
❯ insight-plugin server
2
[2023-08-22 14:39:58 +0000] [1] [INFO] Starting gunicorn 20.0.4
3
[2023-08-22 14:39:58 +0000] [1] [INFO] Listening at: http://0.0.0.0:10001 (1)
4
[2023-08-22 14:39:58 +0000] [1] [INFO] Using worker: threads
5
[2023-08-22 14:39:58 +0000] [8] [INFO] Booting worker with pid: 8

Then you can hit its run or test endpoint with a web request containing the JSON input as payload:

 
1
curl -d @tests/test.json http://127.0.0.1:10001/actions/test

Shell

You can also interact with the plugin via command line by running the plugin and Docker and entering the shell:

insight-plugin shell

Shell Examples

shell
1
❯ insight-plugin shell
2
$ pwd
3
/python/src
4
$ ls
5
Dockerfile  cisco_umbrella_destinations_rapid7_plugin.egg-info	icon.png			  setup.py
6
Makefile    dist						icon_cisco_umbrella_destinations  venv
7
bin	    extension.png					plugin.spec.yaml
8
build	    help.md						requirements.txt
9
$

Validate

The subcommand validate will run validation against the plugin, ensuring that all files including schema, plugin.spec.yaml and help.md are valid to the constraints applied by InsightConnect.

insight-plugin validate

Validate Examples

shell
1
❯ insight-plugin validate
2
[*] Running Integration Validators...
3
[*] Executing validator HelpValidator
4
[*] Executing validator ChangelogValidator
5
[*] Executing validator CloudReadyConnectionCredentialTokenValidator
6
[*] Executing validator RequiredKeysValidator
7
[*] Executing validator UseCaseValidator
8
[*] Executing validator SpecPropertiesValidator
9
[*] Executing validator SpecVersionValidator
10
[*] Executing validator FilesValidator
11
[*] Executing validator TagValidator
12
[*] Executing validator DescriptionValidator
13
[*] Executing validator TitleValidator
14
[*] Executing validator VendorValidator
15
[*] Executing validator DefaultValueValidator
16
[*] Executing validator IconValidator
17
[*] Executing validator RequiredValidator
18
[*] Executing validator VersionValidator
19
[*] Executing validator DockerfileParentValidator
20
[*] Executing validator ProfanityValidator
21
[*] Executing validator AcronymValidator
22
[*] Executing validator JSONValidator
23
[*] Executing validator OutputValidator
24
[*] Executing validator RegenerationValidator
25
[*] Executing validator HelpInputOutputValidator
26
[*] Executing validator SupportValidator
27
[*] Executing validator RuntimeValidator
28
[*] Executing validator VersionPinValidator
29
[*] Executing validator EncodingValidator
30
[*] Executing validator ExampleInputValidator
31
[*] Executing validator CloudReadyValidator
32
[*] Executing validator SupportedVersionValidator
33
[*] Executing validator UnapprovedKeywordsValidator
34
WARNING: Unsupported keywords found: vmware, cloud. The following keywords will not be searchable by the Extension Library. Please remove or update the invalid keywords from the keywords array in the plugin.spec.yaml file.
35
[*] Executing validator HelpExampleValidator
36
[*] Executing validator Version Increment Validator
37
[*] Executing validator NameValidator
38
[*] Executing validator ExceptionValidator
39
[*] Executing validator CredentialsValidator
40
[*] Executing validator PasswordValidator
41
[*] Executing validator PrintValidator
42
[*] Executing validator ConfidentialValidator
43
[*] Executing validator DockerValidator
44
[*] Executing validator URLValidator

Semver

The subcommand semver will update the plugin files to a newly specified version as supplied as an argument in semantic versioning format.

insight-plugin semver 1.0.1

Semver Examples

Export

The subcommand export will export the Docker image as a .tar file which you can import into InsightConnect. The output of this command is what you will import into InsightConnect as a Custom Plugin to begin using it. Note that this command will build the plugin first before exporting it.

insight-plugin export

Export Examples

 
1
❯ insight-plugin export
2
Beginning export..
3
Export process complete!

View

This subcommand allows a user to get a general overview of a plugin.

insight-plugin view

View Examples

console
1
❯ insight-plugin view
2
Name:        Cisco Umbrella Destinations
3
Vendor:      rapid7
4
Version:     4.0.0
5
Description: Cisco Umbrella Destinations allows users to manage, block, and allow network destinations based on policies within their organization
6

7
Actions (10):
8
└── dAdd (Add a destination to the destination list)
9
└── dDelete (Delete a list of destinations from a destination list)
10
└── dGet (Get a list of destinations related to a destination list)
11
└── dlCreate (Create a destination list)
12
└── dlDelete (Delete a destination list)
13
└── dlFilterAll (Filter and Retrieve all destination lists of organization)
14
└── dlGet (Get a destination list)
15
└── dlGetAll (Retrieve all destination lists of organization)
16
└── dlGetByName (Get destination list by name)
17
└── dlPatch (Rename a destination list)

Analysis

This subcommand runs the static code analysis check on the plugin to analyse the source code for quality, reliability and security without executing the code. The plugin must succeed with no errors before it can be contributed to the public repo.

console
1
❯ insight-plugin analysis
2
Messages
3
========
4

5
komand_base64/actions/decode/action.py
6
  Line: 26
7
    pylint: logging-too-many-args / Too many arguments for logging format string (col 12)
8

9
komand_base64/connection/connection.py
10
  Line: 9
11
    pylint: signature-differs / Signature differs from overridden 'connect' method (col 4)
12

13

14

15
Check Information
16
=================
17
         Started: 2024-05-13 10:19:20.916095
18
        Finished: 2024-05-13 10:19:23.286805
19
      Time Taken: 2.37 seconds
20
       Formatter: grouped
21
        Profiles: /Users/comcnally/dev/insightconnect-plugins/prospector.yaml, no_doc_warnings, no_member_warnings, no_test_warnings, strictness_medium, strictness_high, strictness_veryhigh
22
      Strictness: from profile
23
  Libraries Used:
24
       Tools Run: bandit, mccabe, pyflakes, pylint
25
  Messages Found: 2

Linter

This subcommand runs the linter on the code to let the user know if a file needs to be reformatted using the black tool.

shell
1
❯ insight-plugin linter
2
would reformat /dev/insightconnect-plugins/plugins/rapid7_insightidr/komand_rapid7_insightidr/actions/add_indicators_to_a_threat/action.py
3

4
Oh no! 💥 💔 💥
5
1 file would be reformatted, 87 files would be left unchanged.

Checks

This subcommand runs the analysis, linter and validate commands against the plugin. This is useful for mocking the same checks that are usually run against the plugins via GitHub actions. Especially useful for 3rd party contributors.

shell
1
❯ insight-plugin checks
2

3
Analysis
4

5
Messages
6
========
7

8
komand_base64/actions/decode/action.py
9
  Line: 26
10
    pylint: logging-too-many-args / Too many arguments for logging format string (col 12)
11

12
komand_base64/connection/connection.py
13
  Line: 9
14
    pylint: signature-differs / Signature differs from overridden 'connect' method (col 4)
15

16

17

18
Check Information
19
=================
20
         Started: 2024-05-13 10:40:06.302222
21
        Finished: 2024-05-13 10:40:08.493606
22
      Time Taken: 2.19 seconds
23
       Formatter: grouped
24
        Profiles: /Users/comcnally/dev/insightconnect-plugins/prospector.yaml, no_doc_warnings, no_member_warnings, no_test_warnings, strictness_medium, strictness_high, strictness_veryhigh
25
      Strictness: from profile
26
  Libraries Used:
27
       Tools Run: bandit, mccabe, pyflakes, pylint
28
  Messages Found: 2
29

30

31
Linter
32

33
All done! ✨ 🍰 ✨
34
4 files would be left unchanged.
35

36
Validate
37

38
[*] Running Integration Validators...
39
[*] Executing validator HelpValidator
40
[*] Executing validator ChangelogValidator
41
[*] Executing validator CloudReadyConnectionCredentialTokenValidator
42
[*] Executing validator RequiredKeysValidator
43
[*] Executing validator UseCaseValidator
44
[*] Executing validator SpecPropertiesValidator
45
[*] Executing validator SpecVersionValidator
46
[*] Executing validator FilesValidator
47
[*] Executing validator TagValidator
48
[*] Executing validator DescriptionValidator
49
[*] Executing validator TitleValidator
50
[*] Executing validator VendorValidator
51
[*] Executing validator DefaultValueValidator
52
[*] Executing validator IconValidator
53
[*] Executing validator RequiredValidator
54
[*] Executing validator VersionValidator
55
[*] Executing validator DockerfileParentValidator
56
[*] Executing validator ProfanityValidator
57
[*] Executing validator AcronymValidator
58
[*] Executing validator JSONValidator
59
[*] Executing validator OutputValidator
60
[*] Executing validator RegenerationValidator
61
[*] Executing validator HelpInputOutputValidator
62
[*] Executing validator SupportValidator
63
[*] Executing validator RuntimeValidator
64
[*] Executing validator VersionPinValidator
65
[*] Executing validator EncodingValidator
66
[*] Executing validator ExampleInputValidator
67
[*] Executing validator CloudReadyValidator
68
[*] Executing validator SupportedVersionValidator
69
[*] Executing validator UnapprovedKeywordsValidator
70
WARNING: Unsupported keywords found: utilities. The following keywords will not be searchable by the Extension Library. Please remove or update the invalid keywords from the keywords array in the plugin.spec.yaml file.
71
[*] Executing validator HelpExampleValidator
72
[*] Executing validator Version Increment Validator
73
[*] Executing validator NameValidator
74
[*] Executing validator ExceptionValidator
75
[*] Executing validator CredentialsValidator
76
[*] Executing validator PasswordValidator
77
[*] Executing validator PrintValidator
78
[*] Executing validator ConfidentialValidator
79
[*] Executing validator DockerValidator

JSON to Spec

A script called json_to_spec.py converts a JSON object to InsightConnect plugin spec data. It takes a path to a file containing JSON as an argument. It’s located in the open-source plugins repo.

shell
1
$ curl -s https://ifconfig.co/json > ifconfig.json
2
$ tools/json_to_spec.py ifconfig.json
3
types: {}
4

5
output:
6
  city:
7
    description: ENTER DESCRIPTION
8
    title: ENTER TITLE
9
    type: string
10
  hostname:
11
    description: ENTER DESCRIPTION
12
    title: ENTER TITLE
13
    type: string
14
  ip:
15
    description: ENTER DESCRIPTION
16
    title: ENTER TITLE
17
    type: string
18
  country_iso:
19
    description: ENTER DESCRIPTION
20
    title: ENTER TITLE
21
    type: string
22
  ip_decimal:
23
    description: ENTER DESCRIPTION
24
    title: ENTER TITLE
25
    type: integer
26
  country:
27
    description: ENTER DESCRIPTION
28
    title: ENTER TITLE
29
    type: string

You can then copy and paste the output into your plugin spec rather than having to type it out manually.