Regenerating Plugins

Regeneration occurs when you want to update the plugin schema to add, remove, or modify any actions, triggers, improved title and descriptions, etc. without loss of code and manual updates to the JSON schema.

Compatability

The latest plugin architecture is V2, which refers to SDK and insight-plugin tool improvements. The improvements split plugin components such as action and triggers into their own directories. For each directory, a code (e.g. action.py) and schema (e.g. schema.py) file exists. The schema files are generated directly off plugin.spec.yaml (source of truth) and should not be modified by the user. In this design, developers only need to modify the code files.

An example of a Python plugin that is V2 compliant:

 
1
base64
2
├── Dockerfile
3
├── Makefile
4
├── bin
5
│   └── icon_base64
6
├── icon_base64
7
│   ├── __init__.py
8
│   ├── actions
9
│   │   ├── __init__.py
10
│   │   ├── decode
11
│   │   │   ├── __init__.py
12
│   │   │   ├── action.py
13
│   │   │   └── schema.py
14
│   │   └── encode
15
│   │       ├── __init__.py
16
│   │       ├── action.py
17
│   │       └── schema.py
18
│   ├── connection
19
│   │   ├── __init__.py
20
│   │   ├── connection.py
21
│   │   └── schema.py
22
│   ├── triggers
23
│   │   ├── __init__.py
24
│   │   ├── trigger.py
25
│   │   └── schema.py
26
│   └── util
27
│       ├── __init__.py
28
│       └── api.py
29
├── plugin.spec.yaml
30
├── requirements.txt
31
├── setup.py
32
└── tests
33
|   ├── decode.json
34
|   └── encode.json
35
└── unit_tests
36
    ├── __init__.py
37
    ├── test_encode.json
38
    └── test_decode.json

• Plugins developed after October 1, 2016 have the ability to be regenerated using the older git method.
  • V1 method of Regenerating Plugins
• Plugins developed after January 1, 2018 or plugins that have been ported to V2 have the ability to be regenerated using the newer method.
  • The plugin V2 architecture provides regeneration natively with insight-plugin refresh.
  • All our plugins have been ported to V2 but your custom plugins may have not, if they haven't, it's best to port them first to make development easier for the long term.

Generation Command

As described in the SDK Spec document, plugin skeletons are generated by running:

insight-plugin create plugin.spec.yaml

Legacy Content

icon-plugin generate python plugin.spec.yaml

Regeneration Command

If you decide to modify your plugin’s scheme, by editing plugin.spec.yaml, you’ll need to regenerate the plugin skeleton using insight-plugin refresh inside the plugin’s directory to apply the changes to all the files throughout the plugin.

Legacy Content

If you decide to modify your plugin’s scheme, by editing plugin.spec.yaml, you’ll need to regenerate the plugin skeleton using make regenerate inside the plugin’s directory. The make regenerate target is a shortcut that calls the full regenerate command which is icon-plugin generate python --regenerate.

Process

The following process will automatically put the new schema changes from plugin.spec.yaml into effect:

1. Update the plugin.spec.yaml file
2. Run insight-plugin refresh
3. Update any code files needed (e.g. action.py)

Example

Here's an example of plugin regeneration by updating the schema (plugin.spec.yaml) to support:

• A new action
• A new plugin description
• A schema for an existing action

 
1
❯ insight-plugin refresh
2
Refresh process complete!

Here's a list of the changes with git status. You can see that a few files were modified as well as the new addition of the malware_lookup action directory.

 
1
$ git status
2
On branch master
3
Your branch is up to date with 'origin/master'.
4

5
Changes not staged for commit:
6
  (use "git add <file>..." to update what will be committed)
7
  (use "git checkout -- <file>..." to discard changes in working directory)
8

9
      modified:   .CHECKSUM
10
      modified:   bin/komand_cymon
11
      modified:   komand_cymon/actions/__init__.py
12
      modified:   plugin.spec.yaml
13
      modified:   setup.py
14

15
Untracked files:
16
  (use "git add <file>..." to include in what will be committed)
17

18
      komand_cymon/actions/malware_lookup/

We can see the exact changes using git diff. Notice that the description and version were updated in multiple files, specifically: cymon/bin/komand_cymon and setup.py. The new action resulted in the creation of new directory komand_cymon/actions/malware_lookup/ which contains an action.py and schema.py file. The rest of the output are the references to the new action.

 
1
diff --git a/cymon/bin/komand_cymon b/cymon/bin/komand_cymon
2
index ccd4c109d..7aed3ab00 100755
3
--- a/cymon/bin/komand_cymon
4
+++ b/cymon/bin/komand_cymon
5
@@ -6,8 +6,8 @@ from komand_cymon import connection, actions, triggers
6

7
 Name = 'Cymon'
8
 Vendor = 'rapid7'
9
-Version = '0.1.4'
10
-Description = 'Cymon Open Threat Intelligence'
11
+Version = '1.0.0'
12
+Description = 'Cymon is the largest tracker of open-source security reports about phishing, malware, botnets and other malicious activities'
13

14

15
 class KomandCymon(insightconnect_plugin_runtime.Plugin):
16
@@ -31,6 +31,8 @@ class KomandCymon(insightconnect_plugin_runtime.Plugin):
17

18
         self.add_action(actions.DomainLookup())
19

20
+        self.add_action(actions.MalwareLookup())
21
+
22
         self.add_action(actions.UrlLookup())
23

24

25
diff --git a/cymon/komand_cymon/actions/__init__.py b/cymon/komand_cymon/actions/__init__.py
26
index 41ac95ec7..44a025ccb 100755
27
--- a/cymon/komand_cymon/actions/__init__.py
28
+++ b/cymon/komand_cymon/actions/__init__.py
29
@@ -3,4 +3,5 @@ from .address_blacklist.action import AddressBlacklist
30
 from .address_lookup.action import AddressLookup
31
 from .domain_blacklist.action import DomainBlacklist
32
 from .domain_lookup.action import DomainLookup
33
+from .malware_lookup.action import MalwareLookup
34
 from .url_lookup.action import UrlLookup
35
diff --git a/cymon/plugin.spec.yaml b/cymon/plugin.spec.yaml
36
index a435d9eaf..23dda41a8 100644
37
--- a/cymon/plugin.spec.yaml
38
+++ b/cymon/plugin.spec.yaml
39
@@ -1,8 +1,8 @@
40
 plugin_spec_version: v2
41
 name: cymon
42
 title: Cymon
43
-description: "Cymon Open Threat Intelligence"
44
-version: 0.1.4
45
+description: "Cymon is the largest tracker of open-source security reports about phishing, malware, botnets and other malicious activities"
46
+version: 1.0.0
47
 vendor: rapid7
48
 @@ -529,3 +529,14 @@ actions:
49
       url:
50
         type: "[]string"
51
         title: Cymon URL References
52
+  malware_lookup:
53
+    description: Lookup malware hash
54
+    input:
55
+      hash:
56
+        type: string
57
+        description: MD5 or SHA1 hash
58
+        required: true
59
+    output:
60
+      found:
61
+        type: boolean
62
+        title: Malware Found
63
diff --git a/cymon/setup.py b/cymon/setup.py
64
index e72761182..6e8117680 100755
65
--- a/cymon/setup.py
66
+++ b/cymon/setup.py
67
@@ -3,8 +3,8 @@ from setuptools import setup, find_packages
68

69

70
 setup(name='cymon-komand-plugin',
71
-      version='0.1.4',
72
-      description='Cymon Open Threat Intelligence',
73
+      version='1.0.0',
74
+      description='Cymon is the largest tracker of open-source security reports about phishing, malware, botnets and other malicious activities',
75
       author='komand',
76
       author_email='',
77
       url='',

In addition, you will see changes to the plugin.spec.yaml file itself, since it’s the source of truth and we created those changes manually.

Now, add your code to the new komand_cymon/actions/malware_lookup/action.py file, test, and finalize your changes with:

 
1
$ git add .
2
$ git commit -m "Regenerate Cymon with new description and new malware lookup action"