Unit Test Generation

Before you can successfully generate unit tests using the insight-plugin tool, you must do the following:

• Create a correctly formatted plugin.spec.yaml file.
• Generate a plugin skeleton.
• Create sample JSON files with valid values and store them in a /tests directory.

Legacy Content

Before you can successfully generate unit tests using the icon-plugin tool, you must do the following: In the root directory run

• icon-plugin generate unittest

Quick Start

You can generate unit tests based on the actions and triggers you defined in plugin.spec.yaml. Unit tests are placed in the /unit_tests subfolder at the root of the plugin.

The Unit Test Generator creates an integration test and a unit test for each trigger and action. The integration test will make a “live” connection if available in your plugin and return real data from the action or trigger. The unit test will be a skeleton intended to test the logic of your action and trigger.

Unit tests are generated automatically when a plugin is created using insight-plugin create plugin.spec.yaml.

To generate new unit tests for your plugin, in the root directory run:

insight-plugin refresh

Run your unit tests

Do one of the following: Use the IDE of choice (VS Code and Pycharm have right-click options to allow you to run unit tests)

OR

Run the following command in the unit tests directory:

python3 -m unittest *

Your tests will fail the first time through, but will generate error messages that provide you with guidance around your next steps. There are also comment blocks in the unit tests themselves to help you along the way.

To run a specific unit test:

Run the following command in the Unit Tests directory:

python3 -m unittest test_my_action.py

Unit Test Templates

You can generate two types of unit tests: Action Unit Tests, and Trigger Unit Tests.

The following sections provide a high-level overview of the two templates.

NOTE: You may find that some tests that don’t fit neatly into these templates (util functions are a good example). You can create more unit test files in the unit_test directory.

Also, if needed, feel free to separate the test data into its own file(s). This makes using payloads from curl or Postman as test data much easier. Similar to how the test data is read in from the samples directory, you can read in test data from external files to use real API payloads in your tests.

Unit Test Action Template

At their core, the action unit tests that are generated will look something like this:

python
1
import sys
2
import os
3
sys.path.append(os.path.abspath('../'))
4

5
from unittest import TestCase
6
from icon_base64.connection.connection import Connection
7
from icon_base64.actions.decode import Decode
8
import json
9
import logging
10

11

12
class TestDecode(TestCase):
13
    def test_decode(self):
14
        """
15
        DO NOT USE PRODUCTION/SENSITIVE DATA FOR UNIT TESTS
16

17
        TODO: Implement test cases here
18
        """
19

20
        self.fail("Unimplemented Test Case")

The intent of this skeleton is to get you up and running quickly with the unit test. Then you can use the results of the unit test to help you generate further unit tests that test the logic in your action.

Action Integration Test: The action integration test looks like this:

python
1
from unittest import TestCase
2
from komand_base64.actions.encode import Encode
3
import logging
4

5

6
class TestEncode(TestCase):
7
    def test_encode(self):
8
        test_encoder = Encode()
9
        test_conn = Connection()
10
        log = logging.getLogger("Test")
11
        test_encoder.logger = log
12

13
        input_params = {"content": "Rapid7"}
14

15
        results = test_encoder.run(input_params)
16
        
17
        self.assertEqual("UmFwaWQ3", results.get("data"))

The first few lines of this test is to set up your action class and your connection class. The test then reads from the appropriate sample file to get the action parameters and connection settings. We then set the test connection as the connection in the action class. Finally, the action is run with its parameters and the results are returned.

As a unit test developer, you will need to remove the self.fail() line, and make self.assertEquals a reasonable assertion to validate the output of your action.

What is very helpful at this point is an IDE where you can break on the test_action.run() function and explore the data in your plugin. (Author’s note: This is where PyCharm is the preferred editor for our Python plugins, however, VS Code does admirably here. If you are doing all this from the command line, you can use print statements to follow the logic of your plugin.)

Action Unit Test(s): The action unit test is a simple skeleton of an unimplemented test case.

python
1
def test_{{.ActionName}}(self):
2
     self.fail("Unimplemented Test Case")

Here you will need to create unit tests that test the logic of your action. Here's an example of a real unit test from the Microsoft Teams plugin:

python
1
def test_compile_message_content(self):
2
    nmr = NewMessageReceived()
3
    regex = nmr.compile_message_content(".")
4
    self.assertTrue(regex.search("stuff"))
5

6
    with self.assertRaises(PluginException):
7
        nmr.compile_message_content("[")

What this test does is instantiate our action object NewMessageReceived() then tests one of its functions, in this case ‘compile_message_content’. We then assert on the output of that function to validate it’s working as expected.

Something to note, this is a bad unit test because I’m testing both the negative and positive results in one test. Try not to do that. I wanted to show the ‘with self.assertRaises’ functionality as that’s a slick way to do negative unit tests.

Unit Test Trigger Template

The trigger unit tests are more complicated than the action tests. The reason for that is the triggers generally run in an infinite loop, only sending information back to the orchestrator when it meets a specific condition. Since there’s no “return” value for us to assert on, these tests are more difficult to write.

python
1
<imports …>
2
from unittest.mock import patch
3
import timeout_decorator
4

5

6
# This will catch timeout errors and return None. This tells the test framework our test passed.
7
# This is needed because the run function in a trigger is an endless loop.
8
def timeout_pass(func):
9
    def func_wrapper(*args, **kwargs):
10
        try:
11
            return func(*args, **kwargs)
12
        except timeout_decorator.timeout_decorator.TimeoutError as e:
13
            print(f"Test timed out as expected: {e}")
14
            return None
15

16
    return func_wrapper
17

18
# This mocks insightconnect_plugin_runtime.Trigger.send
19
# We need this to fake out the plugin into thinking it's sending output in the trigger's run function
20
class fakeSender():
21
    def send(params):
22
        print(params)
23

24
# Test class
25
class Test{{.ActionClassName}}(TestCase):
26

27
    @timeout_pass
28
    @timeout_decorator.timeout(30)
29
    @patch("insightconnect_plugin_runtime.Trigger.send", side_effect=fakeSender.send)
30
    def test_integration_{{.ActionName}}(self, mockSend):
31
        log = logging.getLogger("Test")
32

33
        try:
34
            with open("../tests/{{.ActionName}}.json") as f:
35
                data = json.load(f)
36
                connection_params = data.get("body").get("connection")
37
                trigger_params = data.get("body").get("input")
38
        except Exception as e:
39
            <message …>
40
            self.fail(message)
41

42
        test_connection = Connection()
43
        test_connection.logger = log
44
        test_connection.connect(connection_params)
45

46
        test_email_received = {{.ActionClassName}}()
47
        test_email_received.connection = test_connection
48
        test_email_received.logger = log
49

50
        test_email_received.run(trigger_params)
51

52
        self.fail() # If we made it this far, the run loop failed somehow
53

54
    def test_{{.ActionName}}_some_function_to_test(self):
55
        self.fail("Unimplemented Test")

In general, it breaks down like this:

• Import stuff
• Mock stuff
• Creat our test class
• Create an integration trigger test
• Setup the skeleton for the rest of your tests

I left a couple of the import lines above because they are important. The timeout_decorator allows us to time out test cases. The patch import allows us to do formal mocks on functions.

The opening functions in the template overwrite the timeout behavior and create a mock for insightconnect_plugin_runtime.Trigger.send.

The trigger integration test looks like this:

 
1
@timeout_pass
2
@timeout_decorator.timeout(30)
3
@patch("insightconnect_plugin_runtime.Trigger.send", side_effect=fakeSender.send)
4
def test_integration_{{.ActionName}}(self, mockSend):
5
    log = logging.getLogger("Test")
6

7
    try:
8
        with open("../tests/{{.ActionName}}.json") as f:
9
            data = json.load(f)
10
            connection_params = data.get("body").get("connection")
11
            trigger_params = data.get("body").get("input")
12
    except Exception as e:
13
        <message … >
14
        self.fail(message)
15

16
    test_connection = Connection()
17
    test_connection.logger = log
18
    test_connection.connect(connection_params)
19

20
    test_email_received = {{.ActionClassName}}()
21
    test_email_received.connection = test_connection
22
    test_email_received.logger = log
23

24
    test_email_received.run(trigger_params)
25

26
    self.fail() # If we made it this far, the run loop failed somehow

From top to bottom:

@timeout_pass changes the timeout behavior for the @timeout decorator. Without that the test will fail on timeout. Using the function wrapper allows us to pass on timed out unit tests. Our triggers are infinite loops, so in this case, we didn’t hit an exception which means the loop is probably doing what we expect it to do.

@timeout_decorator.timeout(30) simply tells the test to timeout after 30 seconds. You are free to change this delay as needed depending on what your trigger is doing.

@patch(“insightconnect_plugin_runtime.Trigger.send”, side_effect=fakeSender.send) - This tells the python unittetst framework to overwrite the behavior of insightconnect_plugin_runtime.Trigger.send with the class we provide, fakeSender. In other words, we intercept anything the trigger tries to send to the orchestrator and print it out (using the fakeSender() class). This is a mock.

The rest of the test is very similar to the action test. We setup our trigger class and connection class with values from the appropriate sample. Then we set the trigger’s connection to our test connection. Finally, we run the trigger.

The run() function for a trigger is an infinite loop, so there’s not really anything concrete we can test other than run does not return an exception. Thus, if this test gets through the run function and hits the self.fail() line, the trigger likely did something unexpected.

Trigger Unit Test:

python
1
The trigger unit test skeleton is pretty basic:
2
   def test_{{.ActionName}}_some_function_to_test(self):
3
        self.fail("Unimplemented Test")

Here you will need to create tests to test the logic of your trigger. This is not an easy task in some instances. A general rule of thumb is if you can’t test lines of code in your trigger, it needs to be refactored as you’ve got too much logic in your run loop. In simple triggers, this might be fine, but if you do any data manipulation or transformation, it’s strongly recommended to move that behavior into their own methods or even a utils file so it can be unit tested.