Unit Test Primer

When creating a unit test for a plugin, the recommended practice is to store them in the root level of the plugin directory in a unit_test folder.

Root Folder

For larger plug-ins, we recommend separating your actions, connections, payloads and triggers into sub-folders. This will enable you to run separate tests as needed. Creating sub-folders in your unit test directory impacts the unit test import process.

Unit Test Layout

Create a Unit Test

Unit tests must adhere to a specific format. Note the following as you prepare to write your unit test:

  • The TestEncode class must inherit from TestCase
  • All tests are functions within the TestEncode class, and they must start with test_

Here is an example of a correctly written unit test:

python
1
from unittest import TestCase
2


3
class TestEncode(TestCase):
4
    def test_run(self):
5
        pass

Run the Unit Test

Once you write the TestEnclose class, you can run it in PyCharm in one of the following ways:

In PyCharm, click the green arrow that appears to the left of the class.

Run the test from the command line. For more information, see here.

Note: If you're running from the command line, you'll have Python path problems. To fix this I used the following code.

python
1
from unittest import TestCase
2
import sys
3
import os
4


5
working_path = os.getcwd()
6
sys.path.insert(0,"../")
7


8
from icon_azure_ad_admin.actions import DisableUserAccount

This code adds the root directory of the plugin structure to your Python path. That allows you to import from the icon_<plugin_name> directory.

Unit Tests WIthout External Calls

Once your unit tests are written and running, you can begin testing your code.

Before you can test your code you must have the following:

  • A valid plugin.spec.yaml file
  • A generated plugin
python
1
from unittest import TestCase
2
from komand_base64.actions import Encode
3
import logging
4


5
class TestEncode(TestCase):
6
   def test_run(self):
7
       log = logging.getLogger("Test")
8
       test_encoder = Encode()
9


10
       params = {
11
           "content": "Look mom, unit tests!"
12
       }
13


14
       test_encoder.logger = log
15
       actual = test_encoder.run(params)
16


17
       expected = {'data': 'TG9vayBtb20sIHVuaXQgdGVzdHMh'}
18


19
       self.assertEqual(actual, expected)

Next, you will need to configure the following settings:

  • Logger
    • The plugin will need a logger. We do this by importing logging and using logging.getLogger(). Thanks to the magic of Python, you can simply inject that in the class with < plugin >.logger = < log >
  • Parameters
    • You will need parameters if we want to run the run function. To create these quickly you can copy them from the .json test files. (If you don’t have those, generate them with ./run.sh -c samples)
  • Expected
    • This setting will save time if you’re working with complex output. Set expected to “” then set a breakpoint on assertEqual. Debug your unit test. It should break on the assert statement and you’ll have all your variable values in a watch window in the debug pane. If the actual value is correct, simply copy the value and paste it in expected.

Here are all the available assert statements: https://docs.python.org/3/library/unittest.html#test-cases

In general, you will use assertEquals, but there are several test asserts that are useful, such as assertTrue and assertRaises.

Tests With External Calls

A way to rapidly develop with external calls is to use Postman to make sure you have your API call correct, copy that payload into your plugin project and mock the call to the API. For large complex calls, this can lead to huge time savings.

Assumptions: You know how to use Postman and can make API calls with that tool. cURL will work too if you’re more familiar with that.

The following example is from the Office 365 Email plugin. This is a ton of code, but it’s much easier than it looks.

This simply reads payloads in as text, which makes mocking an API call much easier.

python
1
This simply reads payloads in as text, which makes mocking an API call much easier.
2


3
# Get a real payload from file
4
def read_file_to_string(filename):
5
    with open(filename) as my_file:
6
        return my_file.read()

This block of code mocks the requests.get() methods. The requests.get() method simulates a Python command, making it an effective way to run your tests.

python
1
# This method will be used by the mock to replace requests.get
2
def mocked_requests_get(*args, **kwargs):
3
    class MockResponse:
4
        def __init__(self, json_data, status_code):
5
            self.json_data = json_data
6
            self.status_code = status_code
7
            self.text = "This is some error text"
8


9
        def json(self):
10
            return json.loads(self.json_data)
11


12
    # Since this is folder down from the base unit_test folder, the base path may change on us if we're
13
    # running the whole suite, or just these tests.
14
    actual_path = os.path.dirname(os.path.realpath(__file__))
15
    actual_joined_path = os.path.join(actual_path, "payloads/get_messages_from_user.json")
16
    get_messages_from_user_payload = read_file_to_string(actual_joined_path)
17


18
    if args[0] == 'https://login.microsoftonline.com/test_tenant_id/oauth2/token':
19
        return MockResponse({"access_token": "test_api_token6"}, 200)
20
    if args[0] == 'https://graph.microsoft.com/v1.0/test_tenant_id/users/bob@hotmail.com/messages?$search="from:From"&$top=250':
21
        return MockResponse(get_messages_from_user_payload, 200)
22


23
    print(f"mocked_requests_get failed looking for: {args[0]}")
24
    return MockResponse(None, 404)

Note the following as you configure your settings:

  • Mock Object:
    • Really, this class is mocking two things. The behavior of requests.get(), and it’s response object. In the first few lines of this block, you’ll see that we create an inline class contained within this method. That’s going to be the thing that’s returned when our mock is called.
  • Python Path Note
    • Below that, you’ll see us reading in the payloads. I have to do a trick with the path to make sure Python can find our payload regardless of where this script is being run. If you run this script from the root of unit_tests when running all the tests, your Current Working Directory (CWD) will be different than if you run just this suite or just a test in this suite.
  • ARGS
    • I look to see what argument we have and see what we are trying to call. I return our mock response if the argument matches one of our expected calls.
  • Errors
    • If the argument that comes in doesn’t match any of my expected URLs, I print that URL and bailout with a 404. Printing comes in handy so you can quickly see what failed. The 404 is handy because it allows us to test error handling in our code.

In this block of code, I create a fake connection class that I can pass into my action class.

python
1
# Mock the connection
2
class MockConnection:
3
    def __init__(self):
4
        self.tenant = "test_tenant_id"
5
        self.auth_token = "test_api_token"
6


7
    def get_headers(self, api_token):
8
        return {
9
            "thisisaheader": "thisisavalue",
10
            "api_token": self.auth_token
11
        }
12


13
    def get_auth_token(self):
14
        return self.auth_token

This code is pretty specific to the O365 plugin, but I wanted to make a note that you'll have to create mock classes for some of the common classes that our plug-ins use.

Finally... the actual test:

python
1
@mock.patch('requests.get', side_effect=mocked_requests_get)
2
def test_get_messages_from_user(self, mock_get):
3
    get_messages = GetEmailFromUser()
4
    get_messages.connection = MockConnection()
5
    actual = get_messages.get_messages_from_user("bob@hotmail.com", '$search="from:From"&$top=250')
6


7
    self.assertTrue(len(actual), 3)
8
    self.assertEqual(actual[2].get('id'), 'AAMkADI3Mzc1ZTg3LTIzYWEtNDNmNi1hZDQ5LTBiMjAzYzA3ZThhYwBGAAAAAAAxDvrPc8q6SqGLTJ9iB-SGBwC8UQDN7ObVSLWQuxHJ-dDTAAE-L_Q7AAC8UQDN7ObVSLWQuxHJ-dDTAAE-MJpHAAA=')
  • Mock Patch
    • The important thing to note here is the @mock.patch decorator. This is a slick function that will replace the behavior of one function with the method you specify. In this case, we’re taking requests.get and replacing that functionality with our mocked_requests_get method from above.

Here's more information about mocking

Unit Tests with Exceptions

One of the most important things we can do with non-traditional unit tess is exception handling and exotic error scenarios. (Want bad JSON from a response with a 200 status code? This is how to test it!)

python
1
def test_make_kql_for_request_throws_exception(self):
2
    get_messages = GetEmailFromUser()
3


4
    with self.assertRaises(PluginException):
5
        get_messages.make_kql_for_request("", "", "", 250)

In this example, we generate bad input for make_kql_for_request , which causes it to throw an exception. We want to make sure it throws a PluginException specifically and not some other error that the user may not know what to do with.

Opening a context with with allows us to capture the exception and verify we have the correct exception.

Unit Tests on Triggers

It’s very difficult to test our triggers since they’re essentially an endless loop. We recommend that you move the logic out of the “main loop” of the trigger into separate functions and test them via direct unit tests.

Ultimately, you’ll still have to verify that your trigger “main loop” works. The following example demonstrates how to set up the calls and a timeout decorator to make sure you don’t hit any exceptions.

python
1
# The timeout decorator allows us to kill the 'while True' loop without failing the test
2
@timeout_pass
3
@timeout_decorator.timeout(3)
4
@mock.patch('requests.get', side_effect=mocked_requests_get)
5
def test_microsoft_message_received(self, mock_get):
6
    mr = EmailReceived()
7
    params = {
8
        "mailbox_id": TEST_MAILBOX_ID,
9
        "subject_query": TEST_QUERY,
10
        "interval": 1
11
    }
12
    mr.connection = mockConnection()
13
    mr.log_stream = MockDispatcher()
14
    mr.dispatcher = MockDispatcher()
15
    mr.logger = logging.getLogger("TestLogger")
16
    mr.current_time = maya.MayaDT('2019-08-05T14:57:40Z')
17


18
    mr.run(params)

@timeout_pass - tells the test framework that a timeout is passing testing

@timeout_decorator.timeout(3) - tells the test to timeout after 3 seconds. With our interval set to 1, that should give us two loops which are enough to verify we've passed this test.

Appendix