Plugins Overview

Plugins are Python packages that are loaded when an Cloud Security (InsightCloudSec) process starts. These are primarily used for registering custom filters and packaging frontend assets to extend the user interface.

Plugin Distribution and Loading

All workers, schedulers, web-servers and other Cloud Security (InsightCloudSec)-related processes pull plugins from a centralized database. The database is used as a central point for distributing plugins to other processes, similar to a file server. To avoid contention during plugin changes, ONLY the scheduler is allowed to update the plugin contents on the database. This includes new plugins, deleting plugins, and updating plugins.

Plugins can be discovered in two manners: either locally on the scheduler’s file system or remotely to other sources such as HTTP or S3. It is for this reason that all plugin management should be done on the scheduler process space and not on workers and web-server processes. Plugins are currently only loaded on startup. If changes to plugins do occur or new plugins are added, then those processes will need to be restarted to pull in new changes.

Plugins are currently only loaded on startup. If changes to plugins do occur or new plugins are added, then those processes will need to be restarted to pull in new changes.

Implementation Details

The following sections outline how Cloud Security (InsightCloudSec) schedules and loads events in the background of the application.

Event registration

A job is registered as a scheduled event using the register method of the ScheduledEventManager class, which acts as a class decorator The scheduled event can then be referred to using the unique identifier defined in the register decorator.

Example: Register a StartResourceJob scheduled event

from DivvyWorkers.Processors.ScheduledEvents import ScheduledEventManager
@ScheduledEventManager.register('divvy.start_resource')
class StartResourceJob(MultiResourceScheduledEventJob):
    ...
 

Example: Register a scheduled event in a Bot action

from DivvyBotfactory.scheduling import ScheduledEventTracker
 
@registry.action(
    uid='divvy.action.start_resource_example',
    bulk_action=True,
    accepts_complement=True,
    ...
)
def start_resource_example(bot, settings, matches, non_matches):
    with ScheduledEventTracker() as context:
        for resource in matches:
            context.schedule_bot_event(
                bot=bot, resource=resource,
                description='Start a resource.',
                event_type='divvy.start_resource'
                schedule_data=schedule.Once(when=datetime.utcnow() + timedelta(hours=12))
            )
 

Dynamic event loading and unloading

When developing a plugin that Cloud Security (InsightCloudSec) dynamically loads and unloads, it is necessary to unload the plugin’s scheduled events using the ScheduledEventRegistryWrapper class, which is similar to standard event registration.

Example: Unload plugin events using the ScheduledEventRegistryWrapper class

from DivvyWorkers.Processors.ScheduledEvents import ScheduledEventRegistryWrapper
 
# Initialize the registry wrapper
events = ScheduledEventRegistryWrapper()
 
# Register a scheduled event job
events.register('divvy.start_resource')
class StartResourceJob(MultiResourceScheduledEventJob):
    ...
 
# Handle plugin loading/unloading
def load():
    events.load()
def unload():
    events.unload()
 

Plugin loading and unloading

To load a plugin in a non-Docker deployment, log in to a scheduler instance and copy the plugin to the plugins directory. It will be automatically be loaded into the software when the process starts.

Example: Copy custom plugin to the plugins directory

dev-box$ cp -r plugin_instance_type_blacklist plugins/
dev-box$ ls -alh plugins/
total 48
drwxr-xr-x   9 username  staff   306B Sep 12 07:57 .
drwxr-xr-x  74 username  staff   2.5K Sep 12 07:27 ..
-rw-r--r--   1 username  staff   2.0K Sep 12 07:57 plugin_instance_type_blacklist

For docker-compose deployments, the location of the plugins directory on your local filesystem is defined by mounting it as ./plugins inside the container.

Example: Mount plugins directory inside a Docker container

  volumes:
    - ./plugins:/plugins

To remove a plugin from the system you can either remove the file from the plugins directory or delete the directory.

Example: Remove plugins directory

dev-box$ rm -rf plugins/plugin_instance_type_blacklist
dev-box$ ls -alh plugins/
total 48
drwxr-xr-x   9 username  staff   306B Sep 12 07:57 .
drwxr-xr-x  74 username  staff   2.5K Sep 12 07:27 ..

Updates

Copy or rsync your updates into the plugin directory and restart all processes.

Warnings (UI)

For users with configured plugins, the landing page (available under the Settings > Integrations > Plugins) displays warnings surfaced during plugin loading. Individual plugins have a section containing the actual warnings on the Manage Plugin page, which you can access from the Action menu.