Installing Remote Plugins

To ease plugin deployment, you can configure InsightCloudSec to download plugins from a remote location on system startup. The application can download these files from S3 or from arbitrary HTTP hosting, optionally with password protection.

Prerequisites

You must have InsightCloudSec plugins in .zip files. Each .zip archive should either contain exactly one plugin or a directory consisting of multiple plugins.

That .zip file must be hosted in:

• an S3 bucket that your InsightCloudSec installation already has access to via the s3:GetObject permission
• an S3 bucket that is either protected by basic auth or is open to the world
• an arbitrary HTTP-accessible location, protected by basic auth or open to the world
• access to the all EC2 instances hosting the Interface Server, Scheduler and Worker containers

Interface

The current interface consists of a collection of environment variables, shown below:

The DIVVY_REMOTE_PLUGIN environment variable is a list of JSON objects, each object representing a remote plugin store configuration described in this table:

AWS S3 Bucket

If the prefix of the url starts with s3:// the application will support the following behavior:

• If no username or password are specified, the application will assume that the InsightCloudSec scheduler is properly set up to support Instance AssumeRole to access that bucket.
• If the username and password are specified, the application will attempt to authenticate with those credentials where username is the API Key and password is the Secret Key.

HTTP/HTTPS storage

If the url starts with http:// or https:// and a username/password are supplied, the application will use basic auth to attempt to authenticate with the endpoint. This means that the application will support fetching a private git repo.

Example

For example, you could specify the following:

bash
1
export DIVVY_PLUGIN_REMOTE='
2
[{"url":"s3://bucket_name/plugin.zip"},
3
{"url":"https://test/many_compressed_plugins.zip", "multi_plugins":true}, 
4
{"url": "http://github.com/example/Plugins/archive/master.zip", "multi_plugins":true, "username": "xxxxxx", "password": "yyyyyy"}]
5
'

Using this configuration:

• fetches an archive of a single plugin from S3 using Instance AssumeRole ({"url":"s3://bucket_name/plugin.zip"}),
• fetches, unauthenticated, a .zip file consisting of many plugins, from a non-GitHub endpoint ({"url":"https://test/many_compressed_plugins.zip", "multi_plugins":true}),
• fetches, using basic authentication, an archive of many plugins hosted in a GitHub repo ({"url": "http://github.com/example/Plugins/archive/master.zip", "multi_plugins":true, "username": "xxxxxx", "password": "yyyyyy"}).

Steps to Implementing

This change should be added to prod.env on all EC2 instances.

1. SSH to the EC2 instance, locate the directory where prod.env is saved, stop the docker container before updating:

   bash
   1
   docker-compose down

2. Ensure the containers are stopped.

3. Update/Modify the prod.env file:

   vim
   1
   vi prod.env
   2
   
   3
   # MySQL 5.7 Primary database
   4
   DIVVY_DB_HOST=mysql
   5
   DIVVY_DB_PORT=3306
   6
   DIVVY_DB_USERNAME=divvy
   7
   DIVVY_DB_PASSWORD=divvy
   8
   
   9
   # MySQL 5.7 Secure database
   10
   DIVVY_SECRET_DB_HOST=mysql
   11
   DIVVY_SECRET_DB_PORT=3306
   12
   DIVVY_SECRET_DB_USERNAME=divvy
   13
   DIVVY_SECRET_DB_PASSWORD=divvy
   14
   
   15
   # Redis
   16
   DIVVY_REDIS_HOST=redis
   17
   DIVVY_REDIS_PORT=6379
   18
   
   

   becomes

   vim
   1
   # MySQL 5.7 Primary database
   2
   DIVVY_DB_HOST=mysql
   3
   DIVVY_DB_PORT=3306
   4
   DIVVY_DB_USERNAME=divvy
   5
   DIVVY_DB_PASSWORD=divvy
   6
   
   7
   # MySQL 5.7 Secure database
   8
   DIVVY_SECRET_DB_HOST=mysql
   9
   DIVVY_SECRET_DB_PORT=3306
   10
   DIVVY_SECRET_DB_USERNAME=divvy
   11
   DIVVY_SECRET_DB_PASSWORD=divvy
   12
   
   13
   # Redis
   14
   DIVVY_REDIS_HOST=redis
   15
   DIVVY_REDIS_PORT=6379
   16
   DIVVY_PLUGIN_REMOTE='[{"url":"s3://bucket_name/plugin.zip"}]'
   17
   
   

4. Start up the containers to apply the change:

   bash
   1
   docker-compose up

5. Verify the variables have been set in the container:

   bash
   1
   docker ps
   2
   
   3
   CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS                         PORTS                     NAMES
   4
   0a512082abfe        divvycloud/divvycloud:latest   "/entrypoint.sh divv…"   35 minutes ago 
   5
   
   6
   docker exec -it 0a512082abfe bash
   7
   
   8
   root@1b5bb7bebc37:/# env | grep DIVVY_PLUGIN_REMOTE
   9
   DIVVY_PLUGIN_REMOTE=[{"url":"s3://bucket_name/plugin.zip"}]

Advanced Configuration for Plugin Developers

The following environment variables are for use if you are developing plugins locally: