Skip to Content
Release NotesInsightcloudsecDev Upgrade Guide: 18.6 to 18.7

Jan 10, 2019

This guide highlights the breaking changes when upgrading from 18.6 to 18.7.

18.7 introduces a new scheduler with major performance improvements, a consolidated python package, a new unified command line interface, and a simplification from five worker types to a single unified worker process.

These major improvements introduce several breaking changes to deployment, configuration, and the API for scheduling custom jobs.

Unified New Command Interface

Prior to 18.7, the commands for running the webserver, scheduler, and workers were independent commands delivered as separate Python packages. These are now unified in a single package and available under the divvycloud command.

The command is self-documenting and is the best reference for options and configuration. View the help info by passing --help option to either the main command or any subcommand. For example, divvycloud --help or divvycloud scheduler --help.

When using docker, output of any help command can be retrieved by running docker run -it -rm divvycloud/divvycloud:latest divvycloud --help.

Configuration

18.7 drops support for the divvy.json in favor of directly configuring the divvycloud command via options or environment variables. As described in the --help document, all arguments can be substituted with environment variables prefixed by “DIVVY_” and by replacing dashes with underscores. For example:

  --redis-host -> DIVVY_REDIS_HOST

  For subcommands, add the subcommand name as an infix. Using the scheduler 
  as an example:

  --ha-enabled -> DIVVY_SCHEDULER_HA_ENABLED

All option defaults can be seen in the --help output.

For customers using Docker with an environment variable file, the variable names have changed to match the new option names. For backwards compatibility we do support the old names, but for clarity moving forward we do recommend renaming the environment variables.

For example DIVVY_MYSQL_DB_NAME is now named DIVVY_DB_NAME or passed as an option of --db-name.

Old:

# MySQL 5.7 Primary database
DIVVY_MYSQL_HOST=mysql
DIVVY_MYSQL_PORT=3306
DIVVY_MYSQL_USER=divvy
DIVVY_MYSQL_PASSWORD=divvy

# MySQL 5.7 Secure database
DIVVY_SECURE_MYSQL_HOST=mysql
DIVVY_SECURE_MYSQL_PORT=3306
DIVVY_SECURE_MYSQL_USER=divvy
DIVVY_SECURE_MYSQL_PASSWORD=divvy

# Redis
DIVVY_REDIS_HOST=redis
DIVVY_REDIS_PORT=6379

# Divvy Required - do not modify
VIRTUAL_ENV=/
DIVVY_MYSQL_DB_NAME=divvy
DIVVY_SECURE_MYSQL_DB_NAME=divvykeys

For legacy installs

add export in front of each line and place in your ~/.bash_profile

New:

# MySQL 5.7 Primary database
DIVVY_DB_HOST=mysql
DIVVY_DB_PORT=3306
DIVVY_DB_USERNAME=divvy
DIVVY_DB_PASSWORD=divvy

# MySQL 5.7 Secure database
DIVVY_SECRET_DB_HOST=mysql
DIVVY_SECRET_DB_PORT=3306
DIVVY_SECRET_DB_USERNAME=divvy
DIVVY_SECRET_DB_PASSWORD=divvy

# Redis
DIVVY_REDIS_HOST=redis
DIVVY_REDIS_PORT=6379

# Divvy Required - do not modify
VIRTUAL_ENV=/
DIVVY_DB_NAME=divvy
DIVVY_SECRET_DB_NAME=divvykeys

# Uncomment and adjust the below values if behind a proxy. Please note that
# 169.254.169.254 are used for AWS STS AssumeRole.
# http_proxy=http://proxy.acmecorp.com:8000
# https_proxy=http://proxy.acmecorp.com:8000
# no_proxy=mysql,redis,169.254.169.254
# May be needed if proxy is performing MITM operations. This tells the local container where to expect the update to the CA bundle.
# REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

Note:

Position of the command option matters. The following pattern should be used: divvycloud [OPTIONS] COMMAND [ARGS]. Only the scheduler subcommand has additional options.

Scheduler

Change the following:

divvyjobscheduler -n ..."

to

divvycloud [OPTIONS] scheduler

Webserver

Change the following:

divvyinterfaceserver -n ..."

to

divvycloud [OPTIONS] webserver

Simplified Worker

Prior to 18.7, a deployment would consist of multiple worker types that would pull jobs from their respective queues. There is now only one worker type with no configuration.

This has the benefit of simplifying deployment and scaling to meet capacity demands.

Simply remove the worker definitions using divvycloudworker.

divvycloudworker -n -t "harvester|harvester-long|processor|etc..."

And replace with a single worker.

divvycloud [OPTIONS] worker

Scale the number of workers to the sum of all individual worker types.

Removal of DivvyCloudProcessManager

For traditional installs the divvycloudprocessmanager is no longer available. Instead we recommend using systemd to manage processes.

Create a service file for webserver, scheduler, and worker. It’s important to note that the logs, plugins, and pid directory are assumed to be located in the same directory where the process is started. In the context of systemd this means we need to configure each of the three directories with the --logs, --plugins and --pid options respectively.

When providing the ExecStart command, use the full path to the executable in the virtual environment. This ensures that the Python path is correct and all dependencies are available.

# divvy-scheduler.service
[Unit]
Description=DivvyCloud Scheduler
After=multi-user.target

[Service]
EnvironmentFile=/home/divvycloud/.bash_profile
ExecStart=/home/divvy/divvycloud/bin/divvycloud --logs /path/to/logs --pid /path/to/pid --plugins /path/to/plugins scheduler
Restart=always
RestartSec=0

Note that only one webserver can run on each system.

# divvy-webserver.service
[Unit]
Description=DivvyCloud Webserver
After=multi-user.target

[Service]
EnvironmentFile=/home/divvycloud/.bash_profile
ExecStart=/home/divvy/divvycloud/bin/divvycloud [OPTIONS] webserver
Restart=always
RestartSec=0

For workers we want to allow for a parameterized system unit file. We do this by adding the @ to the file name.

# divvy-worker@.service
[Unit]
Description=DivvyCloud Worker, instance %i
After=multi-user.target

[Service]
EnvironmentFile=/home/divvycloud/.bash_profile
ExecStart=/home/divvy/divvycloud/bin/divvycloud [OPTIONS] worker
Restart=always
RestartSec=0

Then start the service systemctl start divvy-worker@1.service.

To start multiple workers use the range expansion syntax like systemctl start divvy-worker@{1..5}.service.

To stop the service run systemctl stop divvy-worker@n.service where n is the identifier.

To learn more about parameterized system commands, read this post .

Deployment-specific Guides

Docker

Kubernetes deployments and docker-compose configurations should be updated to reflect the new commands and worker model. The only process types should be scheduler, worker, and webserver. Note the changes to configuration via environment variables.

⚠️

Note

This Docker-compose file does not include MySQL or Redis information. Be sure to replace only the sections identified in “Before” with those identified in “After”.

Docker-compose Changes

Before

version: '2.2'
services:
  # Web Server
  interfaceserver:
    image: divvycloud/divvycloud:latest
    env_file:
      - ./prod.env
    environment:
      VIRTUAL_ENV: /
    restart: always
    volumes:
      - ./plugins:/plugins
    ports:
      - 8001:8001/tcp
    command:
      - divvyinterfaceserver
      - -n

  # Scheduler & Workers
  scheduler:
    image: divvycloud/divvycloud:latest
    env_file:
      - ./prod.env
    environment:
      VIRTUAL_ENV: /
    restart: always
    volumes:
      - ./plugins:/plugins
    command:
      - divvycloudjobscheduler
      - -n
      - --do-db-upgrade

  on-demand:
    image: divvycloud/divvycloud:latest
    scale: 1
    env_file:
      - ./prod.env
    environment:
      VIRTUAL_ENV: /
    restart: always
    volumes:
      - ./plugins:/plugins
    command:
      - divvycloudworker
      - -t
      - on-demand
      - -n

  harvester:
    image: divvycloud/divvycloud:latest
    scale: 4
    env_file:
      - ./prod.env
    environment:
      VIRTUAL_ENV: /
    restart: always
    volumes:
      - ./plugins:/plugins
    command:
      - divvycloudworker
      - -t
      - harvester
      - -n

  processor:
    image: divvycloud/divvycloud:latest
    scale: 1
    env_file:
      - ./prod.env
    environment:
      VIRTUAL_ENV: /
    restart: always
    volumes:
      - ./plugins:/plugins
    command:
      - divvycloudworker
      - -t
      - processor
      - -n

  longharvest:
    image: divvycloud/divvycloud:latest
    scale: 2
    env_file:
      - ./prod.env
    environment:
      VIRTUAL_ENV: /
    restart: always
    volumes:
      - ./plugins:/plugins
    command:
      - divvycloudworker
      - -t
      - harvester-long
      - -n

  events:
    image: divvycloud/divvycloud:latest
    scale: 1
    env_file:
      - ./prod.env
    environment:
      VIRTUAL_ENV: /
    restart: always
    volumes:
      - ./plugins:/plugins
    command:
      - divvycloudworker
      - -t
      - events
      - -n

After

version: '2.2'
services:
  # Web Server
  interfaceserver:
    image: divvycloud/divvycloud:latest
    env_file:
      - ./prod.env
    environment:
      VIRTUAL_ENV: /
    restart: always
    volumes:
      - ./plugins:/plugins
      # Inject proxy certificate/chain into container for enrollment in trusted CA bundle
      #- ./proxy_cert.crt:/usr/local/share/ca-certificates/proxy_cert.crt
    ports:
      - 8001:8001/tcp
    command:
      - divvycloud
      - webserver

  # Scheduler
  scheduler:
    image: divvycloud/divvycloud:latest
    command:
      - divvycloud
      - scheduler
      - --db-upgrade
    env_file:
      - ./prod.env
    environment:
      VIRTUAL_ENV: /
    restart: always
    scale: 1
    volumes:
      - ./plugins:/plugins

  # Workers
  worker:
    image: divvycloud/divvycloud:latest
    command:
      - divvycloud
      - worker
    env_file:
      - ./prod.env
    environment:
      VIRTUAL_ENV: /
    restart: always
    scale: 12
    volumes:
      - ./plugins:/plugins
      # Inject proxy certificate/chain into container for enrollment in trusted CA bundle
      #- ./proxy_cert.crt:/usr/local/share/ca-certificates/proxy_cert.crt

Traditional Install With Egg/Wheel Files

The change from egg to wheel packages requires rebuilding the virtual environment and reinstalling dependencies.

From the divvycloud virtualenv directory, make copies of the config, plugins, and logs directories if needed.

cd ~
cp divvycloud/config/divvy.json divvy.json
cp -r divvycloud/plugins plugins

Delete the divvycloud directory and recreate the virtual environment.

rm -rf divvycloud
virtualenv divvycloud

Reinstall divvycloud package and restore the plugins directory.

mv plugins divvycloud/plugins
source divvycloud/bin/activate
export INSTALL_URL='https://packages.divvycloud.com/packages'
pip install -f $INSTALL_URL divvycloud

Upgrading to a specific version

If you need to identify a specific version of DivvyCloud to which you are upgrading. You can do that by browsing our repository at http://packages.divvycloud.com/simple/divvycloud  to see the egg files for DivvyCloud with the most recent release at the bottom.

export DIVVY_VERSION='<major.minor.point | e.g., 17.03.33>'
pip install  -f ${INSTALL_URL} divvycloud==${DIVVY_VERSION}

Create the bash_profile:

# MySQL 5.7 Primary database
DIVVY_DB_HOST=mySQLHostHere
DIVVY_DB_PORT=3306
DIVVY_DB_USERNAME=divvy
DIVVY_DB_PASSWORD=divvy

# MySQL 5.7 Secure database
DIVVY_SECRET_DB_HOST=mySQLHostHere
DIVVY_SECRET_DB_PORT=3306
DIVVY_SECRET_DB_USERNAME=divvy
DIVVY_SECRET_DB_PASSWORD=divvy

# Redis
DIVVY_REDIS_HOST=RedisHostHere
DIVVY_REDIS_PORT=6379

# Divvy Required - do not modify
VIRTUAL_ENV=/
DIVVY_DB_NAME=divvy
DIVVY_SECRET_DB_NAME=divvykeys

# Uncomment and adjust the below values if behind a proxy. Please note that
# 169.254.169.254 are used for AWS STS AssumeRole.
# http_proxy=http://proxy.acmecorp.com:8000
# https_proxy=http://proxy.acmecorp.com:8000
# no_proxy=mysql,redis,169.254.169.254
# May be needed if proxy is performing MITM operations. This tells the local container where to expect the update to the CA bundle.
# REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
⚠️

If you receive permissions errors when running this script, chmod +x start.sh

Recreate the start.sh and stop.sh scripts
# !/bin/bash

# Read our env variables
source ~/.bash_profile

echo "Starting InterfaceServer"
divvycloud webserver > /dev/null 2>&1 &
sleep 10
echo "Starting Scheduler"
divvycloud scheduler --db-upgrade > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #1"
divvycloud worker > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #2"
divvycloud worker > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #3"
divvycloud worker > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #4"
divvycloud worker > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #5"
divvycloud worker > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #6"
divvycloud worker > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #7"
divvycloud worker > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #8"
divvycloud worker > /dev/null 2>&1 &
echo "Done!"
# !/bin/bash

kill $(ps aux | grep 'divvycloud' | grep -v grep | awk '{print $2}')

If you receive an error after upgrading

destroy the virtualenv and recreate it:

rm -rf divvycloud

virtualenv divvycloud

Now reinstall the divvycloud package:

cd divvycloud source bin/activate pip install -f $INSTALL_URL divvycloud

Custom Jobs

Prior to 18.7 the scheduler implicitly loaded and scheduled jobs based on functions being defined in the module. Developers must now explicitly register the job class with the worker and send a message to the scheduler requesting to be scheduled.

For those with developer licenses we encourage reviewing the plugins documentation shipped with the tool. Also, using the ImageWhitelist  available on github for reference, compare the harvester.py file between master and feature/18.7-support branches.

Changelog

  • added: new scheduler with major performance improvements
  • added: a consolidated python package
  • added: unified command line interface
  • improved: simplification from five worker types to a single unified worker process