Kubernetes Automated Agent Deployment

The Kubernetes automated Agent deployment uses a Lambda/function to provide a seamless cluster discovery, scanner deployment, and upgrade experience. The deployment consists of the following components:

  • AWS Lambda/Azure Function/Google Cloud Function -- The Lambda/function will be responsible for initiating the deployment of the Guardrails Agent to Kubernetes clusters on the associated cloud provider(s). It will use Cloud Provider APIs to identify new clusters and trigger the installation and/or upgrade process; before the installation begins, the function will validate the discovered cluster to ensure it meets the necessary requirements for deployment. This function will be periodically invoked or can be triggered by cloud provider-specific events, e.g., a new cluster is created. You will only need to deploy the function type for the CSP containing the Kubernetes clusters you wish to monitor.
  • Agent -- The desired Agent will be deployed on the target Kubernetes clusters using Helm charts. The deployment can be configurable and other agents may be deployed. Currently, this deployment method only supports Kubernetes Guardrails.
  • Cloud Provider APIs -- Each cloud provider (AWS Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), GCP Google Kubernetes Engine (GKE)) offers APIs for cluster management. The Lambda/function will leverage these APIs to detect new clusters and obtain necessary information for deployment.
  • Permissions and Security -- The Lambda/function will require appropriate IAM roles and permissions to access the Cloud Provider APIs and install Helm charts on target clusters. These permissions should be tightly controlled to ensure security and prevent unauthorized access.

Prerequisites

There are a few general prerequisites as well as CSP-specific ones because deploying the Lambda/function to your environment requires special, explicit permissions.

  • Terraform
  • Local administrator permissions (to run Terraform)
  • Administrator or equivalent permissions in AWS, Azure, and/or GCP
    • AWS -- The Terraform script will be creating and attaching policies, setting up Cloudwatch, setting up a Lambda function, and allowing role assumption; these actions will need the requisite Admin or equivalent permissions inside AWS.
    • Azure -- The Terraform script will be setting up a resource group, storage account, and an Azure Function; these actions will need the requisite Admin or equivalent permissions inside Azure.
    • GCP -- The deployment scripts will be building a Docker image and pushing it to Google Cloud Repository, setting up Eventarc, setting up a Pub/Sub Topic, and setting up a Cloud Run Function; these actions will need the requisite Admin or equivalent permissions inside GCP.
    • For each CSP you configure, a scheduler equivalent will be created to invoke the Lambda/function automatically every 24 hours to perform an upgrade or install based on the version that is defined in the configuration file
  • The relevant CLI (aws, az, gcloud) for each CSP you plan to configure for the local scanner
    • If setting up for GCP, you'll also need the docker CLI

Step 1: Generate an API Key

Generating an API Key is required to identify and authenticate the local scanners (one on each cluster) and allow the scanner to report inventory and Guardrails assessment findings to the InsightCloudSec platform.

  1. Navigate to your InsightCloudSec installation and click to open Cloud > Kubernetes Clusters.
  2. On the top of the page click the Manage Kubernetes API Key button.
  3. Click Add API Key.
  4. Provide a name for the Key and ensure the Activate this API Key checkbox is selected, then click Create API Key.
  5. Copy the newly generated API key and store it in a safe place.

Save the Key!

This will be your only opportunity to save this information.

Step 2: Setup a Configuration File

Before setting up the Lambda/function to deploy the Guardrails Agent, you should first setup a JSON configuration file. Your configuration file will need to have the following:

  • A release to use for the configuration; this should be left as the default (k8s-guardrails)
  • A Helm configuration object that specifies the Agent repo, release name, URL, and version
  • A Guardrails configuration object that specifies an API key, InsightCloudSec URL, and specification version
  • (Optional) Cluster denylists for each CSP you plan on using

Configuration File Storage

InsightCloudSec highly recommends storing the configuration file in GitHub or an AWS S3 bucket for source control and improved revision. The configuration file will be passed as a parameter to the Terraform scripts that will then create an S3 bucket for the file so that it can be stored in a private bucket.

Review or copy the sample configuration file to get started:

Sample Configuration File
json
1
{
2
"configs": [
3
{
4
"release": "k8s-guardrails",
5
"helm_config": {
6
"repo_name": "r7-cloudsec-repo",
7
"release_name": "k8s-guardrails",
8
"repo_url": "https://helm.rapid7.com/cloudsec",
9
"version": "4.0.3"
10
},
11
"guardrails_config": {
12
"api_token": "someToken",
13
"base_url": "https://rapid7.com",
14
"spec_ver": "2.0"
15
},
16
"denylist_eks": [
17
{
18
"region": "us-west-2",
19
"clusters": [
20
"cluster-1",
21
"prod"
22
]
23
}
24
],
25
"denylist_aks": [
26
{
27
"resource_group": "rg-group",
28
"clusters": [
29
"my-k8s-1",
30
"prod"
31
]
32
}
33
],
34
"denylist_gke": [
35
{
36
"location": "us-central-1",
37
"clusters": [
38
"prod"
39
]
40
}
41
]
42
}
43
]
44
}

Step 3: Agent Deployment

The final step of the process is creating a Terraform plan and applying it to build out the necessary infrastructure inside the relevant CSP. Before applying the Terraform plan, you must be authenticated as a user with prescribed permissions using the relevant CLI inside a local terminal.

Only Configure the Necessary CSPs!

You only need to perform the configuration steps for the types of Kubernetes clusters you wish to monitor.

AWS EKS

For EKS clusters, a Lambda function will use the Amazon Elastic Kubernetes Service API to discover and validate clusters. For each valid cluster, the Helm chart for the Guardrails agent will be fetched from the appropriate Helm repository, then a CloudWatch Event Pattern will be created and will invoke the Lambda function.

To setup automatic deployment for AWS EKS:

  1. Download the Auto Deployer (r7-ics-deployer-tf) from InsightCloudSec and extract the files.
  2. Change into the AWS Auto Deployer directory (this example assumes the Auto Deployer was added to the Downloads directory): cd ~/Downloads/r7-ics-deployer-tf/auto-deployer-aws
  3. Update the terraform/terraform.tfvars file as necessary using terraform/variables.tf as a guide.
    • At a minimum, you'll need to provide a value for deployer_config_url (references the location of the configuration file you created in Step 2).
  4. Create and verify a terraform plan: terraform plan -out "auto-deployer-aws-plan"
  5. Apply the plan: terraform apply "auto-deployer-aws-plan"
  6. Optionally, if you would like to enable event-based deployment, e.g., each time a new EKS cluster has been created, you will need to configure permissions per cluster using eksctl before deploying the agent. Review the AWS EKS Remote Scanner Setup for details (the process is the same for the automated deployment).

Known EventBridge Issue

There is a known issue that the deployment returns an error even though it actually succeeded. Here is an example of the error: Error: creating EventBridge Target (r7-dev-scheduler-terraform-20231026183711756400000001): ResourceNotFoundException: Rule r7-dev-scheduler does not exist on EventBus default.

Azure AKS

For AKS clusters, an Azure Function will interact with the Azure Kubernetes Service API to discover and validate clusters. For each valid cluster, the Helm chart for the Guardrails agent will be obtained from the Helm repository, then an Event Subscription will be created to invoke the Function. Event-based deployment, e.g., each time a new AKS cluster has been created, is enabled out-of-the-box with no additional configuration.

To setup automatic deployment for Azure AKS:

  1. Download the Auto Deployer (r7-ics-deployer-tf) from InsightCloudSec and extract the files.
  2. Change into the Azure Auto Deployer directory (this example assumes the Auto Deployer was added to the Downloads directory): cd ~/Downloads/r7-ics-deployer-tf/auto-deployer-azure
  3. Update the terraform/terraform.tfvars file as necessary using terraform/variables.tf as a guide.
    • At a minimum, you'll need to provide a value for deployer_config_url (references the location of the configuration file you created in Step 2) as well as the following variables (these values can be recycled from an existing, onboarded Azure account):
      • client_id
      • client_secret
      • subscription_id
      • tenant_id
      • object_id (this can be found on the Overview page for the App Registration associated with InsightCloudSec/the deployer)
  4. Create and verify a terraform plan: terraform plan -out "auto-deployer-azure-plan"
  5. Apply the plan: terraform apply "auto-deployer-azure-plan"
GCP GKE

For GKE clusters, a Cloud Run Function will utilize the Google Kubernetes Engine API to list and validate clusters. For each valid cluster, the Helm chart will be fetched from the official Helm repository or a private Helm repository if required, then a new Cloud Pub/Sub trigger will be created that will invoke the function. Event-based deployment, e.g., each time a new GKE cluster has been created, is enabled out-of-the-box with no additional configuration.

To setup automatic deployment for GCP GKE:

  1. Ensure Docker and its CLI are installed on your machine.

  2. Download the Auto Deployer (r7-ics-deployer-tf) from InsightCloudSec and extract the files.

  3. Change into the GCP Auto Deployer directory (this example assumes the Auto Deployer was added to the Downloads directory): cd ~/Downloads/r7-ics-deployer-tf/auto-deployer-google

  4. Update the terraform/terraform.tfvars file as necessary using terraform/variables.tf as a guide.

    • At a minimum, you'll need to provide a value for image (this can be a custom value) and deployer_config_url (references the location of the configuration file you created in Step 2) as well as the following variables (these values can be recycled from an existing, onboarded GCP account):
      • secret_file

    Storage Bucket Owner Permissions Required

    The role associated with InsightCloudSec that will be used to run the deployer inside the relevant GCP Project will need to have the following GCP roles:

    • Storage Bucket Owner
    • Kubernetes Engine Admin
    • Eventarc Event Receiver
    • Cloud Run Invoker
  5. Run the build script with a custom Image name and a valid GCP Project ID: ./build-deploy.sh <project-id> <image-name>

  6. Create and verify a terraform plan: terraform plan -out "auto-deployer-google-plan"

  7. Apply the plan: terraform apply "auto-deployer-google-plan"