Kubernetes Remote Scanner
InsightCloudSec’s new Kubernetes Remote Scanner expands our existing Kubernetes Security Guardrails and Container Vulnerability Assessment capabilities by improving InsightCloudSec’s agentless approach for better usability and simplified operation of harvesting Kubernetes entities that exist within different Kubernetes clusters running across different cloud accounts. This solution currently only works with managed clusters.
Feature Overview
- InsightCloudSec harvests the Kubernetes services via the cloud provider API and creates a matching cloud account for each Kubernetes cluster.
- Cluster access is generated using the account access credentials provided by the user.
- Cluster resources are harvested and associated with the parent cloud account that is created to model the Kubernetes service.
- InsightCloudSec automatically performs a scan, repeating scans periodically at a defined interval.
- InsightCloudSec automatically synchronizes the badges associated with the cloud account that contains the Kubernetes service and the tags associated with the Kubernetes cluster into the cloud account that represents the cluster. This allows the user to filter the Kubernetes resources and insights using these badges and tags.
- To migrate between scanners, refer to the instructions on the Clusters Account Setup & Management page.
Prerequisites & Deployment
The Kubernetes Remote Scanner requires the appropriate configuration and permissions for your desired Cloud Service Provider (CSP) accounts (e.g., EKS, AKS) in order to harvest and populate cloud accounts built on the data associated with your clusters. Before getting started with the Kubernetes Remote Scanner you will need to ensure you have the following:
- An existing InsightCloudSec installation (version 22.10.26 or later)
- Appropriate permissions for your desired Cloud Service Providers in order to harvest the resource information specific to each provider
- An existing Kubernetes cluster with network access available to InsightCloudSec
- Ports
443
and6443
are used for communications from InsightCloudSec to Kubernetes clusters - In general, we recommend that Network access be provided through a dedicated access list for InsightCloudSec
- Ports
Configuring Permissions
The following permissions must be provisioned by the user to enable the Kubernetes Remote Scanner.
Subjectaccessreview Details
For more information on the subjectaccessreview permission, see the Kubernetes Scanners Overview FAQ.
AWS
The following permissions are required for InsightCloudSec to access the Elastic Kubernetes Service (EKS):
1"eks:DescribeCluster",2"eks:ListClusters"
Azure
The following roles are required for InsightCloudSec to access the Azure Kubernetes Service (AKS):
- Azure Kubernetes Service Cluster User Role
- Azure Kubernetes Service RBAC Reader
The following write
permission is also required to create subjectaccessreviews:
Microsoft.Kubernetes/connectedClusters/authorization.k8s.io/subjectaccessreviews
GCP
The following permissions/roles are needed for a service account in order to scan a Kubernetes cluster in the Google Kubernetes Engine (GKE):
- Kubernetes Engine Cluster Viewer (
roles/container.clusterViewer
)- Provides access to get and list GKE clusters and get cluster credentials.
- Note: This role is a subset of the Kubernetes Engine Viewer role (
roles/container.viewer
) - Provides permissions:
container.clusters.get
,container.clusters.list
,resourcemanager.projects.get
,resourcemanager.projects.list
- Kubernetes Engine Viewer (
roles/container.viewer
)- Provides read-only access to resources within GKE clusters, such as nodes, pods, and GKE API objects
- Review the GCP documentation for more information about the permissions this role provides
- Permission needed for secrets visibility and validation:
container.secrets.list
- Permission needed for subjectaccessreview creation:
container.subjectAccessReviews.create
Onboarding Roles
One of the recommended roles for onboarding a GCP account, the Viewer role (roles/viewer
), includes most of the permissions above.
Configuring Kubernetes Access
Currently, InsightCloudSec offers support for the following CSPs and their Kubernetes service equivalents:
- AWS (Elastic Kubernetes Service(EKS))
- Azure (Azure Kubernetes Service (AKS))
- GCP (Google Kubernetes Engine (GKE))
AWS (EKS)
To enable access for EKS you will need to either:
- Update your existing AWS IAM Role/Read Access Permissions (InsightCloudSec Harvester role) to include EKS
OR
- Configure a new (dedicated, separate) AWS role for InsightCloudSec that provides read access to EKS
New Onboarding
As of InsightCloudSec v. 23.4.11, the new universal onboarding experience for AWS accounts uses CloudFormation Templates (CFTs) to automatically provision relevant accounts with the necessary policies and roles. This means it is easiest to perform Container Vulnerability Assessment (CVA) configuration while onboarding an account/organization. Review AWS Cloud - Onboarding for more information.
The majority of customers will use their existing AWS IAM Role for InsightCloudSec (the "harvesting" role) and update the configuration to provide read access to EKS clusters resources. To provide the role access to the Kubernetes clusters you will need to configure the role ARN in the cloud account and allow InsightCloudSec the permission to assume this role. Otherwise, you will create a new role and provide the harvester role with read access permissions to the EKS cluster.
For both options the role needs to be configured in the aws-auth ConfigMap
in each EKS cluster. The specific permissions are listed above.
Update Existing Role
Provisioning access permission for reading the Kubernetes cluster resources from EKS requires the following steps:
Define the access permissions in the EKS cluster RBAC:
- Create a ClusterRole using the provided YAML file that allows read access to all Kubernetes resources
- Create a ClusterRoleBinding to a Kubernetes RBAC group; this will bind the role to all service accounts that are under the provided group
MAP an IAM role using
eksctl
or by directly editing the ConfigMap. Refer to the AWS Docs for details
Open a command shell within the desired directory and run the following command to connect to your Kubernetes cluster in the command line, ensuring you replace the placeholder values appropriately:
aws eks --region <region-name> update-kubeconfig --name <cluster-name>
.
Review the AWS EKS Documentation for more information.Copy the following example file to a local file to the desired directory and save it as
ClusterRole-Create.yaml
:yaml1apiVersion: rbac.authorization.k8s.io/v12kind: ClusterRole3metadata:4name: cluster-reader-role5rules:6- apiGroups:7- '*'8resources:9- '*'10verbs:11- get12- list13- apiGroups:14- authorization.k8s.io15resources:16- subjectaccessreviews17verbs:18- createWithin the desired directory, run this command to create the cluster role in Kubernetes:
kubectl apply -f ClusterRole-Create.yaml
In the same directory, copy the following example file to a local file and save it as
ClusterRoleBinding-Apply.yaml
:yaml1apiVersion: rbac.authorization.k8s.io/v12kind: ClusterRoleBinding3metadata:4name: cluster-reader-binding5roleRef:6apiGroup: rbac.authorization.k8s.io7kind: ClusterRole8name: cluster-reader-role9subjects:10- kind: Group11name: cluster-reader-group12apiGroup: rbac.authorization.k8s.ioIn the same command shell, run this command to create the cluster role in Kubernetes:
kubectl apply -f ClusterRoleBinding-Apply.yaml
The final step is to map the existing IAM role to the cluster-reader-group
by configuring aws-auth
ConfigMap
. There are two ways to perform this configuration:
- Using
eksctl
, an open source tool that simplifies the configuration of EKS. - Directly editing the ConfigMap
Recommended Implementation
We encourage you to use the standard harvesting role by default because it already has the established trust relationship to the harvesting role (e.g., arn:aws:iam::123456789123:role/DivvyCloud-CustomerName-Install-Role
).
As a prerequisite to this step, you will need to know the ARN for the harvester role for making this mapping (e.g., arn:aws:iam::050283019178:role/rapid7-readonly
).
You can find this role under Cloud > Clouds on the individual cloud settings page in the Account Information card.
Using eksctl
- If you haven't already, download and install
ekstctl
. - Use the following command to perform the configuration:
shell
1eksctl create iamidentitymapping \2--cluster my-cluster \3--region=region-code \4--arn arn:aws:iam::111122223333:role/rapid7-readonly \5--group cluster-reader-group \6--no-duplicate-arns
Directly edit the ConfigMap
- Run the following command to open the
ConfigMap
for editing:kubectl edit -n kube-system configmap/aws-auth
.
This will openvim
by default; if you prefer to use a different editor, likenano
, use the following command instead:KUBE_EDITOR="nano" kubectl edit -n kube-system configmap/aws-auth
- Adjust the
maproles
section to look something like the example below, ensuring you replace therolearn
with the harvesting role mentioned above:
yaml
1apiVersion: v12data:3mapRoles: |4- groups:5- cluster-reader-group6rolearn: arn:aws:iam::111122223333:role/rapid7-readonly7username: my-role
(Optional) Configure a New Role
The following steps outline the details for the configuration of a separate role that provides access to EKS.
Note that the completion of these steps only create the new separate role and you will still need to grant access through the process outlined above.
Trust Policy Required
If you use a different role than the standard harvesting role, that role must have the same AssumeRolePolicy (Trust policy) to the ICS Installation role (e.g., arn:aws:iam::123456789123:role/DivvyCloud-CustomerName-Install-Role
).
- From InsightCloudSec navigate to Cloud>Clouds and select your target account from the Clouds Listing tab.
- Select the Settings tab and click the Update EKS Scanner Role button.
- Select or create your new separate role as desired and complete the form with the required account details.
- Click OK to finalize your changes.
- Refer to the steps outlined in the Provide an existing IAM Role With Read Access Permissions to EKS section and complete them as outlined for your new role.
Once you complete the steps to provide Read Access Permissions to EKS, Kubernetes access will be configured and the cluster will begin harvesting (assuming it is not in a paused state within InsightCloudSec).
Azure (AKS)
To enable access for AKS you will need to update your existing InsightCloudSec Azure Application (from Azure - Onboarding) to include the permissions/roles mentioned in Configuring Permissions.
Existing Azure Application
If you do not have an Azure account onboarded with InsightCloudSec, you will need to follow those instructions to setup an application and apply the necessary roles.
Login as an Admin to the Azure Dashboard for the account that hosts your InsightCloudSec Application.
Open the Subscription Access Control Dashboard.
- Navigate to the Subscriptions page.
- From the desired subscription's menu panel on the left, select Access control (IAM).
Attribute the Azure Kubernetes Service RBAC Reader role to the existing InsightCloudSec-associated Application registration.
- From the Access control (IAM) panel, click Add > Add role assignment.
- Search for Azure Kubernetes Service.
- Select the Azure Kubernetes Service RBAC Reader role.
- Click Next.
- Leave the Assign access to field as the default value (User, group, or service principal).
- Next to Members, click + Select members.
- In the Select panel, begin typing the name of the InsightCloudSec Azure Application. Select that application once it appears, then click Select.
- Click Review + assign to add the role.
Repeat the previous steps for the other required role.
- From the Access control (IAM) panel, click Add>Add role assignment.
- Search for Azure Kubernetes Service.
- Select the Azure Kubernetes Service Cluster User Role role.
- Click Next.
- Leave the Assign access to field as the default value (User, group, or service principal).
- Next to Members, click + Select members.
- In the Select panel, begin typing the name of the InsightCloudSec Azure Application. Select that application once it appears, then click Select.
- Click Review + assign to add the role.
Once you complete the steps to provide Read Access Permissions to AKS, Kubernetes access will be configured and the cluster will begin harvesting (assuming it is not in a paused state within InsightCloudSec).
GCP (GKE)
To enable access for GKE you will need to update your existing InsightCloudSec GCP Project (from GCP - Onboarding) to include the permissions/roles mentioned in Configuring Permissions.
Existing GCP Project
If you do not have an GCP project onboarded with InsightCloudSec, you will need to follow those instructions to setup an application and apply the necessary roles.
- Within your GCP console navigate into the project that is associated with InsightCloudSec.
- Navigate to IAM & Admin>Roles.
- Edit the role associated with InsightCloudSec.
- Find the role and click the vertical three dots to expose the menu.
- Click Edit.
- Click + ADD PERMISSIONS.
- Filter for the
"container.secrets.list"
permission and select the checkbox next to it. - Click ADD.
Once you complete the steps to provide Read Access Permissions to GKE, Kubernetes access will be configured and the cluster will begin harvesting (assuming it is not in a paused state within InsightCloudSec).
What's Next?
Refer to the Kubernetes Security Guardrails Overview page for an overview of this feature and a summary of the prerequisites.
Jump to the Using Kubernetes Security Guardrails page to view details on using the feature and exploring the imported data.