Working with Bots (Best Practices & Examples)
As a best practice, InsightCloudSec recommends creating Bots from Insights. The benefits of creating a Bot from an Insight are numerous but revolve around two key advantages: visibility and updates.
Bots that are created from a source Insight are linked to that Insight; whether you choose to build the Bot from an existing InsightCloudSec Insight or a custom Insight, you will always be able to see the relationship between the two.
A Bot created from an Insight:
- Enables you to keep track of how many Bots are tied to a specific Insight. Bots built ad-hoc are only traceable via individual Filters (at last count, InsightCloudSec has over 1300).
- Include Exemptions (which are directly linked to Insights) in your Bot's scope or behavior. Bots built ad-hoc will not support exemptions.
- Benefit from updates around Insights or filters that may apply to the associated Bot (e.g., a filter includes an improvement to provide better results, which your Bot can then take action around).
- For example, if any updates are made to the Insight (e.g., to add/modify/remove a filter, or changes to exemptions) the Bot will inherit these changes through the linked Insight.
Caution: Before Getting Started
Since every environment is different, we can offer no guarantees on the results, performance, or outcomes of any Bot example provided here. This includes performance specific to any Cloud Service Provider (e.g., some example Bots may perform well for GCP or Azure but not AWS and vice versa).
If you have access to a testing environment, in general we'd recommend verifying that your Bot performs as expected there before deploying to production. In addition, if you have general questions or concerns we are always happy to assist! Contact us at through the Customer Support Portal.
Exploring Bot Implementation
One way to familiarize yourself with Bots and learn more about using InsightCloudSec's automation through BotFactory is by reviewing and understanding both Resources and Insights.
Starting from Resources
Beginning from the Resources section of InsightCloudSec, you can focus on a specific resource type and examine all of the filters associated with that resource. Building any Insight involves selecting from hundreds of filters to help identify resources based on characteristics and filters.
For example, if you want to view all instances not in the Continental United States, you could view instances on the Resources page, apply and then appropriately configure the Resource not in Region filter.
- This scoped resource view can be saved as an Insight.
- The newly saved Custom Insight can be configured with additional parameters and used as the basis for creating a Bot.
A Bot created from this example Insight could be configured with actions to do any or all of the following:
- Notify the Org Admin when one of these instances is created
- Send the instance owner a reminder of corporate policy on resource location
- Pause or delete the instance
Starting with Insights
Another method of Bot creation takes advantage of an existing Insight. For many users the best approach to creating a custom Bot begins by using an existing InsightCloudSec Insight or even a custom Insight that is relevant to your needs.
In either scenario, a one-click Bot creation will build a Bot template focused on the details defined by that specific Insight.
For example, if you want to use an existing Insight, you can simply locate the Insight by name and click on the actions menu to launch the Create Bot process.
If you want to use an InsightCloudSec Insight as a starting point to customize and then build automation through a Bot, the best way is to locate the Insight and open the Impacted Resources column (this provides a scoped Resources view, which you can then edit and save as a Custom Insight).
Understanding Example Bots
This section includes a number of Bot examples and their underlying architecture. To be clear, this list demonstrates just a few of the possibilities for deploying Bots.
- We want you to explore examples through their components and behaviors to provide a closer look at the mechanics of a Bot.
- By reviewing these examples we hope you can get a better sense of some of the possibilities for how to build, deploy, and customize Bots and automation in your environment.
Important Notes - Recommended Approach
For each example provided below we recommend that you:
- Use Filters defined in the Bot to create a Custom Insight
- Use this Custom Insight as a starting point (via the actions menu) to build the Bot
If you choose to build any of the Bots outlined below (or any Bot for that matter) without an Insight as your starting point you will have to manually maintain the Bot including any additions/modifications/updates to the Filters or Scope.
In addition, Bots that are not created from an Insight will not include Exemptions because those are defined at the Insight level.
Permissions & Entitlements In order to create Bots, basic users will require Editor or Admin rights under Entitlements check out Basic User Groups, Roles, & Entitlements or the User Entitlements Matrix for additional information.
Reach out to your administrator or to support through the Customer Support Portalto address permission or entitlement issues.
Understanding Example Bot Formatting
The table below explains the format for the sample bots that follow.
| Bot Configuration Component | Configuration Details |
|---|---|
| Scope | Defines the scope of the Bot, typically the specified Resource Type (e.g., Access List, Instances, Databases, etc.) AND the cloud accounts that the bot will take action in. In general we recommend using the Badges functionality to manage scope. Details are available in the Badges documentation. |
| Filters | Filters are applied to the Bot (or if you are following the recommended approach, you will select your desired filter(s) and save these as your Custom Insight) |
| Action (1) | Defines the first thing that your Bot does e.g., send a Slack message, send an email, etc. |
| Jinja2 Template | Jinja2 Templating to craft your message. Check out Jinja2 for more information about using this feature. |
| Action (2) | Defines the additional thing that your Bot does e.g., stop an Instance, disable a user, etc. |
Permissions
Permissions are included where applicable but vary based on the behaviors of the Bot. Most permissions for users revolve around access to cloud resources.
- For example, if you only have View permissions, your Bot (which will inherit permissions of the creator) will not be able to take any lifecycle actions (e.g., start, stop, edit, etc.) on resources.
- If you have Modify permissions, some additional actions are available.
- In order to Delete, specific delete permissions are required.
In general, groups and roles are how users interact with cloud resources. Refer to User, Groups, and Roles (Administration) for details on permissions around users.
Entitlements manage feature usage within the InsightCloudSec platform. Details on Entitlements are available under our documentation on Basic User Groups, Roles, & Entitlements and detailed through the User Entitlements Matrix.
Example Bots
Refer to the details on each individual Bot below. These are recommendations but can always be adjusted to suit your environment or requirements. If you need help with the specifics or other configuration details, reach out to us through the Customer Support Portal.
Cleanup Resource Access Policy
Quite a few AWS resources are supported by the Bot action Cleanup Resource Access Policy, which removes public access to a target resource when its public access is enabled by its resource-based access policy. The Bot action removes public access to a resource by updating its resource-based access policy with one stripped of public access-enabling statements. This approach is more refined than detaching or deleting the resource-based access policy, which may have broader access effects than desired.
The action can be taken on individual resources or more systematically as a Bot action, which enables continuous monitoring and enforcement of private-only resource-based access policies.
It is currently available for the following AWS Resources:
- AWS Backup Vault (ICS Backup Vault)
- AWS Cloud Event Bus (ICS Service Event Bus)
- AWS Cloudsearch Cluster (ICS Search Cluster)
- AWS ECR (ICS Container Registry)
- AWS Glacier (ICS Cold Storage)
- AWS Lambda (ICS Serverless Function)
- AWS Opensearch (ICS Elasticsearch Instance)
- AWS S3 Buckets
- AWS Secret (ICS Secret)
- AWS SNS topics (ICS Notification Topic)
- AWS SQS message queues
- AWS VPC Endpoint/Private Link (ICS Network Endpoint)
Security Rules Exposing Public Access to Vulnerable Services
This Bot identifies access lists with (non web) ports open to the world. Examples of services and ports that are inspected include SSH (22), Redis (6379), MySQL (3306), and Microsoft RDP (3389).
You can customize this Bot to look at access list action (allow/deny), direction (ingress/egress), IP protocol (tcp, udp, icmp), list type (security group/NACL), and specific open ports.
- Permissions Required: Delete
| Bot Configuration | Configuration Details |
|---|---|
| Scope | Access Lists |
| Filters | - Access List Rule Ports - Access List Rule Open To World - Access List Rule IP Protocol - Access List Rule Direction - Access List Rule Action |
| Action (1) | Send Slack Message |
| Jinja2 Template | {{resource.parent_resource_id}} is exposing public access to vulnerable services. It is recommended that these rules be cleaned up, or this ACL/SG will be deleted in 72 hours. |
| Action (2) | Instance Stop/Start By Tag Value, or Schedule Deletion in 72 hours |
Cloud Users Without Multi-Factor Authentication (MFA) Enabled
This Bot identifies cloud user accounts that are not set up to leverage multi-factor authentication (MFA) when accessing the native cloud portal.
Allowing access to the native portal with standard credentials can expose the organization to attacks and access by malicious users via brute force, social engineering, and other techniques. MFA can reduce risk by requiring a second form of authentication.
- Permissions Required: Manage
| Bot Configuration | Configuration Details |
|---|---|
| Scope | Cloud User Accounts |
| Filter(s) | Cloud User Without MFA Enabled |
| Action (1) | Send Slack Message |
| Jinja2 Template | {{resource.get_resource_type()}}: {{resource.get_resource_name()}} in account {{resource. get_organization_service_name()}} with account # {{resource.get_organization_service().account_id}} does not have MFA enabled. Please enable Multi-factor Authentication, or this user will be disabled in 12 hours. |
| Action (2) | Disable Service User: 12 hours |
Clouds Without Global API Accounting (CloudTrail)
This Bot inspects Cloud Accounts for API Accounting services (such as AWS CloudTrail) across all regions. When enabled, this Bot ensures that all cloud activity within the native cloud console and via the programmatic API are captured for audit and tracking purposes.
- Permissions Required: View
| Bot Configuration | Configuration Details |
|---|---|
| Scope | Clouds |
| Filter(s) | Clouds Without Global API Accounting |
| Action (1) | Send Slack Message |
| Jinja2 Template | Please Enable CloudTrail in {{resource. get_organization_service_name()}} with account # {{resource.get_organization_service().account_id}}Steps to Enable: 1. Login to AWS Console (with appropriate permissions to View Identity Access Management Account Settings) 2. Click on Trails on the left navigation pane 3. Click Get Started Now, if presented - Click Add new trail - Enter a trail name in the Trail name box - Set the Apply trail to all regions option to Yes - Specify an S3 bucket name in the S3 bucket box - Click Create 4. If 1 or more trails already exist, select the target trail to enable for global logging - Click the edit icon (pencil) next to Apply trail to all regions - Click Yes - Click Save |
| Action (2) | N/A |
Snapshots Publicly Available
This Bot identifies Snapshots that are accessible to the public (snapshot, database snapshot). Many dramatic data leaks are related to snapshots and/or storage containers with buckets available to the public.
This Bot will discover potential data exposures as they occur and can be customized to send notifications to appropriate individuals or organizations.
- Permissions Required: Delete
| Bot Configuration Component | Configuration Details |
|---|---|
| Scope | Instances |
| Filters | - Resource is Exposed to Public - Snapshot Accessible to Public (works with Snapshot and DB Snapshot) |
| Action (1) | Send Slack Message |
| Jinja2 Template | Snapshot {{resource.snapshot_id}} has been identified as public. This violates our security policy. This snapshot resides in {{resource. get_organization_service_name()}}:{{resource.get_organization_service().account_id}} account. Please lock down this snapshot, or this will be deleted in 72 hours. |
| Action (2) | Schedule Deletion: 72 hours |
Storage Containers Exposing Public Permissions
This Bot identifies storage containers exposing data with permissive access lists. You can specify what permissions you’re looking for (full control, read-only, write) and customize this Bot to meet your needs.
To be slightly more aggressive, this Bot can be configured to use the Scheduled Deletion action.
- Permissions Required: Manage, and/or Delete (if using scheduled deletion)
| Bot Configuration Component | Configuration Details |
|---|---|
| Scope | Instances |
| Filters | - Resource is Exposed to Public - Storage Container Exposing Specific Permissions: read, write, full control, read acp, write acp - Storage Container Public Access |
| Action (1) | Send Slack Message |
| Jinja2 Template | An insecure storage container was identified with public permissions to the world. The details of this resource are: {{resource.serialize(indent=2)}}This storage container will have the public permissions removed in 4 hours. |
| Action (2) | Cleanup Exposed Storage Container: Set time * 4 hours. |
Instance Image ID Audit (a.k.a. Golden Image): Security
Use this Bot to Identify instances running an image that is not known by the system. This is typically the result of the parent image being deleted. Note that the filter does not search public images and only will search Shared/Private images.
- Permissions Required: Provision, Manage, Delete
| Bot Configuration Component | Configuration Details |
|---|---|
| Scope | Instances |
| Filters | Instance Image ID (you will need a data collection of the image IDs that you want to use as the gold standard, e.g., your allow list) |
| Action (1) | Send Slack Message |
| Jinja2 Template | Starting and stopping: {{event.resource.get_resource_name()}}. Check scheduled events under bot Schedule -* Instance Scheduler to confirm the events are scheduled. This bot will start and/or stop instances based on the tag value of a specified tag key. Note, tag value must be in UTC and follow HHMM format. For example, "1835" would schedule the lifecycle action to run at 6:35PM UTC. This resource has the following tags:{{resource.serialize_tags(indent=2)}} |
| Action (2) | Instance Stop/Start By Tag Value |
Instance Scheduler (a.k.a. Sleeper Bot)
This Bot periodically starts and stops instances at a set time each day. It reduces spend on instances by shutting down and (optionally) restarting instances that do not need to be running 24/7.
You can identify these instances by tagging them with key/value pairs to denote what time they should be shut down or started, daily.
- Permissions needed: Manage
| Bot Configuration Component | Configuration Details |
|---|---|
| Scope | Instances |
| Filters | Resource Contains Tag Keys: shutdown_time and/or startup_time: Schedule stop/start lifecycle actions on instances based on tag value of a specified tag key. Note: tag value must be in UTC and follow HHMM format. For example, "1835" would schedule the lifecycle action to run at 6:35PM UTC. |
| Action (1) | Send Slack Message |
| Jinja2 Template | Starting and stopping: {{event.resource.get_resource_name()}}. Check scheduled events under bot Schedule -* Instance Scheduler to confirm the events are scheduled. This bot will start and/or stop instances based on the tag value of a specified tag key. This resource has the following tags:{{resource.serialize_tags(indent=2)}} |
| Action (2) | Instance Stop/Start By Tag Value |
Instance Daily Backup
This Bot creates backups of database instances (snapshot) on a daily basis.
You can identify database instances to back up, and using a separate Bot, you can delete backups older than "x" days. This is an important best practice because when a database instance in AWS gets deleted, all automated backups also get deleted. With a manual backup, you’ll lose your data.
- Permissions needed: Provision, Manage, Delete
Additional Notes: For the Cleanup Bot, you can evaluate Database snapshots, Resource Age Exceeds (e.g., 30 days), and then use scheduled deletion with a Slack message letting users know it’s going to be deleted because a new snapshot is taken by the Bot every 14 days.
| Bot Configuration Component | Configuration Details |
|---|---|
| Scope | Instances |
| Filters | Instance Manual Backup Age: 14 days |
| Action (1) | Send Slack Message |
| Jinja2 Template | Database instance {{resource.name}} has had no manual backup within 14 days. This database instance resides in {{resource. get_organization_service_name()}}:{{resource.get_organization_service().account_id}} account and has an endpoint of {{resource.endpoint_address}}.This resource will have a snapshot taken. |
| Action (2) | Create Database Instance Snapshot: 3 hours |
Instances Averaging Low CPU
This Bot identifies compute instances that have been averaging less than user-defined CPU over the user-defined time window.
For example, you can find instances that are using less than 4% CPU over a period of 14 days. The Bot can execute nightly and send notifications about those resources. In addition, the Bot can schedule an instance downsize the next day and, if not stopped, execute the instance downsize to reduce costs.
- Permissions Required: View
| Bot Configuration Component | Configuration Details |
|---|---|
| Scope | Instances |
| Filters | Instance Averaging Low CPU: Match Instances with an Average CPU usage percentage greater or equal to a user defined value, for user defined number of days. |
| Action (1) | Send Slack Message |
| Jinja2 Template | Instance has been averaging low CPU for X (user-defined in filter above) days. Here are the details: {{resource.serialize(indent=2)}} Please downsize this instance for cost savings. |
| Action (2) | Downsize instance |
Resources with TTL (a.k.a. Temp Resources): Optimization
This Bot works in coordination with a tagging policy that allows resource owners to specify a time to live (TTL).
The Bot inspects resource tags, and when it finds a resource with a TTL tag key, it schedules its deletion in the future for the number of hours in its TTL tag value. An effective tagging policy in conjunction with this Bot can improve your organization’s management of cost.
- Permissions Required: Delete
| Bot Configuration Component | Configuration Details |
|---|---|
| Scope | Big Data Instances, Big Data Snapshot, Content Delivery Networks, Database Instance, Database Snapshot, Hypervisor, Instance, Reserved Instance, Load Balancer, Cache Instance, Snapshot, Network Interface, Private Image, Private Network, Private Subnet, Public IP, Access List,Access List Rule, Route Table, Cloud User, Snapshot, SSH Key Pair, Storage Container, Volume |
| Filters | Resource Contains Tag Keys: TTL Note: TTL value must follow #d#h#m format for days, hours, and minutes. For example, "2d3h0m" would schedule the Bot to run in two days and three hours. |
| Action (1) | Send Slack Message |
| Jinja2 Template | {{resource.get_resource_type()}}: {{resource.get_resource_name()}} in account {{resource. get_organization_service_name()}} with account # {{resource.get_organization_service().account_id}} has a Time to Live Tag. This resource will be deleted based upon TTL value associated with TTL key. Resource Tags: {{resource.serialize_tags(indent=2)}} |
| Action (2) | Time To Live Deletion: (Schedules deletion based upon TTL value associated with TTL key. Note, TTL value must follow #d#h#m format for days, hours, and minutes. For example, "2d3h0m" would schedule the Bot to run in two days and three hours.) |
Resources Orphaned: Best Practices
Identifies resources that do not appear to be in use. It looks for detached volumes, access lists without instances, detached public IPs, disconnected network interfaces, detached Internet gateways, abandoned load balancers, SSH key pairs not in use, and more. By identifying these resources, you can clean up your environment and reduce costs.
- Permissions Required: Delete
| Bot Configuration Component | Configuration Details |
|---|---|
| Scope | Supports: Volume, Access List, Hypervisor, Load Balancer, SSH Key Pair, Route Table, Public IP, Network Interface, Internet Gateway, Service Group |
| Filters | Resource Orphaned: Supports: Volume, Access List, Hypervisor, Load Balancer, SSH Key Pair, Route Table, Public IP, Network Interface, Internet Gateway, Service Group |
| Action (1) | Send Slack Message |
| Jinja2 Template | {{resource.get_resource_type()}}: {{resource.get_resource_name()}} in account {{resource. get_organization_service_name()}} with account # {{resource.get_organization_service().account_id}} is orphaned. Please log in to clean up, or this resource will be deleted in 72 hours. |
| Action (2) | Schedule Deletion: 72 hours |
Insights That Make Great Bots
Below are just a few of the InsightCloudSec out-of-the-box Insights we offer that work as a great starting point for your Bot creation. To use any of the Insights mentioned below, simply locate the desired Insight with the name and use the actions menu to create your Bot.
- In the image below, the search for "SSH" has provided several results; Click on the actions menu to the left of the Insight name to launch the Bot creation process for the Insight you want to use.
- Details are available for Creating Bots and Managing Bots
Available Insights include:
- Access List Exposing SSH to World (Security Group): Identifies security groups exposing port 22 to the world (0.0.0.0/0).
- API Accounting Log Exposed: Identifies storage containers, e.g., AWS S3, that expose API accounting logs, e.g., AWS CloudTrail logs.
- Cloud Account Password Policy Not CIS-compliant: Identifies cloud accounts with password policies not compliant with Center for Internet Security (CIS) guidelines.
- Cloud Account Without Global API Accounting: Identifies accounts without API accounting, e.g., AWS CloudTrail, enabled across all regions.
- Cloud Root Account API Access Key Present: Identifies accounts with API access keys present on the root account.
- Cloud User Inactive: Identifies cloud users who have not logged in for a period of time.
- Database Instance Publicly Accessible: Identifies database instances that are accessible to the public.
- Database Instance Without Recent Snapshot: Identifies database instances without a recent manual snapshot.
- Instance Exposing SSH to World: Identifies compute instances exposing port 22 to the world.
- Instance Running 24x7: Identifies instances that have been running 24x7 over a given period of time.
- Load Balancer Without SSL Listener: Identifies load balancers without an SSL listener.
- Network Without Traffic Logging: Identifies cloud networks, e.g., AWS VPCs, that do not have network logging enabled.
- Resource Orphaned: Scans a variety of resources, e.g., volumes, load balancers, public IPs, Internet gateways, etc., to identify orphans.
- Resource Tagging Policy 101: Identifies resources without the Name, Environment and Owner tag keys.
- Snapshot Publicly Available: Identifies snapshots that are accessible to the public.
- Storage Container Exposing Access to World: Identifies storage containers, e.g. AWS S3 buckets, exposing data with permissive access lists.
- Volume With PHI Unencrypted: Identifies volumes with Protected Health Information (PHI) tags that are not encrypted.