Harvesting & Event-Driven Harvesting Overview

Harvesting

Harvesting is the term we use to describe the process of data collection from a selected cloud service provider (CSP) within InsightCloudSec. Harvesting is performed on a standard polling schedule that can be customized by service, region, account, or cloud badge. This approach allows the InsightCloudSec platform to provide flexibility for customers around data collection based on their organization’s requirements.

For example, you can lower the harvesting cadence in regions where you do not have infrastructure deployed to reduce redundant API calls, but still maintain some harvesting to monitor for unauthorized usage in those regions. Administrators have the ability to fine-tune configurations on harvesting strategies for each resource type, either per region or globally, for any cloud provider.

Event-Driven Harvesting (EDH)

In addition to our basic harvesting, InsightCloudSec offers advanced harvesting through our Event-Driven Harvesting (EDH). This capability is used to augment standard polling-based harvesting.

  • For AWS it pulls data from AWS CloudWatch Events and AWS CloudTrail into a central Eventbus
  • For Azure it uses EventGrid for InsightCloudSec's consumption
  • For GCP it uses Cloud Asset Inventory for InsightCloudSec's consumption

By using events to initiate resource-specific harvesting, EDH improves the timeliness of resource updates and provides opportunities for rapid notification and/or remediation. The harvested events also enrich the data by identifying how resources enter noncompliant states. This detail enables auditing capabilities that function at scale.

Check out the following individual pages for details specific to that cloud service provider.

Frequently Asked Questions (FAQ)

Where can I view the harvesting status of a particular resource?
  • Each resource includes a details view with a Last Full Harvest field. Navigate to the Resources page, select your resource type, click on the actions menu and select View Details to open the detailed resource view.
    • This field may be blank because a resource was not recently harvested or the Redis cache (AWS-Only) was updated and the status is no longer stored in the database.
How can I manually trigger job harvesting?
  • Select an individual cloud account under Cloud > Clouds > Listing, and select the Harvest Info tab. Scroll to the right to select Enqueue Now to trigger harvesting for a single job or select multiple jobs under the Cloud Account and then click Enqueue Selected.
How long are EDH Events Retained?
  • EDH Events are retained for five days.
What happens if AWS CloudWatch/Azure/GCP sends all events to InsightCloudSec?
  • InsightCloudSec uses rules to process specific events for supported services. Other events are dropped.
How do rules get updated?
  • This varies depending on your deployment and configuration. InsightCloudSec can manage these rules automatically for you, or you may have elected to do so yourself in a manual way.
Can I manage the set up of EDH on my own and not give InsightCloudSec write permissions?
  • Yes for AWS this requires support to assist with a custom configuration.
  • For Azure this can be configured within the Azure Console

Reach out to us through the Customer Support Portal for assistance in either scenario.

Where should I look to start troubleshooting issues with messages not showing up?
  • SQS (AWS)
  • Cloud Collector Logs (Azure)
  • Docker logs
  • In the InsightCloudSec platform, as some logs are visible in the EDH section and in the admin logs section
When enabled EDH (for Azure) how long does auto-detection take for a new EDH Setup?

For Azure EDH the consumer should be detected on the first run of the collector and any producers will be detected upon the collector receiving an event.

Using Resource MetaData from EDH

This feature is currently only supported for AWS.

Beginning with 21.3, resource Event Driven Harvesting (EDH) metadata was expanded to allow the attachment of resources harvested using EDH to include the IP address of the resource’s creator.

Earlier, we introduced the ability to auto-tag provisioned resources with a divvy.creator tag, which is useful for tracking resource ownership and maintaining accountability. Now, we can also auto-tag provisioned resources with a divvy.creator_ip tag, which captures the IP address of the user who created the resource.

As part of these updates we added a new filter, Resource Provisioned From Unauthorized Network (AWS), that compares resource creator_ip addresses with authorized IPv4 networks to identify resources provisioned from outside of those networks.

To make the filter more useful, we recommend first creating a data collection for the filter called Authorized Networks. The data collection should have internal approved networks supplemented with AWS networks to surface true concerns.

Next, we recommend creating a Bot that runs using the Resource Created hook-point. The Bot can send notifications, take corrective actions, or both.

Product name to be replaced

You may observe that some components, screen captures, or examples use our former product name, DivvyCloud. This doesn't affect the configuration or the product's functionality, and we will notify you as we replace these component names.