Compliance Exporting

InsightCloudSec offers the export of compliance information from Insight packs. The instructions below include steps to set up a report that is generated daily in the early hours of the morning UTC time and uploaded to the user's AWS or GCP storage bucket of choice using the credentials of an organization service account.

The contents of this report provide low level data mapping of resources to compliance issues. The content is intended for teams wanting to use the data for custom reports and integrations with external business intelligence and analytics tools. Linking one or more packs to an export configuration selects the organization service account for credentials and provides a bucket name and filename prefix for objects.

Create an Export Configuration using a storage container

1. Select a pack to link to an export configuration by clicking the link icon on the pack card.
2. Click Add Link. If there are no export configurations or if all of your links are already associated with configurations, you will be prompted to create a new configuration.
3. Click "Create Configuration" to access the "Manage Configurations" tab. This form can be used for creating or editing configurations by selecting an existing configuration or by completing the form to create a new one.

Once created, the configuration will be automatically linked to the selected pack.

Run your report

Once your configuration is created, you have the option to run the export report immediately. This is rate limited at one request per five minutes. We recommend running the report at least once to verify the bucket configuration is correct. Depending on the size of the installation and number of non-compliant resources, the report can take several minutes to run and upload to the bucket.

To run the report immediately, click the blue Run button.

Edit, unlink, and add configuration

From the Manage Export Configurations Links modal, you can also edit or unlink the configuration from the pack. To edit the configuration, select the blue button with the pencil. To unlink the configuration, select the middle button with the link.

Once a configuration is complete, you may link other packs to it. From the Insight Packs page, select the link icon on another pack you would like to link to an existing configuration. The following illustrates the modal that appears. The list of your existing configurations appears in the drop-down list box. Select one of these and select 'Add'.

Report contents

The report consists of two types of files. The first is a single report.json file which acts as a header file that supplies all of the meta data about packs, insights, resource types, and cloud accounts. The remaining files are the json-serialized non-compliant resources by insight.

 
1
.
2
└── myprefix-data-export-2019-02-21T18:40:18Z
3
    ├── backoffice-11.json
4
    ├── ... clip
5
    ├── backoffice-90.json
6
    ├── custom-91.json
7
    ├── custom-98.json
8
    └── report.json

Header file: report.json

The report.json file provides all the metadata necessary to programmatically understand the output in the insight files. The file is formatted as a single json object with six keys:

• name: Name of the report
• version
• resource_types: object where keys are resource_type name and value is an object of property names used in serialization and their type.
• organization_services: list of the name, organization_service_id, and account_id for all organization services.
• packs: list of packs. Each pack contains compliance metadata from their respective framework (e.g., HIPPA). It also lists the insights by ID that make up the pack. Insight IDs are grouped by source in two lists under the keys backoffice and custom. The source and ID are needed to look up insights.
• insights: object where key is <source>:<insight_id> and value is a serialized insight. The insight also has a count key that breaks down results and totals by organization_service_id.

Insight resources

A file is created for each insight and resources are serialized and written per line. This allows for lazy line-by-line reading for large datasets. Each line consists of a json object with two keys. The key resource_type is the InsightCloudSec internal resource type identifier and can be used to look up the properties of the serialization in the report.json file at the path report["resource_types"]["bigdatainstance"]. The resource key contains the resource serialization following the format in the header file.

 
1
{
2
    "resource": {
3
        "instance_flavor_resource_id": "instanceflavor:123:ap-southeast-1:bigdata:ds2.xlarge:",
4
        "db_name": null,
5
        "organization_service_id": 123,
6
        "name": "bi",
7
        "endpoint_address": "bi.cwqmjcawzudl.ap-southeast-1.redshift.amazonaws.com",
8
        "resource_id": "bigdatainstance:123:ap-southeast-1:bi:",
9
        "encrypted": false,
10
        "availability_zone": "ap-southeast-1a",
11
        "ssl_required": false,
12
        "instance_id": "bi",
13
        "instance_type": "ds2.xlarge",
14
        "state": "available",
15
        "version": "1.0",
16
        "publicly_accessible": false,
17
        "master_username": "root",
18
        "backup_retention": 1,
19
        "endpoint_port": 54xx,
20
        "key_resource_id": null,
21
        "id": 1,
22
        "tags": {
23
            "customer": "sample-international",
24
            "application": "sample-apac",
25
            "contact-email": "first.last@sample.com",
26
            "environment": "prod",
27
            "team": "sampleteam-international"
28
        },
29
        "region_name": "ap-southeast-1"
30
    },
31
    "resource_type": "bigdatainstance"
32
}

Export via email notification

Manage subscriptions (Notifications)

Email notifications (or subscriptions) associated with Insight packs are available for administration through the Insights landing page.

To read more about setting up SMTP, configuring email notifications, and creating pack-level notifications, check out our documentation on SMTP (Email Notifications).

After selecting the specific Insight Pack, you can select the envelope icon (Manage Subscriptions). This icon opens a page that provides a list of all subscriptions associated with the Insight Pack.

From this page you can add a new subscription, or modify an existing subscription by clicking on the ellipsis menu to display the following options:

• Send Now - Send a new report based on your Insight Pack immediately (On-Demand Report)
• Edit - modify or update your subscription
• Delete - remove your subscription