Compliance Exporting

Compliance reporting provides low-level data mapping of resources to compliance issues. The content is intended for teams wanting to use the data for custom reports and integrations with external business intelligence and analytics tools.

You can export compliance reporting for built-in and custom compliance packs using two methods: email subscriptions and export configuration links (also called cloud storage subscriptions).

ℹ️

Use Misconfigurations instead

While Compliance Exporting is still supported, Rapid7 strongly recommends using the Misconfigurations feature instead for improved subscription management and improved data visibility, reporting, and context around compliance.

Option 1: Create an export configuration link

A export configuration link (or cloud storage subscription) sends compliance reporting for a pack to a cloud storage service, AWS S3 for example, on a schedule.

⚠️

Bucket name and permissions

The bucket name provided must exist and the selected account must have object write permissions for that bucket.

To create an export configuration link for an Insight pack:

From the Command Platform, go to Controls & Compliance > Insights.
Click Compliance Packs or Custom Packs.
Click a pack name, then click Open Pack Details.
Click Options > Export Configuration Links.
Click Create Configuration.
Enter a Configuration Name.
Select a Cloud Account.
Enter an existing Bucket Name.
Enter a Directory Prefix.
Click Create.

Once your configuration is created, you have the option to run the export report immediately using the Run button. You can run the report on-demand every five minutes. We recommend running the report at least once to verify the bucket configuration is correct. Depending on the size of the installation and number of non-compliant resources, the report can take several minutes to run and upload to the bucket.

Example export configuration contents

Report contents

The report consists of two types of files. The first is a single report.json file which acts as a header file that supplies all of the meta data about packs, insights, resource types, and cloud accounts. The remaining files are the json-serialized non-compliant resources by insight.


.
└── myprefix-data-export-2019-02-21T18:40:18Z
    ├── backoffice-11.json
    ├── ... clip
    ├── backoffice-90.json
    ├── custom-91.json
    ├── custom-98.json
    └── report.json

ℹ️

Storage Buckets

Blob storage services like Amazon’s S3 and Google’s Storage Buckets do not have directories/folders for organizing objects. By convention, object names can have prefixes that look like file system paths. For example, /reports/year/month/day/report.json. This report names all objects with the standard format of <prefix>-data-export-<timestamp>/<file_name>.json, where prefix is supplied by the user.

Insight files are named with the following convention; <insight_name>:<insight_source (backoffice | custom)>:<insight_id>.

Header file: report.json

The report.json file provides all the metadata necessary to programmatically understand the output in the insight files. The file is formatted as a single json object with six keys:

name: Name of the report
version
resource_types: object where keys are resource_type name and value is an object of property names used in serialization and their type.
organization_services: list of the name, organization_service_id, and account_id for all organization services.
packs: list of packs. Each pack contains compliance metadata from their respective framework (for example, HIPPA). It also lists the insights by ID that make up the pack. Insight IDs are grouped by source in two lists under the keys backoffice and custom. The source and ID are needed to look up insights.
insights: object where key is <source>:<insight_id> and value is a serialized insight. The insight also has a count key that breaks down results and totals by organization_service_id.

⚠️

Insight Sources

Cloud Security (InsightCloudSec) has a repository of Insights we serve from an internal service that customer installs harvest down. These are referenced by their source, backoffice, and their ID, which is an integer.

Insights created in the customer install have the source of custom and an ID which is an integer.

Insight IDs are unique to their source. Use caution to ensure you are referencing the correct source. In the report.json header file packs list their insights IDs by source.

Insight resources

A file is created for each insight and resources are serialized and written per line. This allows for lazy line-by-line reading for large datasets. Each line consists of a json object with two keys. The key resource_type is the Cloud Security (InsightCloudSec) internal resource type identifier and can be used to look up the properties of the serialization in the report.json file at the path report["resource_types"]["bigdatainstance"]. The resource key contains the resource serialization following the format in the header file.


{
    "resource": {
        "instance_flavor_resource_id": "instanceflavor:123:ap-southeast-1:bigdata:ds2.xlarge:",
        "db_name": null,
        "organization_service_id": 123,
        "name": "bi",
        "endpoint_address": "bi.cwqmjcawzudl.ap-southeast-1.redshift.amazonaws.com",
        "resource_id": "bigdatainstance:123:ap-southeast-1:bi:",
        "encrypted": false,
        "availability_zone": "ap-southeast-1a",
        "ssl_required": false,
        "instance_id": "bi",
        "instance_type": "ds2.xlarge",
        "state": "available",
        "version": "1.0",
        "publicly_accessible": false,
        "master_username": "root",
        "backup_retention": 1,
        "endpoint_port": 54xx,
        "key_resource_id": null,
        "id": 1,
        "tags": {
            "customer": "sample-international",
            "application": "sample-apac",
            "contact-email": "first.last@sample.com",
            "environment": "prod",
            "team": "sampleteam-international"
        },
        "region_name": "ap-southeast-1"
    },
    "resource_type": "bigdatainstance"
}

ℹ️

Option 2: Create an email subscription

An email subscription sends compliance reporting for a pack to a list of emails on a schedule.

⚠️

Requires SMTP

Email subscriptions require SMTP to be enabled in your Cloud Security organization. Review Managing Organizations for more information.

To create an email subscription for an Insight pack:

From the Command Platform, go to Controls & Compliance > Insights.
Click Compliance Packs or Custom Packs.
Locate a pack, then click Action (…) > Create Email Subscription.
Enter a Subscription Name.
Optionally, enter a Description.
Select Recipient Email Addresses.
Configure the Email Frequency as needed.
Optionally, select Resource badges to include.
Optionally, select Include all resource tags.
Click OK.

Manage subscriptions and export configuration links

Subscriptions and export configuration links are associated with an Insight pack. As such, you can only manage an existing subscription or export configuration link from the Insight pack details.

To manage subscriptions:

From the Command Platform, go to Controls & Compliance > Insights.
Click Compliance Packs or Custom Packs.
Locate a pack, then click Action (…) > Manage Subscriptions.
Click Action (…) next to a subscription.

You can run the subscription on demand, edit its settings, or delete it.

To manage export configuration links:

From the Command Platform, go to Controls & Compliance > Insights.
Click Compliance Packs or Custom Packs.
Locate a pack, then click Action (…) > Manage Cloud Storage Subscription Links.
Click Action (…) next to a subscription.

You can run the subscription on demand, edit its settings, or delete it.