AWS CloudTrail Integration

AWS CloudTrail is a service that continuously monitors your AWS account activity and records events. It tracks user activity, API usage, and changes to your AWS resources, so that you have visibility into the actions being taken on your account. CloudTrail stores the information, like who made the request, the services used, the resource accessed, and actions that were taken. You can view, search, and download this information from your CloudTrail console.

To help you store, track, and respond to events, you can create trails from your console, the AWS CLI, or the CloudTrail API. The goal is to streamline your event history, so that you can oversee your AWS infrastructure and ensure compliance with your company policies and regulatory standards.

You can set up AWS CloudTrail to stream your AWS account activity directly into InsightOps. With the integration, you can use InsightOps to search your AWS account activity for specific actions, users who initiated specific actions, and resources impacted by them.

To set up the AWS CloudTrail integration, you’ll need to:

1. Enable AWS CloudTrail.
2. Create a new trail.
3. Create an SQS queue and subcribe it to the SNS topic.
4. Set up the privacy and trust relationship for the IAM policy and role.
5. Add the integration in InsightOps.

AWS CloudTrail is automatically enabled when an AWS account is created. All activity is recorded as an event and archived for 90 days. To help you store, analyze, and manage changes to your AWS resources, and extend the record of events beyond 90 days, you can create a CloudTrail trail.

A trail is a long-term record of your AWS account activity. It allows you to filter and continuously deliver the trail’s log files to an Amazon S3 bucket, which will be sent to InsightOps.

In order to set up the AWS CloudTrail integration with InsightOps, you will need to create a new trail.

To create a trail:

1. Log in to your AWS Management Console using your IAM user configured for CloudTrail administration. You’ll need to specify the region where you want to create your trail.
2. From the navigation pane, go to CloudTrail > Trails > Create new trail.
3. In the Trail name field, enter a descriptive name that helps you easily identify the purpose of the trail.
4. In the Storage Location section, choose whether you want to create a new Amazon S3 bucket or use an existing one, and then follow the instructions. The S3 bucket will be used to store your CloudTrail logs.
5. After you specify your S3 bucket, click the Advanced link to expand the menu.
6. Find the Send SNS notification for every log file delivery option and select Yes.
7. To create a new SNS topic, select Yes for the Create a New SNS topic option. Otherwise, to use an existing SNS topic, select No and choose the SNS topic you want to use from the dropdown.
8. Create the trail.

After you create a trail, you’ll need to:

1. Create an SQS queue.
2. Subscribe the queue to your SNS topic.

Step 1: Create an Amazon SQS queue

1. Log in to your AWS Management Console.
2. Go to the Simple Queue Service console and click the Create New Queue button.
3. On the Create New Queue page, verify that you have the correct region selected, and enter a name in the Queue Name field.
4. Choose the type of queue you want to create, which will determine the order and delivery of the messages : standard or First-In-First-Out (FIFO).
5. Click Quick-Create to create the queue with default parameters.

Your new queue will be available from the queue list, and you can subscribe the SNS topic you specified for your trail to it.

Step 2: Subscribe an Amazon SQS queue to the SNS topic

1. Log in to your AWS Management Console and go to the Simple Queue Service console .
2. From the list of queues, select the queue you want to which you want to subscribe an SNS topic.
3. Click the Queue Actions dropdown and choose Subscribe Queue to SNS Topic from the menu.
4. From the Subscribe to a Topic window, click the Choose a Topic dropdown and select the 6. SNS topic you want to subscribe to. Make sure you’re choosing the topic you specified for your trail.
5. Subscribe the queue to the topic.

Access to AWS is managed through IAM policies that are attached to IAM identities or AWS resources. An IAM policy defines the permissions for an identity, like a role or user, or an AWS resource. These policies are used to assess whether requests are allowed or denied.

To give an account access and permissions to an SNS topic and SQS queue, you’ll need to:

1. Create your IAM policy.
2. Create your IAM role.

Step 1: Create an IAM policy

1. Log in to your AWS Management Console and go to the IAM console .
2. From the left menu, choose Policies and choose Create policy.
3. Set up a policy that grants permissions to the SQS queue you set up earlier and the Amazon S3 bucket you created to to store your CloudTrail logs.
4. Configure your S3 permissions to be List and Read.
5. Configure your SQS queue permissions to be List, Read, and Write.
6. Create the policy.

Now you’re ready to create an IAM role and attach this policy to it.

Step 2: Create an IAM role

1. Log in to your AWS Management Console and go to the IAM console .
2. From the left menu, choose Roles and choose Create role.
3. When the Create Role page appears, choose the Another AWS account role type option as your trusted entity type.
4. Enter the account ID you want to grant access to your resources to.
5. Click Next: Permissions. On the Permissions section, attach the IAM policy you created earlier.
6. Click Next: Review. On the Review section, enter a name and description for the role. You will also see the policy attached to the role.
7. Create the role. It will be listed in the roles table.
8. From the roles table, find and click the role you just created to open the Summary page.
9. Go to the Trust relationships tab and click Edit Trust Relationship.
10. In the Trust Relationship policy document, you’ll need to update the AWS account in the Principal element to include the following Rapid7 ID: 336818582268. Your policy document might look like:

 
1
{
2
    "Version": "2012-10-17",
3
    "Statement": {
4
        "Effect": "Allow",
5
        "Principal": {"AWS": [
6
            "arn:aws:iam::336818582268:root",
7
        ]},
8
        "Action": "sts:AssumeRole"
9
    }
10
}

11. From the Trust Relationships tab, you will also need to set up an external ID that serves as an extra authentication layer. Select the Require external ID option. The external ID can be any word or number. You just need to remember it when you set up the integration from InsightOps.
12. Apply your changes.

You’ll need the name of the IAM role you created when you set up the CloudTrail integration with InsightOps.

1. Log in to InsightOps.
2. From the left menu, choose Data Collection.
3. When the Add Data Source page appears, click CloudTrail Integrations.
4. Click the Add New Integration button.
5. When the AWS CloudTrail panel appears, enter a name and description for the integration. Click Next to continue.
6. Under the Logs section, choose the destination log and log set you want to use to view your CloudTrail logs. Click Next to continue.
7. Under the AWS section, enter the following information:
   • AWS Account ID - To locate your account ID, log in to your AWS Management Console and go to your IAM console.
   • SQS Queue URL - To find your SQS Queue URL, log in to your AWS Management Console and go to your Simple Queue Service console. You can find the URL on the Details tab.
   • IAM role name - To find a list of available IAM roles, log in to your AWS Management Console and go to your IAM console. From the left menu, choose Roles. The resulting table will display all of the roles available.
   • IAM role external ID - The external ID designates that InsightOps can assume to role. You can find the external ID in the Trust Relationship policy document. The format will be: “sts”: ”ExternalID”.
8. Create the integration.

After you successfully add the integration, you will see it listed under the System Data section on the Add Data Source page. You'll be able to choose it as a source for sending data.