AWS CloudTrail API

Amazon Web Services, or AWS, is a cloud service integration that allows you to track how your corporate cloud services are being used. Whether you are using Amazon’s Standard or GovCloud regions, you can configure AWS CloudTrail to send logs to InsightIDR.

To set up this event source:

1. Configure your AWS Settings
2. Set up this event source in InsightIDR

Configure your AWS Settings

Prior to sending logs from AWS CloudTrail to InsightIDR, you must enable access to your AWS regions, create an IAM policy, group, and user, and set up an Amazon S3 bucket policy. To configure your AWS Settings:

1. Enable access to your AWS regions
2. Create IAM Policy
3. Create IAM Group
4. Create and configure IAM User
5. Set up S3 bucket policy

Step 1: Enable Access to your AWS Regions

InsightIDR supports standard AWS regions and GovCloud regions.

AWS Regions

Depending on the region your CloudTrail logs are stored in, the collector will need to reach the following URL to collect the logs:

Enable CloudTrail in all Standard Regions

To get maximum coverage of CloudTrail monitoring, you should enable CloudTrail in all your standard regions, even if you don't have any EC2 instances or other AWS resources running in all regions. Going forward, this helps ensure that if an attacker compromises a resource in your AWS account and they create or modify resources in other regions, you'll be able to monitor and detect on that behavior.

To enable CloudTrail: To configure your AWS settings, you must Create a New Trail. During the trail configuration, you will be required to create a new S3 Bucket. Record the name of the S3 Bucket to a temporary text file, as you will need this when setting up the AWS CloudTrail in InsightIDR.

To create a Trail, follow Amazon's documentation at: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-a-trail-using-the-console-first-time.html

GovCloud Regions

If you use AWS GovCloud, you can send data to InsightIDR for further analysis.

You can send data from the following GovCloud regions to InsightIDR:

Enable GovCloud Regions

For information on enabling GovCloud regions, see Amazon's documentation at: https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/verifying-cloudtrail.html

Step 2: Create IAM Policy

You must create a IAM policy to control privileges and access. You should record the name and description you give to the policy to a temporary text file, as you will need when setting up AWS CloudTrail in InsightIDR.

To create a IAM policy, follow Amazon’s documentation at: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html

Enter a policy using the following Policy template, which is based on the principle of least privilege and allows access only to the specific Amazon S3 bucket you created for your CloudTrail logs.

text
1
{
2
    "Version": "2012-10-17",
3
    "Statement": [
4
        {
5
            "Effect": "Allow",
6
            "Action": [
7
                "s3:Get*",
8
                "s3:List*"
9
            ],
10
            "Resource": [
11
                "arn:aws:s3:::CloudTrailsS3BucketNameGoesHere",
12
                "arn:aws:s3:::CloudTrailsS3BucketNameGoesHere/*"
13
            ]
14
        }
15
    ]
16
}

Step 3: Create IAM Group

To control user access to a group with specific privileges, you must create a IAM group.

To create an IAM group, follow Amazon’s documentation at: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_groups_create.html

Step 4: Create and configure IAM User

You must create the user that can access the group and inherit the privileges.

To create and configure an IAM user, follow Amazon’s documentation at: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html

You should ensure that you:

• Select Programmatic Access for the users access type.
• Select the User Group you created previously to set permissions.
• Copy and save the User Access Key and Secret Key to a secure location for later use.

Step 5: Set up S3 Bucket Policy

Finally, create a policy for the S3 bucket that dictates that the user and S3 bucket data are associated with each other. You can see what region your data lies in by checking your Amazon Resource Name (ARN). For more information on ARNs and how they are formatted for GovCloud, see https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-arn-format.html.

To create the bucket policy:

1. In the AWS Console, navigate to Buckets and select the associated S3 bucket for CloudTrails.
2. Edit the bucket's Permissions.
3. Add List* and GetObject rights to the bucket that match the ARN of the user.
4. Edit the bucket's policy and enter the text below.

Bucket policy:

text
1
{
2
    "Version": "2012-10-17",
3
    "Statement": [
4
        {
5
            "Sid": "AWSCloudTrailAclCheck20150319",
6
            "Effect": "Allow",
7
            "Principal": {
8
                "Service": "cloudtrail.amazonaws.com"
9
            },
10
            "Action": "s3:GetBucketAcl",
11
            "Resource": "arn:aws:s3:::CLOUDTRAILS S3 BUCKET NAME"
12
        },
13
        {
14
            "Sid": "AWSCloudTrailWrite20150319",
15
            "Effect": "Allow",
16
            "Principal": {
17
                "Service": "cloudtrail.amazonaws.com"
18
            },
19
            "Action": "s3:PutObject",
20
            "Resource": "arn:aws:s3:::CLOUDTRAILS S3 BUCKET NAME/AWSLogs/AWS ACCOUNT NUMBER/*",
21
            "Condition": {
22
                "StringEquals": {
23
                    "s3:x-amz-acl": "bucket-owner-full-control"
24
                }
25
            }
26
        },
27
        {
28
            "Sid": "",
29
            "Effect": "Allow",
30
            "Principal": {
31
                "AWS": "arn:aws:iam::AWS ACCOUNT NUMBER:user/IAM USER NAME"
32
            },
33
            "Action": "s3:List*",
34
            "Resource": [
35
                "arn:aws:s3:::CLOUDTRAILS S3 BUCKET NAME",
36
                "arn:aws:s3:::CLOUDTRAILS S3 BUCKET NAME/*"
37
            ]
38
        },
39
        {
40
            "Sid": "",
41
            "Effect": "Allow",
42
            "Principal": {
43
                "AWS": "arn:aws:iam::AWS ACCOUNT NUMBER:user/IAM USER NAME"
44
            },
45
            "Action": "s3:GetObject",
46
            "Resource": "arn:aws:s3:::CLOUDTRAILS S3 BUCKET NAME/*"
47
        }
48
    ]
49
}

Configure InsightIDR to collect data from the event source

After you complete the prerequisite steps and configure the event source to send data, you must add the event source in InsightIDR.

To configure the new event source in InsightIDR:

1. From the left menu, go to Data Collection and click Setup Event Source > Add Event Source.
2. Do one of the following:
   • Search for AWS CloudTrail in the event sources search bar.
   • In the Product Type filter, select Cloud Service.
3. Select the AWS CloudTrail event source tile.
4. Select your collector and select AWS CloudTrail from the event source dropdown menu.
5. Enter the name of your event source.
6. Optionally choose to send unparsed logs.
7. Select your collection method, we recommend that you select IAM User Credential.
8. Select your existing credentials or use EC2 IAM Roles.
9. Enter the Secret Key that you created previosuly.
10. Enter the S3 Bucket Name that you created previously.
11. Optionally, enter the S3 Key Prefix that you created previously.
12. Select the Bucket Region Name.
13. Enter the refresh rate in minutes.
14. Configure your default domain and any Advanced Event Source Settings.
15. Click Save.

Troubleshoot your Event Source

This section covers some common troubleshooting scenarios.

InsightIDR Not Ingesting Logs

If you find that InsightIDR is not ingesting logs and data is not appearing, please do the following:

1. Check that your IAM policy is correct.
2. Check that you've used the right region.
3. Ensure there are actually logs in the S3 bucket.
4. Ensure that the S3 region of your event sources matches the S3 region used by your CloudTrail.

301 Error

If you encounter this error, this means that the S3 region in the event source does not match the region of the CloudTrail logs. Make sure that both the event source and your CloudTrail use the same S3 region.

Difficulty with S3 Key Prefix

Note that key prefixes are only necessary in the event source configuration if you configured one in AWS CloudTrail.

A normal structure without a key prefix is as follows: bucket_name/AWSLogs/Account ID/CloudTrail/region/YYYY/MM/DD/file_name.json.gz

Structures with a key prefix would look like this: bucket_name/prefix_name/AWSLogs/Account ID/CloudTrail/region/YYYY/MM/DD/file_name.json.gz

Adding an S3 Bucket Folder

To add an S3 bucket folder name, simply add / at the end of the bucket name.