Set Up the AWS CloudTrail Event Source in InsightIDR

Amazon Web Services, or AWS, is a cloud service integration that allows you to track how your corporate cloud services are being used. Whether you are using Amazon’s Standard or GovCloud regions, you can configure AWS CloudTrail to send logs to InsightIDR.

You can also configure AWS CloudTrail using SQS

When you configure using the AWS CloudTrail API, InsightIDR queries the API periodically to see what has changed and then downloads the logs. When you configure with SQS, InsightIDR receives messages through an SQS notification when the S3 logs are created and ready to download. Both methods work well, however when you use the SQS method, InsightIDR is often able to gather logs faster because messages are created as soon as the S3 files are ready. For instructions on how to configure AWS CloudTrail with SQS, see the AWS CloudTrail SQS documentation.

To set up this event source:

  1. Configure your AWS Settings
  2. Set up this event source in InsightIDR

Configure your AWS Settings

Prior to sending logs from AWS CloudTrail to InsightIDR, you must enable access to your AWS regions, create an IAM policy, group, and user, and set up an S3 bucket policy. To configure your AWS Settings:

  1. Enable access to your AWS regions
  2. Create IAM Policy
  3. Create IAM Group
  4. Create and configure IAM User
  5. Set up S3 bucket policy

Step 1: Enable Access to your AWS Regions

InsightIDR supports standard AWS regions and GovCloud regions.

AWS S3 Regions

Depending on the region your CloudTrail logs are stored in, the collector will need to reach the following URL to collect the logs:

S3 Region

URL

US_STANDARD

https://s3.amazonaws.com

US_WEST_OREGON

https://s3-us-west-2.amazonaws.com

US_EAST_OHIO

http://s3-us-east-2.amazonaws.com

US_WEST_N_CALIFORNIA

https://s3-us-west-1.amazonaws.com

CA_CENTRAL

https://s3-ca-central-1.amazonaws.com

EU_IRELAND

https://s3-eu-west-1.amazonaws.com

EU_LONDON

https://s3-eu-west-2.amazonaws.com

EU_PARIS

https://s3-eu-west-3.amazonaws.com

EU_FRANKFURT

https://s3.eu-central-1.amazonaws.com

AP_MUMBAI

https://s3-ap-south-1.amazonaws.com

AP_SEOUL

https://s3-ap-northeast-2.amazonaws.com

AP_SINGAPORE

https://s3-ap-southeast-1.amazonaws.com

AP_SYDNEY

https://s3-ap-southeast-2.amazonaws.com

AP_TOKYO

https://s3-ap-northeast-1.amazonaws.com

SA_SAO_PAULO

https://s3-sa-east-1.amazonaws.com

Enable CloudTrail in all Standard Regions

To get maximum coverage of CloudTrail monitoring, you should enable CloudTrail in all your standard regions, even if you don't have any EC2 instances or other AWS resources running in all regions. Going forward, this helps ensure that if an attacker compromises a resource in your AWS account and they create or modify resources in other regions, you'll be able to monitor and alert on that behavior.

To enable CloudTrail:

  1. In the AWS Console, go to CloudTrail → Trails → Create new trail.
  2. Add a name for your trail in the "Trail name" field.
  3. For the "Apply trail to all regions" option, select Yes .
  4. For the "Create a new S3 bucket" option, select Yes.
  5. Add a name for your S3 bucket. Record this for future steps.
  6. Click Create.

GovCloud Regions

If you use AWS GovCloud, you can send data to InsightIDR for further analysis.

Please note that while AWS GovCloud complies with Federal cryptographic requirements, InsightIDR servers are hosted in standard AWS, and any data that you send to InsightIDR will be stored there as well.

You can send data from the following GovCloud regions to InsightIDR:

GovCloud Region

URL

US East (GovCloud)

s3-website.us-gov-east-1.amazonaws.com

US West (GovCloud)

s3-website.us-gov-west-1.amazonaws.com

Enable GovCloud Regions

For information on enabling GovCloud regions, see https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/verifying-cloudtrail.html.

Step 2: Create IAM Policy

Create the IAM policy to control privileges and access.

To create the IAM policy:

  1. In the AWS Console, go to IAM → Policies → Create Policy → Create Your Own Policy.
  2. Add a Name and Description for your Policy. Keep note of these for later use.
  3. Enter a policy using the following Policy template, which is based on the principle of least privilege and only allows access to the specific S3 bucket you created for your CloudTrail logs.
text
1
{
2
"Version": "2012-10-17",
3
"Statement": [
4
{
5
"Effect": "Allow",
6
"Action": [
7
"s3:Get*",
8
"s3:List*"
9
],
10
"Resource": [
11
"arn:aws:s3:::CloudTrailsS3BucketNameGoesHere",
12
"arn:aws:s3:::CloudTrailsS3BucketNameGoesHere/*"
13
]
14
}
15
]
16
}

Step 3: Create IAM Group

Create the IAM group, which controls user access to a group with specific privileges.

  1. In the AWS Console, go to IAM → Groups → Create New Group.
  2. Create a Group Name and select Next Step.
  3. Select the IAM Policy you created earlier and select Next Step.

Step 4: Create and configure IAM User

Create the user that can access the group and inherit the privileges.

To create the user:

  1. In the AWS Console, go to IAM → User → Add user.
  2. Add a User name and select **Programmatic Access **under the "Access Type" section and select Next: Permissions.
  3. Select the Group you created earlier and select Next: Review.
  4. On the "Complete" page, select Show on the Secret Access Key.
  5. Copy and save this User's Access Key and Secret Key in a secure location for later use.

Step 5: Set up S3 Bucket Policy

Finally, create a policy for the S3 bucket that dictates that the user and S3 bucket data are associated with each other. You can see what region your data lies in by checking your Amazon Resource Name (ARN). For more information on ARNs and how they are formatted for GovCloud, see https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-arn-format.html.

To create the bucket policy:

  1. In the AWS Console, go to the Bucket Policy or CORS configuration permissions pages, and find the ARN for the user associated with the acce
  1. Find the bucket configured for the CloudTrail logs.
  2. Go to the bucket properties in "S3" and click* Edit Bucket Policy.*
  3. Add List* and GetObject rights to the bucket that match the ARN of the user.
  4. Click Save.
text
1
{
2
"Version": "2012-10-17",
3
"Statement": [
4
{
5
"Sid": "AWSCloudTrailAclCheck20150319",
6
"Effect": "Allow",
7
"Principal": {
8
"Service": "cloudtrail.amazonaws.com"
9
},
10
"Action": "s3:GetBucketAcl",
11
"Resource": "arn:aws:s3:::CLOUDTRAILS S3 BUCKET NAME"
12
},
13
{
14
"Sid": "AWSCloudTrailWrite20150319",
15
"Effect": "Allow",
16
"Principal": {
17
"Service": "cloudtrail.amazonaws.com"
18
},
19
"Action": "s3:PutObject",
20
"Resource": "arn:aws:s3:::CLOUDTRAILS S3 BUCKET NAME/AWSLogs/AWS ACCOUNT NUMBER/*",
21
"Condition": {
22
"StringEquals": {
23
"s3:x-amz-acl": "bucket-owner-full-control"
24
}
25
}
26
},
27
{
28
"Sid": "",
29
"Effect": "Allow",
30
"Principal": {
31
"AWS": "arn:aws:iam::AWS ACCOUNT NUMBER:user/IAM USER NAME"
32
},
33
"Action": "s3:List*",
34
"Resource": [
35
"arn:aws:s3:::CLOUDTRAILS S3 BUCKET NAME",
36
"arn:aws:s3:::CLOUDTRAILS S3 BUCKET NAME/*"
37
]
38
},
39
{
40
"Sid": "",
41
"Effect": "Allow",
42
"Principal": {
43
"AWS": "arn:aws:iam::AWS ACCOUNT NUMBER:user/IAM USER NAME"
44
},
45
"Action": "s3:GetObject",
46
"Resource": "arn:aws:s3:::CLOUDTRAILS S3 BUCKET NAME/*"
47
}
48
]
49
}

How to Configure This Event Source

  1. From your dashboard, select Data Collection on the left hand menu.
  2. When the Data Collection page appears, click the Setup Event Source dropdown and choose Add Event Source.
  3. From the “Security Data” section, click the Cloud Services icon. The “Add Event Source” panel appears.
  4. Choose your collector and event source. You can also name your event source if you want.
  5. Choose a time zone and optionally display only US time zones.
  6. Optionally choose to send unfiltered logs.
  7. Select your existing credentials or use EC2 IAM Roles.
  8. Enter the Secret Key created in previous steps.
  9. Enter the S3 Bucket Name created in previous steps.
  10. Enter the S3 Key Prefix created in previous steps.
  11. Select the Bucket Region Name.
  12. Enter the refresh rate in minutes.
  13. Configure your default domain and any Advanced Event Source Settings
  14. Click Save.

Troubleshoot your Event Source

This section covers some common troubleshooting scenarios.

InsightIDR Not Ingesting Logs

If you find that InsightIDR is not ingesting logs and data is not appearing, please do the following:

  1. Check that your IAM policy is correct.
  2. Check that you've used the right region.
  3. Ensure there are actually logs in the S3 bucket.
  4. Ensure that the S3 region of your event sources matches the S3 region used by your CloudTrail.

301 Error

If you encounter this error, this means that the S3 region in the event source does not match the region of the CloudTrail logs. Make sure that both the event source and your CloudTrail use the same S3 region.

Difficulty with S3 Key Prefix

Note that key prefixes are only necessary in the event source configuration if you configured one in AWS CloudTrail.

A normal structure without a key prefix is as follows: bucket_name/AWSLogs/Account ID/CloudTrail/region/YYYY/MM/DD/file_name.json.gz

Structures with a key prefix would look like this: bucket_name/prefix_name/AWSLogs/Account ID/CloudTrail/region/YYYY/MM/DD/file_name.json.gz

Adding an S3 Bucket Folder

To add an S3 bucket folder name, simply add / at the end of the bucket name.