AWS CloudTrail SQS

Amazon Web Services, or AWS, is a cloud service integration that allows you to track how your corporate cloud services are being used. Whether you are using Amazon’s Standard or GovCloud regions, you can configure AWS CloudTrail to send logs to InsightIDR.

To set up this event source:

1. Configure your AWS Settings.
2. Set up AWS CloudTrail using SQS in InsightIDR.

Configure your AWS Settings

Before you can send logs from AWS CloudTrail to InsightIDR, you must enable AWS CloudTrail in all of your regions, create a trail, create an SQS queue, subscribe the queue to the SNS topic, and set up AWS permissions.

To configure your AWS settings:

1. Enable AWS CloudTrail.
2. Create a trail.
3. Create an SQS Queue.
4. Subscribe the queue to the SNS topic.
5. Set up permissions.

Step 1: Enable AWS CloudTrail

To get maximum coverage of CloudTrail monitoring, you should enable CloudTrail in all your standard regions and GovCloud regions, even if you don't have any EC2 instances or other AWS resources running in all regions. Going forward, this helps ensure that if an attacker compromises a resource in your AWS account and they create or modify resources in other regions, you'll be able to monitor and detect on that behavior.

AWS S3 Regions

Depending on the region your CloudTrail logs are stored in, the collector will need to reach the following URL to collect the logs:

AWS CloudTrail is automatically enabled when an AWS account is created. All activity is recorded as an event and archived for 90 days. To help you store, analyze, and manage changes to your AWS resources, and extend the record of events beyond 90 days, you can create a CloudTrail trail.

GovCloud Regions

If you use AWS GovCloud, you can send data to InsightIDR for further analysis.

You can send data from the following GovCloud regions to InsightIDR:

Enable GovCloud Regions

For information on enabling GovCloud regions, see https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/verifying-cloudtrail.html.

Step 2: Create a Trail

A trail is a long-term record of your AWS account activity. It allows you to filter and continuously deliver the trail’s log files to an Amazon S3 bucket, which will be sent to InsightIDR. To learn more about CloudTrail trails, read the AWS documentation: https://docs.amazonaws.cn/en_us/awscloudtrail/latest/userguide/cloudtrail-tutorial.html

To create a trail:

1. In the AWS Console, go to CloudTrail > Trails > Create trail.
2. Add a name for your trail in the Trail name field.
3. For the Apply trail to all regions field, select Yes.
4. Click the Data Events section.
5. Select the appropriate S3 buckets or all buckets in your account.
6. Click the Storage Location section.
7. For the Create a new S3 bucket section, select Yes.
8. Add a name for your S3 bucket. Take note of this for later configuration in InsightIDR.
9. Expand the Advanced section.
10. For the Send SNS notification for every log file delivery field, select Yes.
11. For the Create a new SNS topic field, select Yes.
12. Add a name for your SNS Topic. Take note of this for later configuration in AWS.
13. Click Create.

See the AWS documentation for a more detailed explanation of using SNS with CloudTrail: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/configure-sns-notifications-for-cloudtrail.html#configure-cloudtrail-to-send-notifications

Step 3: Create an SQS Queue

1. In the AWS Console, go to Simple Queue Service > Create New Queue.
2. Type in a name for your queue in the Queue Name field.
3. For the What type of queue do you need? field, select Standard Queue.
4. Click Configure Queue if you want to set up the properties of the queue.
5. Click Create Queue.
6. Take note of the URL for your queue for later configuration in InsightIDR.

Step 4: Subscribe the SQS queue to the SNS topic

1. In the list of queues, find the queue you just created.
2. Right click on this queue and choose Subscribe Queue to SNS Topic.
3. Choose the Topic you created earlier from the Choose a Topic dropdown.
4. Click Subscribe.

To read more about subscribing to a SQS queue to a SNS topic, see the AWS documentation: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-subscribe-queue-sns-topic.html

Step 5: Set Up Permissions

Access to AWS is managed through IAM policies that are attached to IAM identities or AWS resources. An IAM policy defines the permissions for an identity, like a role or user, or an AWS resource. These policies are used to assess whether requests are allowed or denied. For more information about IAM policies see the AWS documentation: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/security_iam_service-with-iam.html

You can create an IAM policy or use an existing policy. We suggest that you set up permissions for:

• List and read permissions to the S3 bucket where you store the logs
• ReceiveMessage and DeleteMessage permissions for the SQS queue

Read the AWS documentation for more information on S3 bucket policies: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/create-s3-bucket-policy-for-cloudtrail.html

Create an IAM policy

1. Log in to your AWS Management Console and go to the IAM console.
2. From the left menu, click Policies > Create policy.
3. Set up a policy that grants permissions to the SQS queue you set up in the Create an SQS Queue procedure and the Amazon S3 bucket you created in the Create a Trail procedure to store your CloudTrail logs.
4. Configure your S3 permissions to be List and Read.
5. Configure your SQS queue permissions to be List, Read, and Write. Within each policy, you can select specific permissions like DeleteMessage and ReceiveMessage. At a minimum, you must select ReceiveMessage from Read and DeleteMessage from Write. Here is an example policy with these permissions:

json
1
{
2
"Version": "2012-10-17",
3
    "Statement": [
4
        {
5
            "Effect": "Allow",
6
            "Action": [
7
                "s3:Get*",
8
                "s3:List*"
9
            ],
10
            "Resource": [
11
                "arn:aws:s3:::CloudTrailsS3BucketNameGoesHere",
12
                "arn:aws:s3:::CloudTrailsS3BucketNameGoesHere/*"
13
            ]
14
        },
15
        {
16
            "Effect": "Allow",
17
            "Action": [
18
                "sqs:ReceiveMessage",
19
                "sqs:DeleteMessage"
20
            ],
21
            "Resource": [
22
                "arn:aws:sqs:region:account-id:resource-id"
23
            ]
24
        }
25
    ]
26
}

6. Create the policy.

Create and configure IAM User

Create the user that can access the group and inherit the privileges.

To create the user:

1. In the AWS Console, go to IAM → User → Add user.
2. Add a User name and select Programmatic Access under the Access Type section and select Next: Permissions.
3. Select the Group you created earlier and select Next: Review.
4. On the Complete page, select Show on the Secret Access Key.
5. Copy and save this User's Access Key and Secret Key in a secure location for later use.

Configure InsightIDR to collect data from the event source

After you complete the prerequisite steps and configure the event source to send data, you must add the event source in InsightIDR.

To configure the new event source in InsightIDR:

1. From the left menu, go to Data Collection and click Setup Event Source > Add Event Source.
2. Do one of the following:
   • Search for AWS CloudTrail in the event sources search bar.
   • In the Product Type filter, select Cloud Service.
3. Select the AWS CloudTrail event source tile.
4. Select your collector and select AWS CloudTrail from the event source dropdown menu.
5. Enter the name of your event source.
6. Optionally choose to send unparsed logs.
7. Select SQS Messages for the Collection Method.
8. Select IAM User Credential or EC2 Instance Profile Credential.
9. If you’re creating new credentials, enter the Secret Key and Access Key.
10. Enter the SQS Queue URL for the queue you created in AWS.
11. Select the appropriate Bucket Region Name.
12. Click Save.

Troubleshooting

This section covers common troubleshooting scenarios that you might experience when configuring the AWS CloudTrails event source using SQS.

InsightIDR Not Ingesting Logs

If you find that InsightIDR is not ingesting logs and data is not appearing, please do the following:

1. Check that your IAM policy is correct.
2. Check that you've used the right region.
3. Ensure there are actually logs in the S3 bucket.
4. Ensure that the S3 region of your event sources matches the S3 region used by your CloudTrail.

301 Error

If you encounter a 301 error, the S3 region in the event source does not match the region of the CloudTrail logs. Make sure that both the event source and your CloudTrail use the same S3 region.