Amazon Security Lake
Alternatives to using Amazon Security Lake to configure AWS CloudTrail
Amazon Security Lake is one of three options you have when it comes to ingesting CloudTrail logs from AWS. The other options are to ingest via SQS or via the CloudTrail API. The Amazon Security Lake and SQS integrations will ingest logs as they are generated, meaning that data will often show up in InsightIDR faster than if you were to use the CloudTrail API integration (which queries the API periodically to see what has changed and then downloads the logs). Currently, there is little difference between the SQS and Amazon Security Lake options. In the future, we plan to add support for ingestion of additional logs via Amazon Security Lake, so using this option could give you more flexibility in terms of sending additional data to InsightIDR down the road.
Amazon Security Lake is a security data lake service that allows you to centrally aggregate, manage, and use security-related logs. All logs in Amazon Security Lake are formatted using the OCSF standard. The Amazon Security Lake event source allows Rapid7 customers to integrate with the service and feed logs from AWS CloudTrail into InsightIDR.
Amazon Security Lake customers can send logs to third-party solutions like InsightIDR by creating a subscription in Security Lake. Customers can then select which logs to include in the subscription. The third-party solution consumes logs from the subscription via an SQS queue that is created with the subscription.
To set up Amazon Security Lake, you’ll need to:
- Configure Amazon Security Lake.
- Configure a Collector.
- Set up Amazon Security Lake in InsightIDR.
- Test the configuration.
Configure Amazon Security Lake
Task 1: Enable Amazon Security Lake
If you haven’t already, ensure that you have enabled Amazon Security Lake by following the instructions at: https://docs.aws.amazon.com/security-lake/latest/userguide/getting-started.html#enable-service
If you have (or expect to have) multiple AWS accounts, we strongly encourage you to make use of AWS Organizations and to set up Amazon Security Lake at the Organization level.
Task 2: Add a CloudTrail data source to Amazon Security Lake
Once you have turned Amazon Security Lake on, add CloudTrail as a data source by following the instructions at: https://docs.aws.amazon.com/security-lake/latest/userguide/internal-sources.html
Ensure CloudTrail is switched on for all AWS regions.
When adding CloudTrail as a data source for Amazon Security Lake, to ensure that you have full visibility into activity across your entire AWS footprint, be sure to turn on the CloudTrail data source for all regions in all your AWS accounts.
Task 3: Create a subscriber in Amazon Security Lake
Although Amazon Security Lake started capturing your CloudTrail logs upon completion of Task 2: Add CloudTrail Source to Amazon Security Lake, you must create a subscriber for InsightIDR to access the CloudTrail data in Security Lake.
Add a subscriber to Amazon Security Lake by following the instructions at: https://docs.aws.amazon.com/security-lake/latest/userguide/subscriber-data-access.html. You do not need to create an IAM role for EventBridge as the InsightIDR integration uses SQS, not EventBridge.
To set up the subscriber:
- For the Subscriber Name, enter a descriptive name (for example, Rapid7-InsightIDR).
- Under Log and event sources, select Specific log and event sources > CloudTrail.
- Under Data access method, select S3.
- For the Account ID in the Subscriber credentials section, enter the AWS account ID for the account you are currently logged in to. For the External ID, enter a unique, unguessable combination of letters and numbers.
- Under Notification details (S3 only), select SQS queue.
- Click the Create button. The list of Amazon Security Lake subscribers displays with the new subscriber you just created.
- Click on the name of the subscriber you just created to see more details.
- Make note of the ARN for the AWS role ID and the ARN for the Subscription endpoint. You will need both of these ARNs later in the setup.
Configure a Collector
Task 1: Create an EC2 Collector
As with most InsightIDR integrations, the data from Amazon Security Lake is sent to InsightIDR via a Collector. It is easiest to do this if you have a Collector running on an EC2 instance in your AWS environment. If you already have a Collector installed in your AWS environment, you can use it. If you haven’t installed a Collector already, you can install one on an EC2 instance by following our documentation.
In the Access AWS Resources with EC2 IAM Roles instructions, you can skip the Configure IAM Policy step and return to this documentation when you reach the Update the IAM Role Permissions step.
Task 2: Create policies to access Amazon Security Lake data
Amazon Security Lake automatically creates an IAM role for each subscriber. However, this role is not designed to be assumed by an EC2 instance, so it cannot be used with the Collector. Instead, the permissions must be copied into new IAM policies that can be added to the Collector’s role.
- In Task 3: Create a subscriber in Amazon Security Lake, you noted the ARN for the AWS role ID of the subscriber that was created (for example, arn:aws:iam::123456789012:role/
AmazonSecurityLake-a12bc345-6de7-89f0-ghi1-234jkl5678mn
.) Copy the highlighted portion of the AWS role ID. - In the AWS console, navigate to IAM > Roles, and search for the highlighted portion of the AWS role ID.
- Open the role returned by the search. A permission policy ending in SQS and a permission policy ending in S3 display.
- Expand the policy ending in SQS and copy it.
- Open the Policies section of IAM and click the Create policy button.
- Switch to the JSON tab, and paste the permission policy ending in SQS.
- Click Next to add any tags you might want. Click Next again and add a descriptive name for the policy (for example, r7idrsecuritylakesqs). Create the policy.
- Repeat steps 4-7 for the policy ending in S3.
- Open the IAM role for the Collector and attach the two policies you just created.
Because we are not using the role created by Amazon Security Lake (due to the fact that an EC2 instance can’t assume it), if you disable the InsightIDR subscriber in Amazon Security Lake, InsightIDR can still access Amazon Security Lake. To disable InsightIDR’s access to Amazon Security Lake, you must remove the two policies that were copied to the Collector’s role.
Set up Amazon Security Lake in InsightIDR
Task 1: Find the SQS URL
- In the AWS console, navigate to Security Lake > Subscribers, and open the subscriber that has been created for InsightIDR.
- Find the Subscription endpoint ARN (for example, arn:aws:sqs:us-east-1:123456789012:
AmazonSecurityLake-a12bc345-6de7-89f0-ghi1-234jkl5678mn-Main-Queue
), and copy the highlighted portion of the Subscription endpoint ARN. - In the AWS Console, go to Simple Queue Service (SQS).
- Search for the portion of the Subscription endpoint ARN that you copied, and open the SQS queue that displays in the search results.
- Copy the URL for the queue.
Task 2: Configure InsightIDR to collect data from the event source
After you complete the prerequisite steps and configure the event source to send data, you must add the event source in InsightIDR.
To configure the new event source in InsightIDR:
- From the left menu, go to Data Collection and click Setup Event Source > Add Event Source.
- Do one of the following:
- Search for AWS Security Lake in the event sources search bar.
- In the Product Type filter, select Cloud Service.
- Select the AWS Security Lake event source tile.
- Optionally, name the event source. This name will be used to name the log that contains the event data in Log Search.
- Select the Collector running in your AWS environment.
- Under Collection Method, select AWS Security Lake.
- Under AWS Authentication, select EC2 Instance Profile Credential.
- In the SQS Queue URL field, enter the SQS queue URL that you retrieved in Task 1: Find the SQS URL.
- Click Save.
Test the configuration
To test that event data is flowing into InsightIDR through the Collector:
- From the Data Collection Management page, click the Event Sources tab.
- Find the event source you created and click View raw log. If the Raw Logs modal displays raw log entries, logs are successfully flowing to the Collector.
- After approximately 7 minutes, log entries start to appear in Log Search. From the left menu, go to Log Search.
- In the Log Sources panel, filter for the applicable log sets and log names. The log name will be the event source name or Amazon Security Lake if you did not name the event source. CloudTrail logs sent via Amazon Security Lake will appear in the Cloud Service Activity log set.
- Set the time range to Last 10 minutes and click Run.
The Results table displays all log entries that flowed into InsightIDR in the last 10 minutes. The keys and values that are displayed are helpful to know when you want to build a query and search your logs.
Logs take a minimum of 7 minutes to appear in Log Search
Please note that logs take at least 7 minutes to appear in Log Search after you set up the event source. If you see log messages when you select View Raw Log on the event source but do not see any log messages in Log Search after waiting for a few minutes for them to appear, then your logs do not match the recommended format and type for this event source.
Sample Logs
Cloud Service Activity
1{2"origin": {3"cloud": {4"service": {5"version": "1.08"6},7"region": "us-east-1"8}9},10"time": 1646266344000,11"api": {12"response": {13"error": null,14"message": null15},16"operation": "GetTable",17"request": {18"uid": "c42082bc-a5b2-43da-be2f-e9462b605cde"19},20"version": null,21"service": {22"name": "glue.amazonaws.com"23}24},25"ref_event_uid": "3384b4bb-11a6-419e-8e77-239279983d81",26"src_endpoint": {27"ip": "glue.amazonaws.com",28"uid": null29},30"resources": null,31"identity": {32"user": {33"type": "AssumedRole",34"name": "testUser",35"uid": "AROAZTO5ZAA5WK2KQUOIB:AWS-Crawler",36"uuid": "arn:aws:sts::660279459899:assumed-role/BiggerELK-MooseGlueRole-1JELSDBSGPRFT/AWS-Crawler",37"account_uid": "660279459899",38"credential_uid": "ASIAZTO5ZAA5QMPGJRFV"39},40"session": {41"creation_time": null,42"mfa": null,43"creator": null44},45"invoked_by": "glue.amazonaws.com",46"idp": {47"name": null48}49},50"class_name": "Cloud API",51"class_uid": 5001,52"category_name": "Cloud Activity",53"category_uid": 5,54"metadata": {55"product": {56"version": "1.08",57"name": "CloudTrail",58"feature": {59"name": "Management, Data, and Insights"60},61"vendor_name": "AWS"62},63"profiles": [64"cloud"65],66"version": "0.26.1"67},68"unmapped": {69"map": ""70}71}