Amazon Security Lake

Amazon Security Lake is a security data lake service that allows you to centrally aggregate, manage, and use security-related logs. All logs in Amazon Security Lake are formatted using the OCSF standard. Rapid7 customers can integrate with the service to feed logs from AWS-based event sources into InsightIDR. CloudTrail, GuardDuty, and Security Hub are event sources that generate security-relevant data. When this data is ingested into Amazon Security Lake, they become data sources for security data analysis.

Amazon Security Lake customers can send logs to third-party solutions like InsightIDR by creating a subscription in Security Lake. Customers can then select which logs to include in the subscription. The third-party solution consumes logs from the subscription via an SQS queue that is created with the subscription.

To set up Amazon Security Lake, you’ll need to:

1. Configure Amazon Security Lake.
2. Configure a Collector.
3. Set up Amazon Security Lake in InsightIDR.
4. Test the configuration.

Configure Amazon Security Lake

Task 1: Enable Amazon Security Lake

If you haven’t already, ensure that you have enabled Amazon Security Lake by following the instructions at: https://docs.aws.amazon.com/security-lake/latest/userguide/getting-started.html#enable-service

Task 2: Add CloudTrail, GuardDuty, or Security Hub as a data source to Amazon Security Lake

After you have turned Amazon Security Lake on, add a data source by following the instructions at: https://docs.aws.amazon.com/security-lake/latest/userguide/internal-sources.html

Task 3: Create a subscriber in Amazon Security Lake

Although Amazon Security Lake started capturing your CloudTrail logs upon completion of Task 2: Add CloudTrail, GuardDuty, or Security Hub as a data source to Amazon Security Lake, you must create a subscriber for InsightIDR to access the event source data in Security Lake.

Add a subscriber to Amazon Security Lake by following the instructions at: https://docs.aws.amazon.com/security-lake/latest/userguide/subscriber-data-access.html. You do not need to create an IAM role for EventBridge as the InsightIDR integration uses SQS, not EventBridge.

To set up the subscriber:

1. For the Subscriber Name, enter a descriptive name (for example, Rapid7-InsightIDR).
2. Under Log and event sources, select Specific log and event sources > and choose either CloudTrail, GuardDuty, or Security Hub.
3. Under Data access method, select S3.
4. For the Account ID in the Subscriber credentials section, enter the AWS account ID for the account you are currently logged in to. For the External ID, enter a unique, unguessable combination of letters and numbers.
5. Under Notification details (S3 only), select SQS queue.
6. Click the Create button. The list of Amazon Security Lake subscribers displays with the new subscriber you just created.
7. Click on the name of the subscriber you just created to see more details.
8. Make note of the ARN for the AWS role ID and the ARN for the Subscription endpoint. You will need both of these ARNs later in the setup.

Configure a Collector

Task 1: Create an EC2 Collector

As with most InsightIDR integrations, the data from Amazon Security Lake is sent to InsightIDR via a Collector. It is easiest to do this if you have a Collector running on an EC2 instance in your AWS environment. If you already have a Collector installed in your AWS environment, you can use it. If you haven’t installed a Collector already, you can install one on an EC2 instance by following our documentation.

In the Access AWS Resources with EC2 IAM Roles instructions, you can skip the Configure IAM Policy step and return to this documentation when you reach the Update the IAM Role Permissions step.

Task 2: Create policies to access Amazon Security Lake data

Amazon Security Lake automatically creates an IAM role for each subscriber. However, this role is not designed to be assumed by an EC2 instance, so it cannot be used with the Collector. Instead, the permissions must be copied into new IAM policies that can be added to the Collector’s role.

1. In Task 3: Create a subscriber in Amazon Security Lake, you noted the ARN for the AWS role ID of the subscriber that was created (for example, arn:aws:iam::123456789012:role/AmazonSecurityLake-a12bc345-6de7-89f0-ghi1-234jkl5678mn.) Copy the highlighted portion of the AWS role ID.
2. In the AWS console, navigate to IAM > Roles, and search for the highlighted portion of the AWS role ID.
3. Open the role returned by the search. A permission policy ending in SQS and a permission policy ending in S3 display.
4. Expand the policy ending in SQS and copy it.
5. Open the Policies section of IAM and click the Create policy button.
6. Switch to the JSON tab, and paste the permission policy ending in SQS.
7. Click Next to add any tags you might want. Click Next again and add a descriptive name for the policy (for example, r7idrsecuritylakesqs). Create the policy.
8. Repeat steps 4-7 for the policy ending in S3.
9. Open the IAM role for the Collector and attach the two policies you just created.

Because we are not using the role created by Amazon Security Lake (due to the fact that an EC2 instance can’t assume it), if you disable the InsightIDR subscriber in Amazon Security Lake, InsightIDR can still access Amazon Security Lake. To disable InsightIDR’s access to Amazon Security Lake, you must remove the two policies that were copied to the Collector’s role.

Set up Amazon Security Lake in InsightIDR

Task 1: Find the SQS URL

1. In the AWS console, navigate to Security Lake > Subscribers, and open the subscriber that has been created for InsightIDR.
2. Find the Subscription endpoint ARN (for example, arn:aws:sqs:us-east-1:123456789012:AmazonSecurityLake-a12bc345-6de7-89f0-ghi1-234jkl5678mn-Main-Queue), and copy the highlighted portion of the Subscription endpoint ARN.
3. In the AWS Console, go to Simple Queue Service (SQS).
4. Search for the portion of the Subscription endpoint ARN that you copied, and open the SQS queue that displays in the search results.
5. Copy the URL for the queue.

Task 2: Configure InsightIDR to collect data from the event source

After you complete the prerequisite steps and configure the event source to send data, you must add the event source in InsightIDR.

To configure the new event source in InsightIDR:

1. From the left menu, go to Data Collection and click Setup Event Source > Add Event Source.
2. Do one of the following:
   • Search for AWS Security Lake in the event sources search bar.
   • In the Product Type filter, select Cloud Service.
3. Select the AWS Security Lake event source tile.
4. Optionally, name the event source. This name will be used to name the log that contains the event data in Log Search.
5. Select the Collector running in your AWS environment.
6. Under Collection Method, select AWS Security Lake.
7. Under AWS Authentication, select EC2 Instance Profile Credential.
8. In the SQS Queue URL field, enter the SQS queue URL that you retrieved in Task 1: Find the SQS URL.
9. Click Save.

Test the configuration

To test that event data is flowing into InsightIDR through the Collector:

1. From the Data Collection Management page, click the Event Sources tab.
2. Find the event source you created and click View raw log. If the Raw Logs modal displays raw log entries, logs are successfully flowing to the Collector.
3. After approximately 7 minutes, log entries start to appear in Log Search. From the left menu, go to Log Search.
4. In the Log Sources panel, filter for the applicable log sets and log names. The log name will be the event source name or Amazon Security Lake if you did not name the event source. Event source logs sent via Amazon Security Lake will appear in the Cloud Service Activity log set.
5. Set the time range to Last 10 minutes and click Run.

The Results table displays all log entries that flowed into InsightIDR in the last 10 minutes. The keys and values that are displayed are helpful to know when you want to build a query and search your logs.

Sample Logs

Cloud Service Activity

 
1
{
2
	"origin": {
3
		"cloud": {
4
			"service": {
5
				"version": "1.08"
6
			},
7
			"region": "us-east-1"
8
		}
9
	},
10
	"time": 1646266344000,
11
	"api": {
12
		"response": {
13
			"error": null,
14
			"message": null
15
		},
16
		"operation": "GetTable",
17
		"request": {
18
			"uid": "c42082bc-a5b2-43da-be2f-e9462b605cde"
19
		},
20
		"version": null,
21
		"service": {
22
			"name": "glue.amazonaws.com"
23
		}
24
	},
25
	"ref_event_uid": "3384b4bb-11a6-419e-8e77-239279983d81",
26
	"src_endpoint": {
27
		"ip": "glue.amazonaws.com",
28
		"uid": null
29
	},
30
	"resources": null,
31
	"identity": {
32
		"user": {
33
			"type": "AssumedRole",
34
			"name": "testUser",
35
			"uid": "AROAZTO5ZAA5WK2KQUOIB:AWS-Crawler",
36
			"uuid": "arn:aws:sts::660279459899:assumed-role/BiggerELK-MooseGlueRole-1JELSDBSGPRFT/AWS-Crawler",
37
			"account_uid": "660279459899",
38
			"credential_uid": "ASIAZTO5ZAA5QMPGJRFV"
39
		},
40
		"session": {
41
			"creation_time": null,
42
			"mfa": null,
43
			"creator": null
44
		},
45
		"invoked_by": "glue.amazonaws.com",
46
		"idp": {
47
			"name": null
48
		}
49
	},
50
	"class_name": "Cloud API",
51
	"class_uid": 5001,
52
	"category_name": "Cloud Activity",
53
	"category_uid": 5,
54
	"metadata": {
55
		"product": {
56
			"version": "1.08",
57
			"name": "CloudTrail",
58
			"feature": {
59
				"name": "Management, Data, and Insights"
60
			},
61
			"vendor_name": "AWS"
62
		},
63
		"profiles": [
64
			"cloud"
65
		],
66
		"version": "0.26.1"
67
	},
68
	"unmapped": {
69
		"map": ""
70
	}
71
}