AWS SQS
AWS SQS, or Amazon Simple Queue Services, is a managed queuing service that works with InsightIDR when sending messages as events. You can also access AWS resources with EC2 IAM Roles.
AWS SQS Configuration
Before you begin, you must create an SQS queue for the exclusive use of InsightIDR. You can find instructions on how to do so here: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-create-queue.html
To configure AWS SQS:
- Create a new IAM Policy for InsightIDR to use when making HTTP requests to that SQS queue, such as the following example:
1{2"Version": "2012-10-17",3"Statement": [4{5"Effect": "Allow",6"Action": [7"sqs:ReceiveMessage",8"sqs:DeleteMessage"9],10"Resource": [11"arn:aws:sqs:::SqsQueueNameGoesHere"12]13}14]15}
- Create a new IAM Group that contains the policy you just created.
- Create a new IAM User that belongs only to the newly created IAM group. Make sure to give this user permission to make
receiveMessage
anddeleteMessage
calls from the SQS queue.
This will be used by InsightIDR to access your data, so it is recommended to limit access to the IAM Group.
- Copy the secret key to a secure place for later configuration of SQS.
Configure InsightIDR to collect data from the event source
After you complete the prerequisite steps and configure the event source to send data, you must add the event source in InsightIDR.
To configure the new event source in InsightIDR:
- From the left menu, go to Data Collection and click Setup Event Source > Add Event Source.
- Click Add Raw Data > Custom Logs.
- Alternatively, you can search for Custom Logs or filter by the Rapid7 Product Type, and then select the Rapid7 Custom Logs event source tile.
- Choose your collector and event source. You can also name your event source if you want.
- Choose the timezone that matches the location of your event source logs.
- Select SQS Messages as the collection method.
- Select your credentials, or optionally create a new credential. The credential you enter will be the access key and secret key that corresponds to an AWS IAM role that has permissions to make
receiveMessage
anddeleteMessage
calls from the above SQS queue.- Access Key: the username you created with IAM permissions
- Secret Key: the password you created that corresponds with the username with IAM permissions
- URL: the URL in your AWS console and will look similar to
https://sqs.us-east-1.amazonaws.com/your-queue-name
- Include Attributes: indicates whether or not Rapid7 will also grab the SQS message’s attributes and send them along with the body of the SQS message
- You can also use EC2 IAM Roles.
- Click Save.
SQS Message Structure
There are several different ways that SQS sends information in it's messages:
- with the message Attributes
- without the message Attributes
- with the message Attributes and the message body is a JSON object
- without the message Attributes and the message body is a JSON object
Message Structure with Attributes
When you use SQS messages as a Collection method, you can click the include attributes box. When you do, messages sent will look like the following:
1{2"body": "MyMessageBody",3"message_attributes": [4{5"name": "MyAttribute1",6"value": {7"dataType": "MyDataType1",8"stringValue": "MyStringValue1"9}10},11{12"name": "MyAttribute2",13"value": {14"dataType": "MyDataType2",15"stringValue": "MyStringValue2"16}17}18]19}
Message Body does not include Attributes
If the body of the SQS message is parseable as JSON, it appears in Log Search as JSON, not as escaped JSON within a string.
When you use SQS messages as a Collection method without including attributes, messages will look like the following:
1{2"body": "MyMessageBody"3}
Message Structure is a JSON Object and includes Attributes
When you use SQS messages as a collection method, you can click the include attributes box. When you do, messages sent will look like the following if they are also a JSON object and include attributes:
1{2"body": {3"myField" : "myValue"4"anotherField" : {5"anotherNestedField" :"a nested value"6}7}8"message_attributes": [9{10"name": "MyAttribute1",11"value": {12"dataType": "MyDataType1",13"stringValue": "MyStringValue1"14}15},16{17"name": "MyAttribute2",18"value": {19"dataType": "MyDataType2",20"stringValue": "MyStringValue2"21}22}23]24}
Message Body is a JSON Object without Attributes
If the body of the SQS message is parseable as JSON, it appears in Log Search as JSON, not as escaped JSON within a string.
When you use SQS messages as a collection method but do not include attributes, messages sent will look like the following if they are a JSON object:
1{2"body": {3"myField" : "myValue"4"anotherField" : {5"anotherNestedField" :"a nested value"6}7}8}
Troubleshooting
Use the following information to help resolve your issues:
- Fatal SQS Error
Fatal SQS error: Signature not yet current
If you are seeing the above error, the system time of the collector running this event source may be out of sync with the Amazon servers running SQS. Make sure you properly configure time synchronization on the collector before attempting to reconfigure this event source.