Amazon: S3 via Lambda

AWS Lambda allows you to run code independently of servers and run applications. When connected to AWS S3, events from S3 buckets can trigger Lambda functions.

You can connect your AWS S3 buckets to InsightOps in order to track the events and functions from AWS Lambda.

In order to connect them, you must:

You can also utilize an example working code.

Before You Begin

Ensure that you are using Python’s Certifi. You can read more about certifi here: https://github.com/certifi/python-certifi and https://certifi.io/en/latest/.

Use Cases

You can utilize InsightOps with AWS Lambda in several different situations:

  • Forward AWS ELB and CloudFront logs.
  • Forward OpenDNS logs
  • Forward CloudTrail logs (if you already configured CloudTrail S3 integration)

Obtain a Log Token

After you download the package, you must obtain at least one log token. Token-based input is a single TCP connection where each log line contains a token which uniquely identifies the destination log.

To obtain a log token:

  1. Log in to your InsightOps account.
  2. Follow the instructions here to obtain a log token: TCP Log Tokens.

Or, you can use an existing token to aggregate your logs.

Deploy a Script to AWS Lambda

Now you must create a function in AWS Lambda that will invoke a script to run and trigger an event for AWS S3 to track.

You must complete the following actions:

Configure Functions

Create a new function in AWS Lambda following the directions here: https://docs.aws.amazon.com/lambda/latest/dg/getting-started-create-function.html

To configure a new function:

  1. Log in to your AWS Console.
  2. From the “Services” dropdown, select Lambda from the “Compute” section.
  3. From the left navigation menu, select Functions and then click the orange Create a New Function button on the right.
  4. In the “Blueprints” section, search for and select the s3-get-object-python blueprint.
  1. Click the Next button.
  2. In the “Triggers” section, search for and select S3 as your trigger.
  1. In “Bucket,” select the S3 Bucket that stores the log data you want to use.
  2. For “Event type,” choose the event type Object Created (All).
  1. Check on the Enable Trigger checkbox.
  2. Click the Next button.

Upload the Function Code

  1. Name your function.
  2. Optionally provide a description.
  3. Set your python runtime version from "Runtime" dropdown as Python 2.7.
  4. On your asset, create a .ZIP file, containing r7insight_lambdas3.py and the folder certifi which you can download here: https://github.com/rapid7/r7insight_lambdaS3
    • Make sure the files and certifi folder are in the root of the ZIP archive
  5. Under “Code Entry Type,” click the Upload a ZIP File radio button.
  6. Select the archived zip file created in the previous steps.
  7. In the “Handler” field, set the handler name to r7insight_lambdas3.lambda_handler.
  1. In the “Role” field, assign or create a new role that allows getObjects on your chosen S3 buckets. Creating a role in AWS Lambda automatically grants the required permissions.
  2. Under “Memory,” set memory limit to a high enough value to facilitate log parsing and sending.
  3. Under “Timeout,” set timeout to a high enough value to facilitate log parsing and sending.
  4. Leave the VPC value to "No VPC.” If you choose to use VPC, please consult the Amazon Documentation here: https://docs.aws.amazon.com/vpc/index.html#lang/en_us
  5. In “Environment Variables,” set the following as a key - value pair:
    • Region - your InsightOps region (such as eu, us, etc)
    • Token - Token UUID you configured earlier
  1. Click the Next button.
  2. Review the function configuration before click the Create Function button.
  3. You can optionally test the function configuration in the next screen.
  4. Click the Close and Save button to save the function configuration.

The Lambda function will fire once the S3 bucket ingests log files.

Example Code

The following is an example of working code:

1
(root) - current AWS sets root directory name to your lambda function name automatically
2
├── certifi/
3
│ ├──
4
│ ├── __init__.py
5
│ ├── __main__.py
6
│ ├── cacert.pem
7
│ ├── core.py
8
│ ├── old_root.pem
9
│ └── weak.pem
10
├── r7insight_lambdas3.py`
11
``
12
13
Note: Zip files downloaded via github puts all the files under a subdirectory; therefore, uploading the downloaded zip directly to AWS will not work.

Configure Data in InsightOps

To finish the configuration for AWS Lambda via your S3 bucket:

  1. Log in to InsightOps.
  2. Select the Data Collection page on the left hand menu.
  3. Select the AWS Lambda S3 icon from the “System Data” section.
  4. In the “Log Name” field, enter the name you want to see for this data in Log Search.
  5. Select an existing log set to add this log to, or type in a name to create a new log set.
  6. Click the Done button.

Once the AWS S3 bucket begins to ingest your log file data, the Lambda function will fire and you will see the logs reflected on the Log Search page. Logs appear in approximately 30 seconds and in the form of JSON.