Google Cloud Platform

You can configure Google Cloud Platform to send log data InsightIDR, where they can be used for detection and investigation purposes. Logs from Google Cloud Platform flow into Log Search in the Cloud Services and Ingress Authentication log sets.

To set up this event source:

  1. Review the Before You Begin section to note any requirements.
  2. Configure Google Cloud Platform to forward logs.
  3. Configure this event source in InsightIDR.
  4. Verify the configuration.

Privacy Policy

Rapid7’s InsightIDR product uses OAuth Authentication to retrieve log messages logs from the Google Pub/Sub HTTP API. When configuring the Google Cloud Platform event source, the only permission that InsightIDR requires in the authorization grant is pubsub. InsightIDR uses encrypted Amazon Web Service S3 Buckets to store saved data. InsightIDR uses your logs to track and detect on potentially malicious admin activity, as well as to make logs available in Log Search for reporting and compliance. Any data you share with Rapid7 is NOT shared externally and will only be used for internal purposes such as troubleshooting and new feature development. You can read more about Rapid7’s privacy policy here: https://www.rapid7.com/privacy-policy.

Before You Begin

Before you continue, verify that you have the following accounts, and have performed the following actions:

  • An InsightIDR admin account. An InsightIDR admin is required to complete the configuration in InsightIDR.
  • A Google admin account. A Google admin must sign in to the GCP to set up the Log Sink and Pub/Sub services.
  • A collector installed in your environment. In order to read Google Cloud Platform logs, the collector needs to reach the Google server to pull the logs from the Pub/Sub service, and must be able to connect to https://www.googleapis.com
  • Add the Rapid7 Connector to the Trusted Apps list. Learn more at: https://support.google.com/a/answer/9368756
  • Set the re-authentication policy to exempt Trusted Apps. Learn more at: https://support.google.com/a/answer/9368756

Rapid7 recommends using a private browsing window to set up this event source

This helps verify that your account has the proper access levels. If you are logged into an account that does not have Admin privileges, the OAuth popup will display, prompting you to use that logon instead. If you encounter the OAuth popup, log out of the non-admin account and log in to a Google account that has Admin privileges.

Configure Google Cloud Platform to forward logs

Google Cloud Platform sends messages (i.e. logs) using Pub/Sub, a messaging service that is enabled via an OAuth credential. To send messages or message attributes to InsightIDR via Pub/Sub, you’ll need to export your data using a Log Sink and create a Pub/Sub Subscription for a Pub/Sub Topic. To read more about Pub/Subs, topics, and subscriptions, see https://cloud.google.com/pubsub/docs/overview#data_model.

Set up a Pub/Sub Subscription

You must set up a Pub/Sub Subscription to allow the data source to send logs to InsightIDR.

Note the following important information as you create a Pub/Sub subscription:

  • Be sure to select a Pub/Sub Topic as the destination. The filter allows you to define what logs will be sent to the destination.
  • Make note of the Project ID and the Subscription ID when setting up your subscription as you will need to enter this information in InsightIDR later.

For more information, see https://cloud.google.com/pubsub/docs/admin.

Set up a log sink

Rapid7 recommends that you export your logs from Google Cloud Platform using a Log Sink, which allows your logs to be filtered and routed to a Pub/Sub Topic. For instructions, see https://cloud.google.com/logging/docs/export/configure_export_v2.

As you prepare your logs for export, you should think about the type of data you want to closely monitor, such as audit logs or activity along the Firewall. We recommend against forwarding “noisy” data, such as logs generated by a service account or internal network traffic. These logs can result in large volumes of data, which may impact your storage rates.

(Optional) Use our pre-built queries

When setting up a log sink, you can use our pre-built Log Sink queries to filter out noisy logs so that you get the data that is most valuable to you.

Send all audit logs except those generated by Kubernetes

logName:"cloudaudit.googleapis.com" NOT protoPayload.serviceName:"k8s.io"

Send Firewall or Threat Detection logs

If you have firewall or threat detection logging enabled in Google Cloud Platform, you can forward that data to InsightIDR for investigation and detection purposes. Firewall messages generate Firewall detections in InsightIDR, and threat detection logs generate Third Party Alerts.

To send firewall or threat detection logs to InsightIDR, add the following to the previous query:

OR log_id("compute.googleapis.com/firewall")

OR log_id("threatdetection.googleapis.com/detection")

Configure InsightIDR to collect data from the event source

After you complete the prerequisite steps and configure the event source to send data, you must add the event source in InsightIDR.

To configure the new event source in InsightIDR:

  1. From the left menu, go to Data Collection and click Setup Event Source > Add Event Source.
  2. Do one of the following:
    • Search for Google Cloud Platform in the event sources search bar.
    • In the Product Type filter, select Cloud Service.
  3. Select the Google Cloud Platform event source tile.
  4. Select your collector and Google Cloud Platform from the event source dropdown.
  5. Name your event source.
  6. Optionally choose to send unparsed logs.
  7. Select your Account Attribution preference:
    • Use short name attribution: The system first attempts to attribute data by email address, for example, jsmith@myorg.example.com. If the first attempt is unsuccessful, attribution is attempted by short name, for example, jsmith. If the short name is unsuccessful, attribution is attempted by a user’s first and last name, for example, John Smith.
    • Use fully qualified domain name attribution: The system first attempts to attribute data by email address, for example, jsmith@myorg.example.com. If the first attempt is unsuccessful, attribution is attempted by a user’s first and last name, for example, John Smith. This option is best if your environment has collisions with short names.
  8. Enter the IDs for the Google Cloud Project.
  9. Enter the ID for the Pub/Sub Subscription where the logs have been routed.
  10. Click Begin to set up OAUTH and start the authorization process. A new window or tab will open for you to perform an authorization grant with Google. The grant must be performed by a GCP admin user.
  11. Click Allow to permit InsightIDR to view and manage Pub/Sub topics and subscriptions. Close the window/tab to return to InsightIDR.
  12. Click Allow to permit InsightIDR to view and manage Pub/Sub topics and subscriptions. Close the window/tab to return to InsightIDR.
  13. If you are using a multi-domain environment, configure a default domain. We will apply the default domain you configure to any logs that do not provide a domain for an account that appears in the logs.
  14. Click Save.

Verify the configuration

  1. From the left menu, click Log Search to view your raw logs to ensure events are making it to the Collector. Select the applicable Log Sets and the Log Names within them. The Log Name will be the event source name or “Google Platform Cloud” if you did not name the event source. Google Platform Cloud logs flow into the log set:

    • Cloud Services
    • Ingress Authentication

  2. Perform a Log Search to make sure events are coming through.

Sample Log

The following log reflects the activation of a Google Cloud Platform service by a user.

json
1
{
2
 "insertId": "12utqhgc2li",
3
 "logName": "projects/testproject/logs/cloudaudit.googleapis.com%2Factivity",
4
 "operation": {
5
   "first": true,
6
   "id": "operations/acf.73b6492a-d2a6-4311-973d-c0de964f0570",
7
   "producer": "servicemanagement.googleapis.com"
8
 },
9
 "protoPayload": {
10
   "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
11
   "authenticationInfo": {
12
     "principalEmail": "username@email.com"
13
   },
14
   "authorizationInfo": [
15
     {
16
       "granted": true,
17
       "permission": "serviceusage.services.enable",
18
       "resource": "projectnumbers/123456789012/services/-",
19
       "resourceAttributes": {}
20
     },
21
     {
22
       "permission": "serviceconsumermanagement.consumers.enable",
23
       "resource": "services/compute.googleapis.com/consumers/123456789012",
24
       "resourceAttributes": {}
25
     },
26
     {
27
       "granted": true,
28
       "permission": "servicemanagement.services.bind",
29
       "resource": "services/compute.googleapis.com",
30
       "resourceAttributes": {}
31
     },
32
     {
33
       "granted": true,
34
       "permission": "serviceusage.services.enable",
35
       "resource": "projectnumbers/123456789012/services/-",
36
       "resourceAttributes": {}
37
     },
38
     {
39
       "permission": "servicemanagement.services.bindAll",
40
       "resource": "services/compute.googleapis.com",
41
       "resourceAttributes": {}
42
     },
43
     {
44
       "permission": "serviceconsumermanagement.consumers.enable",
45
       "resource": "services/compute.googleapis.com/consumers/123456789012",
46
       "resourceAttributes": {}
47
     }
48
   ],
49
   "methodName": "google.api.servicemanagement.v1.ServiceManager.ActivateServices",
50
   "requestMetadata": {
51
     "callerIp": "100.10.200.20",
52
     "callerSuppliedUserAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36,gzip(gfe)",
53
     "destinationAttributes": {},
54
     "requestAttributes": {
55
       "auth": {},
56
       "time": "2020-09-03T14:56:50.773372Z"
57
     }
58
   },
59
   "resourceName": "projects/123456789012/services/[compute.googleapis.com]",
60
   "serviceName": "servicemanagement.googleapis.com"
61
 },
62
 "receiveTimestamp": "2020-09-03T14:56:50.982038857Z",
63
 "resource": {
64
   "labels": {
65
     "method": "google.api.servicemanagement.v1.ServiceManager.ActivateServices",
66
     "project_id": "testproject",
67
     "service": "servicemanagement.googleapis.com"
68
   },
69
   "type": "audited_resource"
70
 },
71
 "severity": "NOTICE",
72
 "timestamp": "2020-09-03T14:56:49.891207Z"
73
}

Additional OAuth 2.0 Use Cases

Google APIs use the OAuth 2.0 protocol for authentication and authorization. Google supports common OAuth 2.0 scenarios such as those for web server, installed, and client-side applications. For more information, read this article: https://developers.google.com/identity/protocols/OAuth2.