GCP Event Collection

Heeler event collection allows near real-time updates, analysis, and notification of meaningful changes in your environment. Heeler harvests events from GCP using logging at the organization level and Pub/Sub resources.

As noted under Google Cloud Platform, please install the gcloud CLI utility to follow this guide with installation instructions located here: https://cloud.google.com/sdk/docs/install. Also as noted under Google Cloud Platform, this guide assumes you have created a dedicated project to connect your GCP organization. These instructions will deploy Pub/Sub resources in that project.

Command Line Steps

1. To simplify the remaining gcloud commands, set the dedicated Heeler Security project as default. Assuming its name is heeler-security, use

gcloud config set project heeler-security

1. Create a Pub/Sub topic, e.g,. heeler-event-collection-topic

gcloud pubsub topics create heeler-event-collection-topic

1. Obtain your organization ID.

gcloud organizations list

You should see a response like, from which you want the 12-digit ID

DISPLAY_NAME            ID  DIRECTORY_CUSTOMER_ID
heeler.com    123456789012              ABC123xyz

1. Create an organization-level log sink, e.g., heeler-event-collection-sink, that will capture a filtered set of events. Note the use of the organization ID with the organization flag

gcloud logging sinks create heeler-event-collection-sink pubsub.googleapis.com/projects/heeler-security/topics/heeler-event-collection-topic \
    --description="Routing events to Pub/Sub for Heeler Security" \
    --include-children \
    --log-filter='logName:cloudaudit.googleapis.com AND severity="NOTICE" AND protoPayload.request.@type!="type.googleapis.com/cloud.dns.api.ChangesCreateRequest" AND (protoPayload.request.@type:"add" OR protoPayload.request.@type:"attach" OR protoPayload.request.@type:"bulkInsert" OR protoPayload.request.@type:"create" OR protoPayload.request.@type:"delete" OR protoPayload.request.@type:"deploy" OR protoPayload.request.@type:"detach" OR protoPayload.request.@type:"disable" OR protoPayload.request.@type:"enable" OR protoPayload.request.@type:"insert" OR protoPayload.request.@type:"patch" OR protoPayload.request.@type:"provision" OR protoPayload.request.@type:"recreate" OR protoPayload.request.@type:"remove" OR protoPayload.request.@type:"update" OR protoPayload.request.@type="type.googleapis.com/compute.disks.resize" OR protoPayload.request.@type="type.googleapis.com/compute.instances.reset" OR protoPayload.request.@type="type.googleapis.com/compute.instances.resume" OR protoPayload.request.@type="type.googleapis.com/compute.instances.setMachineType" OR protoPayload.request.@type="type.googleapis.com/compute.instances.setServiceAccount" OR protoPayload.request.@type="type.googleapis.com/compute.instances.setSize" OR protoPayload.request.@type="type.googleapis.com/compute.instances.start" OR protoPayload.request.@type="type.googleapis.com/compute.instances.stop" OR protoPayload.request.@type="type.googleapis.com/compute.instances.suspend")' \
    --organization=123456789012

The log filter is focused on events that create, update, or delete resources. Here is an easier, human-friendly version. We expect to update the log filter over time as GCP adds more resources and more events and Heeler adds more analysis.

logName:cloudaudit.googleapis.com AND 
severity="NOTICE" AND 
protoPayload.request.@type!="type.googleapis.com/cloud.dns.api.ChangesCreateRequest" AND 
(
    protoPayload.request.@type:"add" OR 
    protoPayload.request.@type:"attach" OR 
    protoPayload.request.@type:"bulkInsert" OR 
    protoPayload.request.@type:"create" OR 
    protoPayload.request.@type:"delete" OR 
    protoPayload.request.@type:"deploy" OR 
    protoPayload.request.@type:"detach" OR 
    protoPayload.request.@type:"disable" OR 
    protoPayload.request.@type:"enable" OR 
    protoPayload.request.@type:"insert" OR 
    protoPayload.request.@type:"patch" OR 
    protoPayload.request.@type:"provision" OR 
    protoPayload.request.@type:"recreate" OR 
    protoPayload.request.@type:"remove" OR 
    protoPayload.request.@type:"update" OR 
    protoPayload.request.@type="type.googleapis.com/compute.disks.resize" OR 
    protoPayload.request.@type="type.googleapis.com/compute.instances.reset" OR 
    protoPayload.request.@type="type.googleapis.com/compute.instances.resume" OR 
    protoPayload.request.@type="type.googleapis.com/compute.instances.setMachineType" OR 
    protoPayload.request.@type="type.googleapis.com/compute.instances.setServiceAccount" OR
    protoPayload.request.@type="type.googleapis.com/compute.instances.setSize" OR 
    protoPayload.request.@type="type.googleapis.com/compute.instances.start" OR 
    protoPayload.request.@type="type.googleapis.com/compute.instances.stop" OR 
    protoPayload.request.@type="type.googleapis.com/compute.instances.suspend"
)

After creating the sink heeler-event-collection-sink, you should have seen a response like

Please remember to grant `serviceAccount:[e-mail]` the Pub/Sub Publisher role on the topic.

1. Copy the provided serviceAccount and use it to update the permissions on the Pub/Sub topic you created, e.g., heeler-event-collection-topic

gcloud pubsub topics add-iam-policy-binding heeler-event-collection-topic \
    --role="roles/pubsub.publisher" \
    --member="serviceAccount:[e-mail]"

1. Now that GCP is capturing filtered events and sending them to a topic, create a subscription to read the events from the topic, e.g., heeler-event-collection-subscription

gcloud pubsub subscriptions create heeler-event-collection-subscription \
    --message-retention-duration=1d \
    --enable-message-ordering \
    --topic=heeler-event-collection-topic

1. Obtain the full name of the subscription

gcloud pubsub subscriptions describe heeler-event-collection-subscription

which should provide a response like

ackDeadlineSeconds: 10
enableMessageOrdering: true
expirationPolicy:
  ttl: 2678400s
messageRetentionDuration: 86400s
name: projects/heeler-security/subscriptions/heeler-event-collection-subscription
pushConfig: {}
state: ACTIVE
topic: projects/heeler-security/topics/heeler-event-collection-topic

Heeler Steps

1. Once the resources are created and configured in GCP, you need to add their information to Heeler. Open the URL https://app.heeler.com/administration/connections/organizations or just click on the settings icon at the top right and then click on Connections.

1. Then select the ellipsis to the right of the desired GCP organization in order to edit its Event Collection Settings

1. Then paste the full name of the Pub/Sub subscription from the prior step in the setting

1. Finally, confirm that event collection is enabled in the updated Cloud Organization listing

At this point, it may take some time for the initial harvest of events, but afterwards, events should harvest every 60 seconds.