# AWS Event Collection Heeler event collection allows near real-time updates, analysis, and notification of meaningful changes in your environment. Heeler harvests events from AWS using logging at the organization level. ## Choosing an implementation There are two ways to get CloudTrail events into Heeler. Both deliver the same data; the tradeoff is **cost vs. latency**. | | **Option 1: S3 → SQS** (recommended) | **Option 2: CloudWatch Log Group → SQS** | | --------------- | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | Flow | CloudTrail → S3 → bucket notification → SNS → SQS → Heeler | CloudTrail → CloudWatch Logs → subscription filter → Lambda → SQS → Heeler | | Cost | Lower — can be **\~1/100th** the cost of Option 2 depending on event volume | Higher — driven by per-event CloudWatch Logs ingestion and Lambda invocations | | Harvest latency | \~5-10 minutes (bound by CloudTrail's S3 delivery cadence) | Under 60 seconds | | When to choose | **Default.** Most customers should pick this. | Only if sub-minute latency is a hard requirement. | **We recommend Option 1 (S3 → SQS)** unless near-real-time alerting is a hard requirement for your organization. ## AWS Prerequisite (shared) Both options require an organization-wide CloudTrail, which most AWS Organizations already have in place. Doing so simplifies event collection and ensures complete visibility as your cloud footprint grows. The trail must have the following settings: 1. Enabled for all accounts in my organization 2. Multi-region trail 3. All management events Each option adds its own additional prerequisites — see the relevant section below. *** ## Option 1: S3 → SQS (recommended) ### Additional prerequisite The trail must deliver log files to an **S3 bucket in the payer / org-management account**. Before running the Terraform, note down these values: * **S3 bucket name** * **KMS key ARN (conditional)**. If the bucket is encrypted with a customer-managed KMS key (SSE-KMS, not default AWS-managed SSE-S3), the ARN of that key
### Terraform Implementation This Terraform must be applied in the **payer / org-management account** — the same account that owns the organization-wide CloudTrail trail and its S3 destination bucket. Executing the Terraform plan creates the following: * **SNS topic** (conditional). When the CloudTrail bucket does not already have an SNS topic receiving object-created notifications, the module creates one and wires the bucket's `s3:ObjectCreated:*` events to it. If an SNS topic already exists, change the `sns_s3_event_notification_name` input to its name. * **SQS queue**. Subscribed to the SNS topic; holds events until they are harvested by Heeler. * **Dead Letter Queue**. Captures messages that fail delivery to the main queue (after four attempts). * **KMS key and alias**. Encrypts messages on the SQS queue and DLQ. * **IAM role and policy**. The cross-account role Heeler assumes, and the policy granting it the minimum permissions to receive messages from the queue, decrypt them, and fetch the referenced CloudTrail log objects from S3. * **KMS grant on the CloudTrail bucket's key** (conditional). When the bucket is encrypted with a customer-managed KMS key (`s3_cloudtrail_bucket_kms_key_arn` is set), the module creates a grant on that key so the harvest role can decrypt retrieved objects. The grant is additive and leaves the key's existing policy untouched. ### Terraform Steps 1. Download and unzip the files to an appropriate location for your environment. The zip file contains: * **main.tf**. Defines the resources to be deployed * **output.tf**. Defines the values displayed after the process completes so they can be used to configure Event Collection in the Heeler application * **variables.tf**. The variable definitions populated by **var.auto.tfvars** * **var.auto.tfvars**. The inputs required to deploy resources to your account hosting your organization-wide CloudTrail * **README.md**. Reference documentation for the module {% file src="/files/BrDaxGTdxb1j24i5K7yU" %} Zip file of Event Collection terraform files (S3 → SQS) {% endfile %} 2. Update the `var.auto.tfvars` file You will need to update the input variable values as indicated. * **account\_id** should be the account ID of the payer / org-management account where the organization-wide CloudTrail is located. * **s3\_cloudtrail\_bucket\_name** is the name of the S3 bucket where CloudTrail delivers log files. * **s3\_cloudtrail\_bucket\_kms\_key\_arn** (conditional) is the ARN of the customer-managed KMS key encrypting the CloudTrail bucket. Leave as `"None"` if the bucket uses SSE-S3. * **sns\_s3\_event\_notification\_name** (conditional) is the name of an existing SNS topic already receiving `s3:ObjectCreated:*` notifications from the bucket. Leave as `"None"` and the module will create a new SNS topic and attach the bucket notification automatically. * **heeler\_external\_id** is a security feature to create a secret to be shared between your Heeler installation and the Event Collection IAM role it assumes in your account. * **heeler\_security\_role\_arns** (*provided by Heeler*) is the list of role ARNs provided by Heeler that Heeler will use to harvest events from the SQS queue in your account. ```hcl account_id = "" s3_cloudtrail_bucket_name = "" s3_cloudtrail_bucket_kms_key_arn = "None" # or the bucket's CMK ARN sns_s3_event_notification_name = "None" # or an existing SNS topic name heeler_external_id = "" heeler_security_role_arns = [""] ``` 3. Initialize and plan the deployment 
terraform init
terraform plan
The plan should summarize the resources to be created — SNS topic (if `sns_s3_event_notification_name = "None"`), S3 bucket notification (same condition), SQS queue and DLQ, KMS key and alias, IAM role and policy, SNS-to-SQS subscription, and optionally a KMS grant on the CloudTrail bucket's customer-managed key. No resources should be modified or destroyed. The total count will vary based on the optional inputs, but the output will end with a summary line like: ``` Plan: N to add, 0 to change, 0 to destroy. Changes to Outputs: + heeler_event_queue_url = (known after apply) + heeler_event_role_arn = (known after apply) + heeler_event_role_external_id = "heeler-ext-id" ``` 4. Apply the deployment ``` terraform apply ``` You should see a response like the one from `terraform plan`, but also includes ``` Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: ``` enter `yes` Once complete, you should see something like this ``` Apply complete! Resources: N added, 0 changed, 0 destroyed. Outputs: heeler_event_queue_url = "https://sqs.us-east-1.amazonaws.com/123456789012/heeler_event_collection_queue" heeler_event_role_arn = "arn:aws:iam::123456789012:role/heeler_event_collection_sqs_role" heeler_event_role_external_id = "" ``` You will need each of these output values when configuring Event Collection in Heeler — continue to the [Heeler Steps](#heeler-steps) section below. *** ## Option 2: CloudWatch Log Group → SQS Choose this option only if you need harvest latency under 60 seconds. Otherwise, use [Option 1](#option-1-s3--sqs-recommended). ### Additional prerequisite The organization-wide CloudTrail must have **CloudWatch Logs enabled**. This may be new to your organization. If so, enable CloudWatch Logs to create a Log Group. Then, edit the Log Group retention setting to `1 day` to allow events to expire. Doing so reduces costs, and Heeler does not need the events retained as it harvests every 60 seconds. ### Terraform Implementation To deploy the resources required for Event Collection, you will need Terraform configured to deploy resources in the account hosting the organization-wide CloudTrail. Executing the Terraform plan creates the following: * **CloudWatch subscription filter**. This filter excludes events from processing that are not security-relevant, e.g. `PutObject`. * **Lambda function**. Reads events from the CloudWatch log group, processes them, and sends them to an SQS queue. * **SQS queue**. Holds the processed events until they are harvested by Heeler. * **IAM role and policy**. The role that Heeler assumes and the policy that allows Heeler to harvest events from the SQS queue. * **Supporting resources**: * Dead Letter Queue to capture failed deliveries to the SQS queue * KMS key to encrypt the SQS queue * IAM role, policy, and permission for the Lambda function to read CloudWatch log groups and send encrypted messages to SQS ### Terraform Steps 1. Download and unzip the files to an appropriate location for your environment. The zip file contains: * **main.tf**. Defines the resources to be deployed * **output.tf**. Defines the values displayed after the process completes so they can be used to configure Event Collection in the Heeler application * **lambda/heeler\_event\_collection.py**. The code for the Lambda function that reads events from the CloudWatch Log Group, processes them, and sends them to the SQS queue * **filter\_pattern.json**. The definition of the filter pattern to use for the CloudWatch log subscription filter * **variables.tf**. The variable definitions populated by **var.auto.tfvars** * **var.auto.tfvars**. The inputs required to deploy resources to your account hosting your organization-wide CloudTrail {% file src="/files/8Lf7AsG74bZcyTTskuNJ" %} Zip file of Event Collection terraform files (CloudWatch → SQS) {% endfile %} 2. Update the `var.auto.tfvars` file * **account\_id** should be the account ID where the organization-wide CloudTrail is located. * **log\_group\_name** should be the name of the CloudWatch Log Group receiving events from the organization-wide CloudTrail. * **heeler\_external\_id** is a security feature to create a secret to be shared between your Heeler installation and the Event Collection IAM role it assumes in your account. * **heeler\_security\_role\_arns** is not an input from you, but provided by Heeler. It is the list of role ARNs Heeler will use to harvest events from the SQS queue in your account. ```hcl account_id = "" log_group_name = "" heeler_external_id = "" heeler_security_role_arns = [""] ``` 3. Initialize and plan the deployment 
terraform init
terraform plan
The plan should list only `+ create` actions — no modifications or destructions. It ends with a summary line like: ``` Plan: 14 to add, 0 to change, 0 to destroy. Changes to Outputs: + heeler_event_queue_url = (known after apply) + heeler_event_role_arn = (known after apply) + heeler_event_role_external_id = "" ``` 4. Apply the deployment ``` terraform apply ``` Enter `yes` at the confirmation prompt. Once complete, you should see something like this: ``` Apply complete! Resources: 14 added, 0 changed, 0 destroyed. Outputs: heeler_event_queue_url = "https://sqs.us-east-1.amazonaws.com/123456789012/heeler_event_collection_queue.fifo" heeler_event_role_arn = "arn:aws:iam::123456789012:role/heeler_event_collection_sqs_role" heeler_event_role_external_id = "" ``` You will need each of these output values when configuring Event Collection in Heeler. *** ## Heeler Steps The steps for configuring Event Collection in the Heeler application are the same regardless of which implementation option you chose — the three output values have the same names in both modules. 1. Once the resources are created and configured in AWS, you need to add their information to Heeler. Open the URL or just click on the settings icon at the top right and then click on **Connections**.
2. Then select the ellipsis to the right of the desired AWS organization in order to edit its Event Collection Settings
3. Then paste the output values provided by Terraform at the completion of the `terraform apply` step.
4. Finally, confirm that event collection is enabled in the updated Cloud Organization listing
At this point, it may take some time for the initial harvest of events, but afterwards events will harvest on the cadence shown in the comparison table above. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.heeler.com/mrecEO40m5D6bt7Pq5pE/getting-started/cloud-setup/amazon-web-services/aws-event-collection.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.