AWS Event Collection

Heeler event collection allows near real-time updates, analysis, and notification of meaningful changes in your environment. Heeler harvests events from AWS using logging at the organization level, routing via a Lambda function, and an SQS queue.

AWS Prerequisite

As noted above, Heeler harvests events from AWS at the organization level using a resource that most AWS Organizations already have in place. Doing so simplifies event collection and ensures complete visibility as your cloud footprint grows. The required resource is an organization-wide CloudTrail. It must have the following settings:

1. Enabled for all accounts in my organization

2. Multi-region trail

3. All management events

4. CloudWatch Logs enabled

The last requirement may be new to your organization. If so, enable CloudWatch Logs to create a Log Group. Then, edit the Log Group retention setting to `1 day` to allow events to expire. Doing so reduces costs and Heeler does not need the events retained as it harvests events every 60 seconds.

Terraform Implementation

To deploy the resources required for Event Collection, you will need Terraform configured to deploy resources in the account hosting the organization-wide CloudTrail.

Executing the Terraform plan creates the following:

• CloudWatch subscription filter. This filter excludes events from processing that are not security-relevant, e.g.., PutObject.

• Lambda function. The Lambda function reads events from the CloudWatch log group, processes the events, and sends the events to an SQS queue.

• SQS queue. The SQS queue holds the processed events until they are harvested by Heeler.

• IAM role and policy. The role that Heeler assumes and the policy that allows Heeler to harvest events from the SQS queue.

• Supporting resources. The Terraform plan also deploys supporting resources such as:

  • Dead Letter Queue to capture failed deliveries to the SQS queue

  • KMS key to encrypt SQS queue

  • IAM role, policy, and permission for the Lambda function to assume and use to read CloudWatch log groups and send encrypted messages to SQS

Terraform Steps

1. Download and unzip the files to an appropriate location for your environment.

The zip file contains:

• main.tf. Defines the resources to be deployed

• output.tf. Defines the values displayed after the process completes so they can be used to configure Event Collection in the Heeler application

• lambda/heeler_event_collection.py. The code for the Lambda function that reads events from the CloudWatch Log Group, processes the events, and sends them to the SQS queue

• filter_pattern.json. The definition of the filter pattern to use for the CloudWatch log subscription filter

• variables.tf. The variable definitions to populated by var.auto.tfvars

• var.auto.tfvars. The inputs required to deploy resources to your account hosting your organization-wide CloudTrail.

4KB

heeler_event_collection.zip

archive

Open

Zip file of Event Collection terraform files

1. Update the var.auto.tfvars file

You will need to update the input variable values as indicated.

• account ID should be the account ID where the organization-wide CloudTrail is located.

• log_group_name should be the name of the CloudWatch Log Group that is receiving events from the organization-wide CloutTrail.

• heeler_external_id is a security feature to create a secret to be shared between your Heeler installation and the Event Collection IAM role it assumes in your account

• heeler_security_role_arn is not an input from you, but provided by Heeler. It is the ARN of the role Heeler will use harvest events from the SQS queue in your account.

account_id         = "<replace with your account_id>"
log_group_name     = "<replace with the name of your log group that is receiving events from your organization-wide CloudTrail>"
heeler_external_id = "<replace with your external_id>"

heeler_security_role_arns = ["<values provided by Heeler>"]

1. Initialize and plan the deployment

terraform init
terraform plan

You should see a response like the following:

data.local_file.filter_pattern: Reading...
data.archive_file.heeler_archive_file: Reading...
data.local_file.filter_pattern: Read complete after 0s [id=4cb9a45654a38dbcfea301c50d9a341379c49884]
data.archive_file.heeler_archive_file: Still reading... [10s elapsed]
data.archive_file.heeler_archive_file: Read complete after 17s [id=7e438df298cc6a25f2eedb8b9f4e89d3805dac54]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_cloudwatch_log_subscription_filter.heeler_cloudwatch_log_subscription_filter will be created
  + resource "aws_cloudwatch_log_subscription_filter" "heeler_cloudwatch_log_subscription_filter" {
      + destination_arn = (known after apply)
      + distribution    = "ByLogStream"
      + filter_pattern  = "{$.readOnly IS FALSE && $.errorCode NOT EXISTS && $.eventName!=AssumeRole && $.eventName!=CompleteLayerUpload && $.eventName!=ConsoleLogin && $.eventName!=CreateGrant && $.eventName!=CreateNetworkInterface && $.eventName!=Federate && $.eventName!=GetSigninToken && $.eventName!=InitiateLayerUpload && $.eventName!=PurgeQueue && $.eventName!=PutImage && $.eventName!=PutObject && $.eventName!=RetireGrant && $.eventName!=SendCommand && $.eventName!=StartQuery && $.eventName!=UpdateInstanceInformation && $.eventName!=UploadLayerPart}"
      + id              = (known after apply)
      + log_group_name  = "aws-cloudtrail-logs-123456789012-abcdefgh"
      + name            = "heeler_cloudwatch_log_subscription_filter"
      + role_arn        = (known after apply)
    }

  # aws_iam_policy.heeler_iam_policy_lambda will be created
  + resource "aws_iam_policy" "heeler_iam_policy_lambda" {
      + arn              = (known after apply)
      + attachment_count = (known after apply)
      + description      = "Policy to allow Heeler Event Collection Lambda to read from Cloudwatch logs and send processed messages to Heeler SQS for harvest."
      + id               = (known after apply)
      + name             = "heeler_event_collection_lambda_policy"
      + name_prefix      = (known after apply)
      + path             = "/"
      + policy           = (known after apply)
      + policy_id        = (known after apply)
      + tags_all         = (known after apply)
    }

  # aws_iam_policy.heeler_iam_policy_sqs will be created
  + resource "aws_iam_policy" "heeler_iam_policy_sqs" {
      + arn              = (known after apply)
      + attachment_count = (known after apply)
      + description      = "Policy to allow Heeler to harvest messages from SQS queue."
      + id               = (known after apply)
      + name             = "heeler_event_collection_sqs_policy"
      + name_prefix      = (known after apply)
      + path             = "/"
      + policy           = (known after apply)
      + policy_id        = (known after apply)
      + tags_all         = (known after apply)
    }

  # aws_iam_role.heeler_iam_role_lambda will be created
  + resource "aws_iam_role" "heeler_iam_role_lambda" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = [
                          + "sts:AssumeRole",
                        ]
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "lambda.amazonaws.com"
                        }
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + description           = "Role for Heeler Event Collection Lambda to assume"
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "heeler_event_collection_lambda_role"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + tags_all              = (known after apply)
      + unique_id             = (known after apply)

      + inline_policy (known after apply)
    }

  # aws_iam_role.heeler_iam_role_sqs will be created
  + resource "aws_iam_role" "heeler_iam_role_sqs" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = [
                          + "sts:AssumeRole",
                        ]
                      + Condition = {
                          + StringEquals = {
                              + "sts:ExternalId" = "<heeler_external_id>"
                            }
                        }
                      + Effect    = "Allow"
                      + Principal = {
                          + AWS = [
                              + "<heeler_security_role_arn>",
                            ]
                        }
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + description           = "Role for Heeler to assume to harvest SQS queue."
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "heeler_event_collection_sqs_role"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + tags_all              = (known after apply)
      + unique_id             = (known after apply)

      + inline_policy (known after apply)
    }

  # aws_iam_role_policy_attachment.heeler_iam_role_policy_attachment_lambda will be created
  + resource "aws_iam_role_policy_attachment" "heeler_iam_role_policy_attachment_lambda" {
      + id         = (known after apply)
      + policy_arn = (known after apply)
      + role       = "heeler_event_collection_lambda_role"
    }

  # aws_iam_role_policy_attachment.heeler_iam_role_policy_attachment_sqs will be created
  + resource "aws_iam_role_policy_attachment" "heeler_iam_role_policy_attachment_sqs" {
      + id         = (known after apply)
      + policy_arn = (known after apply)
      + role       = "heeler_event_collection_sqs_role"
    }

  # aws_kms_alias.heeler_kms_alias_sqs will be created
  + resource "aws_kms_alias" "heeler_kms_alias_sqs" {
      + arn            = (known after apply)
      + id             = (known after apply)
      + name           = "alias/heeler_kms_key_sqs"
      + name_prefix    = (known after apply)
      + target_key_arn = (known after apply)
      + target_key_id  = (known after apply)
    }

  # aws_kms_key.heeler_kms_key_sqs will be created
  + resource "aws_kms_key" "heeler_kms_key_sqs" {
      + arn                                = (known after apply)
      + bypass_policy_lockout_safety_check = false
      + customer_master_key_spec           = "SYMMETRIC_DEFAULT"
      + description                        = "KMS key for encrypting SQS messages"
      + enable_key_rotation                = true
      + id                                 = (known after apply)
      + is_enabled                         = true
      + key_id                             = (known after apply)
      + key_usage                          = "ENCRYPT_DECRYPT"
      + multi_region                       = (known after apply)
      + policy                             = (known after apply)
      + rotation_period_in_days            = (known after apply)
      + tags_all                           = (known after apply)
    }

  # aws_lambda_function.heeler_lambda_function will be created
  + resource "aws_lambda_function" "heeler_lambda_function" {
      + architectures                  = (known after apply)
      + arn                            = (known after apply)
      + code_sha256                    = (known after apply)
      + description                    = "Heeler Event Collection Lambda to read from Cloudwatch logs and send processed messages to Heeler SQS for harvest."
      + filename                       = "./lambda/heeler_event_collection_lambda.zip"
      + function_name                  = "heeler_event_collection_lambda"
      + handler                        = "heeler_event_collection_lambda.lambda_handler"
      + id                             = (known after apply)
      + invoke_arn                     = (known after apply)
      + last_modified                  = (known after apply)
      + layers                         = []
      + memory_size                    = 128
      + package_type                   = "Zip"
      + publish                        = false
      + qualified_arn                  = (known after apply)
      + qualified_invoke_arn           = (known after apply)
      + reserved_concurrent_executions = -1
      + role                           = (known after apply)
      + runtime                        = "python3.12"
      + signing_job_arn                = (known after apply)
      + signing_profile_version_arn    = (known after apply)
      + skip_destroy                   = false
      + source_code_hash               = "rd/dDshit0cS7S6gNdZGyRQGiDP1gDte3TjZ50iP3cU="
      + source_code_size               = (known after apply)
      + tags                           = {
          + "Name" = "HeelerEventCollection"
        }
      + tags_all                       = {
          + "Name" = "HeelerEventCollection"
        }
      + timeout                        = 10
      + version                        = (known after apply)

      + environment {
          + variables = (known after apply)
        }

      + ephemeral_storage {
          + size = 512
        }

      + logging_config {
          + log_format = "Text"
          + log_group  = "/aws/lambda/heeler_event_collection_lambda"
        }

      + tracing_config {
          + mode = "PassThrough"
        }
    }

  # aws_lambda_permission.heeler_lambda_permission will be created
  + resource "aws_lambda_permission" "heeler_lambda_permission" {
      + action              = "lambda:InvokeFunction"
      + function_name       = "heeler_event_collection_lambda"
      + id                  = (known after apply)
      + principal           = "logs.amazonaws.com"
      + source_arn          = "arn:aws:logs:us-east-1:123456789012:log-group:<log_group_name>:*"
      + statement_id        = "InvokePermissionsForCW"
      + statement_id_prefix = (known after apply)
    }

  # aws_sqs_queue.heeler_sqs_queue will be created
  + resource "aws_sqs_queue" "heeler_sqs_queue" {
      + arn                               = (known after apply)
      + content_based_deduplication       = true
      + deduplication_scope               = (known after apply)
      + delay_seconds                     = 0
      + fifo_queue                        = true
      + fifo_throughput_limit             = (known after apply)
      + id                                = (known after apply)
      + kms_data_key_reuse_period_seconds = 300
      + kms_master_key_id                 = (known after apply)
      + max_message_size                  = 262144
      + message_retention_seconds         = 7200
      + name                              = "heeler_event_collection_queue.fifo"
      + name_prefix                       = (known after apply)
      + policy                            = (known after apply)
      + receive_wait_time_seconds         = 10
      + redrive_allow_policy              = (known after apply)
      + redrive_policy                    = (known after apply)
      + sqs_managed_sse_enabled           = (known after apply)
      + tags_all                          = (known after apply)
      + url                               = (known after apply)
      + visibility_timeout_seconds        = 30
    }

  # aws_sqs_queue.heeler_sqs_queue_dlq will be created
  + resource "aws_sqs_queue" "heeler_sqs_queue_dlq" {
      + arn                               = (known after apply)
      + content_based_deduplication       = false
      + deduplication_scope               = (known after apply)
      + delay_seconds                     = 0
      + fifo_queue                        = true
      + fifo_throughput_limit             = (known after apply)
      + id                                = (known after apply)
      + kms_data_key_reuse_period_seconds = 300
      + kms_master_key_id                 = (known after apply)
      + max_message_size                  = 262144
      + message_retention_seconds         = 345600
      + name                              = "heeler_event_collection_queue_dlq.fifo"
      + name_prefix                       = (known after apply)
      + policy                            = (known after apply)
      + receive_wait_time_seconds         = 0
      + redrive_allow_policy              = (known after apply)
      + redrive_policy                    = (known after apply)
      + sqs_managed_sse_enabled           = (known after apply)
      + tags_all                          = (known after apply)
      + url                               = (known after apply)
      + visibility_timeout_seconds        = 30
    }

  # aws_sqs_queue_policy.heeler_sqs_queue_policy will be created
  + resource "aws_sqs_queue_policy" "heeler_sqs_queue_policy" {
      + id        = (known after apply)
      + policy    = (known after apply)
      + queue_url = (known after apply)
    }

Plan: 14 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + heeler_event_queue_url        = (known after apply)
  + heeler_event_role_arn         = (known after apply)
  + heeler_event_role_external_id = "<heeler_external_id>"

1. Apply the deployment

terraform apply

You should see a response like the one from terraform apply, but also includes

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: 

enter yes

Once complete, you should see something like this

Apply complete! Resources: 14 added, 0 changed, 0 destroyed.

Outputs:

heeler_event_queue_url = "https://sqs.us-east-1.amazonaws.com/123456789012/heeler_event_collection_queue.fifo"
heeler_event_role_arn = "arn:aws:iam::123456789012:role/heeler_event_collection_sqs_role"
heeler_event_role_external_id = "<heeler_external_id>"

You will need each of these output values when configuring Event Collection in Heeler.

Heeler Steps

1. Once the resources are created and configured in AWS, you need to add their information to Heeler. Open the URL https://app.heeler.com/administration/connections/organizations or just click on the settings icon at the top right and then click on Connections.

1. Then select the ellipsis to the right of the desired AWS organization in order to edit its Event Collection Settings

1. Then paste the output values provided by Terraform at the completion of the terraform apply step.

1. Finally, confirm that event collection is enabled in the updated Cloud Organization listing

At this point, it may take some time for the initial harvest of events, but afterwards, events should harvest every 60 seconds.