How to Setup a Multi-Account CloudWatch Cross Account Observability Using Terraform and CloudFormation Template for Your AWS Organization

Disclaimer: All the views and opinions expressed in the blog belong solely to the author and not necessarily to the author's employer or any other group or individual. This is not a promotion of any service, feature, or platform. 

In my previous article on CloudWatch(CW) cross-account observability for AWS Organization, I provided a step-by-step guide on how to set up multi-account visibility and observability employing a newly released feature called CloudWatch cross-account observability using AWS Console. In this article, I will provide a step-by-step guide on how you can automate the CloudWatch cross-account observability for your AWS Organization using Terraform and a CloudFormation template. 

Please refer to my earlier article on this topic for a better understanding of the concepts such as Monitoring Accounts and Source Accounts.

Monitoring Account Configuration

For monitoring account configuration, a combination of Terraform and CloudFormation are chosen as the aws_oam_sink and aws_oam_link. Resources are yet to be available in terraform-provider-aws as of Feb 26, 2023. Please refer to the GitHub issue. 

Also, the terraform-provider-awscc has an open bug (as of Feb 26, 2023) that fails on applying the Sink policy. Please refer to the GitHub issue link for more details.

Terraform Code That Creates the OAM Sink in the Monitoring AWS Account

Feel free to customize the code to modify providers and tags or change the naming conventions as per your organization's standards.

provider.tf

Feel free to modify the AWS provider as per your AWS account, region, and authentication/authorization needs. Refer to the AWS provider documentation for more details on configuring the provider for the AWS platform.

provider "aws" {
  region = "us-east-1"
  assume_role {
    role_arn = "arn:aws:iam::MONITORING-ACCOUNT-NUMBER:role/YOUR-IAM-ROLE-NAME"
  }
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.53.0"
    }
  }
}

main.tf

/*
 AWS Cloudformation stack resource that runs CFT - oam-sink-cft.yaml
 The stack creates a OAM Sink in the current account & region as per provider configuration
 Please create the AWS provider configuration as per your environment.
 For AWS provider configuration, please refer to https://registry.terraform.io/providers/hashicorp/aws/2.43.0/docs
*/
resource "aws_cloudformation_stack" "cw_sink_stack" {
  name          = "example"
  template_body = file("${path.module}/oam-sink-cft.yaml")
  parameters = {
    OrgPath = var.org_path
  }
  tags = var.tags
}

/*
  SSM parameter resource puts the CloudWatch Cross Account Observability Sink ARN in the parameter store,
  So that the Sink arn can be used from the source account while creating the Link
*/
resource "aws_ssm_parameter" "cw_sink_arn" {
  name        = "cw-sink-arn"
  description = "CloudWatch Cross Account Observability Sink identifier"
  type        = "SecureString"
  value       = aws_cloudformation_stack.cw_sink_stack.outputs["ObservabilityAccessManagerSinkArn"]
  tags        = var.tags
}

variable.tf

variable "tags" {
  description = "Custom tags for AWS resources"
  type        = map(string)
  default     = {}
}

variable "org_path" {
  description = "AWS Organization path that will be allowed to send Metric and Log data to the monitoring account"
  type        = string
}

AWS CloudFormation Template That Is Used in the Terraform “AWS_cloudformation_stack” Resource

The below CloudFormation template creates the OAM Sink resource in the Monitoring account. This template will be used to create the CloudFormation Stack in the Monitoring account. Make sure to put the template and the terraform files in the same directory.

oam-sink-cft.yaml

AWSTemplateFormatVersion: 2010-09-09
Description: 'AWS CloudFormation Template to 
              creates or updates a sink in the current account, so that it can be used as a monitoring account in CloudWatch cross-account observability. 
              A sink is a resource that represents an attachment point in a monitoring account, which source accounts can link to to be able to send observability data.'
Parameters:
  OrgPath:
    Type: String
    Description: 'Complete AWS Organization path for source account configuration for Metric data'
Resources:
  ObservabilityAccessManagerSink:
    Type: 'AWS::Oam::Sink'
    Properties:
      Name: "observability-access-manager-sink"
      Policy:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: "*"
            Resource: "*"
            Action:
              - "oam:CreateLink"
              - "oam:UpdateLink"
            Condition:
              ForAnyValue:StringLike:
                aws:PrincipalOrgPaths:
                  - !Ref OrgPath
              ForAllValues:StringEquals:
                oam:ResourceTypes:
                  - "AWS::CloudWatch::Metric"
                  - "AWS::Logs::LogGroup"
Outputs:
  ObservabilityAccessManagerSinkArn:
    Value: !GetAtt ObservabilityAccessManagerSink.Arn
    Export:
      Name: ObservabilityAccessManagerSinkArn

Apply the Changes in the AWS Provider Platform

Once you put all the above terraform and CloudFormation template files in the same directory, run terraform init to install the provider and dependencies and then terraform plan or terraform apply depending upon whether you want to view the changes only or view and apply the changes in your AWS account. Please refer to the Hashicorp website for more details on terraform commands.

When you run terraform apply or terraform plan command, you need to input the org_path value. Make sure to provide the complete AWS Organization path to allow the AWS account(s) under that path to send the metric and log data to the monitoring account. For example, if you want to allow all the AWS accounts to send the metric and log data to the monitoring account under the Organization Unit (OU) ou-0dsf-dasd67asd (assuming the OU is directly under the Root account in the organization hierarchy), then the org_path value should look like ORGANIZATION_ID/ROOT_ID/ou-0dsf-dasd67asd/*. For more information on how to set the organization path, please refer to the AWS documentation.

Once the org_path value is provided (you can also use the tfvars file to supply the variable values), and terraform apply is successful, you should see the AWS account is designated as a Monitoring account by navigating to CloudWatch settings in CloudWatch console.

Source Account Configuration

For source account configuration, we can use the terraform-provider-awscc as the link resource works perfectly. Also, the aws_oam_sink and aws_oam_link resources are yet to be available in the terraform-provider-aws as of Feb 26, 2023. Please refer to the GitHub issue. 

Terraform Code That Creates the OAM Link in the Source AWS Account

Feel free to customize the code to modify provider and tags or change the naming conventions as per your organization's standards.

provider.tf

Feel free to modify the AWSCC provider as per your AWS account, region, and authentication/authorization needs. Refer to the AWSCC provider documentation for more details on configuring the provider.

provider "aws" {
  region = "us-east-1"
  assume_role {
    role_arn = "arn:aws:iam::MONITORING-ACCOUNT-NUMBER:role/IAM-ROLE-NAME"
  }
}

provider "awscc" {
  region = "us-east-1"
  assume_role = {
    role_arn = "arn:aws:iam::SOURCE-ACCOUNT-NUMBER:role/IAM-ROLE-NAME"
  }
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.53.0"
    }
    awscc = {
      source = "hashicorp/awscc"
      version = "0.45.0"
    }
  }
}

main.tf

/*
  Link resource to create the link between the source account and the sink in the monitoring account
*/
resource "awscc_oam_link" "cw_link" {
  provider          = awscc
  label_template    = "$AccountName"
  resource_types    = ["AWS::CloudWatch::Metric", "AWS::Logs::LogGroup"]
  sink_identifier   = data.aws_ssm_parameter.cw_sink_arn.value
}

/*
  SSM parameter data block retrieves the CloudWatch Cross Account Observability Sink ARN from the parameter store,
  So that the Sink arn can be associated with the source account while creating the Link
*/
data "aws_ssm_parameter" "cw_sink_arn" {
  provider = aws
  name     = "cw-sink-arn"
}

Put both the terraform files in the same directory and run the terraform init and then terraform apply commands to create the link between the source and monitoring accounts.

Steps To Validate the CloudWatch Cross-Account Observability Changes

Now that changes are applied in both source and monitoring accounts, it's time to validate that CloudWatch log groups and metric data are showing up in the monitoring account.

Navigate to CloudWatch Console > Settings > Manage source accounts in the monitoring account. You should see the new source account is listed, and it should show that CloudWatch log and metric are being shared with the monitoring account 

If you navigate to CloudWatch log groups in the monitoring account, you should now see some of the log groups from the source account.

Also, if you navigate to CloudWatch Metrics > All Metrics in the monitoring account, now you should see some of the Metric data from the source account.