CloudWatch Metrics and the Agent

What it is

A metric is a time ordered set of data points, such as CPU utilization every minute. Amazon CloudWatch is the AWS service that stores metrics, graphs them and raises alarms on them.

Most AWS services publish metrics automatically. For things AWS cannot see, such as memory use inside an instance or your own business counters, you must publish custom metrics.

How it works

Namespaces, dimensions and resolution

A metric is identified by its namespace, name and dimensions. Namespaces for AWS services start with AWS/, such as AWS/EC2, and custom namespaces cannot use that prefix.

A dimension is a name and value pair such as InstanceId. Each unique combination of dimensions is a separate metric.

Not all EC2 metrics are visible to the hypervisor. Memory use, disk space use and swap are not published by default, so the CloudWatch agent is required to collect them.

Retention and rollup

CloudWatch keeps data at decreasing detail and merges it over time.

For longer storage, use a metric stream to send metrics to Amazon Data Firehose and then to S3 or another destination for analysis.

Publishing a custom metric

aws cloudwatch put-metric-data --namespace MyApp --metric-name QueueDepth \
  --value 42 --unit Count --dimensions Service=checkout

Call this from a script, or better, use the agent or the embedded metric format.

The unified CloudWatch agent

The agent runs on EC2 instances and on premises servers and collects system level metrics (memory, disk, swap, network and more) as well as log files. It can also receive StatsD and collectd metrics. The instance role needs the CloudWatchAgentServerPolicy managed policy.

A minimal configuration file:

{
  "metrics": {
    "metrics_collected": {
      "mem": {"measurement": ["mem_used_percent"]},
      "disk": {"measurement": ["used_percent"], "resources": ["*"]}
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {"file_path": "/var/log/app.log",
           "log_group_name": "/app/web",
           "log_stream_name": "{instance_id}"}
        ]
      }
    }
  }
}

Store the configuration in Systems Manager Parameter Store and apply it to many instances at once, then use a State Manager association to keep the agent installed and configured.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config -m ec2 -s -c ssm:AmazonCloudWatch-linux

Embedded metric format

The embedded metric format (EMF) lets an application write one structured JSON log line that CloudWatch turns into metrics. This is efficient for Lambda and containers, because there is no extra API call and the same record also supports detailed log queries.

{
  "_aws": {
    "Timestamp": 1700000000000,
    "CloudWatchMetrics": [{
      "Namespace": "MyApp",
      "Dimensions": [["Service"]],
      "Metrics": [{"Name": "OrderLatency", "Unit": "Milliseconds"}]
    }]
  },
  "Service": "checkout",
  "OrderLatency": 142
}

Specialised collection

Aggregation

Choose the right statistic: average hides spikes, while maximum and percentiles such as p99 show them. Use metric math to build ratios, such as errors divided by requests, and Metrics Insights to query many metrics with SQL like syntax.