Troubleshooting and Optimization · Topic 2 of 5
Custom Metrics, Alerts, Tracing and Health Checks
Emit custom CloudWatch metrics, send alarm notifications, enable X-Ray tracing and set up health checks and probes for DVA-C03 monitoring questions.
- 5 min
- PutMetricData needs cloudwatch:PutMetricData; EMF needs only the ability to write logs.
- Standard metrics are one minute; high resolution is one second (StorageResolution 1).
- X-Ray annotations are indexed and searchable; metadata is not.
What it is
AWS publishes infrastructure metrics for you, but it cannot know your business events, such as orders placed or payments declined. Custom metrics are numbers your code publishes.
Once you have metrics, alarms turn them into notifications, tracing shows where a request spends its time, and health checks let the platform stop sending traffic to an unhealthy copy of your application.
How it works
Emitting custom metrics
There are two ways to publish a metric from code.
| Method | How it works | Pick it when |
|---|---|---|
| PutMetricData API | SDK call with a namespace, metric name, value, unit and dimensions | A long running service publishes values directly |
| Embedded metric format (EMF) | Write a JSON log line with an _aws section; CloudWatch Logs extracts the metrics | A Lambda function should publish metrics without an extra API call |
Key facts for PutMetricData:
- The caller needs the cloudwatch:PutMetricData permission.
- A metric is identified by namespace, name and dimensions. Each unique dimension combination is a separate metric, so a dimension such as requestId creates a metric per request. Avoid high cardinality dimensions.
- Standard resolution metrics have one minute granularity. A high-resolution metric is stored at one second, set with StorageResolution of 1. EMF takes the same StorageResolution field.
Notification alerts
A CloudWatch alarm watches one metric against a threshold and is in one of three states: OK, ALARM or INSUFFICIENT_DATA. You choose the period, the evaluation periods and how missing data is treated. Actions on a state change include publishing to an Amazon SNS topic, which then notifies email, SMS, HTTPS endpoints, Lambda functions or SQS queues.
Common alert sources a developer configures:
- Quota limits: quotas that publish usage metrics (namespace AWS/Usage) can drive an alarm, which you can create from Service Quotas. For Lambda, alarm on the ConcurrentExecutions metric before the account limit is reached.
- Deployment events: CodeDeploy, CodePipeline and CodeBuild publish state change events to EventBridge. An EventBridge rule matches the event, such as a pipeline execution succeeding, and targets an SNS topic. Notification rules in the Developer Tools console do the same without writing a rule.
Tracing with AWS X-Ray
A trace follows one request. Each service adds a segment, your code can add subsegments around downstream calls, and the service map draws the result.
- Annotations are key value pairs that X-Ray indexes. You can filter traces on them, for example by customer tier.
- Metadata is extra data that is stored but not indexed and cannot be searched.
- Sampling decides which requests are traced. The Lambda documentation describes a default of one request per second plus five percent of additional requests.
To turn tracing on:
- Lambda: set the tracing mode to Active (TracingConfig Mode Active, or Tracing: Active in SAM). The execution role needs xray:PutTraceSegments and xray:PutTelemetryRecords, which the AWSXRayDaemonWriteAccess policy includes. In PassThrough mode, the default, Lambda only forwards the trace header.
- API Gateway: enable X-Ray tracing on the stage.
- EC2, ECS and on premises: run the X-Ray daemon (or the AWS Distro for OpenTelemetry collector), which relays traces from your code to the service.
- Your code: use an instrumentation library to wrap AWS SDK clients and HTTP calls so downstream calls appear as subsegments. AWS recommends OpenTelemetry (AWS Distro for OpenTelemetry). The X-Ray SDKs and daemon are in maintenance mode, with end of support on February 25, 2027.
The X-Amzn-Trace-Id header carries the trace ID between services.
Monitoring workload health
A health check lets the platform decide whether a copy of your application should receive traffic.
| Where | Mechanism | Result of failure |
|---|---|---|
| Elastic Load Balancing | Target group health check on a path, with interval and thresholds | Target gets no traffic |
| EC2 Auto Scaling | Health check type EC2 (default) or ELB | With ELB type, an unhealthy instance is replaced |
| Amazon ECS | Container health check in the task definition, plus load balancer checks | Task is replaced |
| Amazon EKS (Kubernetes) | Liveness, readiness and startup probes | Liveness restarts the container; readiness removes the pod from service endpoints |
A liveness probe asks whether the process is stuck. A readiness probe asks whether it can serve requests now. A failed readiness probe stops traffic but restarts nothing.
How the exam asks about this
- "A Lambda function must publish a custom latency metric with the least code and no extra API calls" -> write an embedded metric format log line.
- "Notify the team when a CodePipeline execution fails" -> an EventBridge rule on the pipeline state change event with an SNS topic target.
- "Find slow requests for premium customers in X-Ray" -> annotate traces with the tier and filter on it.
What to remember for the exam
- PutMetricData needs cloudwatch:PutMetricData; EMF needs only the ability to write logs.
- Standard metrics are one minute; high resolution is one second (StorageResolution 1).
- X-Ray annotations are indexed and searchable; metadata is not.
- Lambda needs Active tracing plus the X-Ray write permissions on the execution role.
- An Auto Scaling group replaces unhealthy instances for load balancer failures only when its health check type is ELB.
- Readiness failure removes traffic; liveness failure restarts the container.
High cardinality dimensions such as a request ID or user ID create a separate custom metric for every value, which makes dashboards useless and costs grow quickly. Put those values in logs or X-Ray annotations instead.
Common mistakes
Enabling tracing without permissions. Setting Active tracing on a function whose role cannot call X-Ray leaves the trace empty. Add the X-Ray write permissions.
Using the default Auto Scaling health check. The default only sees instance status, so an instance with a crashed application keeps running. Switch to the ELB check type.
Ignoring missing data. An alarm treats missing data in a way you configure. Choose that behavior on purpose.
Check yourself
Which format lets a Lambda function publish metrics through its log output?
The CloudWatch embedded metric format, a JSON log event with an _aws section that CloudWatch Logs extracts into metrics.
What is the difference between an X-Ray annotation and metadata?
Annotations are indexed and can be used in filter expressions; metadata is stored with the trace but cannot be searched.
What does a failed readiness probe do?
It removes the pod from service endpoints so it receives no traffic, without restarting the container.
Topic 2 of 5 in Troubleshooting and Optimization
Test yourself on Troubleshooting and Optimization
The full AWS DVA-C03 question bank has 850 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.