Operational Excellence and Automation · Topic 4 of 7
Observability and Log Aggregation
Collect metrics, logs and traces with CloudWatch, X-Ray and OpenTelemetry, watch ECS and EKS with Container Insights, and centralize logs by account.
- 5 min
- Metrics, logs and traces answer different questions; tracing finds the slow hop.
- ADOT is the standardized, vendor-neutral way to instrument.
- Memory and disk metrics need the CloudWatch agent.
What it is
Observability means you can explain what a system is doing from the data it emits. The three signals are metrics (numbers over time), logs (event records) and traces (the path of one request across services).
At professional scale the questions ask how to instrument many services the same way, how to see containers clearly, and how to bring logs from hundreds of accounts into one place for search and long-term archive.
How it works
Standardized instrumentation
| Tool | What it collects | Why choose it |
|---|---|---|
| CloudWatch agent | OS metrics, custom metrics, log files | Memory and disk metrics EC2 does not publish by default |
| Embedded metric format | Metrics written as structured log lines | High cardinality custom metrics from Lambda without API calls |
| AWS Distro for OpenTelemetry (ADOT) | Metrics and traces through the OpenTelemetry standard | One vendor-neutral SDK and collector, send to X-Ray, CloudWatch or Prometheus |
| AWS X-Ray | Traces, segments, service map | Find which downstream call adds latency or errors |
| CloudWatch Application Signals | Auto-instrumented latency, errors and faults per service and operation | Application performance monitoring without code changes |
ADOT is the answer when a company wants standardized instrumentation across languages and teams, or wants to keep the option of another backend. OpenTelemetry is the direction AWS recommends for new tracing work.
Basic EC2 metrics do not include memory or disk space. Those need the CloudWatch agent.
Cross-account observability
CloudWatch cross-account observability links source accounts to a monitoring account. From there, engineers search metrics, logs and traces of all linked accounts without switching roles. Link accounts through AWS Organizations so new accounts are included.
Monitoring containers
CloudWatch Container Insights collects cluster, service, task, pod and node metrics plus performance logs for Amazon ECS and Amazon EKS, including Fargate.
| Platform | How to turn it on | Notes |
|---|---|---|
| Amazon ECS | Account or cluster setting for Container Insights | Enhanced observability adds task and container level detail |
| Amazon EKS | Amazon CloudWatch Observability add-on | Enhanced observability for detailed control plane and pod metrics; can also enable Application Signals |
| Prometheus workloads | Amazon Managed Service for Prometheus with Amazon Managed Grafana | When teams already use PromQL and Grafana |
Use Container Insights for operational health: CPU and memory reservation, restarts and failed tasks. Use traces for request flow between microservices, because node metrics cannot show which service is slow.
Multi-account log aggregation
The common target is a dedicated log archive account that workload teams cannot modify.
| Need | Design |
|---|---|
| Real-time stream of logs from many accounts | CloudWatch Logs subscription filter to a cross-account destination backed by Kinesis Data Streams or Amazon Data Firehose |
| Cheap long-term archive | Firehose delivers to S3, and lifecycle rules move objects to Glacier storage classes |
| Tamper resistance | S3 Object Lock, bucket policies and SCPs that deny log deletion |
| API activity for all accounts | An organization trail in CloudTrail delivering to the log archive bucket |
| Searchable copy in one account | CloudWatch Logs centralization into a central account for organization members |
A destination is a resource in the receiving account that points to the stream, with a destination policy listing which accounts or which organization may subscribe. The log group and its destination must be in the same Region, though the stream behind the destination can be elsewhere. An account-level subscription filter policy can send all log groups in an account without configuring each group.
Firehose can transform records and batch them before they land in S3, which cuts storage and request cost. For analysis, query the archive in place with Amazon Athena.
How the exam asks about this
- "Engineers in one account must view dashboards and traces from 200 workload accounts without assuming roles" -> CloudWatch cross-account observability with a monitoring account.
- "Teams use several languages and want one tracing standard that is not locked to one backend" -> AWS Distro for OpenTelemetry sending traces to X-Ray.
- "Stream application logs from every account to a central account and keep them seven years cheaply" -> subscription filters to a cross-account Firehose destination, S3 with lifecycle to Glacier Deep Archive.
What to remember for the exam
- Metrics, logs and traces answer different questions; tracing finds the slow hop.
- ADOT is the standardized, vendor-neutral way to instrument.
- Memory and disk metrics need the CloudWatch agent.
- Container Insights covers ECS, EKS and Fargate at cluster, service and task level.
- Cross-account log delivery uses a destination backed by Kinesis Data Streams or Firehose.
- Archive to S3 with lifecycle rules and protect it with Object Lock and SCPs.
Keeping logs only in CloudWatch Logs for years in every workload account is the costly, scattered answer. Centralize them and move old data to S3 archive tiers.
Common mistakes
Expecting memory metrics from EC2 by default. The hypervisor cannot see guest memory. Install the CloudWatch agent to publish it.
Using metrics to find a slow microservice. Metrics show that latency rose, not where. Distributed tracing shows the exact downstream call.
Letting workload accounts own the archive. An administrator who is under investigation could delete evidence. Put the archive in a separate account with deny policies.
Check yourself
Which service lets one account view metrics, logs and traces from many accounts without switching roles?
CloudWatch cross-account observability, with the workload accounts linked to a monitoring account.
What must exist in the receiving account for a cross-account log subscription?
A CloudWatch Logs destination pointing to a Kinesis Data Streams or Firehose stream, with a policy that allows the sending accounts.
How do you get pod and task level metrics for EKS and ECS?
Enable CloudWatch Container Insights, with the CloudWatch Observability add-on for EKS.
Topic 4 of 7 in Operational Excellence and Automation
Test yourself on Operational Excellence and Automation
The full AWS SAP-C03 question bank has 850 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.