Logs, Metrics and Traces

What it is

Three signals describe what a running application is doing. Metrics are numbers over time, logs are timestamped records of events, and traces follow one request across several services.

Each answers a different question. Metrics tell you that something is wrong, traces tell you where in the request path it is wrong, and logs tell you why.

On the exam you are given a symptom and must pick the signal or the query that finds the cause fastest.

How it works

Reading metrics

AWS services publish metrics to CloudWatch automatically, grouped by namespace such as AWS/Lambda. The average often hides problems, so look at maximums and percentiles such as p99 for latency.

Reading logs

Lambda writes everything your code prints to a log group named /aws/lambda/<function name>, with one log stream per execution environment. The execution role needs the logs:CreateLogGroup, logs:CreateLogStream and logs:PutLogEvents permissions, which the AWSLambdaBasicExecutionRole managed policy includes. A function with no logs at all usually lacks them.

Every invocation ends with a REPORT line showing Duration, Billed Duration, Memory Size, Max Memory Used and, on a cold start, Init Duration. When Max Memory Used equals Memory Size, the function is close to running out of memory.

Querying with CloudWatch Logs Insights

Logs Insights runs ad hoc queries over one or more log groups for a chosen time range. It discovers JSON fields automatically, and for Lambda it exposes fields such as @duration, @maxMemoryUsed and @requestId.

fields @timestamp, @requestId, @duration
| filter @type = "REPORT"
| sort @duration desc
| limit 10

The common commands are fields, filter, stats, sort, limit, parse and dedup. A query such as stats count() by bin(5m) after a filter on ERROR shows when errors started.

Two other tools work on log data continuously:

• Metric filter: matches a pattern in incoming log events and turns it into a CloudWatch metric you can alarm on. It only counts events that arrive after the filter exists.
• Subscription filter: streams matching events to a destination such as Lambda, Kinesis or Amazon Data Firehose for processing or long term storage.

Designing a logging strategy

Log in structured JSON with the same keys everywhere, for example level, message, requestId, userId, operation and durationMs. Logs Insights and metric filters can then select on those fields instead of parsing free text.

• Use levels. DEBUG for detail, INFO for state changes, WARN and ERROR for problems. Lambda advanced logging controls can emit JSON logs and set an application log level, so lower levels are dropped before they reach CloudWatch.
• Add a correlation ID. Pass a request identifier through every service call so one user action can be followed across components. The X-Amzn-Trace-Id header does this for X-Ray.
• Never log secrets or personal data. Mask tokens, passwords and personally identifiable information before writing the event.
• Control cost. CloudWatch Logs keeps data forever by default, so set a retention period on each log group. Avoid logging full payloads on hot paths.

Powertools for AWS Lambda provides a Logger that produces structured JSON with the request context added for you.

Reading traces

AWS X-Ray groups the work of one request into a trace made of segments, one per service, with subsegments for downstream calls. The service map shows latency and error rates per node, and a slow or faulting node points at the culprit. The next page covers how to turn tracing on.

How the exam asks about this

• "Find the ten slowest invocations of a function in the last hour" -> a Logs Insights query that filters REPORT lines, sorts on @duration and limits to ten.
• "Alert the team when the application logs more than five ERROR entries in five minutes" -> a metric filter on the log group plus a CloudWatch alarm.
• "Developers cannot filter log entries by customer across several services" -> structured JSON logs with consistent fields, queried with Logs Insights.