Log Analysis, Tracing and Anomaly Detection

What it is

Collecting data is only useful if you can find the cause of a problem quickly. This page covers the tools that turn raw logs, metrics and requests into answers: queries, traces, synthetic tests and machine learning detection.

A good approach has three layers. Metrics tell you that something is wrong, logs tell you what happened, and traces tell you where in a chain of services the time or the error occurred.

How it works

CloudWatch Logs Insights

Logs Insights queries one or more log groups with a pipe style language. It discovers fields in JSON logs automatically and exposes system fields such as @timestamp and @message.

fields @timestamp, @message
| filter @message like /ERROR/
| stats count() as errors by bin(5m)
| sort errors desc

Lambda writes a REPORT line for each invocation, and Logs Insights exposes fields such as @duration, @maxMemoryUsed and @initDuration.

filter @type = "REPORT"
| stats max(@duration) as maxMs, avg(@maxMemoryUsed) as avgMem by bin(10m)

Queries can cover several log groups and, with cross account observability, groups in other accounts. You can save queries and add their results to dashboards.

Contributor Insights and Metrics Insights

Contributor Insights reads log data and shows the top contributors, such as the client IPs sending the most requests or the users causing most errors. Metrics Insights queries metrics with an SQL like language, for example the ten instances with the highest CPU.

SELECT AVG(CPUUtilization) FROM SCHEMA("AWS/EC2", InstanceId)
GROUP BY InstanceId ORDER BY AVG() DESC LIMIT 10

Anomaly detection

CloudWatch can learn the normal pattern of a metric, including daily and weekly cycles, and draw an expected band. An alarm can then fire when the metric leaves the band, so you do not have to guess a fixed threshold. Logs anomaly detection finds unusual patterns in log events in a similar way.

AWS X-Ray

X-Ray traces a request as it passes through services. A trace is made of segments (one per service), which contain subsegments (calls to downstream services or databases). The service map draws the graph and highlights latency and error rates per node.

Instrument applications with the X-Ray SDK or with AWS Distro for OpenTelemetry. On EC2 and on premises servers the X-Ray daemon relays data, and Lambda supports active tracing as a function setting. ECS tasks usually run the daemon or collector as a sidecar container.

aws lambda update-function-configuration --function-name orders --tracing-config Mode=Active

X-Ray finds the slow downstream call in a chain such as API Gateway to Lambda to DynamoDB, which a single log group cannot show.

Synthetic monitoring

CloudWatch Synthetics runs canaries, which are scripts (Node.js or Python) that visit endpoints or workflows on a schedule from AWS. They find outages before real users do and produce metrics, screenshots and logs that alarms can watch.

Machine learning operations insight

Amazon DevOps Guru analyses metrics, logs, events and traces of the resources you choose (by CloudFormation stack, tag or account) and reports insights about anomalous behavior with related events and recommended actions. Choose it for broad anomaly detection with little configuration.

Analysis outside CloudWatch

For Athena, convert logs to a columnar format such as Parquet and partition them by date to reduce scanned data and cost.

A troubleshooting workflow

1. Start from an alarm or user report and note the time window.
2. Check metrics for the symptom: error rate, latency, saturation.
3. Query logs for errors in that window and group them by cause.
4. Follow a failing request through the X-Ray trace to the slow or failing component.
5. Compare with a recent deployment or configuration change from CloudTrail and Config.