Monitoring and Optimization · Topic 2 of 6
CloudWatch Logs and Filters
Collect logs with the CloudWatch agent, turn patterns into metrics with metric filters, stream logs with subscription filters and query with Insights.
- 4 min
- Log groups hold log streams, and retention is set per log group (never expire by default).
- The unified CloudWatch agent collects logs and system metrics from EC2 and on premises servers.
- A metric filter turns matches into a metric, which an alarm can watch.
What it is
Amazon CloudWatch Logs stores log data from servers, containers, Lambda functions and AWS services in one place. You can search it, keep it for a chosen time, and react to patterns in it.
Logs are organised in log groups (one per application or source) and log streams (one per instance, container or function execution environment). Retention is set on the log group, and by default events never expire.
How it works
Getting logs in
Lambda functions write to CloudWatch Logs automatically when their execution role allows it. Many AWS services, such as VPC (Virtual Private Cloud) flow logs, CloudTrail and Route 53 query logging, can deliver there too.
For EC2 instances and on premises servers, install the unified CloudWatch agent. It collects log files and also memory and disk metrics. The instance role needs permission to write to CloudWatch, and the agent configuration can be kept in Systems Manager Parameter Store so many servers share it.
Metric filters
A metric filter scans incoming log events for a pattern and adds to a CloudWatch metric each time it matches. You then put an alarm on that metric.
# AWS CLI: count log lines that contain ERROR
aws logs put-metric-filter \
--log-group-name /app/web \
--filter-name ErrorCount \
--filter-pattern "ERROR" \
--metric-transformations \
metricName=ErrorCount,metricNamespace=App,metricValue=1For JSON logs the pattern can test fields, for example { $.statusCode = 500 }. A metric filter only counts events that arrive after the filter exists, it does not go back over old data.
Subscription filters
A subscription filter sends matching log events in near real time to another destination. Common targets are Kinesis Data Streams, Amazon Data Firehose and Lambda. It is the standard way to centralise logs from many accounts or to feed a search or archive system.
| Need | Use |
|---|---|
| Alarm when a pattern appears | Metric filter plus alarm |
| Stream logs to another system or account | Subscription filter |
| Ad hoc investigation | Logs Insights query |
| Long term archive | Export to S3 or Firehose to S3 |
Logs Insights
CloudWatch Logs Insights is an interactive query tool for one or more log groups. Queries run over a time range you choose, and you pay for data scanned.
fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 20You can also use stats to aggregate, for example stats count(*) by bin(5m) to count events per five minutes.
Retention and export
Set a retention period on every log group that does not need to live forever, because stored data is billed. To archive, create an export task to an S3 bucket, or stream with Firehose. An export task is not real time.
Related logging services
CloudTrail records AWS API calls (who did what), and it can send events to CloudWatch Logs so you can alarm on them. For example, a metric filter on CloudTrail events can alarm on root account sign in.
AWS Config tracks configuration changes, and VPC flow logs record network traffic metadata. Each answers a different question, so match the service to the question.
If the question says "alert when the word X appears in a log", think metric filter plus alarm. If it says "send logs to another account or system", think subscription filter.
What to remember for the exam
- Log groups hold log streams, and retention is set per log group (never expire by default).
- The unified CloudWatch agent collects logs and system metrics from EC2 and on premises servers.
- A metric filter turns matches into a metric, which an alarm can watch.
- Metric filters apply only to events ingested after creation.
- Subscription filters stream events to Kinesis Data Streams, Firehose or Lambda.
- Logs Insights is for interactive queries and is charged by data scanned.
- Exporting to S3 is a batch task, not real time.
- CloudTrail logs can be sent to CloudWatch Logs to alarm on API activity.
Common mistakes
Expecting an instance to send logs on its own. EC2 does not ship log files by default. Install and configure the agent and give the instance role permission.
Using a metric filter on old data. Filters do not process events that were already stored.
Leaving retention at never expire. Logs pile up and cost grows. Set a retention period that matches your policy.
Mixing up CloudTrail, Config and CloudWatch Logs. CloudTrail is API activity, Config is resource configuration state, and CloudWatch Logs is application and system log text.
Choosing an export task for real time needs. Use a subscription filter when you need near real time delivery.
A Lambda function that cannot write logs usually lacks logging permission on its execution role, so check the role before you suspect the code.
Check yourself
Try these three.
How do you alarm when ERROR appears in an application log?
Create a metric filter on the log group for the ERROR pattern, then create a CloudWatch alarm on the resulting metric.
Which feature sends log events to Kinesis or Lambda as they arrive?
A subscription filter on the log group.
Does a new metric filter count events that were logged yesterday?
No, it only counts events that arrive after the filter is created.
Topic 2 of 6 in Monitoring and Optimization
Test yourself on Monitoring and Optimization
The full AWS SOA-C03 question bank has 850 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.