Log Collection and Storage

What it is

Logs record what happened inside applications and infrastructure. Collecting them well means getting every source into a central place, keeping them for the right time, controlling who can read them and making them searchable.

Amazon CloudWatch Logs is the main collection service. It organizes data into log groups (one per application or source) and log streams (one per instance or container inside the group).

How it works

Where logs come from

Choose S3 as the destination for high volume logs that you analyse in batch, and CloudWatch Logs where you need near real time search, metric filters or alarms.

Retention, encryption and cost

By default a log group keeps data forever, which can become a large cost. Set a retention period per group.

aws logs put-retention-policy --log-group-name /app/web --retention-in-days 30

Associate a customer managed KMS key with a log group to control encryption, and use data protection policies to detect and mask sensitive data such as personal identifiers. The Infrequent Access log class costs less for logs you rarely query.

Metric filters

A metric filter scans incoming log events for a pattern and increments a CloudWatch metric, which an alarm can then watch.

aws logs put-metric-filter --log-group-name /app/web --filter-name errors \
  --filter-pattern '"ERROR"' \
  --metric-transformations metricName=ErrorCount,metricNamespace=MyApp,metricValue=1

Metric filters apply only to data ingested after the filter exists, and they do not change old data.

Subscription filters

A subscription filter streams matching log events in near real time to a destination.

aws logs put-subscription-filter --log-group-name /app/web \
  --filter-name to-firehose --filter-pattern "" \
  --destination-arn arn:aws:firehose:us-east-1:111122223333:deliverystream/central-logs \
  --role-arn arn:aws:iam::111122223333:role/CWLtoFirehose

A log group can have a small number of subscription filters, so route to one stream and fan out from there if many consumers need the data.

Central logging across accounts

Large organizations send logs from every account to a central logging account. The receiving account creates a destination, backed by Kinesis Data Streams or Firehose, with a policy that allows the sender accounts. Each sender creates a subscription filter that targets the destination.

Control Tower creates a Log Archive account for this purpose, and CloudTrail and Config also deliver there. Protect it with a restrictive bucket policy, KMS encryption and S3 Object Lock for tamper resistance.

Export versus streaming

Exporting a log group to S3 with a create export task is a batch operation and not real time. Use it for ad hoc archiving. For continuous delivery, use a subscription filter with Firehose, which can compress, convert and partition data by time before writing to S3.

Analysis in place

Once logs sit in S3, Amazon Athena queries them with SQL and the Glue Data Catalog keeps the schema. For interactive search and dashboards across large volumes, Amazon OpenSearch Service is the usual choice. For queries inside CloudWatch, use CloudWatch Logs Insights, covered on a later page.

Lifecycle for archives

An S3 lifecycle rule moves aging logs to cheaper storage classes and then expires them, matching your compliance period. Partition data by date so queries scan less and cost less.