Audit Logging on AWS

What it is

An audit log is a trustworthy record of who did what, when and from where. Auditors, security teams and regulators use it to check that rules were followed and to investigate incidents.

For a data engineer the work is to switch the right logs on, store them safely for the required time, and make them searchable.

How it works

AWS CloudTrail

CloudTrail records API activity in your account. Each event shows the caller identity, the time, the source IP address, the action and the parameters. The last 90 days of management events are visible in Event history without any setup.

For longer retention, create a trail that delivers events to an S3 bucket, and optionally to CloudWatch Logs. A trail can cover all Regions and, with AWS Organizations, all accounts in the organization.

Data events matter for data platforms because they show who read which object. Enable them for sensitive buckets and tables, and filter with event selectors to control cost.

aws cloudtrail put-event-selectors --trail-name org-trail \
  --advanced-event-selectors '[{"Name":"S3 data events for lake","FieldSelectors":[
    {"Field":"eventCategory","Equals":["Data"]},
    {"Field":"resources.type","Equals":["AWS::S3::Object"]},
    {"Field":"resources.ARN","StartsWith":["arn:aws:s3:::my-lake/"]}]}]'

Integrity and protection

Turn on log file integrity validation, which lets you verify with digest files that log files were not changed or deleted. Store logs in a separate, locked down account or bucket, encrypt them with KMS, and use S3 Object Lock for write once retention. Limit who can delete or edit trails.

CloudTrail Lake

CloudTrail Lake stores events in an event data store and lets you run SQL queries on them directly, with a retention period that you choose. It suits teams that want searchable audit data without building their own pipeline.

CloudWatch Logs

Services such as Lambda, Glue, EMR, MWAA, Redshift and API Gateway send logs to CloudWatch Logs. Set retention for each log group to match policy.

Logs Insights queries recent data interactively. Subscription filters stream logs to Kinesis Data Streams, Firehose or Lambda in near real time, for example to ship them to S3 or OpenSearch.

Service specific audit logs

Redshift's user activity log needs the parameter enable_user_activity_logging to be on in the cluster parameter group, and it also needs audit logging switched on. Without it, the SQL text of queries is not recorded in the audit logs.

Analyzing logs at scale

For large volumes, keep logs in S3 as compressed, partitioned files and query them with Athena. Create the table once with partition projection, and the date partitions stay current.

For quick searches use CloudWatch Logs Insights. For dashboards and search, send logs to OpenSearch Service. For real time alerts on a pattern, use a metric filter and an alarm, or an EventBridge rule on CloudTrail events.

Retention and compliance

Decide retention from legal and business rules. Use lifecycle rules to move older logs to cheaper storage classes and expire them at the end of the retention period. Keep a consistent time format and include the account, Region and resource in each record, so investigations can follow an event across systems.

Centralized logging

A common pattern sends logs from many accounts to one log archive account. Organization trails, cross account log delivery and bucket policies make that possible. Central storage also keeps logs away from the people being audited.