Data Lifecycle Management

What it is

Data has a lifecycle. It is hot when new and often read, then cools as it ages, and finally it must be archived or deleted to meet cost goals and legal rules.

Lifecycle management moves data between storage tiers and removes it at the right time, mostly through automatic rules instead of manual work.

How it works

S3 storage classes

Lower cost classes charge more to retrieve data, and some have a minimum storage duration and a minimum billable object size. Archive classes are poor choices for data that queries read often.

Lifecycle rules

An S3 lifecycle configuration holds rules that apply to a whole bucket, a prefix or objects with a given tag. A rule can transition objects to another class after a number of days, expire (delete) them, expire noncurrent versions and clean up incomplete multipart uploads.

{
  "Rules": [{
    "ID": "tier-then-expire",
    "Status": "Enabled",
    "Filter": { "Prefix": "raw/" },
    "Transitions": [
      { "Days": 30, "StorageClass": "STANDARD_IA" },
      { "Days": 90, "StorageClass": "GLACIER" }
    ],
    "Expiration": { "Days": 2555 }
  }]
}

Apply it with aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://lifecycle.json using the AWS CLI. Transitions generally move data to colder classes, not back.

Versioning, replication and protection

Versioning keeps every version of an object, so an overwrite or delete can be undone. Lifecycle rules should expire noncurrent versions or storage grows without limit. MFA delete adds a second factor to permanent deletes.

S3 Object Lock provides write once read many (WORM) protection with retention periods and legal holds, which suits regulatory retention. S3 replication copies objects to another bucket or Region for resilience or compliance, and it needs versioning on both buckets.

Expiring data in other stores

A common warehouse design keeps hot data in Redshift and moves history to S3 in Parquet. Queries can still join the two through Spectrum.

Backup and resilience

AWS Backup centralizes backup plans for Redshift, DynamoDB, RDS, EFS and other services, with schedules, retention and cross Region copies. Redshift takes automated snapshots and supports manual snapshots, and DynamoDB offers on demand backups and point in time recovery.

Use cross Region copies when the requirement mentions disaster recovery. Pick the recovery point objective (RPO) and recovery time objective (RTO) first, since they decide the service and the cost.

Deletion, retention and compliance

A retention policy states how long to keep each kind of data, and a deletion process enforces it. Deleting data that is no longer allowed to exist, for example after a privacy request, may need table formats such as Iceberg to remove rows, and a purge of backups and replicas as well.

Governance rules can pull in two directions. A legal hold can force you to keep data that a lifecycle rule would delete, and Object Lock can block lifecycle expiration until retention ends.

Cost thinking

Match the class to the access pattern. Intelligent-Tiering avoids guessing, but it adds a monitoring charge per object. Many tiny objects can cost more in lower classes because of minimum sizes, so compact them first.