S3: Storage Classes and Lifecycle

In part fifteen we stored everything the default way. But think about your application: a product photo is requested every second, while a five-year-old invoice is opened only during an audit. Why pay the same for both?

Storage classes give you that choice: the less a file is accessed, the less it costs to store, and the more it costs or the longer it takes to retrieve.

Durability and availability #

Two terms you need to tell apart:

• Durability: will the file be lost? 99.999999999% for all classes, noting that classes in a single Availability Zone lose their data if that zone is destroyed.
• Availability: can you reach it right now? This varies by class, such as 99.99% for Standard.

The classes #

Prices are in dollars per gigabyte per month, approximate for US Regions, so check the pricing page for yours.

All these classes keep data in three or more Availability Zones, except One Zone-IA. There's also S3 Express One Zone: very high performance with single-digit millisecond latency, in one Availability Zone, using a special type of bucket, at a higher price. It suits analytics and model training that need great speed.

Intelligent-Tiering: when you don't know #

If you can't tell how often a file will be requested, this class moves it by itself according to use:

You pay a small monitoring fee per object with no retrieval fees, and if a file is requested it returns to the first tier.

Glacier Flexible and Deep Archive: request before reading #

In these classes you can't read a file directly. You must first request its restore, choosing a speed:

Lifecycle #

You don't need to move files by hand. Lifecycle rules do it automatically according to a file's age. Take an application's logs, for example:

A rule has two kinds of action, and you apply it to a specific prefix or tag:

A few things to watch for:

• Moves only go down the stairs, and the minimum before Standard-IA is 30 days.
• Files smaller than 128 KB don't move by default, because moving them costs more than it saves.
• The move itself has a fee per thousand objects, so watch out for millions of small files.

How do you know what to choose? #

You don't need to guess. You have tools that read your actual use:

• Storage Class Analysis: watches access to a prefix or bucket, and suggests when to move from Standard to Standard-IA.
• Storage Lens: a dashboard across all your accounts: sizes, growth, and buckets with incomplete uploads or no encryption. It has free metrics and paid advanced ones.

If you have millions of files and want to do something to all of them at once, such as changing class or copying, S3 Batch Operations runs it on a list from S3 Inventory, with a report of the results.

On the other side, if you share huge data with another party, Requester Pays has them pay the request and download costs instead of you.

And if you upload millions of small files you won't read one by one, such as logs, bundle them into one archive first. Every PUT is a paid request, so one large upload can cost far less than a million small ones.

Hands-on: classes and a lifecycle rule #

We'll carry on with the bucket from the previous part. If your CloudShell session ended, set its name in BUCKET again. Upload a file to a specific class, and check the class:

echo "monthly report" > report.txt

aws s3 cp report.txt s3://$BUCKET/reports/report.txt --storage-class STANDARD_IA

aws s3api head-object --bucket $BUCKET --key reports/report.txt \
  --query StorageClass

Now add the lifecycle rule from the diagram to the logs/ prefix:

aws s3api put-bucket-lifecycle-configuration --bucket $BUCKET \
  --lifecycle-configuration '{
  "Rules": [{
    "ID": "logs-rule",
    "Status": "Enabled",
    "Filter": {"Prefix": "logs/"},
    "Transitions": [
      {"Days": 30, "StorageClass": "STANDARD_IA"},
      {"Days": 90, "StorageClass": "GLACIER"},
      {"Days": 365, "StorageClass": "DEEP_ARCHIVE"}
    ],
    "Expiration": {"Days": 2555},
    "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}
  }]
}'

aws s3api get-bucket-lifecycle-configuration --bucket $BUCKET \
  --query "Rules[0].Transitions" --output table

Here's the fun part: upload a file to Glacier and try to read it:

echo "old invoice" > old.txt
aws s3 cp old.txt s3://$BUCKET/archive/old.txt --storage-class GLACIER

aws s3 cp s3://$BUCKET/archive/old.txt -

It returns an InvalidObjectState error, because the file is stored but can't be read directly. Request a restore:

aws s3api restore-object --bucket $BUCKET --key archive/old.txt \
  --restore-request '{"Days": 1, "GlacierJobParameters": {"Tier": "Standard"}}'

aws s3api head-object --bucket $BUCKET --key archive/old.txt \
  --query "[StorageClass,Restore]"

You'll see ongoing-request="true". After 3 to 5 hours it becomes false, and a temporary copy is available to read for one day.

Summary #

• The less the access, the lower the storage price, and the higher the cost or time of retrieval.
• If you don't know the access pattern, Intelligent-Tiering moves files by itself.
• Watch out for the minimum storage period, the minimum billable size and retrieval fees.
• Lifecycle rules move and delete files automatically, and always add a rule for incomplete uploads.
• Storage Lens and Storage Class Analysis let you decide with numbers, not guesses.

Test yourself #

Before you move on, try these questions to check you've got this part. Each one is a practical scenario, and the explanation appears once you answer.

Your data is now in the right place at the right price, but who can reach it? In part seventeen we cover encryption, signed links and locks against deletion.