More Storage and Transfer Services

In the previous parts we covered S3 in detail. But there are cases it doesn't solve: a Windows application needing a shared folder, servers in your office wanting to reach the cloud, a partner sending files over SFTP, or huge data that has to move.

Here's this part's map:

Amazon FSx: ready-made file systems #

We've seen EFS for a shared Linux file system. FSx runs well-known file systems for you as a managed service, so you don't have to manage them yourself:

FSx for Lustre has two options: Scratch is cheap and fast for temporary data, but the data has no copies, so a failed disk loses it. Persistent is for long-term data, with copies inside the Availability Zone.

Storage Gateway: your office and the cloud #

Have servers in your office and want to benefit from the cloud without changing your applications? Storage Gateway is software you run on-site as a virtual machine. It presents ordinary storage to your applications, while the data actually lives in AWS, with a local cache for recent files.

There's a fourth type called FSx File Gateway that's no longer available to new customers, and AWS advises connecting to FSx directly.

Moving data #

DataSync #

DataSync moves and syncs large data on a schedule:

• From where to where: from NFS, SMB or HDFS servers or object storage, to S3, EFS or FSx, and between AWS services themselves.
• Agent: you run an agent on-site to move from your office, and don't need one between AWS services.
• It moves only what changed on each run, preserves permissions and dates, and verifies data integrity.

Transfer Family #

Have partners or older systems sending files over SFTP, FTPS, FTP or AS2, and don't want to run a server for them? Transfer Family gives you a managed receiving endpoint, and the files go to S3 or EFS. Users come from within the service, Active Directory or your own identity system.

Note that FTP is unencrypted, so it's available only inside a VPC. You pay by the hour for each endpoint, even without use.

When your data outgrows the internet #

Before choosing how to move data, do the math. These are theoretical times at the line's full speed, and reality is slower:

You have these options for large amounts:

• DataSync if your line can take it, with ongoing sync.
• Direct Connect if the transfer is large and continuous, as explained in part fourteen.
• Data Transfer Terminal: physical sites you visit with your own storage devices, connecting them to AWS's network at high speed.

Comparing storage services #

Hands-on: syncing with DataSync #

We'll move files between two buckets, which needs no agent. From CloudShell, prepare two buckets and some files:

SRC=series-src-$RANDOM$RANDOM
DST=series-dst-$RANDOM$RANDOM

aws s3 mb s3://$SRC
aws s3 mb s3://$DST

for i in 1 2 3; do
  echo "file $i" > file$i.txt
  aws s3 cp file$i.txt s3://$SRC/
done

echo "source: $SRC  destination: $DST"

Open DataSync in the console and choose Create task:

1. Source: create a new location of type S3, select the series-src-... bucket, and choose to create an IAM role automatically.
2. Destination: the same with the series-dst-... bucket.
3. Settings: name the task series-sync, choose Enhanced if a mode option appears, and leave the rest as it is.
4. Run: create the task, then choose Start.

When it finishes, check the destination:

aws s3 ls s3://$DST/

Now add one file and run the task again:

echo "file 4" > file4.txt
aws s3 cp file4.txt s3://$SRC/

On the new run's page you'll find one file transferred, because DataSync moves only what changed. That's what sets it apart from a full copy every time.

aws s3 rb s3://$SRC --force
aws s3 rb s3://$DST --force

Summary #

• FSx offers ready-made file systems: Windows, Lustre, ONTAP and OpenZFS.
• Storage Gateway lets your office see the cloud as ordinary storage.
• DataSync is for scheduled sync, and Transfer Family for receiving over SFTP.
• Work out the transfer time before deciding, and note that Snowball is closed to new customers.

Test yourself #

Before you move on, try these questions to check you've got this part. Each one is a practical scenario, and the explanation appears once you answer.

That completes the storage section. You now have a strong application with a database, a network and storage, but its pieces are all wired directly together. In part nineteen we start the modern applications section, separating them with SQS, SNS and Kinesis.