Connecting Services: SQS, SNS and Kinesis

So far the pieces of our application call each other directly. That's fine until a sale starts and the invoice service can't keep up, and suddenly the customer is staring at a spinner because of a service they've never heard of.

This section is about modern applications, and it starts with the idea behind most of them: separate the services, so that each one works at its own pace and a failure in one doesn't bring down the others.

Here's a map of this part:

Why separate services? #

A direct call means both sides must be up, and fast, at the same moment. With a queue in between you get three things:

• Failures stay contained. If the invoice service stops, messages wait in the queue until it comes back.
• Peaks are absorbed. A thousand orders in a minute don't reach the database all at once.
• Each side scales alone. You add invoice workers without touching the store.

SQS: the queue #

SQS (Simple Queue Service) is a fully managed queue with no servers to run. A producer sends messages, they wait in the queue, and a consumer pulls them, works on them, then deletes them.

Note the direction: consumers ask the queue for messages. The queue doesn't push them to anyone.

The last two points affect your code. Since a message can arrive twice, make processing it twice harmless. An order number you check before charging is enough.

For anything larger than a message, store the file in S3 and send its path in the message.

Visibility timeout #

What stops two consumers taking the same message? When one receives it, SQS hides it from the others for a period, called the visibility timeout. The consumer must delete it before that period ends.

This is the mechanism that keeps you from losing work: if the server crashes halfway through, the message comes back by itself.

Choose the period to suit your processing time. It's 30 seconds by default, and it can go up to 12 hours:

• Too short: the message appears again while it's still being worked on, and it gets processed twice.
• Too long: if a consumer fails, the message waits a long time before anyone retries it.

If a job takes longer than usual, the consumer can extend the period for that message with ChangeMessageVisibility.

Long polling #

By default, asking an empty queue returns an empty response right away, so your consumers fill your bill with pointless requests. With long polling, the request waits up to 20 seconds for a message. That means fewer requests, a lower cost and faster delivery, so always turn it on.

Dead-letter queues #

What about a message that always fails, because of a bug or bad data? It will come back forever. Attach a dead-letter queue (DLQ): after a number of failed attempts you choose, the message moves there, so you can inspect it without it blocking anything. Then you can redrive it to the original queue once you've fixed the problem.

Keep the retention in the dead-letter queue longer than the original, so messages don't expire before you see them.

FIFO queues #

When order matters, as with transactions on one account, use a FIFO queue. Its name must end in .fifo.

The order is per message group, so with one group for each customer, different customers' messages can be processed in parallel. If you need more throughput, high throughput mode raises the limit by far, and the figure varies by Region.

Scaling with the queue #

A classic pattern: an Auto Scaling group of workers grows and shrinks according to the queue's length. The ApproximateNumberOfMessagesVisible metric tells you how many messages are waiting, so a growing queue adds workers.

It also protects your database. Instead of a spike of orders hitting RDS directly, they go into a queue, and workers write them at a pace the database can handle. This only suits work that doesn't need an immediate answer, so the customer is told "we received your order", not "your order is complete".

Security #

Messages are encrypted in transit over HTTPS, and at rest with SSE-SQS by default, or with your own KMS key. Who can reach the queue is decided by IAM policies, plus an access policy on the queue itself, which you need for other accounts or for services like SNS and S3 to send to it. We'll write one in the hands-on.

SNS: one message to many #

A queue suits a job that one party does. But when a customer places an order, the shipping, invoicing and analytics services all want to know. You don't want your application to know about each of them, and to change every time you add one.

SNS (Simple Notification Service) works by publishing and subscribing: your application publishes one message to a topic, and SNS delivers it to every subscriber, right away. A standard topic can have up to 12.5 million subscriptions.

Unlike a queue, SNS doesn't store messages. If a subscriber is down it retries for a while, but it's not somewhere messages wait.

Message size is 256 KiB by default, and you can raise it on a topic up to 1 MiB, though this limits the topic to 100 subscriptions and to SQS, Lambda and Firehose subscribers only.

Fan-out: SNS with SQS #

The most important pattern here combines the two services: SNS spreads the message, and a queue for each service keeps it until that service is ready.

Why this combination?

• Nothing is lost: if the shipping service stops, its messages wait in its queue.
• Each service has its own pace, retries and dead-letter queue.
• Adding a service is easy: a new queue subscribes to the topic, with no change to the publisher.

The same goes for S3 events: S3 doesn't accept two rules with overlapping prefixes for the same event type. So to reach several destinations, send the event to one SNS topic and fan it out from there.

Filtering messages #

By default every subscriber receives everything. A filter policy on the subscription lets the shipping queue receive only what concerns it:

{ "type": ["physical"] }

Messages with type set to digital won't reach it, and you don't pay to process them.

FIFO topics #

For order across a fan-out, SNS has FIFO topics, whose names also end in .fifo, with order and duplicate removal like FIFO queues. These are limited to 100 subscriptions per topic, and they suit cases like sending the same sequence of updates to several queues.

Kinesis Data Streams: a stream that stays #

Imagine a million sensors sending readings every second, and three parties who want to read them: one for instant alerts, one for dashboards, and one for archiving. A queue doesn't suit this, because a message is gone once it's processed.

Kinesis Data Streams is a stream: records stay in it for the retention period, whether they've been read or not, and each consumer reads from its own position, so it can go back and replay them.

• Shards: the stream is split into shards. With provisioned capacity, each one takes 1 MB per second of writes and 2 MB per second of reads.
• Partition key: decides which shard a record goes to. Records with the same key stay in order, so a sensor's number works well as a key.
• Retention: 24 hours by default, up to 365 days.
• Record size: 1 MiB by default, and you can raise it up to 10 MiB per stream.

Capacity modes #

The ceiling for on-demand depends on the Region: up to 10 GB per second of writes in the largest Regions, and 200 MB per second in others.

Video has its own service: Kinesis Video Streams takes live video from devices such as cameras, stores it, and lets you watch it or process it, live or later.

Data Firehose: to storage with no code #

Very often all you want is for the stream to land somewhere. Amazon Data Firehose (formerly Kinesis Data Firehose) takes data, collects it, and delivers it with no code or servers:

• Sources: your application directly, a Kinesis stream, or others like SNS and CloudWatch Logs.
• Processing: optionally transforms records with a Lambda function, and can convert them to Parquet.
• Destinations: S3, Redshift, OpenSearch, Snowflake, Apache Iceberg tables, or an HTTP endpoint.
• Buffering: it collects data until it reaches a size or a period, such as 5 MiB or 5 minutes for S3, and whichever comes first triggers delivery. So it suits near-real-time loading, not an instant reaction.

Which one do I choose? #

A quick rule: a job? SQS. An event many want to know about? SNS. A stream of data many read and replay? Kinesis.

There's also EventBridge, which routes events by rules from AWS services and other applications, and we'll cover it in part twenty-five.

Amazon MQ: for what already exists #

SQS and SNS are AWS's own services with their own APIs. If you're moving an application built on open protocols such as AMQP, MQTT or STOMP, you won't want to rewrite it. Amazon MQ runs a familiar broker as a managed service, in two engines:

Note that it runs on brokers you choose the size of, so it doesn't scale without limit like SQS and SNS. For a new application, start with those two.

Hands-on: a fan-out with a filter #

We'll build the design from the diagram with two queues. From CloudShell, create the topic and the queues:

TOPIC_ARN=$(aws sns create-topic --name orders \
  --query TopicArn --output text)

SHIP_URL=$(aws sqs create-queue --queue-name shipping \
  --query QueueUrl --output text)
BILL_URL=$(aws sqs create-queue --queue-name billing \
  --query QueueUrl --output text)

SHIP_ARN=$(aws sqs get-queue-attributes --queue-url $SHIP_URL \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)
BILL_ARN=$(aws sqs get-queue-attributes --queue-url $BILL_URL \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)

Queues are private, so allow only this topic to send to them:

allow_topic() {
  POLICY=$(jq -nc --arg q "$2" --arg t "$TOPIC_ARN" '{
    Version: "2012-10-17",
    Statement: [{
      Effect: "Allow",
      Principal: {Service: "sns.amazonaws.com"},
      Action: "sqs:SendMessage",
      Resource: $q,
      Condition: {ArnEquals: {"aws:SourceArn": $t}}
    }]
  }')
  aws sqs set-queue-attributes --queue-url "$1" \
    --attributes "$(jq -nc --arg p "$POLICY" '{Policy: $p}')"
}

allow_topic $SHIP_URL $SHIP_ARN
allow_topic $BILL_URL $BILL_ARN

Subscribe the queues to the topic: billing gets everything, and shipping gets physical orders only:

aws sns subscribe --topic-arn $TOPIC_ARN \
  --protocol sqs --notification-endpoint $BILL_ARN \
  --attributes RawMessageDelivery=true

aws sns subscribe --topic-arn $TOPIC_ARN \
  --protocol sqs --notification-endpoint $SHIP_ARN \
  --attributes '{"RawMessageDelivery":"true","FilterPolicy":"{\"type\":[\"physical\"]}"}'

Publish two orders, one physical and one digital:

aws sns publish --topic-arn $TOPIC_ARN --message "order 1: a book" \
  --message-attributes '{"type":{"DataType":"String","StringValue":"physical"}}'

aws sns publish --topic-arn $TOPIC_ARN --message "order 2: an e-book" \
  --message-attributes '{"type":{"DataType":"String","StringValue":"digital"}}'

Give it a few seconds, then count the messages in each queue:

for url in $BILL_URL $SHIP_URL; do
  aws sqs get-queue-attributes --queue-url $url \
    --attribute-names ApproximateNumberOfMessages \
    --query Attributes.ApproximateNumberOfMessages --output text
done

You'll see 2 for billing and 1 for shipping, so the filter worked. Now try the visibility timeout. Receive the shipping message, then try again straight away:

aws sqs receive-message --queue-url $SHIP_URL --wait-time-seconds 5 \
  --query "Messages[0].Body" --output text

aws sqs receive-message --queue-url $SHIP_URL --wait-time-seconds 5 \
  --query "Messages[0].Body" --output text

The first prints order 1: a book, and the second waits five seconds and prints None, because the message is hidden. We didn't delete it, so wait 30 seconds and receive it again, this time deleting it to finish the job:

HANDLE=$(aws sqs receive-message --queue-url $SHIP_URL --wait-time-seconds 20 \
  --query "Messages[0].ReceiptHandle" --output text)

aws sqs delete-message --queue-url $SHIP_URL --receipt-handle "$HANDLE"

Now the message is gone for good.

aws sns delete-topic --topic-arn $TOPIC_ARN
aws sqs delete-queue --queue-url $SHIP_URL
aws sqs delete-queue --queue-url $BILL_URL

Summary #

• Separating services contains failures, absorbs peaks and lets each side scale alone.
• SQS is a queue: a message goes to one consumer, who must delete it before the visibility timeout ends.
• Always add long polling and a dead-letter queue, and make processing safe to repeat.
• SNS delivers one message to many, and fan-out with SQS guarantees none of them miss it.
• Kinesis Data Streams keeps data for several readers to replay, and Firehose delivers it to storage without code.

Test yourself #

Before you move on, try these questions to check you've got this part. Each one is a practical scenario, and the explanation appears once you answer.

The services are now separated, but where does the code itself run? So far our answer has been an EC2 server, with all the setup that entails. In part twenty we package applications in containers and run them with ECS, Fargate and EKS.