Skip to content

Simple Queue Service

SQS (Simple Queue Service) is the oldest AWS service. It was the first publically-available service. SQS is a web service that gives you acces to a message queue that can be used to store messages while waiting for a computer to process them. SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that are awaiting processing.

Meme Website Example

Let's say you have a meme website that overlays a meme over an image. You could utilize the following architecture:

  1. The user uploads the meme and the image.
  2. The image is stored in a bucket with meta data defining the meme.
  3. An object appearing in the S3 bucket triggers a Lambda function.
  4. The Lambda function places information about the image and its meme in an SQS queue (as a message).
  5. A fleet of EC2 instance poll the SQS queue, see the message, and process it.
  6. After processing the message, an application sitting on one of the EC2 instances, stores the newly generated meme image in another bucket, which is now availble for download by the front-end app.

Using SQS, you can decouple the components of an application so they run independently, easing message management between components.

Any component of a distributed application can store messages in the queue. Messages can contain up to 256 KB of text in any format. Any component can later retrieve the messages programatically using the Amazon SQS API.

The queue acts as a buffer between the component producing and savind data and the component receiving the data for processing. This means the queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer are only intermittently connected to the network.

Queue Types

There are two different types of Queues with SQS: Standard Queues and FIFO Queues.

Standard Queues

Amazon SQS offers standard as the default queue type. A standard queue lets you have a nearly-unlimited number of transactions per second. Standard queues guarantee that a message is delivered at least once; however, occasionally (because of the highly-distributed architecture that allows high throughput), more than one copy of a message might be delivered out of order. Standard queues provide best-effort ordering, which ensures that messages are generally delivered in the same order as they are sent.

FIFO (First-In, First-Out) Queues

The FIFO queue complements the standard queue. The most important features of this queue type are FIFO (first-in-first-out) delivery and exactly once processing: The order in which messages are sent and received is strictly preserved and a message is delivered once and remains available until a consumer processes and deletes it; duplicates are not introduced into the queue. FIFO queues also support message groups that allow multiple ordered message groups within a single queue. FIFO queues are limited to 300 transaction per second (TPS), but have all the capabilities of standard queues.

SQS Key Facts

  • SQS is pull-based, not push-based
  • Messages are 256 KB in size
  • Messages can be kept in the queue from 1 minute to 14 days
  • Default retention period is 4 days
  • SQS guarantees that your messages will be processed at least once.
  • The visibility Timeout is the amount of time that the message is invisible in the SQS queue after a reader picks up that message. Provided the job is processed before visibility time out expires, the message will then be deleted from the queue. If the job is not processed within that time, the message will become visible again and another reader will process it. This could result in the same message being delivered twice.
    • Default Visibility Timeout is 30 seconds.
    • Increase Visibility Timeout, if your task takes > 30 seconds
    • Maximum Visibility Timeout is 12 hours

SQS Long Polling

  • SQS Long Polling is a way to retrieve messages from your SQS queues
  • While the regular short polling returns immediately (even if the message queue being polled is empty), long-polling doesn't return a response until a message arrives in the message queue, or the long poll times out.
  • As such, long-polling can save you money.

SQS Exam Tips

  • SQS is a distributed message queueing system
  • Allows you to decouple the components of an application so that they are independent.
  • Pull-based, not push based
  • Standard Queues (default) - best-effort ordering; message delivered at least once
  • FIFO Queues (First-In-First-Out) - Ordering strictly preserved, message delivered once, no duplicates. E.g. good for banking transactions, which need to happen in strict order.
  • Visibility Timeout
    • Default is 30 seconds; increase if your task takes > 30 seconds to complete
    • Max 12 hours
    • The ChangeMessageVisibility API call can be used to extend the length of time to process jobs.
  • Short Polling - returned immediately, even if no messages are in the queue.
  • Long Polling - Polls the queue periodically and only returns a response when a message is in the queue or the timeout is reached. The max long poll timeout is 20 seconds
  • Messages can be at most 256 KB in size