AWS SQS, SNS, SES overview

Introduction

In this post, we will look at three different AWS services. Two of which fall under the application engagement category. Those being SQS and SNS and then SES, which sits within the customer engagement category.

  • SQS: The simple queue service relates to the queuing of messages.
  • SNS: The simple notification service relates to alerts and notification of events, and
  • SES: The simple email service provides a means of sending SMTP messages.

SQS

Simple Queue Service. With the continuing growth of micro-services and the cloud-based practice of designing decoupled systems, it’s imperative that developers have the ability to utilize a service, or system, that handles the delivery of messages between components. And this is where SQS comes in.

SQS is a fully managed service offered by AWS that works seamlessly with serverless systems, microservices, and any distributed architecture. Although it’s simply a queueing service for messages between components, it does much more than that. It has the capability of sending, storing, and receiving these messages at scale without dropping message data, as well as utilizing different queue types depending on requirements. And it includes additional features, such as dead-letter queues. It is also possible to configure the service using the AWS Management Console, the AWS CLI, or using the AWS SDKs.

SQS Components

Let me focus on some of the components to allow to understand how the service is put together. The service itself uses three different elements. Two of which are a part of your distributed system, these being the producers and the consumers, and the third part is the actual queue, which is managed by SQS and is managed across a number of SQS servers for resiliency.

Let me explain how these components work together. The producer component of your architecture is responsible for sending messages to your queue. At this point, the SQS service stores the message across a number of SQS servers for resiliency within the specified queue. This ensures that the message remains in the queue should a failure occur with one of the SQS servers. Consumers are responsible for processing the messages within your queue. As a result, when the consumer element of your architecture is ready to process the message from the queue, the message is retrieved and is then marked as being processed by activating the visibility timeout on the message.

Visibility Timeout

This timeout ensures that the same message will not be read and processed by another consumer. When the message has been processed, the consumer then deletes the message from the queue. Before moving on, I just want to point out a little more relating to the visibility timeout. As I said, when a message is retrieved by a consumer, the visibility timeout is started. The default time is 30 seconds, but it can be set up to as long as 12 hours. During this period, the consumer processes the message. If it fails to process a message, perhaps due to a communication error, the consumer will not send a delete message request back to SQS. As a result, if the visibility timeout expires and it doesn’t receive the request to delete the message, the message will become available again in the queue for other consumers to process. This message will then appear as a new message to the queue.

The value of your visibility timeout should be longer than it takes for your consumers to process your messages.

Types of queues

I mentioned earlier that there were different types of queues.

Standard Queues

Standard queues, which are the default queue type upon configuration, support at-least-once delivery of messages. This means that the message might actually be delivered to the queue more than once, which is largely down to the highly distributive volume of SQS servers, which would make the message appear out of its original order or delivery. As a result, the standard queue will only offer a best-effort on trying to preserve the message ordering from when the message are sent by the producers. Standard queues also offer an almost unlimited number of transactions per second, TPS, making this queue highly scalable.

FIFO Queues

If message ordering is critical to your solution, then standard queues might not be the right choice for you. Instead, you would need to use first-in, first-out queues. This queue is able to ensure the order of messages is maintained, and that there are no duplication of messages within the queue.

Unlike standard queues, FIFO queues do have a limited number of transactions per second. These are defaulted to 300 per second for all send and receive and delete operations. If you use batching with SQS, then this changes to 3,000. Batching essentially allows you to perform actions against 10 messages at once within a single action.

Standard vs FIFO

So, the key takeaways between the two queues are for standard queues, you have unlimited throughput, at-least-once delivery, and best-effort ordering. And for first-in, first-out queues, you have high throughput, first-in, first-out delivery, and exactly-once processing. For both queues, it is also possible to enable encryption using server-side encryption via KMS.

Dead Letter

A dead-letter queue differs to the standard and FIFA queues as this dead-letter queue is not used as a source queue to hold messages submitted by producers. Instead, the dead-letter queue is used by the source queue to send messages that fail processing for one reason or another. This could be the result of cloud enabling your application, corruption within the message, or simply missing information within a database that no message data relates to.

If the message can’t be processed by a consumer after a maximum number of tries specified, the queue will send the message to a dead-letter queue. This allows engineers to assess why the message failed, to identify where the issue is, to help prevent further messages from falling into the dead-letter queue. By viewing and analyzing the content of these messages, it might be possible to identify the problem and ascertain if the issue exists from the producer or consumer perspective. A couple of points to make with a dead-letter queue is that is must be configured as the same queue type as the source is used against. For example, if the source queue is a standard queue, the dead-letter queue must also be a standard queue type. And similarly, for FIFA queues, the dead-letter queue must also be configured as a FIFA queue.

SNS

Simple Notification Service (SNS) is used as a publish and subscribe messaging service. But what does this mean? SNS is centered around topics. And you can think of a topic as a group for collecting messages. Users over endpoints can then subscribe to this topic who will then receive messages or events that are published to that particular topic.

When a message is published, all subscribers to that topic receive a notification of that message. This helps to implement event driven architectures within a decoupled environment. Again, much like SQS, SNS is a managed service and highly scalable, allowing you to distribute messages automatically to all subscribers across your environment, including mobile devices.

It can be configured with the AWS management console, the CLI, or the AWS SDK. As mentioned, SNS uses a concept of publishers and subscribers, which can also be classed as consumers and producers, and works in the same principle as SQS from this perspective. The producers, or publishers, send messages to a topic, which is used as a central communication control point. Consumers, or subscribers of the topic, are then notified of this message by one of the following methods: HTTP, HTTPS, Email, Email-JSON, Amazon SQS, Application, AWS Lambda, or SMS. Subscribers don’t just have to be users. For example, it could be a web server and they can be notified of the message via the HTTP protocol.

Or if it was a user, you could use the email notification method and enter their email address. SNS offers methods of controlling specific access to your topics through a topic policy. For example, you might want to restrict which protocol subscribers can use, such as SMS or HTTPS, or only allow access to this topic for a specific user. The policy themselves follow the same format as IAM policies.

SES

The SES service makes it possible to use AWS infrastructure and email service to handle an automated email system to communicate with your customers. This makes it a good choice for marketers and developers. For example, you could use SES to send a confirmation email to customers, notifying them of their new account details that’s setup on your website or an email confirmation sent to a customer detailing their online order placed by your site.

When receiving email, you can architect your applications to respond automatically to incoming emails such as request to unsubscribe to your newsletter or develop applications to receive incoming email from customers and automatically create tickets that can be assigned to your team to resolve their issues.

As expected, it’s a very reliable and cost-effective service that ensures you’re able to maintain a good communication channel to your customers and third parties. This service is easily able to integrate into existing systems such as your SMTP interfaces or existing applications using the AWS SDKs. So, let’s answer the two most obvious questions you may have, how do I send email and how do I receive email? If you are new to SES, then you will initially be placed within a sandbox environment which allows you to set up a single email address to test and understand the service in greater detail. To show you how to do this, I will demonstrate how to set up a single email. When you are ready to move onto a larger and wider scale distribution of emails, you can request to be moved out of the sandbox environment by opening a ticket within the AWS Support Center to increase the limits for SES sending.

FAQs

Tags:

Updated: