AWS SQS is a web service that provides us with a message queue on the cloud to which a sender process (not humans) can send a message and a reader process (not humans) can retrieve/process/delete the message.
So message queue is basically a temporary repository of messages on the AWS cloud.
SQS is asynchronous. Processes are decoupled (or loosely coupled), distributed, hence highly elastic and scalable.
SQS is pull based system (unlike SNS which is push based). One or more reader programs/lambdas/EC2s can pull messages. You can setup auto scaling to add/remove pulling EC2 instances based on the size the queue.
Messages can be 256 KB max. Text only. Billed in 64 KB chunks. You can ASCII encode if you need to send binary files.
Two types of queues:
No order is maintained.
Messages are guaranteed to be sent at-least once. Some rare occasions they are sent twice or more. You need to architect keeping this in mind.
Order of messages is maintained.
Messages are guaranteed to be sent only once.
Multiple ordered messages groups can be hosted in a single queue.
Max 300 transactions per sec.
Messages can be kept in the queue from duration of 1 min to 14 days max. Default is 4 days.
Visibility Timeout (VT) of a message is the amount of time it will be invisible in the queue after a reader pulls that message.
Max visibility timeout is 12 hours.
Default visibility timeout is 30 sec
After timeout expires, the message will be automatically visible again in the queue. So its the responsibility of the reader to delete the message after processing.
Also make sure message processing is finished before the VT or else you as an architect need to increase the visibility timeout.
Two types of polling possible by the reader
Short polling returns immediately. So it could be expensive if the queue is sparse.
Long polling returns only when a message is available in the queue or the long polling timeout (min 1 sec to max 20 sec) whichever is earliest.
Custom SQS Policy can be used to allow access to queues across accounts
If you want to allow Amazon SQS access based only on an AWS account ID and basic permissions (such as for SendMessage or ReceiveMessage), you don’t need to write your own policies. You can just use the Amazon SQS AddPermission action.
If you want to explicitly deny or allow access based on more specific conditions (such as the time the request comes in or the IP address of the requester), you need to write your own Amazon SQS policies and upload them to the AWS system using the Amazon SQS SetQueueAttributes action.
Single request can have 1 to 10 messages with a maximum 256KB payload
Cost: First 1 million request ares free, then $0.50 per million req.
Priority based Design – Create 2 queues, one for higher priority (Eg: Paid Customers) and other for lower priority (free users). Lets EC2 read priority queues first and process then move on to low priority queue.