welcome back to back space Academy in this lecture on simple queue service or sqs 2
for short we'll go through and explain exactly what the sqs service is it's a 3
service for the queueing of messages and how we can use that to the couple our 4
processes we'll look at the queue types and we'll also look at the life cycle of 5
a message as it passes through the sqs process we'll look at dead letter queues 6
what we what do we do with messages that cannot be processed but look at delay 7
queues and message timers and we'll also look at the process of long polling sqs 8
it is a queue for storing messages and it acts as a buffer of data for 9
processing service sqs can smooth out the peak demand up to 10 attributes or 10
name and values can be added to a message additional to the message body 11
and the message size can be set from 1 k to 256 K so just looking at the diagram 12
there we have messages coming in so those messages will have peaks and 13
troughs of demand there will be more coming in at one stage and then there 14
will be gaps where there won't be so much so between those peaks and troughs 15
the instance needs to cope with that and we don't want the instance to be taken 16
out by too much of a demand so we allow those messages to build up in our sqs 17
queue and our instance will process those messages one by one and it will 18
every time it finishes a message it will go back and grab another one by polling 19
the queue and then it just keeps going until the queue is is completed it's 20
curious it allows us to decouple our processes so it allows us to cope with 21
those peaks and troughs but one thing we need to understand is that if the 22
average demand exceeds our processing capacity so over time is those peaks and 23
troughs still average out to be more than what our instance can cope with 24
then that queue will grow indefinitely and it will continue to grow until such 25
point in time that the sqs service can't handle it anymore so sqs it can provide 26
cloud watch metrics that can be used with auto scaling and that's a very 27
important thing because that will allow us to introduce elasticity that is 28
linked to the size of our sqs queue and so that can also allow us to have upper and 29
lower set points so that when our queue gets hit by a large amount of demand and 30
it grows to a significant size that will trigger a cloud watch metric that will 31
then alert our auto scaling group to add more instances and with our lower set 32
point when that is reached it will do the same thing but in Reverse so in the 33
auto scaling group will start to terminate entrances and reduce our 34
capacity to save costs there are two different types of queue types there 35
used to be in the past only the standard queue but now we have a FIFO or first in 36
that first out queue so standard queue is the default queue type and it 37
provides nearly unlimited number of transactions per second so you need 38
speed the standard queue is a way to go it guarantees that a message is 39
delivered at least once although you'd need to be aware that although really 40
duplicates can occur and so your application needs to be aware of that 41
and have checks and balances in place in case a messages processed twice and it 42
does best-effort ordering so although most of the time the messages will come 43
in in the order that they're received it's not always the case so then we look 44
at first in first out queue type and that complements the standard type and 45
it's limited to 300 transactions per second so it is slower than a standard 46
queue type but it provides first-in first-out 47
delivery of our messages and it provides exactly once processing one thing to be 48
aware of its are currently not available in all regions so you need to check your 49
region to make sure that FIFO queues are available 50
it's very important to understand the lifecycle of a message as it passes 51
through the sqs service so that our application can work properly for the 52
first part of the message life cycle is that the message is received by the sqs 53
queue then the message will be received by the processing server when the 54
message is received by the processing server the visibility timeout period 55
starts so at that point it's not visible to the message is not visible to other 56
instances that may be polling that sqs queue so once the message is processed 57
by the processing server the processing server sends a message delete request 58
back to the sqs service at that point the visibility timeout period ends so 59
the visibility timeout specifies the time which some message is invisible in 60
the queue and has not been deleted and then once it has been processed and 61
deleted then the visibility timeout period ends so if a message has not been 62
deleted by the visibility timeout period that the message then becomes visible 63
and can be received again by a processing server so that's another 64
thing that you need to take in consideration with your application 65
making sure that you can process messages within that visibility timeout 66
period dead-letter queues can be used as a target for messages that cannot be 67
processed successfully so that allows us to remove these messages from our queue 68
and at the same time not lose those messages and allows them to be analyzed 69
or reprocessed at a later time you need to remember the dead-letter queues must 70
be in the same region and from the same account as the original cube 71
delay queues and message timers allow for timing changes to be made to a 72
messages lifecycle so delay queues they postpone the delivery of new messages 73
and the message is not visible when it is first added to the queue for a 74
defined period of time so that thought defined period of time is set with 75
delay seconds parameter with the create queue or set queue attributes 76
command for the existing queue you need to remember that there is a 120,000 77
limit for the number of in-flight messages per queue so make sure that if 78
you implement a delay queue that that limit is not going to be exceeded we 79
also have message timers and they implement an initial invisibility 80
invisibility period for an individual message and they can be created using 81
the console or with the delay seconds parameter of the send message command we 82
need to understand the different types of polling so when our instance has 83
completed its process and it is ready to receive another message for processing 84
it will pull the sqs service and there are two ways for an instance to poll the 85
sqs service for new messages and we have short polling and long polling so short 86
polling is very simple it just sends the SQL Server simply sends a response back 87
immediately whether or not a message is in the queue so that obviously will 88
require the instance to regularly keep polling the sqs service until such point 89
in time as a message is available the other option we have available is long 90
polling and that will return a response to the instance and it will wait until a 91
message is available before it does that so we'll wait until a message is 92
available in the queue before sending a response so that significantly reduces 93
the workload on our instant cell instance is no longer sending all of 94
these messages or these requests to the sqs service it will send one request and 95
then on an hour synchronous basis it will have that sent back when there is a 96
message available so it reduces the number of empty responses when there are 97
no messages available to return so we set the wait time seconds parameter in 98
the received message call or they receive message wait time seconds in the 99
create queue or set attribute score and that is a parameter between 1 and 20 100
seconds so that brings us to the end of our lecture on sqs I'll see you in the 101
next lecture