(narrator) Now let's talk
about the SNS plus SQS Fan Out pattern.
The idea is that you want a message to be sent
to multiple SQS queues,
but if you send them individually
to every SQS queue, they can be problems associated with it.
For example, if your application crashes in between,
if they're delivery failures,
or if you add more SQS queues down the road.
Therefore, we want to use the Fan Out Pattern.
The idea is that you're going to push once in SNS topic
and then you're going to subscribe
as many SQS queues as you want to the SNS topic.
These queues are subscribers
and they will all receive the messages sent into SNS.
So for example, we have a buying service
that wants to send messages
into two SQS queues.
What it will do instead, it will send one message
into an SNS topic, and the queues are subscribers
of that SNS topic so that the fraud service
and the shipping service can read all messages
from their own SQS queues.
The idea with this as well is
that it's a fully decoupled model
and there is no data loss.
SQS will give you data persistence, delayed processing,
as well as reties of work.
And with this pattern, we can add more
SQS queues as a subscriber of the SNS topic over time.
For this, we need to make sure
that you SQS queue access policy, as we've seen from before
allows your SNS topic to write
to your SNS, to your SQS queue.
So again yet, another use case
for using Q Access Policies.
Next. So how do we use this pattern
for other purposes?
For example, S3 events into multiple queues.
So there is a limitation of S3 event rules is
that for a combination of event type, for example
an object is being created and a prefix, for example
images/ you can only have one S3 events rule.
But what if you want to send the same S3 event notification
to multiple SQS queues. In that case,
you would use the Fan Out Pattern.
So for example, we have S3 object created
as an event appearing in S3 buckets
and we will send this event into an SNS topic
and we will subscribe many SQS queues
to the SNS topic as a fan out pattern,
but we could also subscribe other type of application,
email, Lambda functions, et cetera, et cetera.
And then what we get from this is that the message
of the events happening
in Amazon S3 is going to go
to many different destinations
thanks to this fan out pattern.
Another architecture is
that you can send data directly
from SNS to Amazon S3 Through Kinesis Data Firehose.
So because SNS has a direct integration with KDF
then your buying service can send data
into an SNS topic,
then Kinesis Data Firehose, KDF,
will receive that information.
And then from Kinesis Data Firehose,
you can send it into your Amazon S3 bucket.
Or for of that matter,
any supported KDF Specific destination
which allows you to be really extensible
in the way you want to maybe persist your messages
from your SNS topic.
So we can apply the final pattern
to FIFO topics as well.
So Amazon SNS has a FIFO or FIFO capability
which is first in, first out
which gives ordering messages in the topic.
So the producers send the messages 1, 2, 3, 4
and the subscribers can only be an SQS FIFO queue for now
which is receiving messages 1, 2, 3, 4 in order.
So the idea is that with SNS FIFO
we get the same features as SQS FIFO.
We get Ordering Message Group ID.
We get Deduplication using a Deduplication ID
or Content Based Deduplication,
and we can have SQS FIFO queues
as subscribers of the FIFO SNS topic.
In terms of throughputs, you're limited,
you get the same throughputs as the SQS FIFO queue
because right now only SQS FIFO queues can read
from SNS FIFO topics.
So why do we need this?
Well, in case you wanna do a fan out using SQS FIFO.
So you need fan out, ordering, and deduplication.
So the bank service will send the data
into an SNS FIFO topic, and then it will fan out
to two SQS FIFO queue which can have the fraud service
and the shipping service read from FIFO queues as well.
One last feature of SNS
that can be really handy regarding the fan out pattern is
that you can do message filtering in SNS.
So what is message filtering?
Well it's JSON policy that is used
to filter messages sent
to the SNS topics as subscription.
So if a subscription doesn't have a filter policy
it will receive a every message
and that's the default behavior,
but let's take an example of what happens
when we set up a message filtering policy.
So we have a buying service,
and it sends transactions into an SNS topic.
For example, the transaction look
like do there's an order number.
There's a product, for example, the pencil
the quantity for, and the states, which is placed.
Now, we want to rate an SQS queue just for placed orders
not all the orders, but just for place orders.
So for this, we will subscribe the SQS queue
into the SNS topic and we'll apply a filter policy in JSON.
And we will specify in a policy
that we want to have state equals placed.
And so only the message is matching the policy will go
into the SQS queue.
But then we can have an SQS queue for canceled orders.
And so we can create our own filter policy
for canceled orders and have those coming
from the same SNS topic into the SQS queue.
And so the placed orders and the cancel orders
in SQS queue will not have the same messages.
We can also use the same filter policy
the canceled one to create an email subscription
for canceled orders.
We could have a filter policy for declined orders
for example, and as another SQS queue
or we can create an SQS queue
without a filter policy to have all messages
from that SNS topic.
So using all these fan out patterns
and message filtering FIFO queues, and FIFO topics,
we get lots of different possibilities
and the exam will try to test you on all of those.
So that's it for this lecture.
I hope you liked it.
And I will see you in the next lecture.