‫So let's have a lecture about
‫how data is being ordered for Kinesis and SQS FIFO.
‫Because even though these technologies look similar
‫and have some similar capabilities, they're actually very,
‫very, very different.
‫So let's have a little case study.
‫Imagine you have 100 trucks on the road,
‫and each truck will have a truck ID.
‫So truck one, truck two up to truck 100
‫and they're on the road and they're going
‫to send their GPS positions very regularly into AWS.
‫So we want to consume that data in order for each truck so
‫that we can track their movement accurately,
‫we wanna know where there've been in order obviously, right?
‫So how should we send that data into Kinesis?
‫Now the answer is you should use a partition key.
‫And the value of that partition key is the truck ID.
‫So the truck one will send it for the batching key truck one
‫and then truck two will send
‫for partition key truck two et cetera, et cetera.
‫Why?
‫Because if we specify the same partition key
‫then the same key will always go to the same shard.
‫Now let's have a look at a diagram
‫to better understand this.
‫So we have our Kinesis Data Stream and it has three shards
‫one, two, and three.
‫And to simplify things, I'm not going
‫to show you 100 trucks, but five should be enough.
‫So we have five trucks, and they're on the road
‫and they're sending the data into Kinesis.
‫As I said, we choose the partition key to be truck ID.
‫So that means that my truck one,
‫when it's sending it's GPS data, it will send it to Kinesis
‫with the partition key, truck one and Kinesis will say,
‫okay, partition key truck one, I will hash it
‫I mean we'll do a computation.
‫And in this instance, it figures out that truck one
‫should go into shard number one.
‫So my data will go into shard number one.
‫Then the truck two will be sending its data as well
‫and will send a partition key of truck two.
‫And Kinesis will look at this partition key
‫and say I've hashed it and now it looks like you should go
‫into shard two.
‫Same for truck three so truck three will be on the road.
‫And it will send truck three as the partition key.
‫But this time, the Kinesis Data Stream service will hash
‫that truck three as the key and say you should go
‫to shard one and that's fine.
‫It just says it doesn't have to be shard three it just says,
‫this partition key should go to shard one.
‫Now for truck four, it will go to shard three
‫and for truck five, it will go to shard two.
‫So this is the idea now we have a repartition
‫and it's called partition hence the name partition key
‫of each truck on each shard based on the partition key.
‫And because truck one
‫keeps on sending the same partition key which is truck one,
‫then the data will always go to the same shard.
‫Hence so on the next data point for the truck one
‫will be in shard one and the next data point
‫for truck three will be in shard one as well and so on.
‫So anytime the truck one sends data, it will be in shard one
‫and anytime the blue truck,
‫the shard three sends data
‫then it will be in shard one as well,
‫because we are specifying
‫to use the same partition key over time.
‫So we see here is that truck one
‫and three will always have the data into shard one.
‫Now if we look at the shard two, then only truck two
‫and five will have the data into shard two.
‫And if you look at shard three, in this example,
‫we only have the truck four that will send its data
‫into shard three.
‫So now imagine you have 100 trucks and maybe five shards,
‫then each shard on average will have about 20 trucks.
‫But there is no linkage directly,
‫you can tell between the truck and each shard.
‫Kinesis will have to hash the partition key
‫to determine which shard to go to.
‫The idea is though that as soon
‫as we have a stable partition key,
‫then each truck will be sending this data to the same shard
‫and therefore we will have the data in order
‫for each truck at the shard level.
‫Make sense?
‫Next we are talking about SQS.
‫So for SQS standard as we know there's no ordering
‫and that's why we have SQL FIFO,
‫which is First-In-First-Out.
‫And so if we don't use a group ID in SQL FIFO,
‫then all the messages will be consumed in the order,
‫they were sent and we can only have one customer.
‫So in this example, we have a bunch of options,
‫and they're being sent into our SQS FIFO queue.
‫And so the order they're being sent in
‫will be the order a consumer will receive them.
‫And as we can see, we only have one consumer here,
‫it consumes two batches of messages,
‫the first one and then the second one.
‫And so as we can see, this is a First-In-First-Out
‫and it's very easy to reason about.
‫And so we can only have one consumer.
‫So if we had trucks,
‫then all the trucks would be sending data into a FIFO queue,
‫but they can only be one consumer.
‫So sometimes you may want to scale the number of consumers
‫and you want the message to be grouped when they are related
‫to each other.
‫So for this, we can use a group ID which is very similar
‫to the concept of a partition key in Kinesis.
‫So now using a group ID, our FIFO queue,
‫will have two groups of FIFO within.
‫And so for each group that you define,
‫you can have a different consumer.
‫So in this example, we have two groups, group A and group B.
‫And two consumers consumer one
‫and two can read independently, the group one
‫and the group two.
‫And so the idea here is that the more group IDs we have,
‫the more consumers we can have.
‫So this is a very different model from Kinesis.
‫Let's have a look, so if we have Kinesis versus SQS,
‫and we have 100 trucks, five Kinesis shards
‫and one SQS FIFO queue.
‫So if we have Kinesis Data Streams, then on average,
‫you'll have about 20 trucks per shard,
‫thanks to the hashing, so each truck
‫will be designated one shard
‫and will stay in that shard forever.
‫And the trucks will have their data ordered
‫within each shard.
‫But the maximum amount of consumers
‫we can have in parallel can be only five
‫because we have five shards
‫and we need one consumer per shard.
‫So the Kinesis Data Stream though because it has five shards
‫can receive up to five megabytes per second of data,
‫which is quite a high throughput.
‫Now in regards to SQS FIFO,
‫you can only have one SQS FIFO queue okay?
‫So you don't define shards or partitions
‫or anything like this, you just have one SQS FIFO queue.
‫And because we have 100 trucks,
‫then we can create 100 group ID, each equal to the truck ID.
‫And that means that because we have 100 group ID,
‫we can have up to 100 consumers, okay?
‫Each consumer will be hooked to one specific group ID.
‫And in terms of scale as SQS FIFO can have
‫up to 300 messages per second, or 3000,
‫if we use batching.
‫So these are different model of consumption,
‫of production, of ordering.
‫And so what you have to remember is
‫that based on the use case, sometimes is going
‫to be better to use SQS FIFO.
‫If you want to have a dynamic number of consumers
‫based on the number of group IDs,
‫and sometimes it could be better to use Kinesis Data Stream
‫if you have say 10,000 trucks and you need
‫to send it lot of data,
‫and also have data ordering per shard
‫in your Kinesis Data Stream. So I hope that was helpful,
‫like I know it can be complicated
‫to understand these things and they're more low level
‫but the exam is starting to ask you questions about that
‫and so I wanted to make sure
‫that you understand very clearly what this would entail.
‫So I hope you liked it
‫and I will see you in the next lecture.