‫Now let's talk about consumers
‫with Kinesis Data Streams.
‫So the consumers get records
‫from the stream and they process them.
‫There is the consumers can be Lambda functions
‫they could be Kinesis Data Analytics can be a Firehose
‫a custom consumer using the SDK in two different modes
‫classic or enhanced fan-outs and the Kinesis Client Library
‫which is a library used to simplify reading
‫from a data stream.
‫I have a dedicated lecture by the way
‫on the Kinesis Client Library.
‫So let's discuss the difference between a classic shared
‫fan-out consumer and an enhanced consumer.
‫So when you write your consumer
‫in the classic shared throughput way
‫you have your Kinesis Data Stream
‫with a lot of shards and you get two megabytes per second
‫per shard across all consumers.
‫That means that if you look at shard, number one
‫just for this example
‫we can write a Consumer Application A that is
‫going to issue a get records API call to get records
‫from the shard one, but as possible
‫for us to have many different applications reading
‫from the same Kinesis Data Stream.
‫So consumer Application B will also issue get records
‫API calls, and the Consumer Application C will also
‫issue get records across all consumers.
‫Now, what happens in this instance is
‫that they are all sharing two megabytes per second
‫per shard across all consumers.
‫That means that in this instance
‫we have three consumers sharing, two megabytes per second.
‫That means that each consumer can get a maximum
‫of approximately 666 kilobytes per second of data.
‫So as you can see, there's a limit
‫to how many consumers we can have
‫in the more consumer applications we add
‫onto Kinesis Data Stream.
‫The more we're put limitation we're going to have.
‫So, which bring us
‫to a new mode of consumption brought by AWS recently
‫which is called the Enhanced Fan-Out Consumer.
‫So in this case we get two megabytes per second,
‫per consumer, per shard.
‫So it's not across all consumers
‫it's per consumer, per chart.
‫So that means that the Consumer Application A
‫will use a new API code called Subscribe to Shard
‫and this will make the shard, send the data
‫push the data into our Consumer Application A
‫at the rate of two megabytes per second.
‫And if the second Consumer Application B issues
‫another subscribed to shard
‫then this Consumer Application B as well.
‫Will get the data pushed by the shard
‫into the Consumer Application
‫at the rate of two megabytes per second.
‫And so will Consumer C.
‫So as we can see here
‫we have three consumer applications and we get six megabytes
‫per second of throughputs for this one chart.
‫So in the first model we have a pull model
‫and the second model, we have a push model.
‫So lets summarize, pull model
‫for the Shared Classic Fan-out Consumer,
‫which it gives you a low, which is good.
‫When you have a low number of consuming applications.
‫And the Ruth reboot is two megabytes per second
‫per shard across all consumers.
‫There's also limits is that per shard
‫you get maximum five, get records, API calls per second.
‫The latency of the API calls around 200 milliseconds.
‫And this is something you want to use when
‫you want to minimize costs.
‫The consumers will pull from Kinesis directly
‫using the get records API call and the return data
‫can be up to 10 megabytes.
‫Then it will float off
‫for five seconds or up to 10,000 records.
‫So if we use Enhance Fan-out Consumer
‫which is a push method
‫we can get some multiple consuming applications
‫from the same stream and each consumer will get
‫two megabytes per second, per shard.
‫The latency is going to be much lower because
‫the Shard itself will push the data into our consumer.
‫So 70 millisecond, the cost is higher.
‫It's a feature that will cost you more on AWS.
‫And the data, as I said
‫is pushed using a streaming method called HDB two.
‫Finally, there's a soft limit
‫of five consumer applications per data stream
‫but you can increase this by putting in a ticket on AWS.
‫Finally, we haven't seen Lambda in depth yet
‫but it is a way
‫for you to consume data without using servers.
‫So we have Kinesis Data Stream and say has three shards.
‫And so we're each going to learn the functions
‫and their role will be to process records
‫and save the record into dynamodb
‫which is a serverless database.
‫So the Lambda functions are going to call
‫get batch onto the Kinesis Data Stream.
‫And the data is going to be sent to Ireland
‫the functions by partition key to be processed.
‫The Lambda functions can then send data into dynamodb
‫and we have a way to process our Kinesis Data Stream
‫using a serverless mechanism.
‫So in this example, Lambda functions support
‫both the classic and enhance fan-out consumer modes.
‫So you can say how you want to consume data
‫from Kinesis Data Streams.
‫It will read the record in batches.
‫You can configure the batch size and the batch window.
‫And in case there's an error that occurs
‫the Lambda will retry until the succeeds
‫or the data will be expired in Kinesis Data Stream.
‫You can also process
‫up to 10 batches per shard simultaneously.
‫So we'll have a look at Kinesis at Lambda
‫in greater detail in the Lambda function?
‫So don't worry too much
‫if you don't understand what I just said, but so that sets
‫we've seen consumers in Kinesis Data Streams.
‫Now let's go ahead and do a hands-on to really
‫get some practice with Kinesis Data Streams.
‫So I will see you in the next lecture.