‫Now let's talk about how to get
‫data in Kinesis and for this
‫we are going to leverage Kinesis Producers.
‫So they're used to send data into our data streams
‫and a data record, as we said, consists of a sequence number
‫which is unique per partition-key within the shard,
‫the partition key as well, which we must specify
‫while we put records into a stream
‫and the data blob, which is up to one megabytes.
‫The producers can be anything from the SDK
‫to create a very simple producer
‫to using the Kinesis Producer Library, the KPL,
‫which supports different languages such as
‫C++ or Java and it is built on top of the SDK
‫but has some advanced capabilities available as API,
‫so for example, batching, compression and retries.
‫The Kinesis Agent is another way to send data into Kinesis.
‫It is built on top of the Kinesis Producer Library
‫and this is going to be used to monitor log files
‫and stream those into your Kinesis data streams.
‫In terms of write throughput, we know this already.
‫We get one megabytes per second
‫or 1,000 records per second per shard.
‫And the API to send data into Kinesis
‫is called the PutRecord API.
‫If we use batching with PutRecord APIs we can reduce costs
‫and therefore also increase throughputs,
‫which is something that's the Kinesis Producer Library
‫does for us already.
‫So if we look at the producer side
‫we have a stream with say six shards
‫and we have some producers, for example, IoT devices.
‫So they're going to send data at a rate of
‫one megabyte per second or 1,000 records per second
‫per shard, and say we have one device ID, 111222333.
‫So it's going to produce some data
‫and we choose to elect the partition key
‫to be the device ID.
‫So as you can see the partition key is the
‫device ID in this case.
‫And so what will happen is that it will go through
‫a hash function, which is a mathematical function
‫that will take as an input the partition key
‫and we'll figure out to which shard to send the data.
‫And so, for example, it goes to shard one,
‫thanks to the hash function and so that means that
‫all the data that will share the same partition key
‫is going to go to the same shard.
‫So all the data of the device ID will end up in shard one.
‫Now, if you have a second device ID
‫with a different device ID, so 444555666,
‫then the data blob is going to have
‫a different partition key but it's going to go through
‫the same hash function.
‫And this time the hash function may decide
‫to send this data into shard two and so that means
‫that this device ID will have all this data
‫sent to the shard number two.
‫And so this is how you produce to Kinesis data streams
‫with a partition key.
‫Now, as you can see, if one device is very chatty
‫and sends a lot of data it may overwhelm a shard.
‫Also, you need to make sure your partition key
‫is very well distributed to avoid the concept
‫of a hot partition, because then you will have one shard
‫that's going to have more throughputs than the others
‫and they will bring some imbalance.
‫So you need to think about a way to have
‫a distributed partition key.
‫For example, if you have six shards and 10,000 users
‫the user ID is very distributed.
‫But if you have six shards and you just look at Chrome,
‫Firefox and Safari as web browser
‫and the name of the web browser as a partition key,
‫then it's going to be very hot maybe for Chrome,
‫because there are many, many Chrome users in the world
‫versus Firefox or Safari.
‫Who knows, right?
‫So you need to make sure that you use a
‫distributed partition key.
‫Which brings us into the area of
‫ProvisionedThroughputExceeded.
‫So when we produce to Kinesis data streams
‫from our applications, we know that we can produce
‫1 megabyte per second or 1,000 records per second
‫and so as long as we do so it goes well.
‫But if we start over-producing into a shard
‫we're going to get an exception
‫because we are going over the provision throughput.
‫So we get the ProvisionedThroughputExceeded exception.
‫And so the solution to this is number one,
‫to make sure you are using a highly distributed
‫partition key because if not, then this error
‫will happen a lot.
‫And we need to implement retries with exponential backoff
‫to ensure that we can retry these exceptions
‫and this is something that comes up at the exam.
‫And finally, you need to scale the shard,
‫maybe it's called shard-splitting
‫to split a shard into and augment the throughputs
‫and so we'll see shard-splitting in a future lecture.
‫But by increasing the number of shards
‫this can help address throughput exceptions obviously,
‫because if you go from six shards to seven shards
‫we have one more megabyte per second available
‫in our stream.
‫So that's it for this lecture.
‫I hope you liked it and I will see you in the next lecture.