‫So the first service you need to know about
‫is Kinesis Data Streams.
‫And Kinesis Data Streams is a way
‫for you to stream big data in your systems.
‫So a Kinesis Data Stream is made
‫of multiple shards,
‫and shards are numbered.
‫Number one, number two, all the way to number N.
‫And this is something you have to provision ahead of time.
‫So when you start with Kinesis Data Streams,
‫you're saying, hey, I want a stream with six shards.
‫And so the data is going to be split across all the shards.
‫Okay?
‫And the shards are going to be defining your stream capacity
‫in terms of ingestion and consumption rates.
‫So, for now, let's just start with this.
‫And then we have producers.
‫So producers send data into Kinesis Data Streams,
‫and producers can be manyfold.
‫They could be applications,
‫they could be clients which has desktop, or mobile clients,
‫they could be leveraging the AWS SDK at a very,
‫very low level, or the Kinesis Producer Library,
‫KPL, at a higher level and we'll have a deep down,
‫deeper dive onto the producers in the next lectures,
‫or it could be the Kinesis Agent
‫inside of the server to stream, for example
‫application logs into Kinesis Data Streams.
‫So all these producers do the exact same thing.
‫They rely on the SDK at a very, very low level,
‫and they're going to produce records
‫into our Kinesis Data Stream.
‫So a record, at its fundamental, is made of two things,
‫it's made of a partition key and it is made of the
‫data blob, or the value that is up to one megabytes.
‫So the partition key will define
‫and help determine in which shard will the record go to.
‫And the data blob is the value itself.
‫So when you have the producers sending data
‫to Kinesis Data Streams, they can send data
‫at a rate of one megabytes per second,
‫or a thousand messages per second, per shard.
‫So if you have six shards, you get six megabytes per second,
‫or 6,000 messages per second, overall, okay?
‫Now, once the data is in Kinesis Data Streams,
‫it can be consumed by many consumers,
‫and these consumers, again, can have many forms
‫and we'll explore them in details in this section.
‫So we have applications and they could be relying
‫on the SDK or at a high level, the Kinesis Client Libraries,
‫so KCL. They could be Lambda functions,
‫if you want to do serverless processing on top
‫of Kinesis Data Streams.
‫It could be Kinesis Data Firehose,
‫as we'll see in this section,
‫or Kinesis Data Analytics.
‫So when the consumer receives a record, it receives, again,
‫the partition key, also a sequence number
‫which represents where the record was in the shard,
‫as well as the data blob, so the data itself.
‫Now we have different consumption modes
‫for Kinesis Data Streams.
‫We have two megabytes per second
‫of throughput shared for all the consumers, per shard, okay?
‫Or you get two megabytes per second, per shard, per consumer
‫if you are enabling the enhanced consumer mode,
‫the enhanced fan-out.
‫So, we will look at it again in this section
‫in greater detail.
‫So again, producers send data to Kinesis Data Streams.
‫It stays in there for a while,
‫and then it is read by many different consumers.
‫Okay, some properties of Kinesis Data Streams.
‫The first one is that retention can be set
‫between 1 day to 365 days.
‫And that means that by default
‫you have the ability to reprocess or replay data.
‫And once data is inserted into Kinesis,
‫it cannot be deleted.
‫That's called immutability.
‫Also, when you send messages to Kinesis Data Streams
‫you add a partition key. And messages that share
‫the same partition key will go to the same shard,
‫and that gives you key based ordering.
‫For producers, you can send data using the SDK,
‫Kinesis Producer Library, KPL, or the Kinesis Agents.
‫And for consumers, you can write your own.
‫So, Kinesis Client Library, KCL, or the SDK,
‫or you can use a managed consumer on AWS,
‫such as AWS Lambda, Kinesis Data Firehose,
‫or Kinesis Data Analytics.
‫Now for capacity modes,
‫you have two options for Kinesis Data Stream.
‫The first one, that's the historic capacity mode,
‫it's called provisioned mode.
‫So you choose a number of shards provisioned,
‫and then you can scale them manually or using an API.
‫And each shard in Kinesis Data Streams
‫is going to get one megabyte per second,
‫or 1000 records per second.
‫And then for the out-throughput
‫each shard will get two megabytes per second,
‫and this is applicable to classic or fan-out consumer.
‫You also pay per shard provisioned per hour.
‫So you need to think a lot in advance,
‫and that's why it's called provisioned mode.
‫But the second mode is a neuro mode called On-demand mode.
‫And in this, you don't need to provision
‫or manage the capacity.
‫That means that the capacity will be adjusted
‫over time, on demand.
‫You get the default capacity provisioned,
‫which is four megabytes per second, or 4,000 records per,
‫and then there will be automatic scaling based on
‫the observed throughput peak during the last 30 days.
‫And in this mode, you're still going to pay
‫per stream per hour, and per data in/out per gigabyte.
‫So a different pricing model.
‫So if you don't know your capacity events, go for On-demand,
‫but if you can plan capacity events,
‫you should go for Provisioned mode.
‫In terms of security for Kinesis Data Streams,
‫it is deployed within a region.
‫And so you have your shards.
‫You can control access to produce and read
‫from the shard using IAM policies.
‫There is encryption in flight using HTTPS,
‫and encryption at rest using KMS.
‫You can implement your own encryption
‫and decryption of data on the client side,
‫which is called client side encryption,
‫and it is harder to implement because you need to
‫encrypt the data yourself and decrypt it yourself.
‫But this enhances security.
‫VPC endpoints are available for Kinesis.
‫This allows you to access Kinesis directly
‫from HTTPS, for instance
‫in a private subject without going through the internet.
‫And finally,
‫all the API calls can be monitored using CloudTrail.
‫So that's it for an overview of Kinesis Data Streams.
‫I hope you liked it.
‫And I will see you in the next lecture for a deeper dive
‫on all the moving parts in Kinesis Data Streams.