So the first service you need to know about
is Kinesis Data Streams.
And Kinesis Data Streams is a way
for you to stream big data in your systems.
So a Kinesis Data Stream is made
of multiple shards,
and shards are numbered.
Number one, number two, all the way to number N.
And this is something you have to provision ahead of time.
So when you start with Kinesis Data Streams,
you're saying, hey, I want a stream with six shards.
And so the data is going to be split across all the shards.
Okay?
And the shards are going to be defining your stream capacity
in terms of ingestion and consumption rates.
So, for now, let's just start with this.
And then we have producers.
So producers send data into Kinesis Data Streams,
and producers can be manyfold.
They could be applications,
they could be clients which has desktop, or mobile clients,
they could be leveraging the AWS SDK at a very,
very low level, or the Kinesis Producer Library,
KPL, at a higher level and we'll have a deep down,
deeper dive onto the producers in the next lectures,
or it could be the Kinesis Agent
inside of the server to stream, for example
application logs into Kinesis Data Streams.
So all these producers do the exact same thing.
They rely on the SDK at a very, very low level,
and they're going to produce records
into our Kinesis Data Stream.
So a record, at its fundamental, is made of two things,
it's made of a partition key and it is made of the
data blob, or the value that is up to one megabytes.
So the partition key will define
and help determine in which shard will the record go to.
And the data blob is the value itself.
So when you have the producers sending data
to Kinesis Data Streams, they can send data
at a rate of one megabytes per second,
or a thousand messages per second, per shard.
So if you have six shards, you get six megabytes per second,
or 6,000 messages per second, overall, okay?
Now, once the data is in Kinesis Data Streams,
it can be consumed by many consumers,
and these consumers, again, can have many forms
and we'll explore them in details in this section.
So we have applications and they could be relying
on the SDK or at a high level, the Kinesis Client Libraries,
so KCL. They could be Lambda functions,
if you want to do serverless processing on top
of Kinesis Data Streams.
It could be Kinesis Data Firehose,
as we'll see in this section,
or Kinesis Data Analytics.
So when the consumer receives a record, it receives, again,
the partition key, also a sequence number
which represents where the record was in the shard,
as well as the data blob, so the data itself.
Now we have different consumption modes
for Kinesis Data Streams.
We have two megabytes per second
of throughput shared for all the consumers, per shard, okay?
Or you get two megabytes per second, per shard, per consumer
if you are enabling the enhanced consumer mode,
the enhanced fan-out.
So, we will look at it again in this section
in greater detail.
So again, producers send data to Kinesis Data Streams.
It stays in there for a while,
and then it is read by many different consumers.
Okay, some properties of Kinesis Data Streams.
The first one is that retention can be set
between 1 day to 365 days.
And that means that by default
you have the ability to reprocess or replay data.
And once data is inserted into Kinesis,
it cannot be deleted.
That's called immutability.
Also, when you send messages to Kinesis Data Streams
you add a partition key. And messages that share
the same partition key will go to the same shard,
and that gives you key based ordering.
For producers, you can send data using the SDK,
Kinesis Producer Library, KPL, or the Kinesis Agents.
And for consumers, you can write your own.
So, Kinesis Client Library, KCL, or the SDK,
or you can use a managed consumer on AWS,
such as AWS Lambda, Kinesis Data Firehose,
or Kinesis Data Analytics.
Now for capacity modes,
you have two options for Kinesis Data Stream.
The first one, that's the historic capacity mode,
it's called provisioned mode.
So you choose a number of shards provisioned,
and then you can scale them manually or using an API.
And each shard in Kinesis Data Streams
is going to get one megabyte per second,
or 1000 records per second.
And then for the out-throughput
each shard will get two megabytes per second,
and this is applicable to classic or fan-out consumer.
You also pay per shard provisioned per hour.
So you need to think a lot in advance,
and that's why it's called provisioned mode.
But the second mode is a neuro mode called On-demand mode.
And in this, you don't need to provision
or manage the capacity.
That means that the capacity will be adjusted
over time, on demand.
You get the default capacity provisioned,
which is four megabytes per second, or 4,000 records per,
and then there will be automatic scaling based on
the observed throughput peak during the last 30 days.
And in this mode, you're still going to pay
per stream per hour, and per data in/out per gigabyte.
So a different pricing model.
So if you don't know your capacity events, go for On-demand,
but if you can plan capacity events,
you should go for Provisioned mode.
In terms of security for Kinesis Data Streams,
it is deployed within a region.
And so you have your shards.
You can control access to produce and read
from the shard using IAM policies.
There is encryption in flight using HTTPS,
and encryption at rest using KMS.
You can implement your own encryption
and decryption of data on the client side,
which is called client side encryption,
and it is harder to implement because you need to
encrypt the data yourself and decrypt it yourself.
But this enhances security.
VPC endpoints are available for Kinesis.
This allows you to access Kinesis directly
from HTTPS, for instance
in a private subject without going through the internet.
And finally,
all the API calls can be monitored using CloudTrail.
So that's it for an overview of Kinesis Data Streams.
I hope you liked it.
And I will see you in the next lecture for a deeper dive
on all the moving parts in Kinesis Data Streams.