Now let's learn about a new service
which is Kinesis Data Firehose.
So it is a very helpful service
that can take data from producers
and producers can be everything we've seen
for Kinesis Data Streams or applications, clients SDK, KPL
all the cases agents can all produce
into your Kinesis Data Firehose
But also a Kinesis Data Stream can produce
into a Kinesis Firehose, Amazon CloudWatch logs and events
can produce into Kinesis Data Firehose or IOT
and all these applications are going to send records
into Kinesis Data Firehose
and then Kinesis Data Firehose can optionally choose
to transform the data using a Lambda function
but this is optional
and once the data is transformed optionally,
then it can be written in batches into destinations.
So Kinesis Data Firehose takes data from sources.
Usually the most common is going to be Kinesis Data Streams
and it's going to write this data into destinations
without you writing any kind of code
because Kinesis Data Firehose knows how to write data.
So there're three kinds of destinations
with Kinesis Data Firehose.
The number one category is AWS destinations
and you need to know them by heart.
So the first one is Amazon S3.
So you can write all your data into Amazon S3.
The second one is Amazon Redshift
which is a warehousing database
and to do so it first writes the data into Amazon history
and then Kinesis Data Firehose will issue a copy command
and this copy command is going to copy data
from Amazon history into Amazon Redshift
and the last destination on AWS
is called Amazon ElasticSearch.
There're also some third party partner destinations.
So Kinesis Data Firehose can send data
into Datadog, Splunk, New Relic, MongoDB
and this list can get bigger and bigger over time.
So it will not update this if there're new partners
but just so you know, there're partners
that Kinesis Data Firehose can send data to
or finally, if you have your own API with an HTTP endpoint
it is for you to send data from Kinesis Data Firehose
into a custom destination.
Okay, so once the data is sent
into all these destinations, you have two options.
You can also send all the data into an S3 bucket as a backup
or just send the data that was failed to be written
into these destinations into a failed S3 buckets.
So to summarize, Kinesis Data Firehose
is a fully managed service.
So there's no administration, automated scaling
and it is serverless.
So no servers to manage.
You can send data into AWS destination such as RedShift,
Amazon S3 and ElasticSearch.
Third-party partners such
as Splunk, MongoDB, Datadog, New Relic, etc
and custom destinations to any HTTP endpoints.
You're going to pay only
for the data going through Firehose.
So this is a very good data pressing model
and it is a near real time, why?
Well, because we write data in batches
from Firehose to the destination.
So there's going to be a 60 seconds latency minimum
for non full batches,
or you need to wait until you have at least 1 megabyte
of data at a time to send the data into the destination
which makes it a near real time service
and not a real-time service.
It supports many data formats, conversions, transformation
and compressions and you can write
your own data transformation using Lambda if you needed to.
Finally, you can send all the failed
or all the data into your backup S3 buckets.
So a question that comes up at the exam usually
is you understand the difference
of when to use Kinesis Data Streams and Kinesis Firehose.
So should be very easy for you now
if you followed closely but let's summarize.
Kinesis Data Streams is just a streaming service used
to ingest data at scale
and you write your own custom code
for your producers and your consumers.
It's real time.
So 200 millisecond or 70 millisecond
and you manage scaling yourself.
You do shard splitting and shard merging
to increase the scale and throughputs.
You're going to also pay
for how much capacity you have provision.
The data storage in the Kinesis Data Stream
can be between 1 to 365 days.
This allows multiple consumers to read
from the same stream and also supports replay capability
whereas Kinesis Data Firehose is an ingestion service
to stream data into S3, Redshift, ElasticSearch
or third party or a custom HTTP.
It is fully managed, no service to manage.
It is near real time.
So remember this, near real-time
is a keyword you need to look at in your exam questions.
There's automated scaling.
So need for you to worry about it
and you're going to pay only
for what goes through Kinesis Data Firehose.
There is no data source so you cannot replay data
from Kinesis Data Firehose.
So yeah, it doesn't support replay capability.
So that's it's for the overview
of Kinesis Data Firehose.
I hope that makes sense
and I will see you in the next lecture.