‫Now let's learn about a new service
‫which is Kinesis Data Firehose.
‫So it is a very helpful service
‫that can take data from producers
‫and producers can be everything we've seen
‫for Kinesis Data Streams or applications, clients SDK, KPL
‫all the cases agents can all produce
‫into your Kinesis Data Firehose
‫But also a Kinesis Data Stream can produce
‫into a Kinesis Firehose, Amazon CloudWatch logs and events
‫can produce into Kinesis Data Firehose or IOT
‫and all these applications are going to send records
‫into Kinesis Data Firehose
‫and then Kinesis Data Firehose can optionally choose
‫to transform the data using a Lambda function
‫but this is optional
‫and once the data is transformed optionally,
‫then it can be written in batches into destinations.
‫So Kinesis Data Firehose takes data from sources.
‫Usually the most common is going to be Kinesis Data Streams
‫and it's going to write this data into destinations
‫without you writing any kind of code
‫because Kinesis Data Firehose knows how to write data.
‫So there're three kinds of destinations
‫with Kinesis Data Firehose.
‫The number one category is AWS destinations
‫and you need to know them by heart.
‫So the first one is Amazon S3.
‫So you can write all your data into Amazon S3.
‫The second one is Amazon Redshift
‫which is a warehousing database
‫and to do so it first writes the data into Amazon history
‫and then Kinesis Data Firehose will issue a copy command
‫and this copy command is going to copy data
‫from Amazon history into Amazon Redshift
‫and the last destination on AWS
‫is called Amazon OpenSearch.
‫There're also some third party partner destinations.
‫So Kinesis Data Firehose can send data
‫into Datadog, Splunk, New Relic, MongoDB
‫and this list can get bigger and bigger over time.
‫So it will not update this if there're new partners
‫but just so you know, there're partners
‫that Kinesis Data Firehose can send data to
‫or finally, if you have your own API with an HTTP endpoint
‫it is for you to send data from Kinesis Data Firehose
‫into a custom destination.
‫Okay, so once the data is sent
‫into all these destinations, you have two options.
‫You can also send all the data into an S3 bucket as a backup
‫or just send the data that was failed to be written
‫into these destinations into a failed S3 buckets.
‫So to summarize, Kinesis Data Firehose
‫is a fully managed service.
‫So there's no administration, automated scaling
‫and it is serverless.
‫So no servers to manage.
‫You can send data into AWS destination such as RedShift,
‫Amazon S3 and OpenSearch.
‫Third-party partners such
‫as Splunk, MongoDB, Datadog, New Relic, etc
‫and custom destinations to any HTTP endpoints.
‫You're going to pay only
‫for the data going through Firehose.
‫So this is a very good data pressing model
‫and it is a near real time, why?
‫Well, because we write data in batches
‫from Firehose to the destination.
‫So there's going to be a 60 seconds latency minimum
‫for non full batches,
‫or you need to wait until you have at least 1 megabyte
‫of data at a time to send the data into the destination
‫which makes it a near real time service
‫and not a real-time service.
‫It supports many data formats, conversions, transformation
‫and compressions and you can write
‫your own data transformation using Lambda if you needed to.
‫Finally, you can send all the failed
‫or all the data into your backup S3 buckets.
‫So a question that comes up at the exam usually
‫is you understand the difference
‫of when to use Kinesis Data Streams and Kinesis Firehose.
‫So should be very easy for you now
‫if you followed closely but let's summarize.
‫Kinesis Data Streams is just a streaming service used
‫to ingest data at scale
‫and you write your own custom code
‫for your producers and your consumers.
‫It's real time.
‫So 200 millisecond or 70 millisecond
‫and you manage scaling yourself.
‫You do shard splitting and shard merging
‫to increase the scale and throughputs.
‫You're going to also pay
‫for how much capacity you have provision.
‫The data storage in the Kinesis Data Stream
‫can be between 1 to 365 days.
‫This allows multiple consumers to read
‫from the same stream and also supports replay capability
‫whereas Kinesis Data Firehose is an ingestion service
‫to stream data into S3, Redshift, OpenSearch
‫or third party or a custom HTTP.
‫It is fully managed, no service to manage.
‫It is near real time.
‫So remember this, near real-time
‫is a keyword you need to look at in your exam questions.
‫There's automated scaling.
‫So need for you to worry about it
‫and you're going to pay only
‫for what goes through Kinesis Data Firehose.
‫There is no data source so you cannot replay data
‫from Kinesis Data Firehose.
‫So yeah, it doesn't support replay capability.
‫So that's it's for the overview
‫of Kinesis Data Firehose.
‫I hope that makes sense
‫and I will see you in the next lecture.