‫Okay, so let's practice
‫using Kinesis Data Firehose,
‫using delivery streams.
‫So I click on delivery streams.
‫And in here,
‫I'm able to create a delivery stream.
‫And we have a detailed diagram
‫of how Kinesis Data Firehose works.
‫So we ingest data from producers,
‫and these producers can be either
‫a Kinesis Data Stream.
‫So this is our use case,
‫or they can be Direct PUTs
‫done through the Kinesis Data Agents,
‫the Kinesis Agents,
‫some other AWS services,
‫such as CloudWatch, IoT Core,
‫EventBridge, et cetera,
‫et cetera, and also your own apps using the SDK
‫that can send data directly into Kinesis Data Firehose.
‫So once we ingest the data,
‫we can outright transform it using a Lambda function.
‫And this can also be used to do many things,
‫such as converting the record format, and so on.
‫And then, we load the data into target stores.
‫So we have Amazon S3, Amazon OpenSearch Service,
‫which is ElasticSearch being renamed,
‫and Amazon Redshift,
‫and various HTTP endpoint destinations.
‫So in this example,
‫our source is going to be a Kinesis Data Stream,
‫and the destination is going to be Amazon S3.
‫But it's very important for you to notice
‫that we have OpenSearch service.
‫So ElasticSearch, RedShift, S3,
‫you need to remember these three for sure.
‫Then we have some,
‫a lot of third-party services,
‫so you don't need to remember them all or at all.
‫But just remember that they are third party services.
‫Or any custom HTTP Endpoint that you can choose as well.
‫So we'll choose Amazon S3.
‫Now for the source,
‫we need to browse and choose our stream.
‫So we have the ARN Demostream entered right here.
‫So this is good.
‫Then, the delivery stream name is automatically generated,
‫so this is perfect.
‫Now, we go into the Transform and convert records part.
‫So this is optional.
‫But we want to transform source records using Lambda.
‫So here we can transform, filter,
‫un-compress, convert,
‫process source records.
‫So these Lambda functions are just pieces of code
‫you can run into AWS,
‫and they can do whatever you want
‫on this data before being delivered
‫by Kinesis Data Firehose.
‫So this could be quite handy.
‫And if you do enable it,
‫then you need to choose a Lambda function.
‫Okay.
‫Next, the Convert record format option.
‫So based on where you are sending your data to,
‫it can be useful to transform these records
‫into Parquet or ORC based on some advanced options.
‫So this is not in scope,
‫but just remember that you can transform the record format
‫using Kinesis Data Firehose.
‫Now, this is more detailed
‫into the data and analytics certification of AWS.
‫Right now, just know at a high level
‫that you can convert record formats.
‫Next, we need to choose a destination.
‫So we can just choose an S3 bucket
‫that we've created from before or create one.
‫So for me,
‫I've already created an S3 bucket.
‫So I'll use this one.
‫So demo-firehose-stephane-V3.
‫I will choose this one,
‫but feel free to create a bucket
‫or choose an existing one as well.
‫Do you want to have dynamic partitioning?
‫So right now, we will say no.
‫S3 bucket prefix,
‫so do we want to prefix our data?
‫And for now,
‫we don't need to.
‫And also, a bucket error output prefix if we needed to,
‫if you wanted to,
‫you have for example, error.
‫But again, we don't want to this right now.
‫We'll keep it very, very simple.
‫And now, more importantly,
‫around the buffer hints,
‫compression and encryption.
‫So the buffer is a way
‫for Kinesis Data Firehose to accumulate records
‫before delivering them into the targets.
‫And so by default,
‫Kinesis will write five megabytes of data into the buffer
‫before delivering it into the target,
‫so Amazon S3.
‫Now, you can make the buffer more efficient,
‫for example more than 128 megabytes,
‫if you wanted to get a bigger buffer size
‫and more efficiency,
‫or a small buffer size,
‫if you wanted to deliver the data as fast as possible.
‫So we'll have it as 1.
‫And then, the buffer interval,
‫so this is how fast,
‫if the buffer size doesn't fill up.
‫How fast should it be flushed into the target?
‫And so, if you choose 300 seconds,
‫you're going to wait five minutes to fill the buffer size.
‫But if the buffer size is not filled after five minutes,
‫then it's going to be flushed nonetheless.
‫So if we set a lower buffer interval,
‫such as 60 seconds,
‫we have the guarantee that at maximum,
‫every 60 seconds,
‫the buffer is going to be flushed into Amazon S3.
‫If we set a really long buffer size,
‫such as 900 seconds,
‫then we need to wait,
‫I think, 15 minutes before the buffer
‫is flushed into Amazon S3,
‫so a lot longer.
‫So for the purpose of this demo,
‫we don't want to be efficient.
‫We want to be fast.
‫So we'll choose 60 seconds,
‫which is the minimum.
‫Next, we want to enable compression and encryption,
‫so we can compress the records in the target,
‫such as with GZIP, Snappy,
‫Zip or Hadoop-Compatible Snappy.
‫And the idea is that you're going to save some space,
‫because we are compressing the data
‫before storing into Amazon S3.
‫So save some cost.
‫And also, do you want you to encrypt your records?
‫Yes or no.
‫There's some advanced settings.
‫But the one that's very important for you to see
‫is this permission right here.
‫So this is going to create automatically
‫an IAM role named this.
‫And this IAM role
‫is going to have all the permissions required
‫to write into Amazon S3.
‫So this is how Kinesis Data Firehose is able to write
‫to the target buckets
‫and to read as well from the Kinesis Data Stream.
‫So let's create this delivery stream.
‫And it is active.
‫So we can have a look at some metrics.
‫So the more data will go through Kinesis Data Firehose,
‫the more these metrics will be populated,
‫which is very helpful in production.
‫You can look at the configuration,
‫but we've already done that.
‫And then, we can look at the destination for the error logs.
‫And right now,
‫this is CloudWatch Logs.
‫Okay, so what we have here
‫is a Kinesis Data Firehose of source Kinesis Data Streams,
‫and we need to test with some data flowing through it.
‫So you could use the test data right here
‫to test it going into Amazon S3,
‫but we don't want to use this.
‫Actually, because we have Amazon Kinesis Data Stream,
‫let's just use that one as well.
‫So my Kinesis Data Stream right here is named DemoStream,
‫and we're going to send more data to it.
‫Because even though we have created
‫the Firehose Delivery Stream,
‫even if some data was sent in the past
‫to the Kinesis Data Stream,
‫you actually need to send new data
‫after setting up Firehose for Firehose to be active.
‫So let's just use CloudShell,
‫and we're going to use the commands right here.
‫So we're going to have to modify this,
‫and make sure that you have the right stream name.
‫So DemoStream is the one that I have today,
‫and user signup.
‫This is good.
‫And then, paste this command.
‫Let's press enter.
‫The data has been sent.
‫So we have user signup.
‫Then, we have user login.
‫And then, we'll have user logout.
‫Okay, so three records have been sent.
‫And what I can do now is go into Amazon S3
‫and see if they have appeared in Amazon S3.
‫So let's go into the S3 console.
‫I'm going to type firehose.
‫I found my bucket.
‫As you can see,
‫currently, there are zero objects in my bucket.
‫That's because the Kinesis Data Firehose
‫has a buffer of 60 seconds.
‫So we need to wait 60 seconds
‫until the data makes it into Amazon S3.
‫So let's just wait.
‫I'll pause the video.
‫Okay, so it's been more than 60 seconds.
‫So I'm going to refresh.
‫And as you can see,
‫updates have appeared into my Amazon S3 bucket.
‫So I can click, click,
‫click, and it says partitioned by date and so on.
‫And here, I have the record,
‫so I can click on it.
‫Click open,
‫and then open this with my text editor.
‫And as you can see,
‫not very fascinating,
‫but we have user signup, user login,
‫and user logout in one text file.
‫So Kinesis Data Firehose is working
‫and working great.
‫So that's it for this lecture.
‫I hope you liked it.
‫And just to clean it up,
‫please make sure to delete first that delivery stream.
‫So you need to type in the name
‫and you have it.
‫And then most importantly,
‫delete the DemoStream itself.
‫Because if you let it run,
‫then it's going to cost you money every hour, okay?
‫So that's it for this lecture.
‫I hope you liked it.
‫And I will see you in the next lecture.