‫So let's go ahead and get some practice
‫for Kinesis Data Streams.
‫So I'm going to open the Kinesis service,
‫and create our first Kinesis Data Streams.
‫As we can see, we have three options here.
‫We can use Data Streams, Data Firehose, or Data Analytics,
‫but we only know about Data Streams so far,
‫so let's go ahead.
‫We get some information around the pricing,
‫which is at per shard, we pay $0.05 per hour.
‫Okay.
‫And then there's a cost for making PUTs or for sending data
‫into Kinesis Data Streams.
‫So we'll create a Data Stream,
‫let's name our stream, DemoStream,
‫and then we have to define the data stream capacity.
‫So as you can see, we have two modes,
‫we have the On-demand mode and the Provision mode.
‫And in the On-demand mode,
‫you don't have to think about the capacity,
‫it's automatically going to scale for you,
‫so there's a maximum throughput of 200 megabytes per second,
‫and 200,000 records per second,
‫and the maximum read capacity of 400 megabytes per second,
‫per consumer, if you're using the enhanced Fan-Out option.
‫So the on demand as a pay-per throughput pricing model,
‫but there is no free tier. Okay.
‫And the Provision mode also doesn't have a free tier.
‫And in the provision mode, you need to provision shards.
‫And so there's a Shard estimator tool,
‫if you wanted to understand how many shards you needed,
‫based on like how many records you send per second,
‫and what is the size of your record,
‫and how many consumers you have. Okay.
‫But in a example, we're going to just set up one shard.
‫One shard gives us one megabyte per second for write
‫and two megabytes per second for read.
‫And obviously, if you put 10 shards,
‫then everything is multiplied by 10. Okay,
‫And so one shard is enough for us to do the demo.
‫It's also the cheapest option we can get.
‫If you do not want to pay any money in this course,
‫then do not do this hands on,
‫because you will pay some money for your shard,
‫although we will deal with it,
‫and then delete it fast enough.
‫So when you're ready, you just click on Create data stream,
‫and you wait for the data stream to be created.
‫And our stream is now successfully created.
‫So in terms of applications, we can see,
‫we have producers, and we are recommended three options.
‫The Kinesis Agents, the SDK,
‫or the Kinesis Producer Library.
‫So they're all available on GitHub, for you to check it out.
‫This is a way to stream data from application servers
‫into Kinesis Data Streams.
‫The SDK is for you to develop producers at a very low level.
‫And the KPL is for you to develop producers
‫at a very high level, with a better API.
‫And in terms of consumers,
‫we get Kinesis Data Analytics, Kinesis Data Firehose,
‫or the Kinesis Client Library, or Lambda, as well,
‫that is not shown as an option right here.
‫Okay. We get some monitoring information
‫about our Kinesis Data Streams.
‫So how many records we are sending into it?
‫We could look at the configuration.
‫So if you wanted to scale the stream,
‫we can say how many shards we want.
‫We can go from one, to say, five,
‫and scale our Kinesis Data Stream.
‫We could add some tags,
‫and then we could use enhanced Fan-Out, and configure it,
‫if we wanted to have some consumer applications
‫leveraging the enhanced Fan-Out capability.
‫But for now, let's keep it simple.
‫We just want to write and read from our stream,
‫and so therefore,
‫we're going to use the SDK for producing and for consuming.
‫So to do so, we want to open a CLI,
‫and let's use CloudShell because it's fun.
‫So I'm going to click on CloudShell right here,
‫which is the icon, next to the bell icon.
‫And this is going to open, for me,.
‫a command line interface in AWS.
‫As an alternative, you could be using your own terminal
‫or CLI if it's preconfigured,
‫but I like to switch things up,
‫and this one, I really like,
‫because there's less configuration involved at least.
‫Creating the environment, the first time,
‫can take a bit of time, and the CloudShell is free on AWS,
‫so no worries here.
‫In the meantime, on Kinesis,
‫open the kinesis-data-streams.sh file,
‫and we're going to use that overall.
‫So there are two types of commands
‫to write to Kinesis Data Stream based on your CLI version
‫for the AWS CLI.
‫So usually, you have version two installed
‫but it is possible for you to somehow,
‫have version one installed.
‫And so you get the version,
‫you just type aws --version, and paste,
‫and then, you will get some information
‫around the version of AWS CLI.
‫So in CloudShell,
‫the CLI version that's going to be installed is version two,
‫as you can see here, it was CLI version 2.1.16,
‫so we're going to use the CLI commands version two.
‫But if you wanted to have the version one,
‫then you would use these comments.
‫Okay. So now we're good to go.
‫The first we want to do is send a record
‫into our Kinesis Data Stream.
‫And to do so, there is an API called put-record.
‫And put-record, we need to specify a stream name.
‫So in this example, I didn't name my stream test,
‫I named it DemoStream,
‫so we'll have to change this in the CLI command,
‫but you get the idea.
‫Then you specify a partition key for the data settings.
‫So user one, and remember the data
‫that shares the same partition key
‫will go to the same shard, but we only have one shard,
‫so that doesn't matter in this case.
‫Then the data itself, so users signup.
‫And finally, because we are writing some text data,
‫we need to say this option cli-binary-format
‫raw-in-base64-out.
‫Okay. So let it paste this command, so copy and paste,
‫but let me just edit the stream name,
‫to make sure that it is DemoStream.
‫And the cloud channel is automatically configured
‫with your own IM credentials,
‫so it will inherit whatever you have for IM credentials,
‫and also, we'll use the region by default,
‫in which it was launched.
‫So us-east-1.
‫I press Enter.
‫And now, we get a successful message.
‫So the message was sent shardId-0000000000000.
‫So our first shard, and the sequence number
‫of the message is here.
‫If I do it again, then I'm going to get a second message
‫with a successful, so we can do user signup.
‫We can mix up the message,
‫then user login, and then, maybe user logout.
‫So we're just, we're just setting a few messages
‫into our Kinesis Data Stream.
‫Perfect. So I'm going to clear this,
‫and if you waited a little bit,
‫and went into monitoring, and look at the stream metrics,
‫I will have it on one hour,
‫you would see a put record right here,
‫but it takes a bit of time for CloudWatch metrics
‫to be updated, but you would see it here. Okay.
‫So next we want to be able to consume
‫from Kinesis Data Stream.
‫So to do so, we're going to first describe the stream,
‫to get some information around what this stream is made of,
‫because we need to be able to consume from a specific shard.
‫So it's DemoStream, I'll press Enter,
‫and as you can see here, we have this StreamDescription.
‫We have one shard called shardId-0000000000000,
‫and so we need to keep this in our mind. Okay.
‫To be able to read from this stream.
‫And so when you use the CLI,
‫the SDK at a very, very low level,
‫you need to specify from which shard you are reading from.
‫But if you are using Kinesis Client Library,
‫all of this is handled for you by the library itself.
‫But we are using the CLI,
‫so we have to specify the shard ID.
‫So I press Q to quit this,
‫and I'm going to consume some data.
‫So I'm going to run this command, right here,
‫and let me clear this.
‫And there's two things to note,
‫so number one, I need to change the name of the stream
‫I'm consuming from.
‫So DemoStream is a stream,
‫and then the shard-iterator-type is TRIM-HORIZON.
‫And this means that you're going to read
‫from the very beginning of the stream,
‫so it will read all the records
‫that were sent for from the beginning.
‫The other option, just make sure to only receive the records
‫from that very moment onwards when from a new launched
‫to CLI command.
‫Anyway, so I'm going to press Enter.
‫And this is going to give me a ShardIterator.
‫And this ShardIterator can be reused to consume records,
‫so the next API comment
‫is kinesis get-records --shard-iterator.
‫And then, we just specify this entire string, right here.
‫So this consumption mode I'm doing right now,
‫by using the low level API,
‫describing the stream, getting a ShardIterator,
‫and getting records is using the shared consumption mode.
‫This is not using enhanced Fan-Out,
‫which in my opinion,
‫should be using the Kinesis Client Library,
‫Consumer Library for you to really leverage,
‫and have a nice API to do so.
‫But this is low level.
‫So let's click, and let's press Enter
‫on the Kinesis get-records,
‫and we get a batch of records out of it.
‫So we used to have record one right here,
‫which is PartitionKey user1.
‫And we have some data right here, but it's base64-encoded.
‫We have, again, another data, base64-encoded.
‫We get some time stamp information,
‫another data, base64-encoded.
‫And then if I press Enter,
‫it's going to go a little bit down.
‫Some more data that is base64-encoded.
‫So to just make sure we can read the data,
‫I can go to websites, base 64 decode online,
‫and I'm going to paste this data right here,
‫into base 64 decode, and click on DECODE,
‫and this gives us user signup.
‫And if I paste the second type of data, this one,
‫copy and paste it in here, is going to give us user login.
‫So we which is that we sent.
‫So that's perfect.
‫Everything is working.
‫And then, as you can see,
‫there's a NextShardIterator argument right here.
‫So the next time we consume,
‫we need to specify this NextShardIterator argument
‫to consume from where we stop consuming from.
‫So this is something you have to iterate from in your code.
‫But at a low level,
‫we've produced data to Kinesis Data Stream,
‫and consume data from Kinesis Data Stream, which is awesome.
‫And we've also used CloudShell in the meantime,
‫which I think is very handy.
‫So that's it for this demo,
‫just keep this stream open as we will be using it
‫for Kinesis Data Firehose in a second.
‫And I will see you in the next lecture.