Welcome back to Backspace Academy. Amazon Kinesis it is AWS service for 2
handling streaming live data content. It consists of Kinesis data streams for 3
pure data and video streams for streaming video content to devices and 4
data files which will direct streams of data to durable storage such as Amazon 5
s3 and finally Kinesis data analytics which enables the processing and 6
analytics of streaming data content. Kinesis data streams they collect and 7
process large streams of data in real time and it's generated continuously by 8
up to thousands upon thousands of data sources. Your applications can process 9
that data in that stream sequentially and incrementally on a record by record 10
basis if you'd like to do every single record or you can do it over a sliding 11
time window for example if you just wanted to do the last five minutes of 12
data to process. It is not available on the free tier and it's billed on a per 13
shard basis and we'll talk more about what a shard is in the next slide 14
it has server-side encryption and it's very simple to implement similar to 15
Amazon s3 where you can create a key in the AWS key management service and 16
associate that with your stream. Kinesis has a lot of unique benefits. It's going 17
to be processing continuously live data and analyzing that data on a live basis 18
it's a managed service it is elastic you can scale it up and scale it down very 19
simply, it's highly durable it's replicated across three availability 20
zones, it's great for implementing process and safety alarms where you want 21
to be alerted based upon evidence to a problem, 22
multiple applications can consume data from the same stream and you can retain 23
that data from a default of one day up to a maximum of seven days which is at 24
an addition cost. A Kinesis data stream 25
implementation will consist firstly of a producer that will collect data and 26
continually push that data that live data into the Kinesis data stream and 27
then on the output of the stream consumers will be continuously 28
processing that data in real-time a stream it's made up of shards and shards 29
are simply a mini streams within that stream so the more shards you have 30
inside a stream the greater the capacity of that stream each shard is uniquely 31
identified and it contains a continuous sequence of data records those data 32
records will have a sequence number to identify where they are located in the 33
shard a partition key which will help to identify which shard within the stream 34
that record is located in and finally it will have the data blob which will 35
contain the data that we're going to be streaming. A stream can be created using 36
the management console or using the create stream software development kit 37
or command-line interface operation. On the left-hand side there we've got 38
producers and they could be an ec2 instance they could be a desktop client 39
they could be a mobile application provided it has an application that can 40
communicate with the Kinesis service such as the AWS software development kit 41
or one of the libraries that are available for Kinesis and that will 42
continuously push data to the Amazon Kinesis stream. Now a lot of people can 43
be confused about the concept of a shard and the reason being is that a lot of 44
the diagrams are a little bit confusing, so a shard it's not a piece of data, it's 45
not a finite piece of data, it's a continuous stream of data, so it is in 46
itself a mini stream, so there you can see you've got your Amazon Kinesis 47
stream in blue and that inside that will be these shards which are themselves 48
mini streams and so that data will pass through those shards so the more shards 49
you have the more data you can get through your Kinesis stream the less 50
shards are obviously you're going to be more 51
restricted and those records that are coming from the producer will be pushed 52
into those shards on the output of our Amazon Kinesis stream we have consumers 53
and they are applications that we'll be reading continuously 54
from our Kinesis stream and they will be then forwarding their information over 55
to either durable storage such as Amazon s3 or to a processing service such as 56
Amazon EMR or we could also have the Amazon Kinesis fire hose which can 57
direct that stream to other AWS services such as Amazon s3. Writing to Kinesis 58
streams each shard it can support up to 1,000 put records per second and it 59
provides a capacity of one megabyte per second so you need to take that into 60
consideration depending on the amount of data that you're going to be putting 61
through your Kinesis stream the number of shards required is specified when a 62
stream is created now when a producer pushes data to a stream it will need to 63
provide a partition key and that's simply a string that you define in your 64
puppy or producer application so it could be xyz1, xyz2 whatever 65
but you organize your data based upon this partition key and then the Kinesis 66
data service will map that partition key to a specific shard, so the next time 67
that you push data with that same partition key prefix for that record it 68
will go to the same shard which makes it easier for your application on the other 69
end to retrieve that data. A sequence number it will be a unique record 70
identifier and it is assigned by Kinesis it's not assigned by your producer it's 71
assigned by the Kinesis applique or Kinesis service when that record is 72
created and we create records normally with the sdk with a put records command 73
and finally a shard iterator is supplied by the Kinesis service and it identifies 74
a position in the shard to start reading data for that record so the process for 75
a producer will collect live data and it will run a putrecords 76
command and that will need to have the name of the stream it will need 77
to supply a partition key and obviously it will need to supply the data that is 78
going into the stream. Reading Kinesis streams. Each shard it can support only 79
five transactions per second so basically we're reading a chunk of 80
records at a time normally one chart it provides a capacity of two megabytes per 81
second of data reads the shard iterator it's required to read a stream and it 82
specifies the position in the shard from which to start reading those data 83
records sequentially we can specify different shard iterator types depending 84
on where in the stream that we want to get this data so we can specify it at a 85
specific sequence number or after a specific sequence number we can specify 86
it in a specific point in time with a timestamp we can do it at the trim 87
horizon which will give us the oldest data in the stream or we can do it at 88
the latest data if we want to get say for example the last five minutes of the 89
latest data now that shard iterator it will be passed to get records so that 90
can be through the software development kit or it can be done for you through a 91
Kinesis client library now get records it will return the milliseconds behind 92
the latest record and it will also give the next shard iterator and it will then 93
finally give an array of the records which will include the timestamp for the 94
records the actual data and then the sequence number of the records so the 95
process if we're using the one of the software development kits or the Kinesis 96
streams API is that we first need to call get shard iterator which will give 97
us that shard iterator we need to supply the ID of the shard the shard iterator type 98
remember if it's at timestamp or it's the latest or it's the oldest or 99
whatever and then the actual name of the stream itself if we've provided 100
information to get shard iterator then we can use that shard iterator with a 101
get records command to actually retrieve those records and so 102
they'll get records again it will return the milliseconds behind the latest 103
and then the next shard iterator so we can use that to then go in and get our 104
next a lot of records and it then of course will supply an array of those records 105
there are a number of tools that can help us to develop and implement our 106
Kinesis applications, the first one there is the AWS software development kits 107
we also have a Kinesis client library which is recommended by AWS and that is 108
available for Java Python Ruby dotnet and nodejs and that is for developing 109
Kinesis consumer applications, we have the Kinesis connector library and they 110
integrate streams with dynamo DB redshift Amazon s3 and elasticsearch 111
we also have the Kinesis producer library and that simplifies the combining of 112
records so if we've got a lot of small records it can combine those into run 113
records so we're going to get the maximum effectiveness and efficiency out 114
of our stream and then it can write that force to those streams so it does 115
simplify a lot of that heavy lifting for it, there is also the Kinesis agent 116
which is a pre-built java application that can be used to collect and send 117
data to a stream so it's great for creating producer applications and we 118
finally have the Kinesis data generator which is great if you want to test your 119
consumer application because it can produce a test data to a stream for you 120
Kinesis it's a fully managed service and makes it really easy to scale up or 121
scale down depending on our circumstances and that process of 122
scaling is called reshoring and we do that with an update shard count 123
command in the software development kit or the command-line 124
interface and that will increase or decrease the number of shards that are 125
in our stream there are two types of reshoring processes a shard split will 126
increase the show count and a shard merge will decrease the shard count 127
now the Kinesis client library it can't initiate a resharding process but 128
it can adapt when it occurs so if the number of chance decreases or increases 129
it can adapt to that now Kinesis it will handle the mapping 130
of partition keys to that new shard count as well. It is possible to automate the 131
process of scaling Kinesis if you can implement an SDK application that can 132
monitor using cloudwatch and use a cloud watch alarm to initiate that 133
resharding process. During the resharding process a shard will go through a number 134
of different statuses, before resharding occurs the shard it will be in the open 135
state so we can read and write to that shard, after a retarding operation the 136
shard will then transition to the closed state and we can no longer add records 137
to that shard and any new records will be added to the new shards that have 138
been created, after our streams retention period has expired for example after one 139
day that parent will no longer be accessible for reading or writing and it 140
will not contain data, so it's very important that when you have a shard 141
that has been closed make sure that it doesn't still have data left in it if 142
it's if that data is important to you so this is how the process works so we 143
start off with a shard and it will be in the open state we can't read and we can 144
write to that shard not a problem then we go through the process of rewriting 145
by doing the update shard count command when that occurs two new shards will be 146
created and we've got their shard B and shard C and they will be in the open 147
state so that means that they will be accepting records from producers but our 148
original shard, shard a, will now be closed so we can no longer add any 149
records to shard a but we can still read records from shard a so when we go 150
through this process there may still be records inside shard a so those records 151
are important to you you need to continue 152
to read from that close parent until all those shards have been exhausted from 153
that stream and finally after the retention period be it one day or a 154
maximum of seven days whatever you have defined for your stream the Shard a there 155
will actually no longer be accessible and you will have lost any data that was 156
in Shard a so again that data is important to you to make sure that you 157
get it before the retention period expires. Now when we implement Kinesis 158
data streams we need to first off understand what capacity we need and how 159
many shards we need for this Kinesis stream so there is a very quick process 160
that we can go through to identify the number of shards that we need so first 161
off we need to identify what our average record size is rounded to the nearest 162
one kilobyte next we need to identify how many records per second will be 163
going into our stream and then from there we can work out our incoming right 164
bandwidth in kilobytes per second and that will be the size or the average 165
size of those records rounded up to the nearest kilobyte multiplied by the 166
number of Records per second next we need to identify the number of Kinesis 167
applications that will concurrently be consuming data from the output of that 168
stream and so we're going to call that numbers here and from there we can 169
identify our read bandwidth and that will simply be the number of these 170
consumer applications multiplied by our write bandwidth now if we remember from 171
our previous slides the capacity of a shard in write will be one megabyte per 172
second and in read will be two megabytes per second so the number of shards that 173
we require will be the maximum of either our write bandwidth divided by 1000 174
being 1,000 kilobytes or a megabyte or the read bandwidth divided by 2,000 175
Kinesis video streams is another offering in the Kinesis service and that 176
streams live video from devices to the AWS cloud and not only video it can also 177
stream non video time serialized data to the AWS cloud as well. It enables 178
data encryption at rest but it has a limitation in that he only supports MKV 179
video format. Kinesis video streams are made up of fragments and those fragments 180
are self-contained sequences of video frames and we use a putmedia command to 181
write those fragments to the video stream the Kinesis service will assign a 182
fragment number and timestamps to our fragments for sequential organizing of 183
our fragments in that video stream and then we can use a get media command to 184
read those fragments from the stream at a specific start selector location which 185
we supply and it will then return the fragments along with the timing 186
information for those fragments so here we can see there are a couple of 187
applications that we can use it with first off there we've got a mobile 188
device that can be streaming video data to Amazon Kinesis and that video stream 189
could be consumed by ec2 instances and they could be doing that on a continuous 190
basis or they could be doing it on a batch basis so the second example there 191
we've got a CCTV camera on the public internet and again it's streaming data 192
it's producing data to the AWS Kinesis video stream and from there it is being 193
pushed out to the public Internet with a batch or a continuous consumer 194
application that could be running on someone's mobile device it could be 195
running on someone's server but it is pushed out through to the public 196
Internet. We have a number of tools that are available for Kinesis video streams 197
the first one being the producer client which is for Java and Android 198
applications we also have the producer library which is include 199
within that Jabar an Android producer client we have the C++ producer library 200
and we also have a parser library for reading MKV data from these video 201
streams also the AWS software development kits have a Kinesis video 202
media class that can be used for reading the MKV data from a videos video stream 203
but it can only be used for in getting media it cannot be used for putting 204
media into a stream 205
Kinesis data fire hose allows you to deliver your streaming data to a number 206
of different destinations directly including Amazon s3, Redshift 207
Elasticsearch and Splunk so there is no need for you to write applications 208
consumer applications to do that for you or to manage the resources that are 209
needed for that you can simply set up a data file house and it will direct that 210
data through to that destination for you now that data can also be transformed 211
before it's delivered using lambda code that you supplied and if you do that you 212
can also backup the original source data to an Amazon s3 bucket the incoming 213
stream of data will be buffered before delivery and you can define that in 214
megabytes with buffer size or in seconds of data with buffer interval and 215
firehose also supports server-side encryption 216
so here we can see we're going to be getting a stream of records coming into 217
our firehose out Kinesis firehose delivery stream and then we can apply a 218
lander function to transform those records and that will go to our 219
destination bucket and if we apply a transformation we can also save our 220
original source records to a backup s3 bucket or if we're not transforming we 221
can just send that data directly to that destination bucket and remember after 222
being transformed it doesn't have to go to Amazon s3 can go to another service 223
it can go to redshift or elasticsearch it doesn't necessarily have to go to s3 224
but the backup will go to Amazon s3 225
Kinesis data analytics will process and analyze streaming data using standard 226
SQL statements which is great because all of a sudden you can have analytics 227
of live data and you can report that instantaneously it supports a number of 228
different destinations including Amazon Kinesis data fire hose so it can go to 229
s3 redshift elasticsearch but it also can go to an AWS lambda function and it 230
also can go to another Kinesis data stream as a destination so here we can 231
see we've got our inputs on the left there to our Amazon Kinesis analytics 232
application we have our streaming inputs which can come from a Kinesis stream or 233
from a firehose delivery stream and we can also have reference data they're 234
located in an s3 bucket and so our application code which will be our SQL 235
statements will process that and then deliver that out to an application 236
output stream and it will also have an application error stream and those will 237
be inputted to our application output which will be either another Amazon 238
Kinesis stream or it can also be a firehose delivery stream which will then 239
deliver that out to another destination such as an Amazon s3 bucket or redshift 240
or elasticsearch and so that brings us to the end of quite a big lecture and I 241
hope you've learned a lot and I look forward to seeing you in the next one