So continuing on from the last lesson,
in this lesson we are going to pick up,
talk a little bit more about partitions,
and, specifically, talk about what happens
when things go wrong.
So what we're going to be looking at, specifically,
is job recovery from failure.
So when Azure Stream Analytics has a failure,
what happens and how do we recover?
And then we're going to talk about
something called the replay catch-up time,
and we're going to look at how we determine what that is.
And to quote the famous Robert Burns,
"the best-laid schemes o' mice an' men--"
they sometimes co-arise.
So it's a good thing to make some plans.
So what does this look like in Azure Stream Analytics?
Well, we talked about dividing our data into subsets
based upon partitions and partition key (last lesson).
And so what we have is we have a job that is started
in Azure Stream Analytics.
Now, part of that parallel processing,
Azure Stream Analytics is going to break up your work
into nodes automatically.
And it's going to start working on
all of your query in parallel.
Now, sometimes a node can fail.
Basically something bad happens,
a failure occurs in a node,
so what you need to understand
is what Azure Stream Analytics does next.
Well, it is go going to have an automatic recovery for you,
which is a great thing.
It's going to restore from your last available checkpoint.
And it does this through something known as
stateful query logic.
Basically what happens is, when you do
a windowed aggregate or a temporal join
or a function, Azure Stream Analytics is going to
keep information when that job is running,
and it's going to store up to 7 days.
Now that's a pretty long time window,
but if you have longer than that,
you need to keep that in mind.
It stores 7 days worth of data.
And so what it's going to look at
is your last available checkpoint.
And then it's going to create a new, healthy node
and it's going to resume the work
from that last available checkpoint.
So, how do we make that replay process faster?
Because you have some say in how fast
the replay process takes, which is that restart
and catch-up from our last available checkpoint.
Well, in order to do that, we are going to estimate
the replay catch-up time with this formula.
So we're going to take a look at our input event rate,
then we're going to take a look at our gap rate,
and we're going to divide that
by the number of processing partitions.
Now, typically this gap length--and the gap length
is your replay length--the time it takes it
to catch up to where it was,
that is going to typically be a couple of minutes.
So now that you've seen the formula,
let's take a look at how we calculate
that replay catch-up time in practice.
So we're going to look at our window size,
so figure out how big that is.
Then we're going to run sample data through Event Hub.
Once you click Start, you're going to measure how long
it takes until that output first appears.
This is going to be our catch-up time,
it's essentially what you're doing.
So you're doing this formula, through this process.
Now, to fix it--to make it run faster
if it's not running fast enough--
well, it comes down to streaming units.
So if you spend more money
and you have more streaming units available, it's
going to process faster, which means you're going to have
a smaller gap length.
Also, you want to take a look at partitions,
because if you remember from that formula,
the bottom line was the number of processing partitions.
So you can also affect the restoration time
by looking at the number of partitions that you have.
Those are the 2 things that you have control over
to make your replay process faster
in the event of a failure.
So to summarize all this,
basically, if something happens in Azure Stream Analytics
and a node fails, failure is going to be noticed
automatically by Azure, a new node is going to be created,
and then it's going to replay
from your last available checkpoint,
which is stored up to 7 days
worth of streaming data.
To decrease your downtime, you're going to take a look
at properly allocating your streaming units,
which might be increasing that,
and taking a look at your partitions and how many you have.
If you've done those 2 things, you should be good to go.
And with that, we've finished our lesson,
and I'll see you in the next one.