So in this lesson,
we are going to talk about how to handle late-arriving data
in Azure Stream Analytics.
And specifically what we're going to be talking about
is how we define late data.
We're going to take a look at the consequences of tolerance,
and then we're also going to talk about how we adjust
to late data.
Before we do that, there's a few definitions
that you need to understand.
So we're going to start off by talking about watermarks.
There's 2 different ways to define a watermark.
The first way is to define watermarks
when an event is arriving.
So let's jump back in to that movie theater
and I'm getting ready to buy a movie ticket.
When I purchase that movie ticket, I create an event.
That event will then be moving to Azure Stream Analytics
or arriving into an Azure Stream Analytics.
So this is the incoming scenario.
So my watermark would be defined
by the largest event time minus my out-of-order tolerance.
And I set my out-of-order tolerance.
So we're going to take a look at the largest event time
and we're going to subtract the out-of-order tolerance
to get our incoming watermark time.
The other way to define watermarks
is if there's no incoming event, and if that's the case,
we take a look at our current estimated arrival time
and we subtract out our late arrival tolerance.
And so my estimated arrival time
is just going to be whenever the last event occurs to now.
It's going to be that amount of time.
That's how we define watermarks
when there's no incoming event.
And so from there, we can get to our late data.
So, late is defined as taking a look at my arrival time,
subtracting out my event time,
and then comparing that to my tolerance window.
So an example of that would be
if I, at 12 o'clock, buy a movie ticket,
that is my event time. 12 o'clock.
My arrival time is 12:04.
So if it takes 4 minutes from when I bought the ticket
to get into Azure Stream Analytics,
that's going to be this: 12:04 minus 12:00.
So, that's going to be 4 minutes.
And if I had a 5-minute tolerance window set,
I'm going to compare the 2 of those.
5 is greater than 4.
So the event is not late.
If on the other hand, I had purchased the ticket
at 12 o'clock and it didn't get there until 12:06?
6 is greater than 5, and it would be late.
So that's how we define late.
These 3 definitions you do need to understand,
so make sure you that know those formulas
and have a decent handling of how you define watermarks
and how you define late data.
So now let's take a look at that tolerance window
that I've been talking about and see what the consequences
of tolerance are.
So the first consequence is delayed outputs.
If I have a huge tolerance window, it's
going to take forever for that tolerance window to close,
which means I don't get any data.
I don't get any results that I can look at or compare,
which means I'm going to get bad reports.
It also could break processes down the stream.
The second is if I have a tolerance window
that's set too short, I can miss critical events
because I don't have the proper tolerance set.
So if an event is one minute late, for instance,
I might not even look at it,
and I might miss some very critical data, which again,
can lead to broken processes or other missed data.
So looking at this in action.
So let's say that I have a time window down here
at the bottom.
This is my event time going from 12:00 to 12:30.
And then my processing time going from 12:02 to 12:20, okay?
So let's say that I have my first event come in,
and it's just event number 4.
The numbers in these little bubbles mean absolutely nothing.
It's just, I have something come in that's 4. All right?
So it is processed by 12:02.
It arrives at 12:00 and it's processed by 12:02.
The next event comes in at 12:05,
and you can see see there that it's processed at 12:10.
Still within the tolerance window,
which is this large blue box, okay?
So I have my tolerance window,
which would be this large blue box,
and I have my events.
So, so far my event total would be 10: six + 4.
Now let's say that I have an event come in at 12:08,
but it's not processed until, like, 12:25.
Well, that's way late
and it's not going to be in the event window.
So when you look at the sum total
of the data in that window, I'm going to get 10.
I missed my 8 up there at the top.
Moving on, in my next window,
let's just say that I had events
8, 4, 3, and 9,
and all of those are processed in a relatively fast fashion.
You can see here that event 8
is actually processed slower than event 4.
Event 3 is processed much slower than events
8 or 4, but it still just squeaks in
with that window.
And then event 9 is processed pretty close to on time.
12:18 for the event and then 12:18 for the processing time,
so it's processed nearly instantaneously.
All 4 of those events happen within that tolerance window
that we've set.
And so I get a total of 24.
And so then in our third scenario,
let's say that we have events 5 and 7,
and that's going to lead us to 12.
That looks pretty good.
But let's say that we were really concerned
about missing data, and so we increase our tolerance window.
If we do that, we are going to have
a massive tolerance window, and my 7 and 5
are well within the range of the tolerance window.
And in theory, this would also increase to the top as well,
but I can't really show you that
and still have everything fit on the screen.
But you would have then 7 and 5,
and then a massive window.
Now, that massive window may look fantastic,
but the problem is,
let's jump back into our movie theater scenario.
Let's say that we have a theater that only seats 10 people.
I come in and I buy 5 tickets.
This first event here.
Right after me, someone comes in and buys 7 tickets.
So now we've sold 12 tickets
for a theater that only seats 10 people.
The problem is our tolerance window,
this giant, light-blue box is so big
that I don't get data coming out.
And so I can sell 5 seats
and then 7 seats before the system updates,
because it's still waiting on this tolerance window.
So it's really important when we look at tolerance
that we find that sweet spot.
That we don't do what's over here in this dark blue,
and make our tolerance window so short
that we miss critical events.
And that we also don't do what's over here
in this light blue and make a tolerance window
that's so big that we also break processes
or we sell more tickets than we have
because our tolerance window isn't the right size.
So it's important to find that sweet spot.
So now, we're going to take a look
at how we set tolerance in action.
And this may be the shortest demo
in the DP-203, and may be in Azure, but let's take a look.
All right, so we are in Stream Analytics
and I have simply scrolled down
and clicked on Event Ordering here.
And you can see that I have 2 different windows.
I can choose my late-arriving data events,
and I just come in here
and I choose whatever I want for that.
And then I can choose my out-of-order events.
So I can look at the tolerance for that.
And I can just simply set how late
I'm willing to look at late-arriving data
or out-of-order data for my tolerance.
And then finally, I can choose
whether I want to adjust or drop events.
Basically if I drop an event, it's just going to delete it.
If I adjust it, it's going to keep that event
and just change the timestamp.
So you can look at how you want to handle that.
Click on Save, and that's it.
That Stream Analytics job is done
and you are ready to go.
So it's extremely simple to set.
It's really more important to think about that tolerance
and understand what you're actually setting
when you do that.
All right, so in review, we have 3 formulas to look at.
We have our 2 watermark formulas.
The first being our incoming event
which is the largest event minus out-of-order tolerance.
And then our second is if there's no incoming event,
which is just our current estimated arrival time
minus our late arrival tolerance.
And then keep in mind as well, we have our late definition,
which is just our arrival time minus our event time,
and then comparing that to our tolerance window.
We also have the consequences of tolerance.
So make sure that you use metrics to help you set tolerance.
And what I mean by that
is as you start running your Stream Analytics jobs,
you'll be able to see how many events
are late or out-of-order.
And based upon that data,
you can use those metrics
to help you set that tolerance window.
Finally, watermarks and tolerance are extremely important.
Make sure that you don't gloss over that
and that when you make Stream Analytics jobs,
you have a strategy for watermarks and tolerance,
because they can absolutely ruin your day
if you don't have them set appropriately.
All right, that's it.
We are done with late-arriving data.
With that, I'll see you in the next video.