Hey, what's up Gurus?
So in this lesson, we are going to be talking about time,
and specifically, how do we process time series data
in Azure Stream Analytics?
So in this lesson, we're going to talk about what time is,
some ways that we can measure it,
and then we're going to talk about some key concepts
that you need to know for the 203.
This is going to be a fairly short lesson
so let's just dive right in and get started.
To start off with, we need to talk about time
and specifically, how do we calculate or measure time?
So there's a couple of ways that we can do this.
The first way is what's called the event time.
Now, if we're looking at an event time,
this would be when an event reaches the source.
So the arrival.
And a good way to think about this
or I'll give you 2 different examples.
The first way is if I'm writing a letter.
So when I write the letter, I'm actively creating the event.
That would be the event time while I'm writing the letter.
Or for instance, if I want to go to the movies,
when I go up and I purchase my ticket,
that's the event time, when that physical ticket
is first purchased at the counter.
Now, there's another way that we can look at time.
We can also look at processing time.
So this is when the event
is actually observed by the system.
For instance, when I buy that ticket at the movie counter,
there is a lag from when I physically buy the ticket,
and when it goes through to cyberspace and it gets stored
in the database in Azure or some sort of cloud, right?
That point when it's stored in the cloud,
that's actually when it's observed by this system.
And so you got to think about this
as data's coming across in the streaming fashion.
Well, when did the event actually happen?
Did it happen when I bought the ticket
or when I wrote the letter,
or did it happen when the letter was delivered,
or when the ticket was actually recorded in the database
in that processing time and event time?
Lastly, we have the concept of watermarks.
Watermarks are simply an arbitrary point in time,
that simply shows what point the event
has been ingressed or pulled into Azure Stream Analytics.
That's all a watermark is, but it's a very important concept
as we think about how we're looking at data
in Azure Stream Analytics.
So kind of putting everything in order,
we have our watermark
that is determining when the event arrived.
And so this is going to be on the input side of things
as we figure out when the event actually arrived.
And then we have our windowing concept
that is really more involved with querying.
So watermark when it arrived, and then from there,
we're going to take that window,
and we're going to look at a bracket of time
or a period of time, based upon that watermark
to determine what sorts of transformations
that we want to do to the data
as it flows through Azure Stream Analytics.
So in an ideal world, that's all you need.
However, it's not an ideal world, right?
We have early arriving events,
late arriving events, missed events.
And so because of this,
we need to look at something called tolerance.
And we'll talk more about this
as we go over the next couple of lessons,
but essentially what tolerance is,
is just taking a look at your watermark and saying,
okay, well how long are we going to wait for an event?
So if I have an event that arrives
in Azure Stream Analytics, what's the tolerance
for a late-arriving or an early-arriving event?
And so we'll talk more about those strategies
in the next couple of lessons.
What's really important for you in this lesson,
is just simply to understand the 3 key concepts.
What are the definitions for event,
time, processing time, and watermarks?
And then also understand that watermarking
is the key to unlocking time issues.
So the watermark is extremely important.
So we'll be talking more about that
and you're going to need to know the details for the 203.
But if you understand the concepts
of event processing and watermarks,
you should be good to go on to the next lesson.
I'll see you there.