Hey, welcome back.
So in this lesson, we are going to start building our puzzle
and we're going to be looking, specifically, at
how we identify services for stream processing in Azure.
So in this lesson,
we're going to be talking about what stream processing is.
We're going to be talking about how stream processing
is typically used in Azure.
And then, we're going to take a look at
some common stream processing solutions.
Now, much like the last section, we are only going to focus
on solutions that you would find on the DP-203,
and we're going to attempt to focus the majority of our time
on the services that you're most likely to heavily see
on the DP-203.
So just keep that in mind as we move forward.
So from the last section, we talked a lot about batch
and you remember the discussion about the pizza
and the waterfall, or the stream here
and what that looks like.
We talked about batch
being for data that's not required immediately.
It's used for large transformations
or can handle large transformations easier.
And then it was that lower cost, larger data.
Well with stream, it's kind of the opposite.
When we talk about stream, we're talking about,
we need data now, we need those insights immediately,
and that is worth spending more time processing,
spending more money to get those processes done faster,
and dealing with some of the complexities
that you see with stream.
So, picture that stream over here on the right,
and remember, streaming is unbounded data,
which means there's just a lot of unknowns.
We don't necessarily know when that data's going to end.
Most likely, we're not exactly sure
about the size of the data coming in.
So we have to be able to deal with some fluctuations there.
And we have more complexity because as that data moves,
if we have issues with transformations
or issues with our pipelines, that data is still flowing.
And so we have to figure out strategies
to be able to handle that missing data
and catch up with where the stream is now.
So all of this comes down to streaming is more complex
and more expensive.
So where do we see streaming most likely used?
Well, first is in fraud detection.
If I am logging into my bank,
it is very important
that that bank be able to look in real-time
to figure out if I am a high risk for fraud.
So if that's the case, they need to know immediately
to be able to take steps to fix it.
You also see streaming used in stock trading.
Stock trading is something that is extremely fast
and you need to be able to make decisions
and see data immediately.
So stock trading is something
that you would see streaming data in.
Customer activity, so think Amazon here or Netflix.
If I'm watching shows,
we want to be able to recommend things to me.
Or if I'm browsing in Amazon,
we want to be able to show me products
that I might see immediately
and that's also going to be true of Walmart,
and so think customer activity.
Ride-sharing. Ride-sharing is another place
that streaming data is used.
If I log on to my phone for Uber and I request an Uber,
I need to be able to see all of the cars that are around me
and they need to see that I am a potential customer.
So there's a lot of streaming data that's used there.
And then also with marketing, as we look at marketing,
being able to identify potential customers immediately
and/or change the experience of someone
as they come on to a website, for instance,
is something that you would use streaming data as well.
So anything that you need real-time insights
or you need to be able to change things on the fly
for customer experience,
that's where you're going to see streaming.
So now, let's talk about some challenges.
I mentioned the time issue.
So the first is time windows.
We're going to be looking at how do we evaluate
and get insights on the data as it passes.
We can't look at an indefinite period of time,
we need to build windows and we'll talk in this section
about different types of windows and what you can do,
but we have to be able to frame our data
in some sort of window
so that we can get things like summations, or averages,
or other insights that I might want to see on a report.
Next, missing data.
So when we lose that data, what do we do to catch up?
So we'll be talking about that in this section as well
and some different opportunities.
And then transformation challenges.
So, think as we move through with our data,
we have to do transformation on that data.
We have to clean it up.
We have to clean up the sources or convert things.
Well, that cleanup has to happen in near real-time
because we are getting data behind it.
And so we have some transformation challenges
that you don't see in batch
that we have to consider when we look at streaming.
So we'll talk about that as well.
So let's talk about now where streaming lives
when we talk about Azure.
So on the left here, you're typically going to start off
with a Blob source or a Data Lake,
something that the data is coming from,
it might also be like an Event Hub,
but some form of input.
That then takes us into our services.
And the first is Databricks.
Databricks is definitely a streaming solution
and is a frequently used streaming solution,
when we start talking about large data sources
or when we're talking about machine learning--
that's another place you'll see
Databricks used for streaming.
HDInsight. A good way to think about HDInsight
is a control panel with a thousand buttons.
There's a ton of things that you can do in there
but it's very complex to configure.
And so it's not as well used or utilized in Azure,
doesn't mean that it's not good,
it just means it's more complex,
and so it's not as used
because it takes a lot of configuration.
And then Azure Stream Analytics, or ASA,
as you'll see it quite frequently,
because it's very long to say
Azure Stream Analytics all the time.
So ASA is the most common service
that you'll see streaming for, especially on the DP-203.
So we're going to spend the bulk of our time
in Azure Stream Analytics.
Now, the concepts that we talk about
will carry through into other services as well,
but this is where we're going to spend the bulk of our time.
So we take that streaming data
and we pass it through Databricks
or Stream Analytics or HDInsight,
and then we're going to push it out,
and it's going to be pushed out into
either a reporting tool like Power BI
or pushed back into a Data Lake
to have something else happen,
or it might be pushed through an Event Hub
to have another action taken on it or even machine learning.
But that's what happens in this step.
So, Stream Analytics, Databricks,
the streaming service is really
it serves the role of Data Factory.
So it kind of takes that data
and it moves it through the system.
And then it also combines that with transformation as well.
So you have both of those things kind of bundled together
in a pipeline that's going to move your data
and clean it and do some other processing
so that it's ready to go on to the next step.
All right, so in review,
we have talked about stream processing.
You need to make sure that you keep in mind
the characteristics of batch versus stream.
Examples and challenges.
So, when would we stream versus when would we batch?
If you understand those challenges and
you understand the common examples that we're going to see,
you'll do much better when thinking about new projects
and trying to decide which you should use, or defend
which you should use to a stakeholder.
And then finally, Azure technology choices.
So, which services are ideal for stream processing
and which would you see on the DP-203?
Keeping that in mind, we'll also help you maybe knock off
a couple of answers that wouldn't make any sense
as you look through questions on the exam.
All right, that's it for this lesson,
I'll see you in the next.