Hey, what's up, Gurus?
In this lesson, we are going to talk about processing data
by using Spark Structured Streaming.
Specifically, we're going to take a look
at a high-level overview
of Apache Spark Structured Streaming.
And when I say high level, I mean high level.
We could have an entire course on Spark,
but we don't have that much time
nor do we need to go to that much depth.
So we're going to cover just the basics
so that you understand how it functions.
That includes input streams and tables,
and a little bit about how that works.
And then, of course, we're going to review
some simple code scripts to process data
so that you understand how the Apache Spark module
in Data Factory works.
All right, with that,
let's get started with the high-level overview.
So here we have an overview
of the Spark Structured Streaming process.
And we're going to break this down
kind of section by section
so that you understand what's happening.
So our first section here is our input.
We have our TCP socket at the top there
that you could use as a debugging source,
or you have your main options,
which are your message source,
which would be like streaming through Kafka or Event Hubs.
Or you could have a watcher set out
looking at your data lake
or a storage blob or something like that,
and it would be watching for new files.
As those files appear, it would input them
into your Spark Structured Streaming.
Now the streaming process is where the windowing occurs
and normal transformations.
So when we talk about Stream Analytics,
it's very similar to that.
You're going to have your processing
where you set up a window of time that you're looking at,
and as that data flows through the window,
you're doing transformations on that data.
Once you finish that, you're just going to have a sink
and a sink is just an output location for the data.
So you're going to send your data once it's been processed
to a Power BI dashboard or a database
or the data lake store, or something like that.
And then we also, just like we had a debug source,
or an input, we also have a debug sink
that you can use as well if you need to do debugging.
So that's the basics of how structured streaming works.
Essentially, It's just input, query, output.
So keep that in mind.
Now let's talk a little bit about tables
and how the data actually gets processed.
So we actually have 2 different tables.
We have an input table, we run queries,
and then we have our output or our results table.
So our data stream here on the left
would represent our input table.
And we're going to take that data
and we are going to do processing and send that through
to an unbounded or results table.
So now just a couple of things to keep in mind
as we move forward with this.
The first is taking a look at this unbounded table concept.
So an unbounded table is one that doesn't have a boundary,
which means it's going to continuously grow.
And so this is an appending table,
or it's being continuously appended or added to.
So each one of these purple lines coming down
is adding to this unbounded table.
The next thing to keep in mind
is this concept of streaming in batch.
So Spark Structured Streaming
is definitely a streaming service.
However, you can see the lines coming off of this arrow.
We're going to take these individual packets,
and we're going to process them
almost like we would a batch request.
So it's kind of a mixture
of streaming in batch, if you will.
And then finally, incremental processing.
We definitely need to understand incremental processing.
And so this kind of goes to that concept
of this batch processing.
So we're going to take these incremental pieces
and we're going to process those in parallel.
And I'll show you this in the next slide,
but keep in mind those 3 concepts
as we talk about this unbounded table
that continuously grows.
All right, so as we look at this,
you need to understand that there's 3 different modes
for outputting our table,
and keep in mind the concept,
we have the input, we're running processing,
and then we're going to output it
so you can see your input, result, and output, same thing.
There's 3 different choices, though.
We have a complete mode or a complete rewrite.
And so in this mode,
we are going to completely rewrite the result table.
Second option is append mode.
So like we've talked about before,
append mode is only going to update the new
or the appended result table.
It's not going to rewrite everything.
It's just going to write since the last trigger,
so it's just going to continue to add.
And then we also have an update mode.
So this is pretty much the exact same thing
as the append mode, except for the concept of aggregations.
So it's only going to update the rows
that have changed since the last trigger,
whereas the append mode is just going to add new rows.
It's not looking at changing or not changing.
So if you had an update mode that didn't have any changes
or aggregations, it would be the exact same as append.
All right, those are the 3 different types
of table outputs.
And so now let's move over here and look at this chart.
So you can see we have time moving from left to right,
and there's a trigger every second.
So we have our window moving,
and every second we're going to have a trigger.
So it's going to look at that data,
batch that up as a one-second trigger,
and it's going to take that,
and it's going to pull all the input.
So everything for data up to one second, it's going to pull.
It's then going to run our query.
So whatever query language we have, it's going to run that.
Once it's done that, then it's going to give us our result.
So for that, we have our output.
So we get our result, then we push it through in the output.
And in this chart, we're looking at a complete mode table.
So what's going to happen then
is we're going to look at the 2nd second.
And what going to do is we're going to query data
up to 1 and 2 seconds.
So the input doubled in size
because we're completely rewriting that table.
Then we're going to run our query.
We're going to get our result for those 2,
and then we're going to spit our output.
Trigger 3 seconds, same thing.
We're going to pull all the data up to 3 seconds.
Then we're going to run our query, get our result,
and then spit out our output.
So when you look at Spark Structured Streaming,
that's what's happening.
But keep in mind that this table right here
is what that looks like.
So each one of those little arrows
is actually one of those blocks that we're looking at
as we move forward in time.
So to wrap this section up,
let's talk about what's happening.
So we have our input data stream. Data comes in,
and it moves into an input table.
Once it hits that input table,
it's then going to be pulled out,
and a query is going to be run on that data.
It's then going to be filtered into our results table.
And there's really a couple of different ways.
And there's 3, but we're only going to talk
about the complete and append,
because the update mode is very similar.
So in append mode, let's say that we have an IoT sensor,
which is a very common application for Spark.
And with this IoT sensor, we are looking at temperature.
Let's say we have a manufacturing line,
and we have a IoT sensor
that is going to detect the temperature of the machine,
and so it's going to help to spot any abnormalities.
With that temperature sensor,
we can just simply append that data into a results table.
So that's a fantastic use of append.
Now complete, one question you may be thinking
in your mind is, well, what happens
if my data goes from 1 to 10,000?
Am I eventually going to run out of memory?
And the answer is, yeah, you definitely will.
If you are simply continuing to do a complete mode,
over time as that grows bigger and bigger and bigger,
you're eventually going to run out of memory.
So a better use of complete mode is to look at windows.
So let's say we have a window,
and we are going to sum up all of our data.
And you'll see this a little in the script
that we do in a second,
but we're going to sum up all of our data
for a 5-second window,
and then we're going to write all of that.
Well, so that would be a good use of a complete mode.
It's really designed to be used with aggregations,
not just to continue to add up over and over and over
more and more data.
So keep that in mind for your complete mode,
generally with aggregations,
and keep in mind which you use the append mode for.
To kind of finish out our discussion,
I want to take a look at a script
that we could use in Data Factory
and kind of break down what's happening here.
So what we have is we have our input,
which is these first 2 blocks.
We're going to import the pieces
that we need to make this script function.
And then we are going to provide the input data.
So that's the input path.
That's the schema that we're going to use,
and then that is our streaming information as well.
So we have all of that defined as our input.
And so then we do work, and you can see that,
in this case, we are aggregating our data together
based upon time and temperature,
so this is probably like an IoT sensor
or something like that.
And then finally, we have our output here.
And so this is defining our output sink.
And then finally we are going to get going,
which means we're going to actually start the query.
So this is the command that you would use
to actually start the query.
So this is the Spark Activity
in Data Factory at a high level.
Now for the DP-203, what do you actually need to know?
Well, not a whole lot, as far as the code is concerned.
Basically, if you understand the blocks,
the input, the query, the output block,
and you could recognize kind of those blocks
and where they sit, then you're just fine.
I think it's wildly unlikely
that you're going to need to know individual functions
or things like that.
So really just being able to glance
and understand kind of what's happening at a high level
should be just fine for this specific slide.
So we've talked about quite a bit.
Let's sort of sum up what you need to know for the DP-203.
The first is Spark is for big data
or specialized applications.
That's really where you're going to see Spark.
Next, you're querying off of a table.
Don't forget about that unbounded table
where we're are pulling specific pieces of time
in almost like a batch process,
and we're going to pull out those data stream pieces,
run processing, and append them to that unbounded table.
So don't forget about that.
And then finally, it really comes down
to--still--input, query, output.
We define our input sources.
We define the work that we want to do in Spark,
and then we define where it goes.
So if you keep these 3 concepts in mind,
you should be good to go for the DP-203.
Again, don't spend a lot of time focusing on this
because it is not a major component of the DP-203,
but you do need to understand
the highlights of what Spark is.
Hope this has been helpful.
Let's jump on to the next lesson.