So in this lesson, we are going to talk
about a very important concept in data engineering.
We are going to talk about
how we process one or more partitions,
which is known as repartitioning.
Specifically in this lesson,
we are going to talk about
how we understand partitions and repartitions,
and then we're also going to talk about
how we use repartitions in Azure Stream Analytics.
And the first little tidbit is,
when you think of partitions, think of buckets
and--there you go--cute dog in a bucket.
(Brian chuckles) So, as you think of partitions,
think about just building buckets
and putting all your stuff into various buckets.
So the basics of partitioning is just simply,
we divide our data into subsets based on a partition key.
So we're going to divide our data into those different
buckets, based on some sort of metric that we define,
and let's talk about why that matters.
It matters because those subsets,
that data in a bucket, it makes searching faster.
I'll give you a little side-story here.
So, my son loves LEGOs,
and he took all of his LEGOs one day
and broke them into pieces, and it looked kind of like this.
Everything was mixed up and together,
and so I helped him put all of them back together
and build all of his sets again.
In order to do that, I partitioned the LEGOs
into various colors, red, yellow, blue,
and that really helped because it made searching
for the different pieces a lot faster.
Same thing is true in data.
And we use partition keys
to help determine what buckets the data goes into.
So think about this for partition keys.
Let's say that we have a database that has customer name,
order number, order date, order country, and total sale.
What would be a good key for this data?
Well, the first thing that we want to look at is static.
We don't want something that changes.
It's not going to help us
if we have something that changes frequently.
So another way to think about static
(using data engineering terminology)
would be reasonably consistent
and bounded number of records.
So thinking about something that is bounded,
it has a beginning and an ending
and it's going to be reasonably consistent
in that it's not changing.
The other thing we want to look at is high cardinality,
and when we talk about high cardinality,
this just means a big range.
We want something
that's going to have a lot of different values.
For instance, if I chose order country as my partition,
but everyone purchases within the United States,
that's a terrible key,
because we're going to have one giant bucket
with all the data in it and our partitioning was useless.
So when we look at our partition keys
and we look at this data here,
there's a couple of different choices,
and the partition key that you would choose,
would be based upon how you want to divide up the data.
For instance, you might look at order date.
If you have a reasonably smooth flow or even flow
of orders per day, you could choose order date
and use that as a partition key, or you might,
and I know I mentioned before
that you might not want to choose order country,
but if you are in a global sales
and all of your orders come from across the world,
order country might be a good solution for you,
if there's a reasonable distribution where it's pretty even
and there's a lot of different countries,
again, that high cardinality.
So when we look at partition keys,
it's important that we kind of think about
how we're going to make those buckets work
and that we think about what we want to use
to help find the data that we're looking for
within those buckets, i.e., our partition key.
And keep in mind, this is the basics of partitioning.
We could talk for several sections just on partitioning
and the different ways that you can do it,
but I want to give you an overview of some things
that you can do and how it works at a high level.
So when we talk about Stream Analytics,
we're going to talk about embarrassingly parallel jobs.
Yes, that is a real thing.
Microsoft defines embarrassingly parallel jobs
as the most scalable scenarios in Azure Stream Analytics,
i.e., the easy button over here on the right.
So, embarrassingly parallel jobs connect one partition
of the input, to one instance of the output.
That is the most scalable scenario
for Azure Stream Analytics.
You do need to understand that.
Embarrassingly parallel jobs, 1 input to 1 output.
So here's our key.
We have our Azure Stream Analytics query
and we have an output.
So if we had multiple partitions,
we would just have multiple sets of this.
So when we look at the script for this,
you can see that we have our select count as count,
just selecting all of our data from our input,
and then this is the part that's important for partitioning.
We're going to partition by partition ID,
which is whatever our partition key is,
and then below that you can see
that we simply tack on the standard hopping window
or whatever our window is going to be.
So that's our group by statement.
So this is pretty typical.
We have our select,
then we have our from,
then we have our partition by partition key,
and then we have our group by statement.
And so that is the code that we would use
to run an Azure Stream Analytics query utilizing partitions.
So what happens when it's not embarrassingly parallel
and what would that even be?
So if it was not embarrassingly parallel,
we would have a mismatched partition count, possibly.
In other words, our input partition counts
don't match our output partition counts.
Another scenario would be,
if we are moving into something that doesn't have partitions
so our input has partitions
but output does not have partitions.
So that would not be embarrassingly parallel
because it's not 1 input to 1 output with partitions.
So if that's the case,
what you would do is you would run additional query steps
and you would utilize something known as shuffling,
and for that, let's go ahead and jump over
and take a look at shuffling.
So this is repartitioning.
We've now transitioned from partitioning
into repartitioning, or reshuffling.
So this is for those scenarios
that aren't embarrassingly parallel.
So what we can do is,
we can process all of the partitions independently.
So in order to do repartitioning,
we have basically 2 choices.
The first choice is,
we create a separate Stream Analytics job,
that does all the repartitioning,
or we create multiple steps in one job,
and we do all of the repartitioning first, or the shuffling,
and then we do everything else
that we actually wanted to do.
So those are the 2 different choices that you have.
So now let's take a look at a script,
so we can see what this looks like.
So, we have here 2 different steps
and what we're doing is we're going to repartition our data
and then we're going to do something with it.
So basically what we do is,
we're going to create a repartitioned input
and we're going to select everything,
and then we are going to create a partition system,
that's going to partition by device ID in this case,
and that partition by could be whatever you want--
could be that customer number, could be a device ID,
whatever you want.
So once that first step is done,
we have repartitioned our data,
and then what we're going to do is,
we're going to select our partition
and then we're going to perform some simple averages.
So we're going to do an average
and then we're going to move that into our output,
and we're going to use that repartitioned input
that we just did,
that reshuffled and created those partitions,
as our source data.
And then at the very bottom,
you can see that we've added in a hopping window,
so that Azure Stream Analytics has a good idea
of how it should look at the data
as it's coming through the stream.
So this is a pretty simple script,
but it gives you a good idea of what that 2-step process
of reshuffling and then doing something, looks like.
You can also use this to join multiple streams
if you need to.
So in this case what we have,
is we have our step1 that we're creating
and then we're also creating a step2,
and those are going to be from 2 different input sources.
And then what we can do is,
we can select our step1 and step2,
those 2 different partitions that we've created,
and we can join those together into 1 output.
So that is what you see here.
So just a couple of simple scripts that we can use
to give you an idea of what repartitioning looks like
in Stream Analytics.
All right, so in review.
First, you need to know
what an embarrassingly parallel job is.
Basically, the input and output partitions? They match.
Those numbers match.
Repartitioning is just reshuffling.
So that's what we do with jobs
that aren't embarrassingly parallel,
and you can see how we do that with those sample scripts.
And you need to know that this affects the query section
of Azure Stream Analytics.
That is where all of this would come into play.
All right, so that is it for this lesson.
I'll see you in the next one.