In this lesson,
we are going to talk about "Tuning Shuffle Partitions".
And I'm going to tell you right off the bat,
I'm having a little bit of a struggle,
because when I think of tuning shuffle partitions,
I'm kind of torn between shuffling cards
and shuffling as a dance move.
So, I don't know.
I threw them both on there.
I'll let you figure it out.
In this lesson, though,
we're actually going to talk about Spark shuffle.
And I'm going to start by talking to you about
what Spark shuffle is at a high level,
because again, I just want to focus on the DP-203 here,
and not get lost in the weeds.
Then I'm going to talk to you about
how do we set a shuffle partition size.
Going to be a relatively short lesson,
but let's go ahead and jump on in.
So what is a Spark shuffle?
Well, at a core, it's a mechanism for repartitioning data.
And we've talked about repartitioning several times
in this course.
When we talk about Spark shuffling,
it looks like this over here on the right.
So, we've talked about the driver, cluster manager,
and then the executors.
And then within those executors,
we have individual tasks, right?
That is the standard setup for Spark.
Now, what if we wanted to copy or move data?
What does that look like?
Well, we introduce a shuffle step.
And basically what this is going to do
is it's going to take the executor data from those tasks,
-: -and those are going to be in different partitions--
and it's going to pull all of those together
into a shuffle step, or maybe multiple shuffle steps.
And then it's going to reduce that data down.
So essentially, we're moving it here.
And we would go from maybe 5 partitions
down to 2 partitions.
And that happens in that shuffle and reducer step.
So, that's what's happening behind the scenes
when we do a Spark shuffle.
And what we're doing is,
essentially, we're grouping data differently
across partitions, okay?
This can be beneficial or it can be harmful.
Now, it can be beneficial because,
hey, we're reducing things down. When you reduce things
down, generally you have less storage,
and it's generally easier and faster to run queries.
It can also be harmful, however,
because this shuffle step can be a very costly step
if you don't do it correctly.
And so, when we run shuffling,
anytime you do a transformation operation in Spark,
-: -so a groupBy, a join, a union, that kind of stuff--
you are going to initiate a shuffle behind the scenes.
And like I said before, shuffling is an expensive operation,
so it's important that we set it correctly.
And what makes it expensive is it's requiring
disk I/O and network I/O when we do this shuffle.
Because what it's doing is it's storing data
and it's moving data.
And so that can be a very expensive operation
when we talk about Databricks,
which is generally done on massive, big datasets.
And it also involves data serialization
and deserialization.
And what that means is,
we are just transforming data and the format of it
into a series of bytes,
and then pulling it back out of bytes.
And so basically, it's involving a lot of transformation
and it's involving a lot of movement and storage.
All of that is expensive.
All right. So, how do we set
the shuffle partition size then?
Well, the challenge with this is
finding the right shuffle partition number.
And it actually gets even a little more complicated,
because we have multiple steps, sometimes, of shuffle.
And when we run a query,
and if we are doing multiple query steps,
the shuffle partition size, it may change
or it may need to change in order to have it optimized.
The answer to this is using AQE,
or adaptive query execution.
This is something that you can do within Databricks.
And basically, you set the shuffle partition number,
and then as you move forward,
AQE is going to automatically adjust
your shuffle partition number at each stage of your query.
And this is going to be based
on the size of your shuffle output.
So basically, as your data grows or shrinks
over those different stages,
your AQE is going to be adjusting
your shuffle partition number
to make sure that you have the right task size.
Now, a critical part to this is
you need to set the initial shuffle partition number.
And you do that with a script just like this one here.
So basically, you are setting your shuffle partitions.
And in this case, we'd be setting it to 100.
So you can see that in that very top line there.
Now, that's it for this lesson.
Let me give you just a few parting key points to remember.
First, you need to remember that Spark shuffle
can be very costly if you don't optimize correctly.
AQE can assist with this.
You need to use adaptive query execution.
You also need to set that initial shuffle number.
And don't forget, we set that spark number
with a script just like this.
And so, for the DP-203, this is as far as we need to go.
You need to remember that Spark shuffle exists.
You need to remember that it can be very costly
if you don't set it correctly.
And AQE, or adaptive query execution, can assist with that
when you're looking at Databricks.
So for the DP-203, that's all you need to know.
So I'm actually going to stop here,
and we're going to jump on to the next lesson.
All right. I'll see you there.