Welcome back.
So in this lesson,
I'm going to talk to you about "Compacting Small Files".
And we do that with a tool known as auto optimization.
So we're going to talk a little bit
about auto optimization, and what that actually means.
And then I'm going to show you
just a couple of quick scripts,
so that you know how to use auto optimization.
So with that, let's dive in and get started.
First off, auto optimization is an Azure Databricks tool.
So keep that in mind for the DP-203.
Auto optimization, Databricks.
So how does it work?
Well, it allows you to automatically compact small files.
And what it's going to do is it's going to write those
into a Delta table.
So let's take a look at a traditional write process
over here on the right.
So in a traditional write process,
we're going to have our large blue box,
which is our executor,
and then we're going to have our data pieces.
And in a traditional write,
we're just going to take those data pieces
and we are going to move them
into Delta tables and partitions.
So what can happen is you can wind up with a large number
of small files in a partition.
Optimization can fix this.
So with optimization, what you're going to do
is you're going to go through and look at the files,
and you're going to combine all of those files
into 128-megabit sizes.
Now, with that,
you're going to get less files in the partitions,
which means you're going to be able
to query your data faster.
Now, with auto optimization,
we're just simply turning that process on
so that Azure Databricks, as it runs,
is going to automatically look and compact those files
into that larger file size.
Now, how do we turn that on?
With an auto optimization script.
So you have to enable Optimized Writes and Auto Compaction.
It is not on by default.
You do that by, first, writing something like this script,
and this piece in orange here that I've highlighted,
this is the actual piece that turns on Auto Optimize.
So you can see that we're turning on
autoOptimize.optimizeWrite = true,
and autoOptimize.autoCompact = true.
So this is where we're turning those on.
So you need to turn on Auto Optimize in Databricks.
And then, if you're going to alter a table, same process.
So the top one is for creating a table,
and the bottom one is for altering a table.
But again, that piece in orange,
that's the piece that we actually care about,
at least as far as auto optimization is concerned.
Should I always auto compact?
The answer is: not necessarily.
It depends on latency first.
So if we have a ton of files being written very quickly,
we might care about latency and how fast it takes us
to actually get those files into the Delta tables.
Obviously, if we're going to optimize,
that's going to add an additional step,
which is going to slow the process down just a little bit.
It also depends on your concurrent operations.
So if you are deleting or merging data or things like that,
it can break the auto optimization process,
and so you might not want to use it.
Some key points to remember as we wrap up this lesson.
First, don't get lost in the weeds.
This is not about Databricks.
This is about auto optimization and Auto Compaction.
You need to know that it exists, and you need to know
that you do need to turn it on in Databricks.
It can be extremely helpful when queries are important
and latency isn't.
This is when we're going to use this Auto Compaction.
So keep that in mind.
And with that, we finished this lesson.
On to the next.
I'll see you there.