Hey, Cloud Gurus.
Welcome to Partitioning Best Practices: Part 1.
In this lesson,
we're going to discuss partitioning in Azure Data Lake.
We're also going to look at performance considerations,
scalability considerations, and availability considerations.
Much of the content within those concepts will apply to both
Azure data lakes, as well as Azure Synapse.
But we're going to be focused mostly
in this lesson on Azure Data Lake.
And then, in part 2,
we'll extend that into some specifics about Azure Synapse.
At the end of this lesson, as usual,
we will wrap it up in a review.
Of course, before we can talk about best practices
for Azure Data Lake, we need to kind of understand
how the partitioning is working behind the scenes.
The partition key for Blob and Data Lake storage is actually
comprised of the account, plus the container, plus the blob.
So, in essence, the full blob name.
For example, if we had our Data Lake account,
named acdatalake,
you can see it represented in orange here.
And then, our container for our landing zone in purple,
and finally, the blob itself, the actual file,
our Parquet file in this case.
And that gives you the full partition key
when working with Azure Data Lake.
Behind the scenes, it uses range-based partitioning.
So the data is split into ranges,
which are load balanced across the storage system.
With this type of approach,
if you're using a naming convention
that has lexical ordering,
that means your partitions are more likely to be located
on the same partition server.
And then, once load increases to a certain point,
they are split into smaller ranges.
Let's get a visual idea of what that looks like.
If our convention for Awesome Company is prefixed with AC,
we have our AC engineering department,
AC HR, AC IT, and AC sales.
And to start with, they're all in order,
and they're all served by the same partition server.
But over time, as the load increases,
it will be split out into smaller ranges,
each served by their own partition server.
Having these 2 key pieces of information is important
when considering your partitioning strategy
with Azure Data Lake.
Be aware that you're naming at the account, container,
and blob level affects the partition key.
And that, in the background,
it's using that to perform this range-based partitioning.
Let's dive into some performance considerations.
First, based on what we just learned,
it's important to name with care.
Create a naming convention that sets you up for success.
Remember that when we say naming convention,
that includes accounts, containers,
as well as the blobs themselves.
Also, be careful if you're using numeric
or timestamp identifiers.
If you're not careful,
you could dump everything into 1 bucket.
For instance,
if you're using today's date, or you're using a specific ID,
you could dump a large amount of records
into just that bucket.
So consider using a hashing function
to prefix names with a 3-digit hash.
Remember that one of the primary goals in these systems
is parallel operations.
The more reads and writes that can happen in parallel,
the better.
Think of a highway system.
If you have a 6-lane highway, you're crippling yourself
if you're only using 2 of those lanes.
Design your partitioning so that operations can happen
in parallel, using all 6 lanes.
Embrace pruning.
Pruning data means that your application has to look
in fewer places for smaller amounts of information.
Usually this is going to be based on dates,
but consider carefully when you can prune data
out of your system.
Less is more.
Keep your partition sizes smaller,
and that will help keep your query response times lower.
Think of yourself when you're traveling through the airport
with baggage.
If you took everything in the kitchen sink with you,
you're going to be loaded down and not very mobile
as you're trying to get to your plane.
Keep it lean and mean,
and that will enable you to move quickly.
For scalability, plan for the future.
Estimate the size and workload of each partition
with an aim toward balance.
And so, you really need to analyze your application's needs
to project those scalability requirements.
Stay within the limits of the system.
Be aware of Azure infrastructure limits
on a single partition store, and be sure not to exceed them.
If it looks like you're going to exceed them,
plan for more and smaller partitions.
And then follow up.
Don't just assume that the partition scheme you've created
is going to work as expected.
Monitor it to make sure the distribution
is working optimally.
Reality doesn't always match our predictions,
and that's okay.
Everybody's wrong sometimes.
Hope for the best, but plan for the worst,
and follow up afterwards to make sure you got it right.
Lastly, a couple of availability considerations.
Prioritize by partition.
Apply availability and backup plans
according to a partition's level of criticalness.
You know your data, or you should.
And so, you know what is critical
and what weighs more heavily against other data.
Also be mindful of time.
Know the best times for partitions to be taken offline
for maintenance, and keep them small
to make sure that maintenance completes
within your planned window.
Again, this all comes back to knowing your data
and knowing when something can be taken offline
and how long operations against it will take.
By way of review,
the naming convention you use will greatly affect
how data is partitioned.
And I can't stress that enough.
Be mindful of how you plan your account,
container, and blob names.
This is going to be your partition key, and therefore,
what affects your partitioning in that range-based system.
For best performance and fewer administrative headaches,
keep partition sizes smaller and prune.
I know the tendency is always to hang on
to everything possible,
but really, that's only going to hurt you in the end,
not help you.
And lastly, I know you hear me say it a lot,
but know your data well
in order to plan your partitions well.
And then, monitor afterwards to verify your success.
That's it for this lesson.
I hope this helps you understand better
good partitioning practices,
especially when applied to Azure Data Lake.
In our next lesson,
we're going to continue discussing
how to have a great partitioning strategy
and also focus a little more on Azure Synapse.
I look forward to seeing you there.