Hey, Cloud Gurus.
Welcome to this lesson on sharding data.
In this lesson, we're going to take a look
at what sharding is,
some sharding strategies available to you,
some considerations when you're planning out your
sharding strategies, and wrap everything up with a review.
So let's jump in.
When we say sharding,
it's an all-for-one-and-one-for-all effort.
And by that, we mean multi-computer collaboration.
This is different from partitioning
in that sharding spreads your data
across multiple computers.
They all have the same schema,
but they hold their own distinct subset of the data,
whereas partitioning groups subsets
of data within a single database instance.
And so when you hear
that multiple computers are involved, think sharding.
Sometimes vertical scaling is not enough.
You can only add so much to 1 server.
And so sharding allows us to take advantage
of horizontal scaling,
in which you can expand outward almost indefinitely.
You simply add more nodes or shards.
It's high class, but low cost.
This pattern allows you to use commodity hardware
instead of really expensive monolithic servers.
And it allows you to keep your data close and available.
Using sharding, you can locate the relevant data close
to the users or applications that need it.
And by spreading it across multiple zones,
you, of course, increase your availability.
There are some primary strategies
for how to go about sharding data.
One is the lookup strategy,
and this uses a map to route requests
to the appropriate shard based on the shard key.
Another strategy is the range strategy,
grouping similar items together in the same shard,
and ordering them sequentially by the shard key.
And finally, there's the hash strategy,
and this is similar to hashing
that we talked about previously.
One or more attributes are put through a hash function
to determine the shard in which an item should be placed.
Taking a deeper look at these strategies.
First up, the lookup strategy.
Again, this uses a map in order to guide queries
to the correct shard.
An example of this is
if you have a multi-tenant application.
You can put all of the data for a particular tenant together
in 1 shard.
And then the application can find that shard
by referencing the tenant ID.
Let's take a look at this visually.
So we have multiple shards, shard 1, 2, 3, and 4.
And our application out there submits a query looking
for tenant 77.
Our sharding logic has a map that tells exactly
which shard tenant 77 resides in.
And so it's able to point the application
right where it needs to go.
The advantage of this approach is that it offers
more control over how shards are configured and accessed.
As you can see,
we're able to map out exactly where we want them to go.
Next up is the range strategy.
And this follows the old saying
of, "Birds of a feather flock together."
It groups related items into a shard
and orders them by the shard key.
For example, if you have an application
that frequently pulls all of the orders
for a particular month,
it's useful to have them all right there
in one shard, together and ordered sequentially.
Again, if we look at our example environment from before,
our application is submitting a query,
this time for orders placed in December.
And the sharding logic knows which shard contains orders
for that range, and points it directly there.
And so then you can easily pull
all of the orders for December.
Advantages of this strategy are,
it's the easiest to implement and manage.
I mean, your ranges practically create themselves.
It's easy to divide them up by dates and other ranges.
It's also easier to manage and work with.
For example, if you have all of your users in 1 region,
and therefore in the same shard,
you can easily schedule maintenance against that shard,
based on the time zone of that region.
It works well with range queries, of course,
and can retrieve multiple data items from a single shard
in a single operation.
Like we said before, it goes straight to that shard
and pulls everything from the month of December,
right there in a single operation.
Our final strategy is the hash strategy,
and this is all about balance.
One or more attributes of the data are sent
through the hashing function
to determine the item placement.
And so it's trying to distribute the data across the shards
in a way that achieves balance,
both for the size of the shards
and the load that each shard will encounter.
For example, in order to balance both the size
and load across your nodes,
you use a hash of tenent IDs to evenly spread the data.
Back to our example environment.
This time, our application is submitting again,
a query for tenant 77, but this time,
instead of using a map to go to a particular shard,
it goes through a hashing function,
the output of which directs it to shard 4,
and this will allow it to spread the data
by tenant ID more evenly across the shards.
The advantage here is its distribution of the data
and load is likely to be more even across the shards.
Now that you know the strategies, let's take a look
at some considerations for implementing sharding.
One, range may not be the most balanced.
When you're using the range strategy,
there's a risk of creating hotspots
because it's grouping by relationship,
not by even distribution.
You can mix and match sharding with other techniques.
Its complimentary to other forms of data separation,
such as vertical and functional partitioning.
So you might want to work out a strategy for using them
in conjunction with one another to achieve the best results.
Keep it steady.
Using stable data for the shard key goes a long way.
If the data changes often,
you could be causing yourself an unnecessary amount
of performance degradation.
Best is the enemy of good.
No shard key is going to be perfect.
I'm sorry. It just doesn't happen.
So aim to benefit the most frequent queries,
and then if necessary, create secondary index tables
to support queries that retrieve data using criteria based
on attributes that aren't part of the shard key.
But as much as possible,
try to design it in a way that is most impactful
for your highest priority queries,
and don't sweat it if it doesn't make
your lower priority ones blazing fast.
Strive for more rather than larger.
It's often helpful to have a large number
of small shards rather than a small number of large shards.
This increases the opportunities for load balancing.
And temper your expectations around consistency.
Evaluate whether you really need strong,
immediate consistency, or if eventual consistency will do,
especially in the cloud.
It's really difficult to maintain referential integrity
and consistency across shards.
So one, make sure you are planning your expectations
for consistency within the bounds of real life,
and also try to minimize operations
that affect multiple shards.
By way of review, sharding allows you
to horizontally scale your data across multiple computers.
And remember, that's the differentiating factor.
This is across multiple computers,
whereas partitioning separates the data
within a single database instance.
The lookup, range, and hash strategies offer compelling
options for sharding your data.
And of course, which one is best for you will depend
on your business and application needs.
And sharding allows you
to achieve not only greater performance,
but also greater availability.
There's a lot to be said for being able to spread your data
across multiple zones,
and even to locate your data closer
to your applications and users.
That's it for this lesson.
Thank you for joining me.
And when you're ready, I'll see you in the next video.