Hey Cloud Gurus, welcome to the section recap.
You made it! I'm so proud of you.
I know this has been a lot of information,
and so let's take a second to review some of the main points
from this section and take a look at where we go from here.
Azure Data Lake Storage Gen2 allows scalable, flexible,
and highly available storage for a variety of data formats.
And remember, this technology is a central piece
of almost every data engineering solution.
Of course, there are many other pieces,
including Azure Synapse Analytics, Databricks,
but for our storage section, it almost always comes back
to Azure Data Lake Storage Gen2.
Distribution, partitioning, sharding, and pruning
are all unique methods of breaking up data
into workable subsets.
Remember that distribution has to do
with those 60 underlying databases
that Azure Synapse Analytics creates for you automatically.
Partitioning is dividing up a single database instance
into different chunks.
Sharding differs from that in that it spreads the data
across multiple computers.
And pruning allows you to selectively pull back
only specific pieces, even within a single partition
according to your query filters.
Know your data well in order to make wise decisions
on folder structure, file formats, and partition keys.
Each of your situations will be different,
and how you answer exam questions may be different
depending on the needs of the scenario.
Keep in mind our zones for landing,
curated production, and so forth,
the different file formats,
and each of their advantages and disadvantages,
and how to wisely choose partition keys.
Not necessarily the perfect key,
because there is no perfect key,
but the one that gets you the best results
for your situation.
All of these concepts coming back
to one central theme, going fast.
Remember, properly designed storage lays a foundation
for achieving maximum query performance
and data availability.
All of the methods we've discussed
have different ways of going about this,
but it's pretty much all to accomplish this 1 goal,
making things performant and making them available.
And so at this point in the course,
you're familiar with what data engineering is.
Thanks to Brian's spectacular introduction
in his crash course, you now have a foundation
about data storage.
From here, you're going to move on
to learning about data ingestion and transformation
before we get into how to create
different processing solutions.
I hope that you have enjoyed this section.
If there's any part that you're shaky on,
feel free to go back and review that video,
practice the hands-on labs, and I would encourage you also
to look up Microsoft's documentation
on that particular subject.
Approach learning from multiple avenues
to get the best result possible.
Thank you so much for joining me,
take a break if you need, get a little coffee,
absorb all that you've learned, and when you're ready,
proceed on to the next section.
Keep being awesome, Gurus.