Hey Cloud Gurus.
Welcome to this lesson on using Azure Data Lakes.
In this lesson, we'll be doing a quick recap
on what Azure Data Lake Storage Gen2 is.
I know that Brian has already covered that
in the previous section, so that will be just a quick stop.
From there, we'll talk about where it fits
in the modern data warehouse.
We're going to do a demo on creating and interacting
with these accounts live in the Azure portal,
and then we'll wrap everything up with a review.
So let's get started.
What exactly is an Azure Data Lake Storage Gen2 account?
Well, it's built on Azure Blob storage.
That was kind of our first iteration and gave us several
benefits in the way of storage,
including being a low-cost solution.
They had various tiers we could take advantage of,
and it was highly available.
And so these great benefits are still at the core
of what Azure Data Lake Storage is.
But now with the advent of storage Gen2 accounts,
it builds on top of that and combines the capabilities
of Azure Blob Storage and Azure Data Lake Storage Gen1,
giving us additional abilities
such as file system semantics,
file-level security, and scale.
Really what this does for us in the end is
enhances the solution for analytics.
And there are a few key areas in which it does this,
namely, performance, because you have no need to copy
or transform the data before analysis,
compared to just the flat namespace of Blob storage.
It also benefits us in the area of management.
Having that hierarchical namespace allows us to organize
by directories and subdirectories,
making management much easier.
And finally it benefits us with security,
allowing POSIX permissions on directories
or individual files.
And so, as you can ascertain,
and as Brian pointed out earlier,
this all comes back to the hierarchical namespace.
That's really what makes it stand apart,
and really what makes it enhanced for analytics.
That being the case, let's talk for a second
about how it fits into our modern data warehouse.
These days, we have all kinds of data of all kinds of types
coming from all different directions.
We have logs, media files, relational database data,
all kinds of structured and unstructured data.
We use a component such as Azure Data Factory
to ingest that.
And then we need a place to store it,
namely Azure Data Lake Storage Gen2.
And this is a great repository for all of our data
in our analytics solution.
From there, it kind of becomes the central hub
from which all the fun breaks out.
You could have Azure Databricks in your prep and train step
and maybe services such as Azure Synapse Analytics,
Azure Analysis Services,
or Power BI downstream in your model and serve step.
But each of these is going to be relying
on Azure Data Lake Storage Gen2.
We could be pulling information from there,
prepping and training it into Databricks,
and then sending it over to Synapse,
or Synapse could be pulling directly from our data lake.
Those services can further pass it over to Analysis Services
and Power BI, but no matter what services you add
into your analytics solution,
be it Cosmos DB, Stream Analytics, or what have you,
it's all tying back to the Azure data lake
as its central hub for this data.
With that in mind, let's jump over to the Azure portal
and take a live look at this crucial analytics component.
Here we are in the Azure portal, and I'm utilizing
the cloud sandbox feature available
through A Cloud Guru subscriptions.
This is a fantastic tool for getting hands-on experience
with Azure services.
To get started, it's the same as creating
any other storage account.
We could search for storage accounts in the search box,
or click our 3-line hamburger menu
and go to Storage Accounts.
Let's Create Storage Account.
Since I'm using the cloud sandbox,
it's already filled in my subscription
and resource group for me. Of course,
you can make those any that you want.
We can set our storage account name, and it's already taken.
So I'll add some numbers to the end.
East US is fine with me, and I'll leave everything else
default for now because what we are really concerned about
with a data lake is on our Advanced tab.
And let's come down to Data Lake Storage Gen2.
This one setting is what sets this apart
from creation of a standard storage account,
enabling the hierarchical namespace. As mentioned earlier,
all of those performance management and security benefits
stem off of having this hierarchical namespace.
I leave everything else at default for now
and come to Review + Create.
It's going to verify that the setup looks good,
and now I can click Create.
That's going to take a couple of moments
to spin up this resource.
I'll pause the video here and be right back with you
when it has completed.
Alright, that resource is ready.
It actually spun that up very quickly.
And now we can click Go to Resource. Of course,
lots of helpful information right on our Overview page.
Let's come over under Data Storage and look at Containers,
and containers are really how you're going to be dividing
up a lot of your analytics files,
which we'll talk about in another lesson,
but it's easy as clicking on the Create Container button,
and we can name this something like landing
for our landing zone.
And maybe we have another that has our curated data.
And so you can already begin to imagine the benefits
of having this hierarchical namespace and easily managing
different zones and those files within the zones.
We can manage access control and ACLs per folder
and per file,
huge advantages over just your traditional blob storage.
We can also manage things like the networking,
which networks have access to this Azure data lake,
as well as shared access signatures
and other security features.
We'll go further into those in a future lesson.
For now, I just want you to start imagining
how we can create this Azure data lake,
start storing our files there,
and set it up to be that central hub
in our analytical solution.
By way of review, Azure Data Lake Storage Gen2
combines the benefits of Azure Blob storage
and ADLs Gen1.
Those added features make a great solution for analytic
workloads or when a human-readable hierarchy is needed,
because let's be honest, it's just easier for us to read
sensible directory and file structures.
ADLs Gen2 is a foundational component
of almost every Azure analytics solution.
So you're going to be circling back around to this a lot.
No matter how you build your solution,
it's usually going to be a key component of it.
And finally, even though we're going to go over security
in more depth later,
I want you to have in mind that it's recommended
to turn on the firewall and only allow access
from other Azure services.
That's really just in your best interest
for keeping that data secure.
Thank you for joining me for this lesson.
If you have any questions,
please feel free to reach out to me
and I'll see you in the next one.