Hey, Cloud Gurus. Welcome back.
Now that you know about creating your Azure data lake
and some basics about container creation,
let's talk about getting its folder structure right.
Not every data lake will be structured the same way,
but there are some common practices
that I'd like to go over with you.
In this lesson, we'll start by talking about
data lake zones, and then we'll talk about
some folder structure strategy within those zones.
From that general strategy,
we'll construct some examples for you
and then we'll wrap everything up in a review.
And so let's start with data zones.
And zone sounds really intimidating,
but really all this means is a folder
such as the ones you saw me create
in the last lesson's demo.
There are usually 3 to 5,
but everybody does it differently.
Some are optional, but we'll get to that in a moment.
Starting out, we have our landing zone,
and this is where our raw data lands in its original state.
It's always immutable and not usually fit for consumption.
It can be organized with folders per source.
And as you can imagine,
this can become a very expensive zone
because you're keeping the full set of data
from a lot of different places.
And so you might want to programmatically move it
to a cool access tier at an interval of your choosing.
From there, we have staging.
And this is the first step to refinement
by adding basic structure.
There's usually an automatic process to bring raw data in
and provide a better structure for it
to prepare it for being curated.
But even at this stage, it provides an increased value
over the raw data.
From there, we have our more curated zone,
where it's transformed into consumable datasets.
This could be files or tables.
This zone can be used to feed a data warehouse,
but it's not a replacement for a data warehouse.
The speeds aren't suited to end user dashboards or reports.
Instead, it's focused more on larger internal analytics
that don't have time requirements such as ad hoc queries.
But it differentiates itself from the staging
in that it contains only consumer data
that has been quality checked
and combined with like sources.
We use that curated data to feed our production zone.
And this provides an easy access point for consumers
that includes business logic.
And that business logic may be something like surrogate keys
or other application-specific needs.
We can also use this zone to feed our
machine learning models.
But overall, it's beneficial because it provides data
to consumers in a friendly format.
Lastly, we may have an experimental zone.
And this is more of a sandbox where our data scientists
can combine multiple datasets
both from within the data lake and from outside of it.
And they can mix and match and experiment
to their heart's content.
As I said earlier, not every organization is going to have
these exact zones, and even if they do,
they may name them differently.
That's completely okay.
What's important is understanding the general philosophy
behind each.
And as I said, not everywhere will have all of them.
The most important and commonly shared among them
is our landing zone, curated, and production.
Almost every analytic solution is going to have
at least those, and everything else just kind of adds
niceties around them.
But once you have your zone structure,
within there you begin to develop your folder structure.
So let's go over some strategy for that.
First, you want to keep it simple.
Your naming convention should be human readable,
easily understood, and self-documenting.
It's always tempting to over-engineer things
and make it super complex, but don't fall into that trap.
The simpler you make it,
the easier it is to scale and to manage.
Speaking of management, it should be appropriately granular.
Design for effective permissions
without generating unnecessary maintenance overhead.
You can name it in such a way
that you're going to create problems for yourself
by having too many subdirectories
on which you have to manage too many permissions.
Be thoughtful in your partitioning.
Align it with your partition strategy for the purpose
of the zone you're working in.
For example, aim for optimal retrieval on the curated zone.
And don't feel you have to have the same strategy
across all of the zones.
Design it intentionally.
We'll get to more on partitioning in a later lesson.
Also, group similar items.
In general, folders should have files
of the same schema and format.
It just makes it easier to work with them
in a consistent way.
Now that you have some strategy,
let's go over some concrete examples.
We can create structures by source system, by departments,
by projects, and really whatever makes sense
for your business.
Some examples of these types of filtering include
for our source system, maybe we have one in our raw zone
divided by data source.
Like I mentioned earlier, you can have a folder
within your raw zone for each of your data sources.
That allows write permissions to be granted
to source systems at the data source level
with default ACLs.
And then permissions will be inherited
to new folders under there.
For department, perhaps you have
within your production zone a sales organization
or a marketing or an internal IT one.
All of those can be divided up by department.
And then you can point various systems
with different application logic
at those different folder structures.
A common thing to do is to use dates.
So maybe we have things down a few levels
divided into years, month, and day,
with the day being our final folder
that holds our actual files.
If you put that all together,
it might look something like this:
our raw zone with our data source folder,
whatever entity is applicable,
and then our year, month, day structure.
If you put that year, month, and day at the front
instead of the data source,
you could cause problems for yourself
because it would be more difficult to inherit permissions,
and therefore to maintain them.
It's much easier to set it at the data source level
and let that trickle down to all of your
years, months, and days.
Having the dates also makes it easy to prune later
if you need to go back and delete certain ranges of dates.
Aside from this filtering,
we can also use sensitivity levels.
Perhaps in your raw zone, you can divide it into
general before splitting it into data sources.
And for your sensitive data in your staging zone,
again, you may have a separate sensitive folder.
But this allows us to add different security controls
on different folder structures.
By way of review, zone and folder structure
help represent various levels of data transformation.
And really that's all you're trying to do here.
You're trying to create a structure that is both beneficial
for your automated workflow and for the humans
trying to work with it.
Folder structure can affect everything
from query performance to security,
data pruning, and administrative maintenance.
And so making smart choices up front
will help you down the road in each of those areas.
And lastly, each solution has to be crafted
according to individual business requirements and data type.
No 2 businesses are going to have the same
analytical needs.
And so don't feel like you have to use the exact same zone
or folder structure as someone else out there
on the internet.
Take advice with a grain of salt
and design it in a way that makes sense for your business.
That's it for this lesson. Thank you for joining me.
I look forward to seeing you in the next.