In this lesson, we are going to talk about
"Purging Data Based upon Business Requirements".
Well, what does that mean? We're going to start off
by talking about what data purging is.
Then we're going to talk about some reasons
that you want to purge data.
Followed by how we choose the data that we want to purge.
And then we'll finish up with some best practices.
So with that, let's get started.
First up, what is data purging?
Well, at the core,
it's the process of deleting data from the database.
However, there's a bit more to it than just deleting stuff,
and so I want to start with a discussion
of data lifecycle management.
Data lifecycle management is a core concept in
data engineering, and it starts with the creation of data.
So think about this as a point of sale.
As soon as that sale is created, we have data.
That data then moves through processing.
This is where it's ingested into the system
and transformed, and eventually winds up
in a database where we can then use that data
to do reporting or a whole host of different things.
After awhile though, that data is no longer useful.
And once that data is no longer useful,
we will often archive that data,
or store it for a variety of reasons.
And we'll talk about that here in a few minutes as well.
And then once we're done with that archiving phase,
then we destroy or purge the data.
This is data lifecycle management,
and it's an important concept not just for this lesson,
but for the entire course, to think about your data
as it travels through these 5 stages.
So another point, data archiving
or retention is not data deletion.
This data archiving is going to be set
based upon your business needs.
It could be that we want 5 years of transaction data.
Or it could be because of compliance requirements
or regulatory requirements, we need 7 years of data.
And so this data archive or data retention
is going to be based upon your business needs.
So why, then, would we purge the data?
Well, again, it could be regulatory requirements.
Due to right to erasure laws, or GDPR, or other reasons,
we might have a need, because of regulatory requirements,
to delete the data at a given time period;
3 years, 5 years, whatever that is.
We also want to reduce our footprint, because it saves money
and it reduces the risk.
The less data that we have,
the better off we're going to be
if that data is not being used,
because again, it's going to save us money in the long term,
not storing and managing it,
and it also reduces the risk to our organization.
So you want to think through and make sure that the data
that you're storing actually has a purpose.
So then how do we choose the data that we want to purge?
Well, first we'd look at outdated.
So based upon our data retention policy,
we might have data that is past the due date,
and we need to delete it because of that. Or errors.
If we have inaccurate data, we need to clean that up,
either by transforming and conforming the data
to the norm of the rest of the dataset,
or by deleting that extra data.
Duplicate data. So if we have duplicate data,
hey, we don't need 2 sets of data in most cases,
and we don't want to make a data dump.
So again, make sure that your data has a purpose
if it's duplicated.
Regulatory requirements, we've already talked about that.
It is important that, when you look
at regulatory requirements, that you make a footprint map.
We should know where the regulatory requirements are
and what data those regulatory requirements touch on.
So what are some best practices?
Well, first you need a data retention policy.
I mentioned that in the last slide.
And the data retention policy is going to include
your lifetime and your regulatory requirements.
Data needs to move from active, to archive, to purge.
That should be the process as you move data
through that system. Think data lifecycle management there.
Watch preconfigured backups from services.
So when we look at Azure services, quite a few of them
have preconfigured backups where it's automatically
going to store your data.
That may be fine, but you also need to weigh that
against any sort of regulatory requirements that you have
about purging your data.
Plan your purging strategy for off hours.
Purging or archiving, either one,
generally isn't something that's time-critical.
And so if you start at the morning, or the afternoon,
or the evening, it generally doesn't make any difference
for a purging strategy or your data retention policy.
So you want to plan your purging strategy in off hours
where you're going to have less traffic
and less weight on the system
for people that are actively running queries.
5: assess and run cost management analysis.
So as you look at your data footprint,
you should be running a cost management analysis,
including that storage of the data in archive,
and also the movement as we move it through
those various stages. Especially if we have more than
1 archive or we're moving data around a lot.
So it's important that you have
that cost management analysis done.
And last, for multi-cloud and hybrid environments,
you need to map your storage.
If we have environments that are not in Azure,
you need to know where you're going to archive,
and where that data lives, and how you're going to move data
between all of those environments
so that, again, you know where everything is
and you can reduce your footprint and your risk.
So wrapping up, just a couple of points to remember.
The key here, have a plan. And one that's written down,
not just one that's in your head.
I know that seems really easy, but there's a lot
of business groups that don't actually
write these things down, so make sure that you do that.
And then finally, look at your standard application storage
and your backup.
Standard backups might cause regulatory issues,
we talked about that.
So make sure that you understand the services
that you're using, how they back up,
and where they back up to,
so that you can avoid any regulatory issues.
All right, that's it for this lesson.
I'll see you in the next.