In this lesson,
we are going to take some of the concepts
that we learned about data flow,
and we're going to dive just a little bit deeper
to talk about how we handle schema drift.
Specifically, we are going to talk about
what schema drift actually is.
Then we're going to talk about
how we handle schema drift in data flow
by talking about the what, when, where, and why.
And then we're going to finish up
with a live look in the Azure portal
so you can see how we would set for schema drift.
Alright, so what is schema drift?
Well, you can have changes that are fast or slow,
and so take a look at this boat over here on the right.
Think about currents.
So over time, that boat is going to slowly move
if you don't have it anchored to the dock.
Same thing is true for that race car.
It's just really fast.
As it goes around the corner,
it's going to drift,
or move off of the line that it intends to go on.
And again, this drift, when we talk about data or schemas,
it's going to lead to a breakdown of your pipelines.
So in data engineering,
what we talk about when we look at schema drift,
we're thinking about changes to our fields
or our columns or our data types.
That is what schema drift looks like.
And again, that leads to,
when we talk about Data Factory,
a breakdown of pipelines if we don't correctly account
for that schema drift.
So make sure that you think about the stability
of your schema as you are building pipelines
and as you are using data flow.
You need to be considering,
"Well, is my schema likely to change over time?"
And if your schema is likely to change over time,
you need to think about that stability
and make sure that you have accounted for that
in your data flow and in your Data Factory pipelines.
So what about the what, when, where, and why?
Well, summarizing,
the problem is schema drift causes broken pipelines,
and that can lead to bad data.
This issue occurs when we look at incoming data sources,
and you need to take care of this
as you're building the data flow,
not down the road in production.
And you need to take care of it,
because if you don't,
it can lead to a complete breakdown of your pipelines.
So now that we have that out of the way,
let's take just a couple minutes
and look at how we set that in Data Factory.
So let's jump over to the portal
and take a look at a data flow.
You can see here that we have our incoming source,
and then we have our middle step,
which is whatever transformations,
in this case it's an aggregate,
but it could be a whole bunch of transformation steps here,
and then we finish with our sink.
The sink is just simply the finishing product, the output.
So we define our source, the input,
and we define our sink, the output.
When we look at setting up for schema drift,
we start off in the input source.
We're going to come down to our configuration panel
under Source Settings,
and we're going to make sure
that we have allowed for schema drift.
Then we're going to come over here to our sink,
and we are going to make sure
that we have schema drift enabled here,
and also, that under Mapping, we are Auto Mapping.
So what's going to happen is,
if we have selected for this,
your data flow is going to look
at all of the incoming fields,
and then it's going to look at all of the outgoing fields,
and it's going to attempt to automap through these steps.
So that's what we're doing,
we're attempting to automap every single time
we run that data flow and not relying on a fixed schema.
So if our schema changes from the input
or changes as it's going into the output,
we should be able to account for that.
So at its core, it's that simple.
Basically, what we're doing is,
as we look at the changing of our metadata,
which again, that's the fields, the columns,
that is the schema drift that we're looking at.
And as those change over time,
we account for that by using data flow
to enable our schema drift,
which allows for late binding.
So we're not going to have a fixed schema.
We're going to actually run the data flow
and attempt to bind those input and output columns
as data flow runs.
And this is important
pretty much anytime you have frequently changing sources.
So as you build your data flow and your data pipelines,
you need to be considering,
are those sources likely to change over time?
And if so, are you accounting for that change?
If you have, awesome.
That's it for this lesson,
and I'll see you in the next.