Congratulations.
You made it through section 5.
In this lesson,
we're going to be looking through the rear-view mirror
at batch processing
and just recovering a few of the concepts
that you need to know for the DP-203.
So specifically, hey, it's all a review.
We're just going to be looking at things
that we have already talked about in section 5
and highlighting things that you need to know
for the DP-203.
If you come across something that is confusing
or you don't quite remember,
you need to go back through and rewatch those lessons.
Second, it is a focus on the DP-203.
Everything in this course is designed to get you prepped
and ready to pass that exam.
So there are going to be topics that are glossed over
or other things that are missed
because they're not a main focus for the 203,
but things you might want to know for data engineering.
Finally, if you don't know something, review.
Don't forget that. It's extremely important.
With that, let's jump in and start the review.
First up, we talked about batch processing.
We talked about it being used in banking and retail
and hospitals and marketing.
We went through examples
of why batch processing is used there,
and typically, just like with a bank,
it comes down to being able to run everything
in one large lump or a batch.
Also, we talked about some of the challenges.
One of them is date formatting.
We talked about encoding being a challenge as well.
And then we talked about what happens with missed runs,
so batch processing being accomplished all at once.
What happens if you miss a run?
How are you going to go back and fix or catch up the data?
Or do you catch up the data?
Things to consider when you look at batch.
We also talked about full data loading
versus incremental data loading.
If you remember, full data loading
is where we just literally dump the entire data set
and we start over afresh with everything replaced.
There's no other requirements, that's it.
Incremental data loading is where we're not dumping anything
from the current database.
We're just looking at the differences
and we're loading those differences in.
We talked about that process,
creating lookup activities,
creating a copy activity in Data Factory,
and then creating a stored procedure activity
to update your watermark.
And your watermark was measuring what was last updated.
We talked about mapping data flows.
If you remember, a data flow is the visual no-code solution
to developing and implementing transformational logic
in data factory.
So keys here, no code solution
for adding transformational logic.
So if you want to do some transforming, it's somewhat light,
data factory and mapping data flows
might be a fantastic solution.
Those data flows are then created
and added into your Data Factory pipelines.
And if you remember the basic steps,
create a data flow in Data Factory.
You pick your source, or where the data is coming from.
You choose the modifier, which is the things
that you want done,
the transformations you want done on that data.
And then you always finish up
with choosing your destination.
So those are the basic steps for creating those data flows
in Data Factory.
And, of course, there are a ton of scripts that you can use.
And there's the link again. It's also found in the lesson,
but make sure you take a look at that.
Because as you actually jump into data engineering,
those scripts can be very helpful.
Upserting.
We talked about upserting being the operation
that allows you to insert rows into a database table
if they don't exist, or update them if they do.
The key to this is your alteration activity.
So you'll grab that, and you'll use that
in order to create upserting in Data Factory.
We also talked about the key considerations for that
being activity, fault tolerance, and retry.
And if you remember, the activity was setting your success,
failure, completion, and skipped.
So what happens after your activity completes or fails?
Making sure you have a strategy for that.
Fault tolerance was deciding what errors you can ignore.
And then the retry was just simply setting
how many times you're going to retry an activity
before you decide the pipeline has failed.
We also talked about execution triggers. If you remember,
we can manually execute a pipeline.
That's just the on-demand.
We're going to click the button and start it.
And if you remember,
there's 4 methods for configuring. There's .NET,
PowerShell, REST, Python, and, of course, the portal.
In addition to that, you can also set up triggers,
which are going to execute via a schedule--
that was like our simple wall clock, or our tumbling window,
which is more complicated, but allows us to look back
or forward in time.
Remember, the tumbling windows are fixed
and non-overlapping,
and those are contiguous time intervals.
And contiguous just means that they're touching each other
as we move forward in time.
And, of course, we have our event-based
that we can also set up for execution triggers.
So don't forget about those.
And don't forget about Databricks.
Just like our elephant over here on the right,
you need to remember that the DP-203
is going to look at Databricks primarily for transformation,
and you need to remember that demo
where we went through, created,
and then launched a notebook.
Finally, don't forget
that Databricks can be implemented into other services
like Data Factory or Synapse.
Spark in Data Factory.
We talked a little bit about this, and if you remember,
we can execute Spark activities using an HDInsight cluster.
We talk through a whole bunch of code over here
on the right.
So make sure that you remember that.
I'm not going to go back through all of that code,
but really what you need to know
is the basics of what's happening
and how we create those activities.
In summary, this is not all.
This was a huge section, and what I've covered here
wasn't even all of the slides
and it wasn't even all of the lessons.
It was the lessons that I felt were most important,
but that's not all.
So don't just gloss over the other topics.
Also, don't forget the big picture.
As you go through this course,
you need to be constantly thinking about how this fits
into data engineering pipelines,
how you can use these individual activities
and use them for your larger projects.
That's going to help you in your career
and it's going to help you on this exam.
Also, don't forget to focus
on those Microsoft exam requirements.
If you remember, I've showed this in several lessons,
so we're not going to jump in and do that again,
but make sure
that you're looking at those exam requirements.
And if you need a refresher on that, back in section 1,
we talked about what those are
and I show you how to find them.
Don't forget about the labs.
You also need those for your exam and career.
And I'm going to keep mentioning that
because the labs are very important for your success.
And finally, if you would do me a huge favor
and you would just smash that thumbs up button
on those lessons as you go through,
I would greatly appreciate it.
I hope you're finding this content helpful,
and congratulations again on finishing section 5.
I'll see you in section six
and we'll talk a little bit about streaming.