Congratulation, Gurus,
on finishing this section.
In this lesson, we're going to take a look
in the rear-view mirror,
and we are going to review some of the highlights.
In the lesson, we're going to talk about 2 things.
First, we are going to say, "Hey, thanks, Landon.
"Greatly appreciate all the work you've done
"through this section, it's been extremely helpful."
And 2, we are going to jump in,
and take a look at the review process.
Having looked back,
I kind of want to go skiing now.
However, can't do that because we need to jump in
and complete this section.
So here we go.
First, we talked about Data Factory,
and some of the core concepts there.
We talked about pipelines.
Remember those are a logical grouping of activities,
and we talked about those activities
being the processing steps in a pipeline,
there are 3 main types of activities,
data movement, and data transformation,
and if you remember,
we really focused on data movement, and data transformation,
we'll jump in later on in the course,
and talk a little bit about control,
but that's important for activities.
We talked about data sets being where the data that you need
for those inputs or outputs for those activities,
where that data lives,
and then the linked services being the connection string
that allows us to connect to that data.
We also talked about T-SQL.
If you remember, that is Transact-SQL,
and it's a powerful language that lets us do,
move, and transform data activities.
T-SQL can be used in machine learning,
as well as data engineering, and if you remember,
there was a couple of uses over here on the right.
We can use it to create tables for results
or save data sets,
we can use it for custom transformations
on a whole bunch of different things,
or we could use it to filter and alter data
and return those query results as data tables.
So don't forget about T-SQL,
it is a very important concept for data engineers.
We also talked about Azure Data Factory,
and how it compares to Synapse.
If you remember,
we looked through and we said that synapse
misses some of the features that ADF has,
but it also adds other features on its own.
We also talked about Azure Data Factory
being used primarily for migration work or ETL.
However, if you have a need for analytics projects
or more complex solutions involving databases,
Synapse is a fantastic choice.
Next up we talked about Scala.
Scala is a programming language that is used heavily
in data engineering and in machine learning,
and we talked about just the basics for that,
for this course, those basics are going to be just fine,
you don't need to know a whole lot about Scala
to pass the DP-203, so don't dive too far into that.
We also talked about Apache Spark.
And if you remember,
we talked about Data Factory and that Data Factory
uses Apache Spark as an on-demand service,
it creates Spark clusters for just-in-time processes,
and it processes the data that you need,
and then it deletes that cluster once it's done,
so that just-in time-processing is very important,
and we talked about Spark jobs being extensible,
which means you can provide multiple files,
such as Python scripts, or jar packages.
And so it's a very flexible and helpful concept
for you to understand.
We'll talk more about Apache Spark
as we go through the course,
but for now just getting that overview in mind
will be very helpful.
We also talked about notebooks,
and if you remember, we talked about Jupyter Notebooks
being those open source web tools
that we can use to write scripts,
and it's really kind of a conglomeration of everything.
So it allows us to do instructions, code, visual output,
whole bunch of different stuff,
and we can use Python and Scala code in Databricks
and machine learning to help us with data transformations.
We also talked about testing, and if you remember,
we built this really nice workflow to write tests,
publish pipelines, count activities,
and then check our row counts,
and so this is something that's helpful
for you to understand.
Just, again, DP203 is not going to be heavy on testing,
but understanding the basics of what's going on
is going to be helpful for you.
And then we talked about tools.
If you remember,
there were a couple of different tools,
we talked about data quality services,
the clean missing data module and mapping data flows,
so this is going to be helpful for you to just understand
what is available and what you could use
to go about cleaning your data.
So don't forget about those 3 different options.
Next up, we talked about conditional splits,
and so this is basically making new branches,
and what happens is we're actually routing our data rows
to particular streams based upon conditions that we specify.
And this is very similar to the case statement
in traditional programming.
And we talked about our code example,
and so if you see this code here,
you should immediately think about--
you should say shredding JSON here.
So when you see this code snippet,
you should be thinking about shredding JSON.
Next up we talked about continue on error.
Errors happen, but it's what we do about them that counts.
And the default is normally to fail on first error,
but we can enable continue on error,
get some new options--so we can do transact commit,
or output rejected data, or success on error.
And so, understanding those different types of options
and what we can do to just avoid that
fail on first error can really help you
in your pipeline development,
and help pipelines to go through,
even if there's errors in the pipeline.
We also talked about exploratory data analysis, or EDA.
And if you remember, this is a process
for performing initial investigations on data,
and what we're doing is we're applying summary statistics
and graphical representations to test hypothesis.
So the important bit here is the initial investigations
in summary statistics and graphical representations.
That's what you need to remember about EDA.
And of course,
don't forget that it is used to make sense of the data
that we have before we go too deep with it.
In summary, this is the foundation
we're going to continue to build upon these concepts
as we move through the course.
It's also a focus on the DP-203,
so there's a lot of things that we could be covering,
but we're focusing on the DP-203 only,
so we might gloss over a few concepts
that are important for data engineering,
but we don't need to go into that much depth
in order to get you to pass the 203.
Finally, don't forget to look at
those Microsoft exam requirements as well.
We talked about that in the very first section,
so if you want to go back and see what that is,
you can jump back to the first section and review on that,
and of course, don't forget about the labs.
You're going to need those for your exam and your career.
Finally, Landon and I would both appreciate it
if you're enjoying this course, do us a huge favor,
smash that thumbs up button on those lessons,
and with that, congratulations for passing this section,
I'll see you in the next,
and we'll talk a little bit about batch processing.
I'll see you there.