Hey, what's up, Gurus?
Welcome to Section 4: Data Ingestion and Transformation.
Now, if you'll notice down here at the bottom,
you'll see that Landon is on here, as well as myself,
and we're going to kind of work on this section together.
Landon's going to do most of the lessons.
I'm going to jump in a little bit as well.
So just keep that in mind.
We'll see a little bit of change,
but I wanted to introduce us both for this section.
Now with that, let's get started,
and talk about section 4.
If you'll notice, section 4 is just about 1/3
of the way through this course.
So make sure you look at your schedule,
make sure that you're on track,
and have a little celebration
because when you finish this--
hey--you're a 1/3 of the way there.
That's awesome!
In this section, we are going to be breaking down,
of course, our introduction.
We're going to talk about Azure Data Factory
just a little bit.
We actually have almost an entire section coming up on that,
but we'll kind of introduce that as well.
We're going to talk about Transact-SQL.
We're going to talk about Azure Synapse Pipelines.
And again, this will be just a little bit of a deeper dive,
and we're going to re-introduce a lot of these services
over and over again,
teaching new concepts to kind of further what you can do
with them as you move through this course.
We'll talk about Scala, Apache Spark,
and we'll talk quite a bit about creating data pipelines.
In addition to that, we're going to talk about designing
and creating tests for data pipelines,
integrating Jupiter and Python notebooks
into data pipelines.
Cleaning data, splitting data, shredding JSON,
and what that means.
And then of course, we're going to talk about
how we encode and decode data,
and we're going to talk about
configuring error handling for transformations.
We will also talk about normalizing
and denormalizing values,
performing data exploratory analysis,
and finally, our section recap.
So, if you think there's a lot of stuff in this section,
you are correct!
There is a ton of information in this section,
a ton of different concepts,
but they all tie into data ingestion and transformation.
As we talk through those services, keep in mind,
don't get too hung up on the services themselves,
because we'll be going back over those services
throughout the course,
as we've reinforced these concepts.
Finally, don't forget,
we do have a hands-on lab in the section:
Moving and Transforming Data with Azure Data Factory.
So, make sure that you tune into that,
because that is going to be something
that you definitely want to understand.
So as we think about this section,
Data Ingestion and Transformation
is all about taking raw materials
and transforming it into usable data
for downstream consumption.
So in this case, we're going to be talking about
business decision-making.
So this data is incredibly important
because it's going to shape the vision for the business
as it moves forward.
And so really, it's all about taking that coal,
compressing it down into something that's usable,
and turning it into something beautiful, like a diamond.
All right, with that,
we're done with this introduction.
I'm going to turn you over to Landon,
and I will jump back in to see you later
for the section recap.
Enjoy the section.