Hey Cloud Gurus, welcome back.
In this lesson, we're going to talk about
creating data pipelines.
To start off with, we'll do
a little bit of an Azure Data Factory recap.
You've already been introduced
to the components of pipelines.
So that will be a quick stop
before we jump straight into the demo.
And we'll take a hands-on look at these
in order to tie all the concepts together.
The goal is really to run through the process
of constructing a pipeline so that you're familiar with it,
have an idea of what you can accomplish with it
and can see it end to end.
We'll then wrap everything up with a review.
As you will remember when Brian went over
Azure Data Factory,
we have pipelines that are logical groupings of activities.
These activities can consist of many different action types,
including data movement, transformation, and control.
We use data sets to actually interact with the data itself,
and we populate those data sets by using linked services
to the various source types.
In this video, we're looking specifically at the pipelines
where we're tying all of this together
and making it a functional workflow.
So without further ado,
let's jump into the Azure portal
and take a look at constructing data pipelines.
Here we are in the Azure portal
and I've simply gone to my Azure Data Factory
named awesome-company-adf and opened up
Azure Data Factory studio.
We're here on the author tab where we can have a look
at our pipelines, datasets data flows, etc.
You can see I have no pipelines yet.
There are multiple datasets.
The scenario we're going to work off of for this demo
is that we have nightly loads and transformations
from our production environment to our dev environment
for some of our sales data.
To that end, there are a number of datasets
already created some to production tables
and some to development tables.
You can see these link back to Azure SQL Database
linked services,
but let's collapse this for now and let's create a pipeline.
We can name it something like NightlyLoad,
and within here, let's do a couple of copies.
Let's say that we want to copy data from production
to development every night.
Maybe we want to copy customer data.
I'll go ahead and collapse this to give us more room
and under Source, we can specify our dataset.
Like I mentioned, I have several already, but for this one,
let's pull from Customer_Prod and for our sync,
it's going to go into the dataset Customer_Dev.
Since that table does not yet exist there,
I'm going to say auto create table.
And so this one activity is taking care of the task
of loading the customer table from prod
and placing it in dev.
But the beauty of pipelines is that we don't have to manage
every little thing individually.
We can roll up groups of tasks or activities
into one pipeline.
So let's copy another table,
move that over make it nice and neat.
And let's call this one
SalesOrderDetail Copy,
performing a very similar task.
But this time we're choosing a different dataset.
The one for SalesOrderDetail,
pulling from prod and going to dev
and, again, creating the table.
So now every night we're copying both the customer
and SalesOrderDetail tables,
but what if we have more complex scenarios than that
as part of our nightly load?
That's completely okay,
let's grab a data flow over here,
toss it in and notice I don't have these chained together,
which means they'll operate independently of one another,
but I could chain actions together as well.
Or I can grab a data flow like this.
Let's call this one Product Transform.
And for our product table,
we're going to do a little bit of a transformation
instead of just straight copying the data.
Under settings, let's do a new data flow
and I'll dismiss this message.
Let's add the source
and select the dataset Product_Prod.
From there, we can chain new activities such as aggregate.
Let's say, I want to aggregate some values for this table.
Let's choose aggregates.
And for the column; StandardCost.
In my expression, I can use expression builder.
Let's say let's grab a function, such as average,
and then throw in the column
StandardCost.
Save and finish.
And so that's going to average our StandardCost column
and we can pick a group.
I have something like,
Hmm, let's say color.
Of course, that's going to need somewhere to go.
So let's click the + again and go all the way down
to sync.
And for this, I'll choose the dataset Product_Avg.
Let's jump over to the settings for that.
And I'm going to tell it to recreate the table every time
because it doesn't currently exist.
I can give it a better name than sink1.
Let's say Product_Avg
and our source was Product.
Now all right, let's pop out of here for a second
over to our pipeline tab
and take a look at what we have.
So we have 2 different copies of 2 different tables
coming from our production environment
and copying the entire table down to dev.
And then we also have a data flow
that is not only copying the table,
but performing a transformation
by aggregating the standard cost and grouping by color.
And then placing that in another dev table.
Of course, before we can use the pipeline,
we need to publish it.
So let's click Publish All and publish.
That'll take just a moment and it will save
all of our changes: our new pipeline, our new data flow,
and all of the added components.
As mentioned, one of the beautiful things about pipelines
is that these activities can be run together
either on demand or on a schedule.
So we could add a trigger for a later time,
but let's go ahead and run them together now
and click OK.
We can monitor that over here on the monitor tab,
it will automatically update you,
or you can hit refresh to check on the progress.
I'll pause the video here,
and I'll be right back with you when it's completed.
And we're back.
As you can see, it has successfully completed.
It took about 4 minutes and 34 seconds.
And so we have successfully created a data pipeline
and executed it.
We pop back over to our author tab.
Remember you can chain any number of activities together
to perform whatever loads and transformations you want.
But I wanted you to have a full picture
of creating a pipeline,
throwing some activities in there and using your data sets
and linked services as we've talked about.
By way of review,
data pipelines are logical groupings of activities
that together accomplish a task.
No matter what that task is,
whether loading your development environment
or setting up an ETL/ELT solution
for downstream data analysis,
you can configure and chain together activities
in a multitude of ways to create the types of pipelines
that you need.
And this allows you to manage activities as a whole,
rather than individually.
As you saw, we had 3 different tasks being executed
as a unit, copying 2 different tables
and then transforming another.
This is great for deploying and scheduling them as a unit
that makes sense with your business needs
rather than having to tediously do every activity
on its own.
You can use pipelines to construct end-to-end,
data-driven workflows for your data movement
and data processing solutions.
And keep in mind that this doesn't apply
only to Azure Data Factory.
You can create pipelines in a very similar fashion
in Azure Synapse Analytics, as well.
If that's the utility you're working in
for an analytics project.
That's it for this lesson.
Thank you for joining me.
I hope this helps you get familiar
with what it looks like to create a data pipeline.
Be sure to fire up the Cloud playground and go in
and experiment for your own.
Nothing beats hands-on practice.
Create pipelines, datasets, and linked services
to your heart's content until it's second nature.
And before too long,
you're going to be a data transformation master.
All right.
That's it, Gurus. When you're ready,
I'll see you in the next one.