Hey, Cloud Gurus.
Welcome to the first of a series of lessons
on meeting the tools of the trade.
Getting familiar with what you'll be using
to ingest and transform data within Azure.
First up is Azure Data Factory.
So in this lesson, we're going to start off with a recap
on Azure Data Factory.
We're then going to briefly look at native data flows
and external transformations,
just to get you familiar with what they are.
Most of our time is going to be spent in the demo,
taking a hands-on look at transformations
in Azure Data Factory.
And then we'll wrap everything up with a review.
Brian did a fantastic job in section two,
introducing Azure Data Factory.
If you remember this slide from his presentation,
within ADF we have structures called pipelines
they are logical groupings of activities.
Our activities can be data movement,
data transformation and control.
Datasets are what allow us
to actually interact with the data
and linked services is how we connect to the systems,
holding that data.
In this lesson, we're focusing in on activities.
And more specifically,
data movement and data transformation activities.
Let's start by looking at our native data flows.
First of all, we have the Mapping Data Flow,
and this allows us to visually design data transformations
without writing a single bit of code. Pretty awesome, right?
As you'll see in a moment,
you can link different items together
and direct your data into the shape you want it to be.
Think of it kind of like rivers of data coming together
and changing along the way
to produce one combined stream in the end.
Another code-free way
of accomplishing data transformation is data wrangling.
This utilizes Power Query to accomplish, again,
code-free data preparation at cloud scale.
This utilizes Power Query
to accomplish code-free data preparation at cloud scale.
This activity was previously called a wrangling data flow.
It integrates with Power Query Online
and makes Power Query M functions available.
Both of these are executed as activities within the pipeline
because the transformation is really just an activity
that executes in a computing environment
such as Azure Databricks or Azure HDInsight.
Moving past our native data flows,
we have a number of external transformations
and these allow you to bring your own compute, if you want--
such as an HDInsight Cluster.
We have the HDInsight Hive Activity,
which lets you execute Hive queries on your cluster.
The HDInsight Pig Activity,
which lets you execute Pig queries.
The HDInsight MapReduce Activity,
which executes MapReduce programs.
And those are just a few examples.
We have many more, such as: HDInsight Streaming Activities,
HDInsight Spark Activities,
Machine Language Studio Activities,
Stored Procedure Activities,
Data Lake Analytics with U-SQL, Azure Synapse Notebooks,
Databricks Notebooks, Databricks Jar,
and Databricks Python Activity.
And if all of that still isn't enough,
you can create a custom .net activity.
And manually create the transformations that you want.
All of that to say, there is a huge amount of flexibility.
And so whatever your transformation need is,
Azure Data Factory pretty much has you covered.
With that overview in mind,
let's jump into the Azure portal
and take a look at this in action.
Here we are in the Azure portal
and I've opened up Azure Data Studio
within my Azure Data Factory.
If we come over to the pencil icon that stands for author,
we can begin to work with our pipelines.
We have categories here for our pipelines,
datasets, data flows,
and even its own separate category for Power Query.
That is a change in recent years.
And as you know with Azure,
you'll see the interface change along the way
so don't be alarmed by that.
It's just part of the cloud constantly improving.
But to get started, we can click on the + sign
and let's create a new pipeline.
I'll leave it as pipeline1, just for fun.
Really what we want to do here during this lesson
is take a look at the various activities
we were just discussing.
So under Move & Transform, we have different options here.
If you want to simply copy your data and not transform it,
there is an activity for that.
And you can see, we can easily specify our source,
such as our Dev environment and
our Sink--where we're going to--like our Prod environment
or vice versa.
We can change mappings if they're not identical
and a few other general properties,
but that's not going to accomplish a lot for you
if you have a complex scenario.
For that, we have the Data flow activity
that I was just telling you about,
one of our native data flows.
Within here, we can accomplish some more complex tasks.
If we go to Settings and start a new Data flow,
it'll give you lots of helpful hints along the way
to help you learn how to set up your data flow.
We can add a source.
And this is very similar to our copy activity
where we're defining a dataset to pull from.
But here's where things really start to get fun.
If you click on our little + icon,
you can start to see the different directions
you can guide your river in,
to bring about that clean stream at the end.
We can join the multiple datasets.
Say we had a couple of different sources.
We can split it into 2 if we only have one
based on certain values,
and a number of other powerful transformations.
As you start to add these together,
they can branch into more and more.
And eventually, come out to the sink, or target,
with the state that you want it to be in.
And remember, this is just one element within our pipeline.
Once this result is achieved,
we can still add to that in a chain of other activities
for our complete pipeline.
If Power Query is what you're looking for instead,
to do more data wrangling
that has its own category now down at the bottom,
we have Power Query currently in preview.
If we click on Settings, we can do a new Power Query.
I'll select the same dataset, add that in there.
And you can begin forming your Power Query.
You see some similar elements here, such as transformations,
combinations and the like,
but this gives you another option
for how to interact with and transform your data.
Again, still just one activity within this pipeline.
We could even combine them together
and go from one to the other.
But we'll talk more on putting pipelines together
in a later lesson.
Before we pop out of our demo, I wanted to show you
where our other external transformations are.
If you remember, many started with the word HDInsight.
And so those are here under this category, our Hive,
MapReduce, Pig, Spark and Streaming Activities.
There is also under--Data Lake Analytics--U-SQL.
And so I hope you can begin to imagine
how you can mix and match these together
to accomplish very simple
or very complex data transformations.
By way of review, mapping data flows and data wrangling,
or, as it's now called, Power Query,
allows you to build code-free transformations
at cloud scale.
They are natively integrated into Spark Clusters
that automatically scale.
There are a variety of external transformations as well
that enable you to accomplish almost any task.
With these, you have the option
of connecting to your own compute instance,
such as an HDInsight Cluster that you already have set up.
And they really fill out the flexibility
of Azure Data Factory.
Remember that all of these execute as activities
within a pipeline and so they can be standalone
or chained together as we saw
to produce your desired end result.
That's it for this lesson, thank you for joining me
as we talk about how to use Azure Data Factory
for data ingestion and transformation.
Next up, we'll talk about another tool of the trade
that you can have in your arsenal as a data engineer.
When you're ready, I'll see you there.