Hey, what's up, Gurus?
In this lesson, we are going to take a couple of minutes,
and we're going to talk about Data Factory Flow.
So let's get started.
So in the lesson, we are going to discuss
what Data Factory Flow is.
And then we're going to talk about why it's useful.
We'll finish up by taking a look in the portal,
and I'll actually show you how Data Factory Flow works.
So at the beginning, Data Factory Flow
is just a visual way to do data transformation
in Azure Data Factory.
So this is a visual no-code solution
that allows you to do some basic transformations
in Data Factory, which is very helpful.
Now, basically what you do is, you go into Data Factory,
and you can create Data Factory Flows,
which are then added into your Data Factory pipelines
as an additional step.
So why would we want to use a Data Factory Flow,
and why would we not want to use a Data Factory Flow?
Well, the reason we want to use it is because,
hey, it's faster, and it doesn't require code.
If we're already in Data Factory,
and we want to do some simple transformations,
simple being the important word, hey,
Data Factory Flow can be a fantastic solution,
because we can just very quickly visually draw out
what we want to have happen,
and then implement that directly into our pipelines.
So think Databricks Light when we talk about this.
Now, why we wouldn't want it
is if we have a lot of complexity.
If you have a lot of complexity in your transformations,
Data Factory Flow is probably
not going to be the best solution.
Or, if you need something lighter still,
let's say you're pulling your data
into a visualization solution like Power BI,
you might want to do some transformations in Power BI,
some very simple transformations.
So it really comes down to what you're doing.
If you're already using Data Factory,
and you need some basic transformations,
Data Factory Flow is a fantastic solution.
If you have complex transformation needs
or you're using an additional service like Power BI,
you might want to evaluate
and see which solution would be the best one for you.
So that is the basics of Data Factory Flow.
Let's jump on over to the portal
and take a look at it in action.
So here we find ourselves in our Data Factory studio,
and what we're going to do,
is we're going to choose Author,
and I'm just going to create a real quick data flow for you.
So we'll come over here, click on the 3 dots,
do a new Data Factory Flow.
Now, at the beginning,
there's 3 basic things that you want to see.
So, the first is our top bar for this.
Then we have our graph or our pane,
where all the work is going to take place.
And then down below that, we have our configuration area,
where we can configure the different components
that we're dragging into Data Flow.
So let's go ahead and start by adding a source.
So I'm just going to click on the Add Source.
It's going to pull in our first source box,
and I'm going to come down to my configuration panel,
and I'm just going to choose the components.
Now, for this,
I'm not going to go into all the configuration,
because that's way more detailed than we need,
but we're going to go ahead
and just choose a sample dataset that I had created.
Choose source2 by clicking another add source.
And we're going to create a second source,
and we could continue that process on as much as we wanted.
And you can see that I can drag around my canvas
and kind of resize things as needed.
So now let's say that we have our 2 sources,
and the very first thing that we want to do
is we want to join those 2 sources.
So I can click on this little plus button here,
and I can look at all of the different types
of transformations that I could do.
So let's start off by creating a join.
So right now, we're going to join,
and it's only connected to 1 source.
So that's our left stream,
and I can also choose our right stream,
and you can see that it just creates
that nice little bar for me,
and I can choose different types of joins,
and I can create join conditions however I would like to.
I can also come in to Optimize
if I want to go more detailed,
and I can create different types of partitioning
for the different steps.
So you can get pretty detailed with what you want to do.
But essentially, all you're going to do
is click on that plus button,
and you're going to keep going.
So I could create an aggregate.
And I could come down here and I could define the columns
that I want to aggregate together.
So I could do all of my transformations,
and then when I'm done, I can take that data flow,
and I can introduce it into my Data Factory pipeline.
So it's really a very handy toolset
that allows you to do quite a bit.
So let's finish up and kind of review
what we have talked about.
So what is Data Factory Flow?
Well, it's a code-free transformational tool
that we can use in Data Factory.
We use it because it is code free.
It's already integrated in with Data Factory
and allows us to directly connect our flows into pipelines.
But there are a few limitations in what you can do,
so if you have something very complex, again,
you may not want Data Factory Flow.
Kind of depends on what you're doing.
So a very useful example would be,
if we have duplicate data, missing data, things like that.
So, simple transformations.
You want to take a look at Data Factory Flow.
So just make sure you have those concepts in mind.
You understand what Data Factory Flow is
and the basics of how easy it is
to create a flow in Data Factory.
That's it for this lesson. I'll see you in the next.