Hey, Cloud Gurus,
welcome to our lesson on splitting data,
learning how to create and take different paths.
In this lesson, we're going to briefly go over
source branches and conditional splits,
really just introducing them
and then jumping into the demo so that we can see them live.
After that, we'll wrap everything up with a review.
New branches allow you to take multiple paths
to multiple sinks while branching from the same source.
This is a great way to self-join data together,
or even just branch it out to multiple sinks.
This allows you to fully utilize one source
instead of creating multiple data flows
to read from that one source.
You can add these either at the source object
or after a transformation,
and it takes the entire results set
from that particular point into a completely new direction.
Another option is conditional splits.
These route data to particular streams
based on specified conditions.
And so, it evaluates expressions
and then routes the data into the appropriate stream.
This is similar to a CASE statement,
if you've ever used a traditional programming language.
And contrasting to the new branch,
it allows you to take particular subsets of the data
into a new direction
instead of taking the entire stream of data.
With that in mind,
let's jump over to the Azure portal
and take a look at both of these.
Here we are in the Azure portal,
in my Azure Data Factory Studio.
Let's come over to the author tab
and let's create a new data flow.
I'll minimize this just to give us more space
and let's add a source.
For this demo, I'm going to pick the products table.
Let's name this Products, just to keep everything straight,
and start off with let's take a look
at our conditional split.
We have all kinds of products at Awesome Company,
including ones of various colors.
Let's say we wanted to pull out all of the red products.
Let's name this stream RedProducts.
And for our condition, let's say color = red.
We can take that result set and send it over to a sink.
Maybe we'll call that RedTable.
So far that looks like a pretty standard data flow,
just moving the data,
but we can continue to add different splits here.
Let's say that we want, maybe, our black products.
Color = black.
And maybe we want to group all the other products together.
And we can have different sinks for each of those.
Name that one BlackTable,
and then add a sink for our other table.
And this is the power of the conditional split.
You're taking one stream of data
and breaking it in to multiple streams,
depending on the case.
If the product is red, it will go to this table.
Black will go to this table,
and everything else will go to our catch-all table.
And this is different from a new branch,
which we can add here.
If we wanted to use the same product source,
but take it in a different direction,
we can easily do that from this new branch.
From here, we can do completely different things.
For the moment, we'll stick with conditional split
because that's kind of the theme.
But what if we wanted to work
on a completely different field?
Let's bring this back up a little bit.
Maybe we want to just pull out our more expensive products.
We can open up expression builder,
or since I already know the name of the column,
we can just say standard cost greater than 1,000.
And then maybe we'll name our other one, Affordable.
Bring this back down so we can see everything.
This will allow us to use a different sink, still.
We have our Expensive table.
And one more for our AffordableTable.
All of this to illustrate
we have the option of splitting out smaller streams
based on certain cases,
or taking an entire branch and taking it in a new direction.
This allows us to still only have one data flow
that we can call once,
but fully utilize that product's source.
And as mentioned, you don't have to add the new branch
just at the beginning.
You can add that from the transformation level, too.
We could have a new branch off of red products
to do further work with those products,
taking that entire stream in a different direction.
By way of review, the new branch transformation
will allow you to split one source into multiple streams.
The conditional split transformation
will allow you to direct a subset of data into a new stream.
And remember, that's much like your case statement
or maybe a really complex if/then/else statement.
And don't forget that these transformations are available
in both Azure Data Factory and Azure Synapse Pipelines.
That's it for now!
I know you'll pick up on this right away
and be a master of your data traffic,
directing it in multiple streams,
wherever you need it to go
in order to accomplish as much as possible,
as efficiently as possible.
When you're ready, I'll see you in the next video.