Hey, Cloud Gurus.
Welcome to our lesson on integrating
Jupyter Python Notebooks into a data pipeline.
In this lesson, we'll be taking a look
at what notebooks are,
talking about how they fit into data pipelines,
giving a little bit of a demo so you can see it for
yourself, and then wrapping everything up with a review.
To start off with, let's overview notebooks a little bit.
A definition online that I liked says,
"Jupyter is a free, open-source, interactive web tool
"known as a computational notebook,
"which researchers can use to combine software code,
"computational output, explanatory text,
"and multimedia resources, in a single document."
So you can throw a little bit of everything in there
and see it all in line together.
You can throw some instructions, some Python code,
and then even the output right there in the notebook.
Specific to our purposes, it allows you
to use Python and Scala code in Azure Databricks
and Azure Machine Learning for data transformations.
So you can use these powerful languages
to do very customized transformations,
and then even share them collaboratively
with your coworkers.
And when everything's just the way you want it,
you can throw it into an Azure Data Factory pipeline.
To give you a bit of an idea of how this might look--
perhaps we have a copy activity copying from a data lake.
We can send our failed ones to an activity
to fire off an email,
but we can also use an Azure Notebook activity to process
the files from that data lake.
It will go through and perform all of the transformations
exactly as you've programmed it using Python,
and then perhaps send it off to Azure Synapse.
And so this really allows you to operationalize
your Databricks transformations.
Some of the benefits of this include,
easily pulling data from a large number
and variety of sources.
Azure Data Factory allows you to connect
to all kinds of sources.
And so you can now use those with your Databricks Notebooks.
To give you an idea of how great that is,
previously, you would've had to write custom code
to pull from all those sources and even to schedule and run
the notebook in Databricks.
Now you can also easily integrate
with other Azure Data Factory components.
Again, instead of writing custom code
to tie them all together,
you can easily chain one from another
in your Azure Data Factory pipeline.
And it also gives you the ability
to parameterize your notebooks.
In a word, all this means is making things simple,
taking a lot of things that would have required custom code
in the past and easily operationalizing them in a pipeline.
Let's jump over to the Azure portal
and just get a quick visual
of what these activities look like.
Here we are an Azure Data Factory Studio.
If we come back over to our familiar, pencil/author button,
this brings us back to the working area
we've seen in other videos.
Real quick, let's create a new pipeline to test with
and maybe call it DatabricksDemo.
Under our Databricks activity,
here's our notebook.
And I previously set up a linked service.
If you want to view your linked services,
you can come down to manage and Linked Services,
and you can see, I have an AwesomeCompanyDatabricks
linked service.
But back on our author tab,
we can come to this activity, to our Azure Databricks tab,
and select that linked service.
That connects up to a workspace that I've already created
for the sake of time and a couple of notebooks there
that are pre-populated.
If we click on Settings,
we can specify here the notebook we want to use
in this activity.
So if I browse and go to Users,
under my user,
I have a couple of samples here.
So I can select my Awesome Company Sample and click OK.
And it now knows the notebook to use for my activity.
If I bring data in from some source,
let's say Azure SQL database, Azure Data Lake.
I can then connect it to this transformation
and have it run the Python code
within that notebook against it.
And it's just that simple.
What in the past would have taken a lot of custom code:
1) to pull in the source 2) to configure it,
to work with Databricks transformation,
and 3) to output it to a sync,
is now easily operationalized in one, single pipeline.
By just adding some activities
and configuring some settings.
By way of review, Jupyter Notebooks allow you to write
data transformations using Python and Scala
in Azure Databricks.
The notebook activity then enables you
to integrate these transformations
into your Azure Data Factory pipeline.
And this dramatically simplifies the process
of utilizing Databricks Notebooks
in your ETL or ELT workflows.
Again, before you would have had to use
all kinds of custom code to pull in the data,
chain all these things together,
but now you can utilize the wide range of sources available
in Azure Data Factory and the wide range of activities
that you can connect your notebooks to,
and simply chain them together in a pipeline.
I hope this has been helpful and allowed you to visualize
how you can utilize Jupyter Notebooks
in conjunction with your Azure Data Factory pipelines.
That's it for this lesson.when you're ready,
I'll see you in the next one.