Hey, what's up, Gurus?
In this lesson,
we are going to be talking about Azure Databricks.
Specifically, we're going to take a look
at the Azure Databricks notebook,
and I'm going to talk to you
about how you would implement that in Azure.
Now, a couple things to keep in mind.
We are not going to really focus on the code
because you don't need to understand
the code for the DP-203 as it relates to Databricks.
Really, you need to more understand the concepts
and understand what Databricks can be used for,
so don't focus in too much on the code.
Next up, this is more of a demo than the other lessons
that we've been looking at, because, again,
we really just need to understand the basics,
and the quickest way for me to explain the basics to you
is just to hop into the portal and show you how it works.
So with that in mind, let's hop in, take a look.
Alright, so I have opened up
an Azure Databricks workspace.
And the first thing that we want to do
is hop over here into this left panel,
and we're going to make that a little bit bigger,
and let's jump in and take a look at the workspace.
So we can click on Workspace.
Let's go ahead and click on Users,
and then let's go to Create a Notebook.
And I want to show you what happens.
So let's just call this thing test.
Right away, we can choose a default language.
Now keep in mind, this is the default language.
We can actually switch languages as we move in
from cell to cell in the notebook,
which I'll show you here in a minute.
And then we need to select a cluster.
Hey, we don't have a cluster running.
That's going to be important in order to make a notebook.
So we could go ahead and create it.
But if I did that, it's still going to need a cluster.
So we're going to want to set up our cluster.
And so let me go ahead and jump over and create that now.
So we'll go back to this left panel, scroll down to Compute,
and let's go ahead and create an all-purpose cluster.
We'll just call this, test.
Standard is just fine.
And we can choose our runtime version,
but we're going to leave it as just our standard 9.1.
We don't want to enable autoscaling for what I'm doing,
but in a production environment,
you would want to enable autoscaling
so that you don't run into issues as you run your notebooks.
And then we can terminate.
And I'm just going to go ahead
and set this to 30 minutes of inactivity,
but you can set that if you are concerned
about the cost of the cluster, which you should be,
unless you are, again, in a production environment,
in which case you might leave it running all the time.
We're going to choose our worker type,
and we can leave that at the DS3_V2.
And I'm going to go ahead and ramp down the workers,
because, again, this is just more of a demo.
And there we go.
So let's go ahead and create that cluster.
So we'll let that spin up,
and I'm going to go ahead and pause the video here,
and I will resume it
as soon as we have our cluster up and running.
Alright, so we are back
and we have our cluster all up and running now.
So let's go ahead and hop back
into our workspace and our notebook.
So I'm going to go ahead and open up my notebook.
And if you will notice, up here in the left-hand corner,
it says we are detached.
That is referring to our cluster.
So let's go ahead and click on that
and choose the cluster that we spun up.
So now we are connected and ready to go.
So this is your notebook.
And inside of the notebook, we have cells.
Each one of these cells is essentially 1 line of code,
and we can run these cells individually.
So let me just go ahead and grab a few lines of code.
And we're just going to use
a standard Quickstart notebook section from Databricks,
and we'll run a couple of those.
Now at the very beginning, I want to show you as well
that we can actually change the language
by just using that percent sign
and then typing in the language
that we want to use for this cell.
So let's go ahead and run this cell.
And you can see here that even though this is SQL commands
and we're running it in a Python default notebook,
we can run the SQL command by using this %sql.
So we have now completed this cell
from a Databricks sample notebook,
so you can see some code quickly.
So we just simply pulled that out, pasted it in here,
changed the language of this cell to SQL,
and we are off and running.
Now you'll note that since this is SQL,
if I was to paste this in here and try and run this,
it's not going to work
because this cell is going to default to Python,
which we can see in this upper right-hand corner.
I can also click on that,
and I could have chosen a different language here as well.
So if I change the language over to SQL, it would run.
So one last thing that I want to show you here
in the notebook.
Let's go ahead and grab, and we'll kill this cell.
Let's go ahead and just run a simple SELECT statement.
And again, we'll switch it to SQL here.
And if we do that,
we can actually begin to visualize our data.
So you can see here
that it just gives us a standard spreadsheet.
I can actually come down here
and I can begin to visualize this
in different ways if I want to.
And so we can start to visualize in different ways.
Let's go back here to our spreadsheet.
And if I did just a little bit more work here,
I can actually change around
and give myself a little bit better representation
if I want to see things in a bar chart
or a pie chart or something like that.
So this is how we interact with Databricks,
and keep in mind that me changing the languages
is for demonstration purposes only.
If you actually were using every single cell as SQL,
it would make a whole lot more sense
to actually set your notebook to be SQL,
but I wanted to do it this way
to make sure that you could see
how you can change languages.
And so this is a notebook that I have created,
and then we could take this notebook
and I could actually jump into Data Factory,
and I could create an activity in Data Factory
that would tie this notebook,
and it would actually run this notebook
in my Data Factory pipeline.
So we can start to create some very complex scenarios,
and keep in mind, again,
Databricks is really going to be for big data.
That's where you're going to see Databricks used most often.
And primarily you're going to use it,
at least for the DP-203, as a transformation tool set.
It's also very big in machine learning and some other areas,
but again, for the DP-203,
think transformation, think big data,
and that's where you're going to see Databricks.
So with that, let's go ahead
and jump in and talk about the review.
So over here on the right, building a data pipeline
is kind of like an onion.
As we start to shave off more and more pieces,
we get layers, layers upon layers,
and so you're going to be interacting
with a lot of different services.
So it's important to think about the overall architecture
before you jump into the weeds of Databricks or Data Factory
or Synapse, or whatever the services are.
Think about what you're trying to accomplish
and the best way to do that,
and then start to build out your building blocks.
These pipelines are going to be absolutely critical
for data engineering.
Also keep in mind, you'll be using pipelines
in Synapse and Data Factory,
and those are very important for the 203.
Don't forget to set your worker options.
It can be an extremely expensive mistake.
So back here in my cluster,
when I went to Compute,
we went down and we set my workers as 1.
If I was to set that as 10,
I would be getting charged 10 times the amount
of what I'm getting charged now for one.
If you're not careful,
you can make an extremely expensive mistake
by not setting your worker options
and by not setting your terminate
so that your clusters are actually spinning down
after they haven't been used.
Finally, we will take a look at authentication
a little bit later on in the course.
I believe that comes up in section 10.
So don't worry too much
about the security and authentication.
Those are things that we'll be covering later.
The big takeaway for Databricks
is just understanding the impact of clusters
and then understanding the basic principles
of how you would implement a notebook in Azure Databricks.
All right, that's it for this lesson.
I'll see you in the next.