All right, so in this lesson,
we are going to talk about how to load
a DataFrame with sensitive information.
Now, this lesson is very much going to be a blending lesson.
We're going to be combining topics on SAS tokens,
Azure Key Vault, and we're going to be taking
several trips to the portal.
So if you find yourself questioning what a SAS token is
or how to use Azure Key Vault, make sure to jump back
into those lessons. But I'm hoping that, as we move forward,
you're starting to see all of this come together.
Next, we're going to take a look at some code in Databricks
that you need in order to complete this task.
It's time to go back to the Key Vault.
Now, the very first step in this process
is adding a Blob Storage and generating a SAS token.
We need to do that in order to create this secure connection
between Databricks and our Blob Storage.
So let's actually go ahead and hop into the portal
and take a look at how we do that.
So here, we find ourself in my storage account.
And what we need to do is we need to generate a SAS token.
So we're going to come down here to Shared Access Signature,
click on that. And you can see here that I can create
a whole bunch of different options to really specify
what I want this token to do.
So let's just go ahead and allow all resource types for now.
And we'll leave the rest of this as is,
and we're going to go ahead and generate our SAS token.
Now if you're doing this for real, you're going to want
to change your start and expiration time,
because that is same day and not very long.
So, let's go ahead and generate that SAS token string.
So I've clicked on that, and now it has generated my string.
And so if I scroll down, and again, let me point this out.
So these resources are all going away.
So it's not really that much of a security concern,
but you would not want to be sharing this information.
So it gives me my SAS information.
So we'll come back to that here in just a few minutes.
But that's the first step in the process.
The second step is hopping into Key Vault
and configuring a secret to store my SAS token,
because it's got to go somewhere, and we talked a lot
about not hardcoding, but instead,
storing our keys in the Key Vault.
So let me hop into my Key Vault,
which is the second tab here.
So now we find ourself in my Key Vault,
and I have scrolled down and clicked on Secrets.
So now, I can generate or import a secret.
So let me go ahead and click on that.
So let's go ahead and give this a name,
and we'll just call it dp203storagesecret.
And we want to paste our token in,
and this is the token that we got from this last storage.
So we'll go ahead and paste that in.
And then let's go ahead and create that secret.
All right, so now we have our storage secret,
which is step 2.
Then, we need to link our Azure Databricks to the Key Vault.
And we do that by going into Databricks,
under Create Secret Scope.
And you can actually find that by going to the URL
of your Databricks, and then #secrets/createScope.
Or you can access it from the menu here.
But I've opened this up, so let's go ahead and type in
just dp203secret for our name,
and then we need to link our Key Vault.
And so to do that, we're going to come into our Key Vault,
go down to Properties. We're going to grab our Vault URI,
and then we're going to paste that here.
And then we're going to grab our resource ID as well,
and paste that in right here.
Go ahead and create that,
and then that will configure our secret scope.
Let's go ahead and click on that.
All right, so we have now generated a SAS token,
we have created a secret to store our SAS token
in the Key Vault, and then we have linked
our Databricks to our Key Vault.
The next thing we need to do is we're ready to go ahead
and mount our storage container to our Databricks account.
And this is the code that we will be using to do that.
And so you can see, actually, here at the very top,
we are adding the storage account, the container,
and the reference for the secret
in order to pass our SAS token.
So this is the piece that's really important,
up here at the top.
Now, there's 2 different ways to do this.
This is the first way.
Now if we do this, anyone who has access to this workspace
will have access to this data source.
If we just want the data source to be passed or written
directly using our SAS token, which means that not everyone
in the workspace would have access,
then we want to use this.
So this is a code snippet that allows us to use
our SAS token to directly write,
without actually mounting our data source.
Very similar, you can see our container and our SAS token,
and then the source and URI.
So this is what we would use
if we want to just directly write.
All right, so let's go ahead and wrap all this up.
It really comes down to the basic flow that you need
in order to mount and write securely to a data source.
First, we create the SAS token.
Then we create a secret to store the SAS token.
Then we link our Databricks to our Azure Key Vault.
Then we mount the container using the Key Vault
or alternately, we just write directly
using that SAS token with that second code snippet.
And with that, we're all done.
That's what we need to remember to load a DataFrame
with sensitive information for the DP-203.
All right, I'll see you in the next lesson.