So this lesson is going to be
a little bit different.
In this lesson, I'm going to talk to you
about how we manage Spark jobs in a pipeline.
This lesson is going to be a little bit different
in that there's not a whole lot of Spark on the DP-203.
So since it's on here,
I'm going to show you how we implement Spark activities
in Data Factory,
and then I'm going to show you in the Azure portal
how you do that as well,
but we're not going to spend a lot of time here
because it's not a main component
for the DP-203.
Having said that,
I'm going to show you 2 different ways to do this.
The first is using JSON.
And so when we execute Spark activities
using an HDInsight cluster,
we're going to use JSON
that looks like this over here on the right,
and I'm going to take just a couple of minutes
to walk you through what that all means.
So let's start off with the top section here
and talk about the basics.
So we're going to name our Spark activity.
We're going to give it a description, whatever we want.
The type, of course, is going to be HDInsightSpark.
And then our link service name is just simply
where our Spark program is going to run,
so our compute, right.
Below that, we are going to then define the properties.
So we are going to define our sparkJobLinkedService,
and this is going to be where our Spark file is located.
So in Azure Storage, where's our Spark job file located?
And there's only 2 choices--
this is actually important--
Blob storage and ADLS Gen2.
So Data Lake and Blob storage are the only 2 options
currently that you can store your Spark jobs in.
So next up, we're going to do just the basics,
and we're just going to describe where that file lives.
And we can include some config file information
if we have some specific values for properties on Spark
that we want to take a look at.
And then finally, we have our debug info
that we can include.
The default is failure, but we have 3 options:
None, Always, or Failure.
And we're just simply specifying where those Spark log files
are going to be copied and when they're going to be copied.
So that's it for JSON.
Now, let me jump over into the portal
and let's take a look at this in Data Factory.
So here we are in our Data Factory,
and I've opened up a pipeline,
and you can see that we also have a choice
for an activity under HDInsights.
So you would just click on the arrow, scroll down,
grab your Spark activity, drag that onto the canvas,
and then you can see here,
we have very similar information.
We can name and describe our Spark activity.
We do have an option of Retry if we want to.
We can define our HDInsight cluster here.
And when we define that cluster,
you can also see, there's the cluster,
and then you can also see below that
we have our storage link for either Blob
or ADLS Gen2.
So we're going to define that connection there.
Then we're going to define our script here
so we can include where our file is located.
And then after that,
we can include some user properties if we so choose.
And that's it.
So you just simply build this into your pipeline.
And as far as in this lesson,
that's really all you need to know for the DP-203.
Pretty short.
We just need to know how we implement an activity
and what it looks like.
If you've got that,
hey, you're good to go on to the next lesson.
I'll see you there.