-: Congratulations!
You have almost finished this section.
In this lesson,
I'm going to talk to you about interpreting
a Spark directed acyclic graph, hereafter known as DAG.
So when we get started,
we're going to start with what a DAG actually is.
Then I'm going to jump in,
and we're going to talk about how we use DAG in Spark,
and then follow that up by how we execute a job in Spark.
So, the basics of DAG. Let's start with a vertex.
So this is just a point. It's also called a node.
Then we have an edge. There's the edge.
So we have 2 vertices.
Those are the 2 nodes, the 2 blue circles,
and then we have the edge that's connecting them.
Now, if we take a bunch of these,
and we put them in order like this,
you can see that we have a graph,
and we have a direction to that graph.
It's moving down.
Lastly, as we look at DAG,
keep in mind that these are acyclic,
meaning that we go from start to finish,
and we don't cycle back around to the beginning.
So this is the basic of what a directed acyclic graph is.
All right, so moving on,
let's talk about what this looks like in Spark.
To start with, we need to talk just a tiny bit
about resilient distributed datasets, or RDD.
So these are the main logical data units in Spark.
A good way to think about that
is like a partition that is read only,
and inside of that partition,
we have our data that we can perform work on in Spark.
And this is what you use or what you can use in Spark.
And so when we talk about DAG, here's an example
of kind of what this looks like in practice.
So we have our collection of records,
and we're going to perform some work
on these resilient distributed datasets.
So we're going to start off here in stage zero,
and we're going to pull in a text file.
Then what we're going to do
is we're going to split the line of the text into words.
So once we've done that, we're going to take that,
and we're going to map that into word pairs.
And then finally, we're going to sum counts for each word.
And so what we're doing here
is we're developing a DAG visualization.
So basically, we are taking all of the different actions
that we're going to do on those datasets,
and we are putting them in order
in an acyclic-directed fashion.
So that is what a simple DAG looks like
in execution in Spark.
So then finally, what we're going to do is we're going to
jump in, and we're going to take a look a little bit further
into that execution,
and I'm going to show you what's happening behind the
scenes. So we start off with our driver,
and the driver is going to take that request
that's been given to it,
and it is going to interact with the cluster manager,
and it's going to ask for worker nodes to be spun up.
So the cluster manager is going to allocate resources
and spin up some workers.
Now, these workers are just simply nodes
that are going to perform parts of that request in parallel.
And so with inside that worker,
they have an executor and then the individual tasks
that are assigned to that worker.
And so then the other piece to this
is Spark is going to create an operator graph
through this DAG scheduler.
And what's going to happen is it's going to take
all of those tasks,
and it's going to break those down and put them into a DAG.
That DAG then is going to be moved into the workers,
so that the workers know how to perform the different tasks,
or how the tasks get assigned to the workers.
And so this is behind the scenes what's actual happening
as we execute a job in Spark.
All right, we're going to stop here,
and we're going to start talking a little bit
about the wrap up.
First, this is a small topic, don't go overboard.
We did not touch very deeply into DAG.
There is a whole lot of information around Spark
and how resilient distributed datasets work
and how DAG works.
But that is not in the scope of this DP-203.
Really what we need to do is we need to understand
at a high-level what's happening with DAG,
understand that it connects with Spark
and understand a little bit about what's going on
behind the scenes that makes a DAG scheduler or DAG work.
We have covered that.
You don't really need to go much beyond that
so don't waste a ton of time diving into that
just for the sake of the DP-203.
What you really need to understand is what DAG is,
and you need to understand where it's used.
If you understand those 2 things,
you should be good to go for the DP-203 requirements.
You can dive in further
if you really interact with Spark a lot,
or you need to know more.
All right, so with that, we have wrapped up this lesson.
I'll see you in the next.