Hello and welcome back to the course on Machine Learning.
We've already talked about hierarchical clustering and how the algorithm works.
Also we talked about dental grammars and how they're constructed.
Today we're going to put the two together and learn how to get
the maximum value out of our hierarchical clustering algorithms.
So let's get straight into it.
All right so here we've got an example.
The example that we looked
at previously where on the left we've got the points you know scatterplot.
And then here on the right we've got the dendrogram as a which contains the memory
of how the clusters were formed during the hierarchical clustering algorithm.
So here we can tell right away from
first of all Peter and P3 were combined into a cluster.
Then because their
their height is the lowest, the height of this bar is the lowest.
Then we look at the next lowest bar is this one.
So P5 and p6 are the least dissimilar out of the remaining.
And then these are pretty much the same height.
But we first perform cluster A we combine these into one cluster.
So P1 was added to cluster p2 p3.
Then P4 was added to cluster of p5 p6.
And then at the end all of the points were combined into one cluster.
So that's what the dendrogram is telling us.
As you can see right away, is giving us a lot of additional information
on top of the scatter plots.
And, it contains that memory of the hierarchical clustering algorithm.
So how do we use this dendrogram to understand
how to best execute, or get the most value out of the HTK.
So let's have a look.
What we need to do with the dendrogram,
or what we can do is look at the horizontal levels
and set thresholds so we can set height thresholds a distance actually distance
thresholds are also called dissimilarity thresholds
because this vertical axis measures the Euclidean distance between points,
which also represents the dissimilarity between them or points or clusters.
So what we can do is set a threshold for all dissimilarity.
And we can say that, we don't want the dissimilarity to be greater
than this level.
So again, it doesn't matter what the absolute value is, it matters
what's, the relative values and how it looks on this image.
So we we're setting the dissimilarity threshold.
We're saying that anything if we come across clusters
that are above this threshold.
So we don't want within a cluster to have dissimilarity above this threshold.
So what that will do is it'll give us two clusters.
And let's have a look at them.
There's our first cluster and there's our second cluster.
And that's that makes sense.
So what that it's telling us is that within each one of these clusters,
the dissimilarity is always less than our threshold.
So let's say we've got some values here.
Let's say this is 1.5. This is 2.0.
So let's say we want to set the threshold at 1.7.
And what this is doing is it is not allowing any clusters
that would have dissimilarity of greater than 1.7 within them.
And as you can see from the dendrogram, we can tell that over everything below
that level, this cluster and this cluster,
they don't have dissimilarity of 1.7,
because the similarity is represented by these vertical lines.
And that's how the concept of thresholding works.
And the interesting part about dendrogram is you can quickly tell how many classes
you will have at a certain threshold by just looking at how many
vertical lines this horizontal threshold actually crosses.
So here you can see it crosses one two vertical lines.
That means we will have two clusters.
Will be this cluster of all these points p1, p2, p3 and this cluster p45 p6.
All right. So let's have a look at another example.
Let's have a look at a example where we put the threshold at this level.
So somewhere just below where we combined as you remember
we had p5 p6 in one class A, p2, p3 in one cluster before by itself P1 by itself.
And then we combined P1 with this cluster, P4 with this cluster.
So let's say we're setting the threshold at just before
that level of dissimilarity, which allowed us to combine
P1 with this cluster and before with this cluster.
So what that will do is it will give us a certain number of clusters.
So can you tell just by looking at the dendrogram how many clusters will have.
Exactly correct.
We're going to have four clusters
because it crosses four vertical lines one, two, three, four.
Right.
So we're going to have a cluster P1
cluster with P2 and P3 cluster with before cluster five and p6.
Let's have a look for clusters.
And there they are.
So that is what we're going to get if we set the
dissimilarity or distance threshold at that level.
Let's try another one.
Let's say we want to set our dissimilarity threshold very low at 0.3,
meaning that we don't want clusters that have any points
within them that have dissimilarity greater than this threshold.
So we're not going to allow any clusters like that.
And the interesting part here is that we're actually setting the threshold
below our very first cluster that we created over here, P2 and P3.
So we're not even going to allow P2 and P3 to be combined in one cluster.
We're going to say
that dissimilarity level, that distance between them is too great, too high.
We we don't think that based
on our business knowledge or based on our other internal research
or external research, that we don't think that any points with,
dissimilarity greater than this level should be combined into a cluster.
It's just it just doesn't make sense
from a, from a finite line of financial, from a business perspective,
from a perspective of knowledge about what this dataset is about.
And what that will do is it'll create six clusters because we cross six lives
one, two, three, 4 or 5, six, and then they are every single point
will be in its own cluster.
As you can see, we've got six clusters.
So that is how a dendrogram works or how you can get value out of a dendrogram.
And you can set this threshold at different levels to understand
how many clusters you'll get.
Just by looking at the dendrogram, you can tell right away.
And, you can that we find the optimal level for the threshold,
or the optimal number of clusters that suits your project the best.
So but how do you find the actual,
not just an optimal number of clusters that you think is optimal?
What is the dendrogram giving us?
Any ideas about the optimal number of clusters?
Well, what can we tell from the dendrogram?
That might be a good guide for us to select the optimal number of clusters.
Well, there's a great giveaway that the dendrogram contains,
and that is the vertical distance because it is measuring a dissimilarity.
So the one of the standard approaches is just to look for
the highest vertical distance that you can find on the dendrogram.
So basically any line that will not cross any horizontal lines.
So for instance this line can be considered.
This line can be considered.
This line cannot be considered for that research
because it crosses hypothetical horizontal lines.
So what you need to do is kind of like every horizontal line you have.
Just imagine it extends all the way across the dendrogram.
Every single horizontal line you have.
And now find the longest line among yours, among your existing vertical lines
that doesn't cross any horizontal, any of these extended horizontal lines.
So for instance, even this line cannot be considered
for that purpose because it would hypothetically cross this horizontal line
that we have coming from this red line between 5 and 6.
Again, this line cannot be considered because it's crossing this line.
So you would need to look at this line, for example, or this line.
Or if you wanted to use this line, you would need to use only a bit of it,
that part or this part.
So you can only use parts of lines that are between horizontal lines.
So out of all of the lines that you have here,
which is the longest that doesn't cross any extended horizontal lines.
Well that's correct.
This one over here is the longest one.
Or basically in our example, the green and the red
there were about the same height.
So this one or this one are the longest ones.
And so this is the largest distance
and therefore the best or the recommended approach.
Again it's not a set in stone approach.
It's a kind of one of the things that you could do
is take a threshold that will cross this largest distance.
So cross that largest distance with a threshold,
and then you use that threshold to calculate
the optimal number of clusters and actually find them.
So once we've crossed this, largest distance with our threshold,
it doesn't matter what you said, you can set it here,
you can set low or you can set high as long as it crosses this line.
Then now the two clusters are this one and this one.
As you can see, that is considered to be one of the approaches.
or this approach is telling us that the optimal number of clusters are two
and these are them.
And kind of in this case it makes sense.
You can see that indeed these points look that as if they're closer together.
And these points look as if they're closer together.
that rather than getting any clusters in between them or even breaking up
into more classes, wouldn't make as much sense as this makes sense.
And, so there you go.
That's that's one of the approaches that you can use.
You can still look at this whole problem using
a similar approach to K-means, where you use the elbow method.
So you could use something like that.
But in, hierarchical clustering
we're going to focus on this approach with the largest distance.
And now let's quickly have a knowledge test.
So I'm going to I have two charts here which are hidden on the left.
We've got the scatterplot on the right. We've got the dendrogram.
I'm going to show you only the dendrogram.
And I would like you to try to understand or try to assess
very quickly what's going on on the scatter plots.
So for instance, we'd like to know even without seeing the scatterplot
or the data set.
At the moment we'd like to know what is the optimal number of clusters
in this dataset just by looking at dendrogram.
Can you identify that.
So if you like you can pause the video and just look at, these vertical
and horizontal lines and try to find out based on the method that we discussed
what would be the optimal number of clusters.
So in 3 to 1 I'm going to now reveal how I would solve this, challenge.
Well, what I would do is I would look for the
the longest vertical line that doesn't cross any extended horizontal lines.
So if you extend that extend that extend that,
you can see that it's probably this line over here.
And so that's the largest distance.
That means we need to cross it with a horizontal lines with our threshold.
And that will give us the number of clusters
which is three clusters because it crosses three lines here one, two, three.
And if we look at the chart, as you can see, indeed we do have three clusters.
And it does look that, that like that
is the optimal number of clusters for this business problem.
So hopefully you enjoyed this tutorial.
We walk through all of this many so that you have a better intuitive
understanding of how the hierarchical clustering algorithm and dendrogram work.
And next, headlong will show you around in R and Python, and together
you will create some amazing analysis around hierarchical clustering.
And together with him you will solve a business problem
using the hierarchical clustering algorithm.
There you got some fun tutorials ahead of you
and I look forward to seeing you next time.
Until then, enjoy machine learning.