Hi. In this video we'll create our gaming model in Jupyter.
Firstly, uh, please install these packages, which we will use in this module,
xg Boost Large G B M Ticket Plot. We have used this one already.
Uh, then, uh, yellow brick. Uh, this package, uh, helps us with, uh,
clustering evaluation,
and these last four are used for hyper parameter optimization.
Okay, now let's go to our Jupyter and, uh,
open the notebook in the module nine folder. This one
in this notebook, uh, we will use two dataset for binary classification.
Uh, we will use one dataset, and for clustering and multi classification,
we'll use the dataset. Uh, we used B four in, uh, module seven.
If you recall, that was the data set, which had around 500 K observations.
Uh, with statistics about cybersecurity attacks.
There was altogether 20 different types of attacks, including smurf,
Neptune, teardrop, and others.
During the first section, as usual, uh, we imported the packages. Uh,
we can skip this section and we can go right ahead to the data set for
clustering or multi classification.
Uh, we load the data set and we see, uh, it has, uh, 42 variables and,
uh, 500 K rows. Uh, we need to do a little bit of pre-processing,
like adding the right column names, uh, then changing few,
uh, variables to proper data type.
We have to also drop, uh, these two columns, uh,
because they have only one distinct value.
And, uh, we can also check if the data set is, uh,
balanced and we see that, uh, it's highly imbalanced. Uh,
but for clustering, uh, it is not important because I, uh, mentioned,
uh, that clustering is an unsupervised machine learning technique, so it,
uh, does not require any target variable.
Regarding, uh, the training dataset, uh, we need to do a little bit of,
uh, reprocessing. Uh,
we will create our model on a sample of, uh, 20 K observations.
Uh, it's usually big enough sample, uh, depending on what you need,
uh, and also the evaluation will be, uh, a little bit faster.
We also have to remote, uh, these, uh, categorical variables.
I noticed here I also dropped, uh, the target variable.
I also created, um, uh, y clustering variable,
which contains the attack classes. Uh, but this is only for the purpose, uh,
to check if the clustering, uh,
worked and if it identified at least, uh,
the high level patterns in the input data.
And because k means clustering is, uh,
very scale sensitive to it inputs,
and we have to normalize the data to the same scale. Uh,
so let's use the minimax scaler. MinMax scaler scales the data,
so each variable will have its minimum at zero and maximum at
one. The x uh,
MinMax, uh, dataset, uh,
will be the training data set used to train the KMS model.
Now, how to find the best clustering depending on number of clusters.
Imagine that we would not know that this dataset has 21 classes.
What we can do is to make, uh,
multiple cains models or multiple number of clusters and evaluate the quality of
clustering with few metrics. The most common metrics used are,
uh, kalinski, harash index,
index distortion, and Davis balding index.
I'll not go into details, uh, about, uh,
calculations of these metrics and why and when they are more appropriate to use.
But, uh, there are 10 of these kind of measure, uh,
which evaluate the quality of clustering.
And a common approach is to do voting or manually explore their values
depending on number of clusters, which is called the elbow method.
Regarding, uh, kalinsky harash index, the bigger value the better. Uh,
same as for silhouette index. Silhouette index is,
is basically an average silhouette value of each observation and can range
from minus one to one. Where, uh, minus one, uh,
is denoting the worst clustering. And one, the best
distortion is the average squared distance of each observation to
it cluster ate, uh, the lower, the better,
and the lower values of the Davis Building Index also means, uh,
better clustering. Now how to do it,
uh, with the K elbow visualizer,
we can make a charge showing us exactly what we need. Uh, we just specify,
uh, the model and the minimum and maximum, uh,
of the cluster number and also the metric.
With K elbow visualizer, we can make a chart showing us exactly what we need.
Uh, we just specify the model with its, uh, parameters. Uh,
here I said just the initial number of iterations.
Then we specify the minimum and maximum number of clusters
and the metric. Uh, we want to see, uh, behind the scenes, uh,
it creates multiple cames models with varying, uh, number of clusters.
We see that according to Kalinsky Habash index, uh,
it is recommended to go for, uh, four clusters. Uh,
even that the highest value is for 10 clusters. I think,
uh, it is because it is not only about the highest value, but also about, uh,
uh, what is the gain of this metric. Uh, with, uh, added, uh,
additional cluster,
we can do the same for a silhouette index. And silhouette index shows us,
uh, we should go also, uh, for four clusters.
The distortion is the same. We see that at, uh, four or five clusters.
The curve dramatically change each slope.
Okay. Elbow visualizer doesn't support, uh, Davis's balding score. Uh,
so we have to do it manually, uh, using, uh,
the method from sticky learn.
And we see that also according to Davis' building score, we should go, uh,
for four clusters as well.
Now, let's create the final model, uh, and, uh, check the proportions.
Uh, this is also very important.
We see, uh, one big cluster and three smaller ones,
but there are no dramatic disproportions, uh, which is good.
And now, uh, let's check if the clustering captured, uh,
the patterns depending on the types of attack.
We see here that even if we did not use any supervised
learning method,
the chemist clustering was able to capture the high level patterns and was able
to distinguish between the most common attacks and normal connections.
Uh, the first cluster, uh,
is basically all the normal connections or almost all normal collections.
Uh, second cluster is basically the Smurf attack,
and the third cluster, uh, is mostly consisting of the leptin attacks.
Uh, when you run this notebook, the numbering, uh, may change
just a few final words, uh, to CAS clustering. Uh,
Cass clustering is the most common method used for clustering the data,
but this not means, um, it'll work on any dataset.
There are also modifications of cames clustering, like, uh, kss,
for example, or K modes.
You can also specify a different parameters like the distance calculation.
And of course, there are other groups of clustering algorithms,
for example, hierarchical clustering algorithms or density. Based.
In our next video, uh, we will learn a little bit about, uh,
the algorithm XG Boost.