Hello and welcome to this art tutorial.
So now we know how to implement two feature extraction techniques.
These are PCA and LDA.
But these feature extraction techniques work on linear problems.
That is when the data is linearly separable.
And in this section we are going to see one new feature extraction technique.
But this time adapt it
for nonlinear problems where the data is non-linearly separable.
So this technique is called kernel PCA.
Kernel PCA is.
A kernel sized version. Of PCA.
Where we map the data to a higher dimension
using the kernel trick,
and then from there we extract some new principal components,
and we are going to see how it manages to deal with non-linear problems.
So we're not going to work on the same problem
as we did in the previous sections with the one data set,
but we're going to work on the same data set as the one used in part
three classification. Because now we need visuals.
We need to clearly see what happens.
We need.
To see how kernel PCA.
Manages to extract some new independent variables.
The principal components.
Even when the problem is non-linear, that is,
when the data is not linearly separable.
And this data set that we used in part three, the social network
Ads data sets will remember it was clearly a nonlinear problem
because nonlinear classifiers showed much better performance.
So let's take this data set and let's apply.
Konopka to see how it.
Will handle the non-linearity.
So let's find this data set into our working directory folder.
So we'll go to our machine learning A to z folder.
Then part nine dimensionality reduction.
And here we are at the last section of this part nine kernel PCA.
So that's the folder you want to set as a working directory.
Make sure that you have the social network and CSV file.
And if that's the case you
ready to click on this more button here to set the folder as working directory.
And now what we're going to do is take this logistic regression model.
Because you know this logistic regression model is a linear classifier.
Therefore it will not be appropriate for our problem because
our data is not linearly separable.
So what we're going to do is take this linear classifier here.
But we are going to apply kernel PCA inside of it to see how kernel.
PCA will. Save the situation.
And so you will see that even if we apply a linear model well thanks to kernel.
PCA that.
Will manage to extract
new principal components adapted for this non-linearly separable data.
Well, you will see that we will get amazing results.
So right now let's copy the whole model here from the top down to the bottom.
Copy and let's paste it in our common.
PCA. File. All right.
And now basically the only thing that we have to do is to apply.
Kernel PCA. At the right place.
But before we do that I would just like us to visualize again
why this linear model is not appropriate for this data set.
So what we're going to do is take everything from here.
Because, you know, this
will visualize the training set results by plotting the prediction regions
and the prediction boundary.
So we're going to take everything from here up to the top
to you know, import the data set, apply the preprocessing phase,
fit the logistic regression to the training set
and eventually plot the training set results.
So let's do it.
Let's visualize this again very quickly.
And that will give us motivation to apply kernel PCA.
And here we go.
All executed properly.
So as a reminder the points are the real observations.
That is our real customers in the social network
represented by their age and their estimated salary.
So that's our real observation points.
And our predictions are represented by these regions.
The red region here and the green region here.
And basically this red region is where our model predicts
that the customer will not click on the ad.
And this green region
here is the region where a model predicts that the customers will click on the ad.
And by the. SVM.
And so remember, the problem was that this straight line here is
actually the prediction boundary generated by the logistic regression model.
But since the logistic regression model is a linear classifier,
then it has to be a straight line here separating the data.
And therefore remember the problem is that it cannot make some kind
of a curve here to catch these green users.
That should be in the green region right now they're in the red region.
And so this clearly represents the fact that our data is not linearly separable,
because we can clearly see that
this prediction boundary here that plays the role of separator.
And that is supposed to separate the two classes.
Well, it cannot separate the two classes properly because.
As you can see, these.
Users are not in the right region.
And so now what we're going to do is not make a non-linear classifier
like we did in part three.
You know, when we made kernel, SVM, Naive Bayes, decision trees or random forest.
Well, what we're going to do now instead is applied.
Kernel PCA.
So that we keep a straight line as the separator, as the prediction
boundary of a linear classifier.
That is still going to be the prediction boundary of a logistic regression model.
But since we're going to apply. Kernel PCA.
Well, this will manage to apply some trick where the trick is actually
the kernel trick to map the data into a higher dimension and then apply.
PCA to.
Extract new components that will be new dimensions that explain the most variance.
But thanks to this kernel trick, well.
You'll see that we'll manage.
To get some new dimensions in which the data will be linearly separable
even by a linear classifier like logistic regression.
So let's see that right now.
I can't wait to show you this. I'm going to close this.
And now let's apply kernel. PCA.
At the right location.
So you already know what this location is.
It's actually not different than before.
We need to apply. Kernel PCA.
Right after the data preprocessing phase.
And just before fitting our classifier like logistic regression
to our training set.
So basically.
We need to apply. Kernel PCA. Right here.
So use section here. Applying
kernel PCA.
And here we go. Let's do it.
All right.
So first we need to install a new package that is called kernel lab
which I don't think we've installed before.
So let's do it right now.
So we use the command install dot packages.
Here we go.
And in quotes kernel lab.
All right.
So I think I already have it installed.
Let's check it out.
So here it is kernel lab kernel based machine learning lab.
So I will not install it.
But if you want to do it you just select this line and execute.
So I will just put this line as command.
Here we go.
But then since it is not imported I will import it using the library
command kernel lab.
All right.
And that will import it.
All right. Kernel lab will import it.
And now let's start applying kernel PCA.
So as. For PCA.
And LDA we're going to start by creating an object which will.
Be the kernel PCA.
Object that we will use to transform our original data
set into this new data set after using the kernel trick.
So we'll call this object k. PCA.
And then equals.
And then that's where we use the function that will create this kernel.
PCA. Object.
So this function is also k PCA.
Then parentheses.
And then let's input the different arguments.
So let's check it out.
Let's press F1 here to have a look at the arguments.
So the first argument is x.
And this is actually the data matrix of the formula describing the model.
And here I'll give you a little trick to describe the model.
Very simply and very efficiently.
We can simply input here a tilde and dot.
And that will be enough for the.
KPK. Function to understand what the formula is,
because then we will add the second argument which is data,
and that is actually the training set but without the dependent variable.
Because remember kernel.
PCA is just a PCA technique.
Where we use the kernel trick to map the data into higher dimension
and then apply. PCA.
Because indeed, in this higher dimension, the data is linearly separable.
And therefore, since we. Apply PCA.
In this higher dimension. And PCA is an.
Unsupervised technique, well, here for the data argument,
we just need to input the training set but without the dependent variable.
And therefore.
As for PCA, we.
Input here data equal training set then brackets
to remove the dependent variable which is indexed by three.
Because we only have two independent variables.
All right.
And then the next argument is kernel.
So kernel is the kernel you want to use to apply the kernel trick.
Remember when we studied kernel SVM we saw that there were several kernels
to use the kernel trick.
And here we're going to use the most common one which is the Gaussian kernel.
And that is called here RBF dot.
So that's our third argument.
And so here we input kernel equals RBF dot.
All right.
And then what is the next argument.
The next argument is k bar.
We will actually not use this one.
But then we have a very important argument that is at the heart
of dimensionality reduction.
That is features which is the number of features,
the number of principal components you want to end up with.
So here of course, we would like to visualize
the training set results and the test results in two dimensions.
And to have this in two dimensions, we need to keep a number of two
new extracted independent variables.
So here the number of.
Features will be. Two. As for. PCA.
So we will input here features equals to.
All right.
And that's it for our. K PCA. Object.
It is ready to be created
and to be used to transform our original data set into this new data
set with the new extracted features derived from kernel PCA.
So let's select this line and create the object.
Here it is. CPK k well. Created.
And now let's move on to the next step, which is to transform our original data
set into this new extracted data set.
So now things are going to look like what we did with PCA.
But some things are going to change.
So we will do it step by step and we will see where we need to make some changes.
All right. So first as for.
PCA we're.
Going to use the predict function to transform our original training set
into this new extracted training set.
So this new training set with the new extracted features derived from.
Kernel PCA.
We call it training set underscore. PCA.
All right. And then equals.
And then we use the predict function to do the transformation.
And inside this predict function we first input our.
CBC.ca. Object as we. Did for PCA.
And then the training set the original training set.
So let's do it.
Training set the second one.
All right.
And as opposed. To PCA.
And as with. LDA.
This will return a matrix.
And we need it as a dataframe.
So as for LDA we will use the as dot
data dot frame function.
So parentheses here and we close the parentheses here
to set this transform training set the training.
Set PCA. As dataframe.
And as a reminder we're doing this to give what
the next function will use in the next sections expect.
All right. So far so good.
And so now let's select this line and execute this.
And you're going to see what's going to happen.
And you're going to understand why we called this new training set training.
Set PCA.
With a different name than the original training set.
Training set.
All right.
So let's execute.
Here we go. Execute it properly.
And now let's have a look at our training set.
PCA that. We just created.
So I'm going to enlarge this so that we can see which one is trained the PCA.
That's the one.
So let's have a look at this I'm going to click on it.
And here is our training set PCA. So as.
We can see it is.
Composed of only two columns V1 and V2.
So try to guess what these two guns are.
I'm going to tell you right now,
these two columns are the principal components that we obtained
through kernel PCA.
That is these are our two new extracted features.
After all
this mapping into this high dimension using the kernel trick and then applying.
PCA. To the data set mapped into this higher dimension.
But now the problem is that
in this training set PCA, we don't have the dependent variable
and we need it for the next sections, because in our code template
we need to have the independent variables and the dependent variable.
So what is the next step?
Now the next step is to add the dependent variable into this training set PCA.
And so the thing to understand here is that we lost the dependent variable.
But we kept the observations. That is.
This one here corresponds to the first observation
we had in the original training set.
This one.
So this first observation here has the zero label
that is that this first customer didn't buy the SUV.
This was the original independent variables.
And then if we go to our training set PCA, well this first customer
is the same first customer as this training set.
So it will have the zero label in the purchase column.
But then these are new extracted features.
So of course we don't get the same values as for the independent variables
of our original training set.
So what we can do now is simply take the dependent variable column
purchased of this original training set and add it to our training set, PCA.
Because these observations here
are the same observations of our original training set.
And so what we need to do now is very simple.
We just need to take our training set PCA.
Then we're going to add a new column that we'll call purchased.
So by doing this you know I'm just creating a new column
that I also called purchased because this new column is going.
To be.
The purchase dependent variable and then equals.
And then what I have to do now is to take the real purchase
dependent variable column from the original training set.
And we can do that because the training set.
PCA. Contains
the same observations as the observations of the original training set.
So here to take the purchase column of the original training set,
we just need to take our original training set, which is called training set
and then dollars.
And then that's where we take the purchased column.
So by doing this I will add this new column purchased.
And then this new column purchased.
I will include the values of the purchase column of the original training set.
So let's check it out I'm going to select this line and execute.
And now as you can. See.
If I go back to training. Set PCA.
This contains the purchase column of the original training set.
So that's good.
That's the next step done.
And now we need to take care of the test set.
And so to take care of the test set
we're going to do exactly the same as we did for this.
Training set PCA.
So let's copy this. Copy.
And let's paste it here.
And of course what we're going to do now is replace this training.
Set PCA.
By test set. PCA.
And same here.
We take the original test set to make the transformation.
And then we're going to add the purchase column of the original test
set to this new test set.
That is the test set extracted from current PCA.
So test it here. And that should be okay.
So I'm going to select.
These two lines here.
And execute perfect.
Our new test. Set PCA is. Created.
Let's have a quick check.
So that's the test set.
And that's our test set. PCA.
With the two new extracted features and the purchase column.
And now that means that we correctly applied kernel PCA.
So great.
We are ready to move on to the next section.
So let's go back to our kernel. PCA. File.
And let's now fit the logistic regression to the training set.
Now do we need to change anything in this code section.
Well yes of course we do because be careful.
We called our new extractor training set training set PCA.
So here for the data argument we need to specify training set PCA.
So that's the only thing we need to change here.
So we are ready to select this section and execute.
All right classifier ready.
And now let's move on to the next section predicting the test set results.
And of course here that's the same.
We need to replace test set by test set PCA.
You need to enlarge this a little bit right.
And that's it.
We are ready to execute this section to predict the test set results.
And here we go.
We get our vector of predictions.
Why pred for this new test set PCA.
All right. Now let's make the confusion matrix.
We of course need to change test set by test set PCA.
Here we go. And now it's ready.
Now we can execute this line of code to get the confusion matrix.
And here it is.
We can have a quick look and the council by pressing cmd enter.
And we get 57 plus 26 equals 83.
And since we have 100 observations in the test set,
that gives us an 83% accuracy.
So that's pretty good.
And now let's get to the exciting part visualizing the training set results.
So very quickly what do we need to change.
Remember we need to change the names of the independent variables and columns
here. That's compulsory.
So as a reminder the names are v1 and v2.
That's the name of the independent variables.
So here we need to replace age by v1
and estimated salary by v2.
And that's not compulsory.
And anyway we already have two good names.
PC1 and PC2. So don't forget about that.
And of course we need to change the name of the training set
because we called our training set.
Training set. PCA.
So here I'm. Adding training. Set PCA.
And that's perfect.
That's ready to be executed to visualize the training set results.
So we will only visualize the training set results.
But let's make the same changes for the test set
so that you can have a look at it yourself.
So same we're replacing H by v1
estimated salary by v2.
And here we replace test set by test set underscore PCA.
All right.
And now let's have a look I look forward to showing you what's going to happen.
So I'm going to select everything from here up to the top here.
That is the whole section to visualize the training set results.
And let's press Command and Control plus enter to execute.
Here we go. The computations are being run.
All right.
So these are the results of kernel PCA.
Combine to a logistic regression model that we applied on the nonlinear
separable data set.
And so we can appreciate the contrast between the simplicity
of the obtained results and the complexity of what happened behind the scenes,
because indeed we have this very simple results here
with these two classes separated by the straight line.
But what happened behind the scenes is that our original data set
in our original feature space, was mapped to a higher dimension
using the kernel trick to avoid too highly compute intensive computations,
and then by mapping
our data set in the original feature space to this higher dimension.
Well, first, that created some new dimensions,
and mostly that created a new feature space
where our De that was then linearly separable.
But by doing that, we had more dimensions than the original number of dimensions.
So we still needed to apply. The PCA.
Dimensionality reduction technique to end up with a lower number of dimensions.
So then PCA was.
Applied to this new feature space where the data was linearly separable.
And through PCA some.
New extracted independent variables were created
that are nothing else and the principal components.
Of PCA.
And eventually we obtained this new feature space
formed by these two new extracted principal components resulting from PCA,
in which now our data is linearly separable
and much better separated by a linear classifier.
All right. So that's it for kernel PCA.
And that's also the end of this part dimensionality reduction.
And I'll see you in the next part.
Part ten model selection and boosting.
The last part of this course we will cover a very exciting algorithm
in machine learning that is called XGBoost.
So I look forward to seeing you in this next part.
And until then, enjoy machine learning.