Hello my friends, and welcome to the.
Final practical activity of.
This part nine dimensionality. Reduction.
We already built two.
Dimensionality reduction.
Models first principal.
Component analysis and then second linear discriminant.
Analysis.
We got amazing.
Results with both, but slightly better results and actually perfect results.
With linear discriminant analysis.
So now we're. Hoping that with our third tool of.
The dimensionality reduction.
Toolkit that we get at least the same thing as PCA
or that same perfect results as with LDA.
And you might guess that since.
Now we're about to add a kernel, and as we saw with SVM
and kernel SVM, adding a kernel always improves the result.
Well, you might guess that we are about to.
Get amazing results. As well.
All right.
So let's start. Let's build that final model.
But before this let's make sure everyone here is on the same page.
I give you the link to this whole folder right before this tutorial.
So make sure to connect to it.
And now here we go.
Let's enter port.
Nine and then.
Section 45 kernel PCA.
And as usual we're going to start with Python.
And this Python folder contains two files.
First the kernel.
PCA implementation and the. Ipynb format.
And of course the same data set.
Wine. CSV which is a data set of.
Many wines and many different.
Wines. Each row corresponds to a wine.
And for each of these wines,
we have these features from the alcohol level to the proline.
And then for each.
Of these wines, we also have the customer. Segment.
Which is the segment of customers to which
the wine belongs to in the sense that all the customers of each.
Segment, and we have three segments in total.
Have the same preference for such wines.
Okay.
And so now the challenge is.
To build a logistic regression.
Model, combine to some dimensionality reduction techniques.
Apply to this data set so.
That we can end up with.
Less complex data set, which.
At the same time will provide an excellent way.
For the logistic. Regression model to learn.
The correlations between all these features. And.
The dependent variable.
All right.
And then for each new. Wine, we.
Will deploy this.
Predictive model.
To predict the. Customer segment to. Which.
This wine belongs.
So that the owner of.
The wine shop can recommend.
Each new wine to the right customer.
All right.
So that's the exact same case as before.
And now let's open this implementation.
With either Google Collaboratory or.
Jupyter. Notebook.
As you can see, I kept this PCA.
Implementation and this LDA implementation so that we can compare.
And now.
Well as usual this.
Implementation is in read only mode.
Because you all have access to it.
So let's create a copy by.
Clicking. File here.
Then save a copy in drive. Because indeed in this.
Copy we will.
Re-Implement the cell that implements kernel. PCA.
Let's get rid of this so that we can have clearly the three.
Dimensionality reduction techniques.
And now there we go. Let's implement.
Kernel PCA.
But first this time let's upload the data set first so that.
We can, you know.
Get the assistance of Google Colab.
Right now our notebook is connecting to runtime.
There we go. Then let's click upload.
We'll end.
Up in the Linear Discriminant analysis. Folder.
So let's just do the whole path again.
So this is the. Whole machine. Learning is it folder.
Let's go inside and let's go to part nine dimensionality reduction and section.
45 kernel PCA.
Python and wine dot.
CSV. Open.
Okay and there we go.
Now our notebook is connected okay.
So now we're going to do two things.
First we're. Going to remove. That.
Cell.
You know put it in the bin so that we can re-implement it.
But also lets.
You know remove all. The outputs.
By train not to look at them.
You know.
I hope I hope you didn't look at the results.
But anyway, I'm. Sure you expect.
An amazing result. As well.
So let's just remove the output here.
That's the visualization of the training set results.
And this one visualization of the test set results.
All right then let's take the table of.
Contents applying kernel PCA.
And there we go.
We are ready to implement this.
So let's create a new code cell.
And now do we want to re-implement this you know from the very scratch.
Or do we want to be efficient.
And well.
Of course that's really my spirit as a coder.
As a machine learning programmer, I always want to be.
Efficient.
And by. That I mean that, you know.
The kernel PCA.
Implementation is.
Super close to the PCA. Implementation.
Because basically it will be. Almost the same, except.
That we will have to add.
A kernel in one of the inputs.
So what we're going to do, you know, in that spirit.
Of efficiency, is we will go to our PCA.
Implementation. We will say that cell.
Because you're going to see that it's going to be almost the same.
So let's paste it here.
And now the only thing that we have to change.
Is first the. Name of the. Class.
But not the. Module, because the class we're about.
To import to implement. Kernel PCA.
Still belongs to this decomposition module.
By the cyclic library.
And that class is of course kernel PCA just like that.
So that's the class then.
Well let's give a. Different name to the object.
We're not going to call. It PCA but we can call. It you know.
K PCA as. You want. You know.
Then of course here when we call the class to.
Create an instance of.
This object, which will be this CPK variable, well.
Of course we.
Need to call the right class which is kernel PCA.
And now inside this. Class well same.
We have to choose. A number of.
Extracted features which is.
Still given by this argument and components.
But since now we're working with a kernel, you know, we're doing kernel PCA.
Well, exactly the.
Same as when we transitioned from.
SVM to kernel. SVM.
Well, we simply need to. Add a.
Kernel argument here, and we'll actually.
Choose.
The same kernel as with kernel SVM, meaning the.
RBF kernel which is the radial basis.
Function kernel.
So there we go. That's our.
Second argument. Here. Kernel
in. Quotes.
Well r d f radial basis function.
And now let's see let's see what there is left to change.
So this line is good.
The next line of code.
Well same in order to perform.
The kernel PCA.
Dimensionality reduction technique.
Well we only. Need the features of Xtrain.
And not the dependent. Variable y train.
So all good.
You know that's the same as PCA but not the same as LDA which required
the dependent variable y train.
So all good here. However, be careful we.
Renamed our object not.
PCA but k PCA.
So same here. Cbc.ca.
And now my friends, this implementation is over.
That's what. Happens.
You know when we're being efficient,
the implementation is sometimes completed faster than expected.
And that's.
Because as you can see, kernel PCA is very.
Similar to. PCA.
You know, in terms of its implementation.
Okay. So now we're.
Just ready to. Run all again.
We have our data set.
Our implementation is all good.
So let's do this.
Let's click. Runtime here.
And then.
Three to. One run. Oh go.
All right. So now.
All the cells. Are running.
Our logistic regression model is built.
and.
As expected well we get of course an accuracy of.
100%. I've really. Seen some. Cases.
Where you know, the non kernel version of the model.
Beats the kernel. Version of the model.
It can happen. But it's very rare.
There you go.
Here of course kernel PCA manages to beat the PCA model.
Thinks that kernel.
Well we fixed that incorrect prediction which we had.
Remember in. PCA right here.
So all good here we get a one.
Hundred percent accuracy.
And now. Let's have a look at the results. To know.
How kernel PCA was able to separate our classes.
In the test.
Set right, which.
Are new observations on which the model wasn't trained.
Well there you go. That's our two.
Principal components PC1 and. PC2.
And now in a new dimension once again
you know, because the wine's observation points.
Here are.
Arranged in a different way than with. PCA.
Right? We have very different arrangement of the points here.
We can see them more.
Dispersed. Than. Here right with our PCA.
Well that's because. We are in a new dimension.
We are with different. Dimensions pc1. PC2.
Meaning. Different extracted features.
So that's totally normal that.
Our observation. Points, you know the wines here. Are.
Arranged in a very different way.
That's because we are in a different dimension in.
Which, well, the logistic regression.
Model. Was perfectly able to classify our.
Observation points with.
These three prediction regions.
And similar to LDA, the.
Observation points are arranged differently because once again, these.
Are. Some different dimensions.
We are in another dimension here.
Thanks to these extracted features.
So you see. This dimensionality reduction technique is.
Pretty.
Fascinating because it.
Basically allows us to create a new space of.
Dimensions and in some new dimension.
Well, indeed, we can.
Perfectly classify.
Some observations. Like it is the case for.
Linear discriminant analysis and kernel. PCA.
Now what I recommend for you to.
Do is to practice. This on other. Data set.
So I recommend, for example.
To check out the UCI.
ML repository platform and go to.
The classification.
Section and try the kernel. PCA. On other.
Data sets. And you'll.
See you will end up with similar results.
With. Some prediction boundaries. Like that.
Separating well.
The classes. Please share your.
Results in the Q&A, especially the ones where we.
Clearly see.
An improvement.
With kernel PCA. With respect to PCA.
You know, maybe you'll find some data.
Sets where PCA performs poorly, but.
Then by.
Adding a kernel with kernel PCA, you will get much better results.
So please share. This.
I'm actually very interested to see what you get.
All right, thanks in advance.
And now congratulations.
This new chapter on dimensionality reduction is done.
And now we're going to move on to the final.
Chapter of the course, Poisson.
Model Selection and Boosting.
Where you will learn your three.
Last and very important.
Tools which are first k fold.
Cross-Validation to.
Evaluate your machine learning models. The best way.
Then parameter tuning to find the.
Best values of your. Hyperparameters.
And finally. The cherry. On the.
Cake of this course I will teach you and give you the x g boost.
Model, which is one of the best and most powerful machine learning.
Models for regression or classification.
That will complete your master machine.
Learning toolkit the best way.
And until then, enjoy machine learning.