Hello my friends, and welcome to this.
New practical activity.
On this time dimensionality reduction, which is not a branch of machine learning.
Per se, but a. Very important technique.
To. Know how to handle.
When you work with big data sets, you know, huge data sets with.
Many.
Features and you know for which you would like to reduce to complexity by.
Reducing the dimensionality.
And this is exactly what.
Dimensionality reduction is about.
So in this new part, part nine dimensionality reduction, we.
Will build.
Three different models that can perform such a.
Task.
These are first.
Principal component analysis.
The most famous one then linear discriminant.
Analysis. And finally. Kernel PCA.
So we will build these.
Three models, one for each section. And now.
We're about to start with the first.
One principal component analysis.
But before we start let's.
Just make sure everyone here is on the same page.
I gave you the link to this folder right before this.
Tutorial in the article, so make sure to connect to it.
And now we. Should be. All on the same page.
So we're going to go into part nine dimensionality reduction.
All right.
And as I told. You you have the three sections.
Corresponding to each of the models. And we're going.
To start with.
Principal component. Analysis PCA.
And as. Usual we're going to start with. Python.
And then despite and folder you will find two files as usual.
First the.
Implementation in. Ipynb. Format.
Which. Now once again we will be able to run on.
Google Collaboratory because we will work with a classic data. Set.
And speaking.
Of which, here is a data set one. Dot. CSV.
So let's open it and let me.
Explain what this is about.
All right.
So actually first.
You notice indeed that we have many. Features.
I did not take a dataset with hundreds of features
because then we would, you know, get lost in data set.
So I just took a.
Dataset with more than ten features.
And of course. These are all. The features from here.
Alcohol to this one proline.
And as you can.
Guess, each feature gives a.
Certain information.
Of a. Certain. Wine, right?
Each row.
Corresponds. To. A wine.
And for each wine we. Have diverse.
Informations, diverse.
Features, you know, characteristics of the wine, the alcohol level.
The malic acid.
I'm not an expert of wines, but.
These are some. Wines.
Characteristics.
Ash. Ash Alicante. Magnesium.
Total phenols.
Flavonoids.
Anyway, so.
You see you have. Many wine features.
And for each of these wines.
Well there. We go.
I'm about to explain the dependent variable.
For each of. These wines.
We have the customer segment.
You know. That's the.
Last column.
To which. The wines belong. Okay.
So let me explain what happens in terms of business.
First of all, this is a data set I took.
From the UCI ML. Repository.
So all the credits go of course to.
This amazing platform of. Data set.
However, in this.
Data set, I just changed the last.
Column customer segment to make it more business wise.
You know, to make this case. The more a business case study.
Because the scenario is the. Following.
Let's imagine that.
This data set belongs to a.
Wine merchant with many different bottles of wine to sell, and.
Therefore a large base of customers.
And this wine shop.
Owner actually hired you as a.
Data scientist to.
First do a.
Preliminary work. Of clustering.
Meaning that.
At first we had all these features.
Without this last. Column customer segment.
We have all these. Features from alcohol. To.
Proline and this wine shop owner.
Actually asked you to. Perform some.
Clustering to identify diverse.
Segments of customers.
Grouped by similarities which correspond to the ones they prefer.
All right.
So each.
Customer segment.
Here and by the way, there are three of them. Right.
If we scroll. Down we can.
See that we have three.
Different categories or, you know. Clusters.
And each. Of these. Segments.
Will. Correspond to a certain group of customers.
That have. Similar preferences.
For. Similar wines.
And that's exactly what these segments are about.
But that was the first work.
And if you want, you can have fun and.
Do this first work yourself.
But here we want to work on dimensionality reduction.
So there. Goes the.
Second mission that this wine shop owner.
Asks you to do.
This wine shop owner was actually satisfied with your first work.
You know, identifying these three segments.
But now the. Owner would like to.
You know.
Reduce the complexity of this dataset.
By ending up with a smaller amount of features.
And at the same time, this.
Owner would like you to build a. Predictive model.
That will be.
Trained on this data, you know, including.
The features up to here.
And the dependent variable.
So that for each. New wine.
That this owner has in, it's up.
Well, we can deploy. This. Predictive model.
Applied to a reduced.
Dimensionality.
Data set to predict which customer.
Segment this new wine belongs to. Right.
And therefore once we managed to. Predict.
Which customer segment this wine belongs to, then we can recommend this wine.
To the right. Customers. And that's.
Exactly why what we're about to do is like a.
Recommender system.
Because for each. New wine that will be in the.
Shop, well, our predictive.
Model. Will tell us to which.
Customer segment it. Will.
Be the most appropriate.
You know, it will be the most appreciated.
All right. So that's the business case.
And therefore, you know, our.
Predictive model will add tons of.
Value to this owner because therefore if this owner manages to build.
A good recommender system, of. Course it will.
Optimize the sales and therefore the.
Profit of the business.
Okay. So that's what the case study is about.
Now we're going to move on to the implementation of course.
Therefore I'm opening this file principal component analysis.
Which you have the choice to open with either Google Colaboratory.
Or Jupyter Notebook.
As we. Did in the previous section on CNNs.
But there we go.
Let's open it with Google Collaboratory.
And enjoy a brand new implementation on it.
All right.
So here is the.
Implementation principal components analysis.
This is in read only mode.
So as. Usual we will create a copy by clicking.
File here. And then save. A copy in drive.
This will create a copy inside which we will be able to re-implement.
Not the whole.
Implementation this time, because I will explain that most of.
The cells are cells we already did before.
You know, many times in the classification part
and also in the first section of part eight.
So we won't have to re-implement everything.
This would. Be a. Waste of time. And mostly.
We. Rather want to focus on dimensionality reduction.
And therefore.
Here. Is what we're going to do.
I'm going to show you the implementation.
Of course, but the only cell that we will.
Re-Implement will. Be.
This one applying. PCA. So let's.
Remove it. Right away.
Not the text, only this one.
And now I'm going to show you that indeed, you.
Know, all the cells are super familiar to us, right?
Because indeed we. Start.
By importing the.
Libraries that we did 100. Times. Right?
So we have the three essential libraries here.
Then we import the. Data set with the exact.
Same code as the one you have in your.
Data preprocessing. Template.
So of course here I just put.
The right name of the data. Set which is.
Wine. Dot CSV.
Okay.
Then you will recognize the next steps.
Of the data preprocessing. Template.
Which is to split the data set.
Into the training set and the.
Test set executive the same code.
Then we apply feature scaling as it.
Is, you know, most of the time recommended.
So we apply it of course on separately.
The training set and the test. Set.
And that closes the data preprocessing phase.
Then we apply. PCA.
And that's of.
Course the cell we will.
Re-Implement together.
Then let me just remove.
All the outputs here so that. You don't see them.
I hope you close. Your eyes when I just remove them.
But there you.
Now close.
Your eyes a little. Bit.
I'm going to.
Remove that output as well because.
Actually the dimensionality reduction technique.
That we'll use will. Manage. To get us great results.
With only two. Extracted features.
Right.
We're not reducing the number of existing features.
We are creating new extracted features based on these existing features.
So we will get totally. Different.
New features at the end which we.
Call, you know, principal components. So we'll have.
Principal component one and principal component two at the end.
But there we go. So back to.
Our implementation.
After applying PCA which we will redo together.
Well we. Trained the.
Logistic. Regression model on the training. Set.
I chose the logistic regression model
as the first model of our classification toolkit, but I.
Could have chosen any other ones.
But you will see that
we will get great results with this one, but feel free to choose.
Another classification model and we will. Work.
But notice that.
It is important to. Apply PCA before.
Training your.
Classification.
Model on the training set right, you want to reduce. The.
Dimensionality of your. Data set before of course.
Training it on your training.
Set right. The training set basically is the.
Final version. Of your data after you.
Performed all. The data preprocessing.
Phase and dimensionality reduction if you want.
Okay.
So the training happens after applying your dimensionality reduction technique.
And then.
Of course, well we. Will make the confusion matrix.
You know how to do that. We did it many times.
And then since our dimensionality reduction technique.
Will get us great results. With only two.
Extracted features.
Principal component one and principal component two.
Well, that will allow us to visualize the training set results.
In two dimensions. Right?
Because remember.
That each.
Dimension corresponds to one feature.
And we do this. For the. Training.
Set right here.
And the test set okay.
So as you can see. What I did with this.
Implementation is something you can do in less than five minutes right now.
Thanks to your toolkit. Right.
Because you just need to take
the data preprocessing toolkit to make these four cells.
Then you just need to.
Grab the feature. Scaling tool in your.
Data preprocessing toolkit.
Then you.
Just need to grab your logistic regression.
Implementation to implement this cell.
And same for the other ones. You know the confusion matrix.
And same for these last two.
Visualizing the transit results.
And. Visualizing the test results.
These are all cells that you have in your logistic regression implementation.
So absolutely no need.
To do it together again. And therefore.
We can now focus. Directly.
On. This cell.
Applying. PCA.
So there we. Go.
We're going to create a new code cell.
And now let's implement.
PCA. Principal component analysis.
All right.
So you. Could almost.
Press. Pause on the video.
Now and get the. Right. Tool from the scikit.
Learn API to see how. To implement this.
That would be a good exercise.
But if you don't. Want to do it that's fine.
Let's implement this right now.
And as you guessed by what I've just said, well.
We're going to implement PCA.
Using the scikit learn library.
So the first thing we'll do is start from the scikit
learn, from which we're going to get access to a.
Certain module, which we'll find in the cyclone API and which.
Is called decomposition.
Just like that.
Decomposition from which we're going to import.
Of course, a class that will allow us to build this object.
Which will be nothing else.
Then this PCA tool that will.
Apply dimensionality reduction on our data. Set.
And. That class is called very simply okay, okay.
So you can't miss it in the API, PCA.
And now next natural.
Step is of course to.
Create an. Object.
Or you know, an instance of this class.
And guess how we're going to call that object.
Well very simply we're going to call that object okay.
Right. So this is super intuitive.
And now you know the next step next step.
Is to call the PCA class which needs to take.
One essential argument.
You know we only have to input one argument here.
And you can totally guess. What.
This argument will be. Right.
It is.
The final number. Of extracted features.
You want to end up with in your new.
Data set.
And that argument to choose that number is.
Called.
N underscore components and components.
All right.
So now the.
Question is of course which. Number should we.
Choose. Right.
How do we know down to which number of features right.
Extract features.
We want to reduce dimensionality.
Of our data set.
Well I have a very.
Simple answer.
To that question.
What I usually do is start with two.
You know. Two.
Principal components.
Therefore two extracted features.
And see the. Results I. Get in the end.
And thanks to our code, you know, our code template, we can.
Check that very quickly. And easily.
And besides.
We do want to try with two. Because then if we.
Get good results with two,
well we will be able to visualize the training set result.
And the. Test result. In two dimensions.
You know, in this nice. Plot that we had. In part.
Three classification.
So we definitely want to start with two.
And if you know, we get really.
Poor results and. We see on the.
Graphics here.
That we can't.
Separate the three classes properly.
You know, remember
with those different prediction regions and the prediction boundary.
Well if we see that we have poor.
Results on the visualizations, then we can try.
With.
Higher numbers of principal components meaning three than four.
And at some point we'll get, you know, some extracted.
Features that explain. Well enough the variance.
Which is exactly what PCA is about, right, is about extracting.
Some features. That. Explain well enough. The variance.
And once you find them.
Well, you will get good results.
Even with lower dimensionality.
Okay. So let's try with.
Two and let's see what we'll. Get.
But I already told you that will get.
Amazing. Results.
Therefore there you go.
And components. Equals to two.
Principal components.
Or in other words.
To extracted features okay.
So that's for our object.
And now next.
Step of course is to apply this object to our.
Training set to. Reduce the. Dimensionality.
Of our. Training set in. Order to ease.
The learning process.
Of the logistic regression model.
But also we will have to apply it. On the.
Test set.
Because remember.
That the predict.
Method that.
We will call here has to be called.
On the exact same format of. Data.
As the one that was. Used.
For the training set.
So as long as you apply.
Some transformations like data. Preprocessing.
Or dimensionality reduction on your training.
Set, well you have to do the.
Same on your test. Set.
However, be careful exactly as feature scaling.
We will have to apply the fit.
Transform method on the training. Set, but.
Only the transform method on the. Test set.
And that's. Always for the same. Reason.
That's because we want to avoid information leakage on the test set.
Write.
The test set is supposed to be you observations like.
Data on which we deploy our model in production.
And therefore we're not supposed.
To fit our scaler.
Or, you know. Feature extractor object.
On the. Test set.
We can apply them to transform them. Right, because they were.
Fitted on the training set.
But we. Can't fit them again to.
The test set because that would be like.
Trying to get some hints of information from the test. Set.
That we're not supposed. To have. That's exactly.
What information. Leakage is about.
So there you go. I said. Everything. Now you can.
Press pause. On this.
Video to.
Finish this implementation of. PCA.
And in two seconds I'm going to implement with you the solution.
All right.
I hope you did well. Now let's do it together.
So as we said, we want to apply this PCA object separately.
On the training set.
And two sets.
So first I'm going to. Take X. Train.
All right which I'm going to.
Update by applying this PCA object from.
Which I'm going to.
Call the fit.
Transform method.
On this all the.
Version of X train meaning before the transformation of PCA.
And so here.
What. Happens technically is that. The fit part of this.
Fit transform method will. Get.
All the information it needs from X train.
To apply.
Principal component analysis.
And then of course the transform.
Part of this.
Fit transform method. Will apply.
The transformation. Itself to extract the.
Principal component features. Okay.
So that what it means technically and now.
Well let's. Do the same.
Actually for X2'S I'm copying this, pasting it here.
And. Replacing here x train by x test.
Then x train here again by. X test.
And only applying the transform method.
And there we go my friends.
This implementation is already over.