Hello and welcome to this art tutorial.
So in the previous section, this PCA feature extraction technique
reduced the dimensionality of our problem by extracting
the variables that explained the most the variance.
And now in LDA this is quite different.
We are extracting some new independent variables
that will separate the most the classes of the dependent variable.
And therefore, since this time it considers the classes
of the dependent variable, well, that means that it considers
the dependent variable to proceed to this feature extraction technique,
and therefore that makes LDA a supervised dimensionality reduction model.
All right.
So now let's apply LDA on R.
So first very quickly let's set the right folder as working directory.
So we will go to our machine learning A to z folder
for nine dimensional data reduction.
And we are now in this section 44 linear discriminant analysis.
So let's go inside.
And that's the folder you want to set as working directory.
We are still working on the one dot csv file.
So that's exactly the same business problem
as the one we worked with when implementing PCA.
And so that will be a good opportunity to compare this dimensionality
reduction technique LDA to the previous one. PCA.
So now let's not forget
to click on this more button here and set as working directory.
Perfect.
And so since we are working on the same business problem as before,
as for PCA, well implementing LDA here is going to be very easy.
We will just take the PCA code, take everything from here
down to the bottom copy and we'll go back to LDA and paste the whole thing here.
And inside of this code we will just need basically to replace
this applying PCA section by a new section dedicated to applying LDA.
So I'm going to remove all this section.
So let's remove this and let's
replace here PCA by LDA.
And now it's time to implement LDA.
So the package that we're going to use to apply LDA
is the max package Max.
And it's actually a package that is by default in your list of packages.
It's the package right here.
Mass support functions and data sets for the labels and replacements.
All right.
So as you can see this is not imported.
So let's import it by using the library command.
Here we go.
And mess this way in parenthesis.
All right.
We can already select this to import the package.
And now let's implement LDA.
So the first thing that we have to do is as for PCA create an LDA variable
that we will use to transform our original data
set into this new data set composed of the linear discriminant.
And so we're going to call this variable LDA equals.
And now we're going to use the LDA function as simple as that.
And let's add some parenthesis.
And now let's see the arguments by pressing F1.
All right.
So the arguments are here.
The first argument is formula.
So that's exactly the formula of your dependent variable
with respect to the independent variables so far the original ones.
So here as a first argument we will input
formula equals customer segment.
Remember that's the name of the dependent variable.
And tilde.
And we can add a dot.
Don't worry we don't have to write all the names of the independent variables.
The dot is here for us.
So come up and then next argument.
And then the next argument is data.
And as for PCA the data here is going to be the training set.
So let's add here training training set.
Here we go.
All right. And actually that's all.
And that's for a specific reason a very important reason that is directly
related to LDA. It's due to the fact
that LDA is a supervised dimensionality reduction technique.
Remember supervised
means that the LDA model takes into account the dependent variable.
And since it takes into account the dependent variable,
well, it's quite intuitive to understand that the number of linear discriminant
will be related
to the information of the dependent variable, and that information is actually
the number of classes in the dependent variable.
And there is this explicit correlation between the number of linear
discriminant and the information of the dependent variable.
It's that there will be k minus one linear
discriminant where k is the number of classes.
So here, since we have three classes, that means that we'll get at most
three minus one equals to linear discriminant.
So here without specifying the number of linear discriminant equal to two
we will get automatically to linear discriminant.
And therefore that's all.
We don't need to add any other argument.
So the LDA object is ready to be used to transform
our original data set into this new one, composed of the linear discriminant.
We will get to linear discriminant, which is exactly what we want.
As for PCA, we want two new extracted features, and
this time these two new extracted features will separate the most two classes.
So we should get very good results as well.
And then that's the same as for PCA.
We need to transform into training set and the test set
so that we can then use them in the next sections.
So we will use the training set to fit SVM
well actually to the training set to build the classifier.
And then we will make our predictions predicting the test results with this test
set that will be transformed as well and make the confusion matrix.
And most importantly we will visualize the training set and the test results,
something that we'll be able to do because we now have two features.
All right. So let's do it.
Let's do the same to apply LDA on the training set and the test set.
So first let's start with the training set.
Training set.
So remember we're keeping this
name of the training set so that we don't have to change it
in the rest of the sections.
So training set equals.
And then remember we need to use the predict function.
That's actually exactly the same as for PCA.
We will however need to add something to make it work.
And I will explain what it is.
But definitely we are doing this transformation with the predict function.
So parentheses.
And now inside this predict function we need to specify
you know the first argument here which is the object.
So the object is LDA and then comma and then the second argument.
And the second argument
is the data set on which we want to make the transformation.
That is, extract the new features.
And that's the training set here it is.
So as a reminder
this is the original data set composed of the 13 independent variables.
And this will be the new training set composed of the two new extracted features
that are the two linear discriminant.
All right.
So we can already do that to see if we get the right training set as we expect.
So before executing this line of course we need to execute the previous sections
because we need to import the data set and apply data preprocessing.
So let's do it.
We don't have anything to change here.
Everything is already well prepared
thanks to what we did in the previous section with PCA.
So let's execute this.
Here we go. Well executed.
And now let's apply LDA, create the LDA object
and then use this object to transform our original training
set into this new training set composed of the two linear discriminant.
So we already imported the math package.
So we just need to execute this line of code.
Let's do it.
Here we go LDA object well created.
And now we are ready to transform the training set.
But before we select this line and execute it,
we need to add this one more thing that I just mentioned, which is a function
that we will apply to this whole predict LDA training set here.
And that will set this training set that will be transformed
into a data frame.
Because for PCA, when we did this, when we transform
here, the training set to this new training set
composed of the principal components, well we got a data frame.
But for LDA that's not the same.
We will get a matrix
and we need a data frame because then you know we have the next code sections.
And inside these code sections the functions that we use expect a data
frame for the training set here.
For example it expects a data frame.
And so we absolutely need to convert this transform training set.
That will not be a data frame if we execute it this way, but a matrix.
And so to simply convert
this into a data frame, I think we already did it before.
We need to use the function as dot data, dot
frame and parenthesis.
And we close the parenthesis here.
And that will set this transform training set as a data frame.
And now we are ready to execute this line of code to obtain our new training
set composed of the extracted features the linear discriminant.
So let's do it.
Let's select this line and execute.
And as you can see the training set is still a training set.
Otherwise it would be in values.
And when I click on it let's have a look at what we just created.
So first of all the first thing that we see is this first column here.
That is the dependent variable itself.
I know this is no longer called customer segment,
but it's actually the customer segment column.
That's exactly the same one for the same observations
and the same labels one, two and three.
But automatically it was called class by the predict function.
So don't worry about that.
That's the dependent variable.
And then the next interesting thing we see
are the two linear discriminant L1 and L2.
And as I told you that's the number of linear discriminant
we get due to the fact that we have three classes for the dependent variable.
So that's what matters for us now.
And that's the variables that we'll use.
That's the new extracted features that we'll use to train the SVM model
to make the predictions,
to make the confusion matrix and eventually to visualize the results.
And then we have this other three variables posterior one, procedure
two and three.
That is just variables derived from the LDA model equations.
So that's not very important here.
What matters is that we have our dependent variable class
and our two new extracted features the linear discriminant one and two.
And so now what we have to do is set our training set into the right format.
That is we want the training set that is composed of first to two
extracted features to two new independent variables.
And then in last position, the dependent variable class.
That is nothing else than the customer segment.
So basically what we need to do here is the same as what we did for PCA.
That is, play with the indexes to not only set the right order for our columns,
but also to not include
the three columns here posterior one, pursue two and pursue three.
So what we'll do here to be efficient is take our PCA model
and we will take this line here.
Copy and go back to LDA and paste that here.
All right.
And now as for PCA we need to include three indexes.
This first index here will be the index of LDA one.
So that is the index of this column that is 1234 and five.
So that's index five.
So let's add it here.
Replace two by five.
Then the second index here should be the index
of the second new extracted feature that is LDA two.
So that is index six.
This column has index six.
So let's replace here three by six.
All right.
And eventually this should be the index of the dependent variable.
And this index is of course one because this is the first column here
that has index one.
So now when I execute this line here we go.
Look at our new training set.
Well that is this time exactly what we want.
The first two columns are the new extracted features.
And the last column is the dependent variable vector.
Exactly as what is expected in the rest of the code sections.
So perfect.
Our training set is well transformed and ready
to be used to train the SVM model.
All right.
So now we need to do the same for the test set.
So that will be very quick and easy.
We will select these two lines here copy paste
and now we just need to replace training set
by test set here here as well
here and eventually here also.
And now we can just execute these two lines.
But let's execute them one by one.
This is the test set so far
composed of the 13 independent variables the original ones.
Then when we select this line and execute
well we only get two new extracted features
LD1 and 92 and the three variables here of the equation.
And of course the dependent variable class.
And then when we do this again to take the right
indexes in the correct order we execute this.
And now we get the test set composed of the two new extracted features
in first positions L1 and L2 and the dependent variable and last position.
So that's perfect.
Now we are ready to execute the rest of the sections to build our SVM model.
All right so let's do it.
And actually we don't have much thing to change in this section.
Do thing. We need to change something.
Well the answer is yes because remember the dependent variable is no longer called
customer segment even if it is the customer segment variable.
But this time it has a different name which is class.
And that's actually the only thing that we need to change
because the training set still has the same name.
It's the training set that we just transform here.
So that's fine.
And then it's the same type and the same kernel
because we are building a linear SVM model.
All right.
So perfect.
Let's execute this section. Let's do it.
Done model created.
And now we are ready to predict the test set results.
So the test results do we need to change something here.
Well this time the answer is no because we have our test set transformed.
It has the same name and the classifier is this one.
So everything is perfect.
We are ready to execute this line of code.
Perfect and simple. Here.
We don't need to change anything.
We can just make the confusion matrix by executing this line.
Here we go. Confusion matrix created.
Let's see if we also get 100% accuracy.
We will be able to see that in a flashlight,
because if there is one incorrect prediction,
well that means that we will not get 100% accuracy as we obtained with PCA.
So that means that it will not be as perfect as PCA.
So let's do it.
Let's type CM here and press enter.
And unfortunately we get one incorrect prediction here.
But that's not such a big deal
because not only one incorrect prediction is still excellent.
But also remember when we visualize the training set results with PCA, well,
we had incorrect predictions, so we were just a little lucky
to get zero incorrect predictions with PCA.
So all right so that's still excellent results.
And now let's visualize the training set results.
Now do we need to change something in the section.
Well try to figure out if yes we do because it's important to understand this.
In case you use this code
section here to visualize the results on your problem, on your data set.
Well, the answer is this time, yes, because this line of code here call names
expects to have the real name of your independent
variables, the new extracted features L1 and L2.
So here we need to replace PC1 and PC2 by respectively
x dot L1, because that's the name of the first extracted feature,
the first new independent variable and x dot L2.
The second new extracted feature, the second new independent variable.
So let's replace them PC1
by x dot L1
and pc2 by x dot ld two.
So that's very important.
That's the only thing that you will need to change the cool names.
You must have the real names of your independent variables
when you visualize the results.
And then do we need to change something else.
Well this we can also change.
But that's not compulsory.
That's just for the labels of the x axis and the y axis.
So let's do it anyway.
And this time we don't need to specify the real name of the independent variable.
We can just replace PC one by L2 one to specify
that it's the linear discriminant and not the principal component.
And same here we can replace PC2 by L2.
All right.
And now that's done.
Now this code is ready to be executed.
So let's make the quick same changes for the visualization of the test results.
So let's replace PC1 here by x dot LD one
then PC two by x dot LD two
and replace pc1 by LD1 and PC2 by two.
And now everything is ready.
We don't need to change anything more.
We can grab a cup of coffee
and visualize the training set results and test set results.
So let's do it.
Let's hope that everything is okay.
So I'm going to select all the section up to here.
So that's to visualize the training set results.
Let's do it. And here we go.
All right. So it's executing.
It's always taking a little time.
But we will get there.
We can already click on plots.
All right.
So the computations are being made. Almost done.
And here are the results.
Beautiful results.
The three classes were almost well separated.
Perfectly well separated.
We can see the incorrect prediction. But be careful.
That's not the same incorrect prediction we saw with the confusion
matrix because that concerns the test set here.
It's the training set.
So we also have one incorrect prediction on the training set.
That's this one.
But that's almost perfect
which is quite intuitive to understand because as you remember
LDA tries to separate the most the classes of your dependent variable.
So that's why here we can see that the prediction boundary
is kind of equidistant to the majority
of the green points here, and the majority of the blue points here.
So that's perfect.
Each one is in its correct segment of customers.
And therefore this one business owner can feel pretty confident
at predicting for each new wine to which
customer segment he should recommend it.
And not only he can be pretty confident at recommending the new wines
to the right customers, but also thanks to the feature extraction
technique that allows him to visualize its results in two dimensions.
Thanks to this number of two new independent variables
the linear discriminant.
Well, now this wine business owner
can make a clear plot of its different segment of customers,
and putting in each segment of customers the different wines.
So eventually that can be pretty convenient.
Okay, so perfect.
We managed to build a great LDA model, so we'll finish on this good note,
and therefore we'll move on to the next section of this course,
which is going to be another feature extraction technique.
But this time adapt it for nonlinear problems.
So since this problem is obviously
a linear problem because we managed to apply very successfully
linear models as PCA and LDA, well, it's won't be relevant
to apply a nonlinear feature extraction model on this data set.
So we will work on another data set that is of course going to be nonlinear.
And this next new feature extraction technique
that we're going to see is going to be kernel PCA.
So I look forward to studying this in the next section.
And until then enjoy machine learning.