Hello and welcome to this
Python tutorial and welcome to part ten Model Selection and Boosting.
So in this part we will do two things.
First, evaluating. Our model performance.
And second improving our model performance.
And then there. Will be this.
Bonus section about one of the most.
Powerful. Algorithm in machine learning.
Which. Has become more and more. Popular.
And that is called XGBoost.
But first, we want to be able to.
Improve the model performance of all the machine learning models.
We've built in this course.
And improving the model performance can be done
with a technique called model selection that.
Consists of choosing the best parameters.
Of your machine learning. Models.
Because you know, remember, every time we build a machine learning.
Model, well we have two types of parameters.
The first type where the.
Parameters that the model. Learned, that is the parameters.
That were changed and found optimal. Values.
By running the model.
And then the second type of parameters are the parameters that we chose ourselves.
For example, the kernel parameter in the kernel SVM model and these.
Parameters are called the hyperparameters.
So there is still. Room to improve the model.
Because we can still choose.
Some optimal values. For these parameters.
But since these parameters.
Are not the parameters learned by the model,
then we need to figure out another.
Way to choose these optimal values for these parameters.
For the hyperparameters.
And that's one of the powerful thing we'll do in this part ten.
And that will be through a very efficient technique called grid search.
But before we start grid search we need.
To optimize our way.
To evaluate our models, because so far, what we did is split our data set.
Between a training set and a test. Set.
And you know, we trained our model on the training set
and we tested its performance on the test set.
That's the correct way of.
Evaluating the model performance.
But that's not the best one because we actually have the variance problem.
The variance problem can be explained by the fact.
That when we get the accuracy on the test set, well, if we.
Run the model again and test again, its performance on another test. Set.
Well, we can get a very different accuracy.
So judging our model performance only on one accuracy on one test.
Set is. Actually not. Super relevant.
That's not the most relevant way.
To. Evaluate the model performance.
And so there is this technique called k fold.
Cross-Validation that improves.
This a lot because that will fix this.
Variance problem.
And how will it fix it.
It will fix it by splitting.
The training set into ten fold.
When k equals ten.
And most of the time k equals ten.
And we train our model on nine folds.
And we test. It on the last remaining fold.
And since with ten folds we can make ten different combinations
of nine folds to train the model and one fold to test it.
That means that we can train the model and test the model
on ten combinations of training and test sets, and that will.
Already give. Us a much. Better idea of the model.
Performance, because.
What we can do afterwards is take.
An. Average of the. Different. Accuracies of.
The ten evaluations, and also compute the standard deviation to.
Have a look at the variance.
So eventually our analysis will be much more relevant.
And besides, we'll know in which.
Of these four categories will be.
Because if we get a good accuracy.
And a small variance will be on the lower left one.
If we get a large accuracy and a.
High variance, we will be on the lower right one.
If we get a small accuracy.
And a low variance, we will be on the upper left one.
And eventually, if we get. A low accuracy and a.
High variance, we will be on the upper right one.
So this k fold cross.
Validation is very. Useful.
And besides our performance analysis is much more relevant.
So let's start with this k fold cross-validation.
Our first technique of model selection.
So since we already built a.
Lot of models we're not going to build another one.
We are going to use one of the model we.
Built and apply k fold cross-validation.
On it.
And so the model. We're going to use is this kernel.
SVM we made in part three. Classification.
And that remember.
We used to predict if the customers.
Are going to click on the ads on the social network.
To buy yes or no the SUV.
So the model is.
Already built and we already have everything.
So what we're going to do is take the.
Whole model and.
We are going to add a new section.
Code inside of. It.
That is going to be, of course.
The section code that will implement k fold cross-validation.
So before we start doing it, let's pick the right folder.
As working directory.
So we go to File.
Explorer machine learning. A to Z.
We are now in the last part of this course.
Congratulations for reaching it for ten models selection and boosting
and section 48 model selection.
All right.
Make sure that you have the social network. Add CSV file.
And if that's the case you're ready to go.
All right. So now.
Where do we apply the k fold cross.
Validation code section.
Well since that. Consists of evaluating.
The model performance.
Well the most relevant location to put.
It is right after we build our kernel SVM.
Model. That is right after we built the model.
And actually in this. Code here we have.
The predictions of the test. Results.
And the confusion matrix.
That is actually a first way of evaluating.
The. Model.
But as I said at.
The beginning of.
This tutorial, this is a correct way of.
Evaluating the model, but not the best one.
And in today's. Tutorial.
We are introducing a much better.
Way to evaluate. Our model.
And so let's put it right after.
This section as in a more advanced performance evaluation method.
And so we're going to call this section applying.
K-Fold cross.
Validation.
All right so now let's start.
So the first thing is of course to import the right class for the job.
And more precisely the right function.
And this function. Is called cross val score.
And it is imported from the model selection module.
That is the. Same module.
That we use here.
To import the Traintestsplit function.
To. Split the data sets into the training set and the test. Set.
So that's the. Same.
So let's. Do it. Let's import.
The crossover score. Function. So from.
Sklearn dot.
Model selection here. We go.
Import cross val score.
All right. That's done.
And so now let's. Apply k.
Fold cross validation on our training. Set.
So before applying it we need to understand what it will return.
Well it's will actually return.
The ten. Accuracies for each one of.
The ten combinations that.
Will be created through k fold cross. Validation.
So you know, since each combination is composed of nine.
Folds. To train the model and one fold.
To test it. Well, as I said before.
We'll get ten combinations and therefore ten.
Accuracies. And so the first thing to do. Is to.
Define a.
Vector that's we're going to call accuracies.
And that is going to be the vector that will get the ten accuracies that.
Will be computed through the ten combinations.
Created through k fold cross validation.
So eventually.
This accuracy vector will.
Be composed of ten elements. And these ten elements.
Will be ten accuracies.
To evaluate our model.
All right.
So now what we have to do is to use this cross val.
Score function.
To apply k fold cross.
Validation very efficiently.
So cross val score.
And then parentheses.
And then we input the different parameters.
So let's see what. Those parameters are.
I'm going to. Press.
Command Eye here to inspect.
This cross. Val score function.
So let's see the first parameter is estimator.
So that's your model.
That's your classifier.
And how did we call our classifier.
We called it classifier.
So here we. Have two inputs.
Estimator equals classifier.
All right.
So first parameter is our estimator classifier.
Then the second.
Parameter is x the data to fit.
So that's actually the training set because k fold cross-validation.
Is applied on the training set.
But that is the matrix of.
Features part of the training set.
So therefore x will be x train.
Because our training set is composed of X.
Train the matrix of features and y train the dependent.
Variable vector.
So here we add x equals x right.
All right.
Now the next argument, as you might have guessed.
Is y and y of course is the.
Dependent variable. Vector of the training set.
As you can see it is set to be the target variable to try to predict.
In the case. Of supervised learning.
And that still corresponds to the training set.
So of course that is Y train.
So here we add y equals y train.
All right.
Now next argument next argument is groups.
But that's not. Actually important here.
Same for scoring.
Not the most important which we are mostly interested in is.
This CV parameter.
Because the CV parameter is actually.
The number of. Fools you want to split your.
Training sets into.
And the most common choice for the CV number.
Is actually ten.
Most of the time
you'll use ten fold cross-validation, because ten fold cross-validation means.
That you'll get ten accuracies.
And ten. Accuracies. Is actually enough.
To get a relevant.
Idea of the model. Performance.
So here we will add CV equals ten.
Perfect. And that's.
Actually all we need to input here to.
Perform an efficient k fold.
Cross validation.
If you. Are working on a very.
Large data set.
What you can do is. Add this.
N jobs. Parameter.
And set its value to minus one, because minus one means that you.
Will use all the.
CPUs on your machine, and therefore your.
K fold cross. Validation will run faster.
But here. That's fine.
The point of. This tutorial. Is not to work on a.
Large data set.
It is rather to learn how to apply k fold cross-validation.
So we are ready to get our ten accuracies, and therefore
we will get a much better idea of our model performance.
So let's do this.
Let's select all. The lines of code from.
Here to the top.
And that will. Run the preprocessing. Phase.
Fit the kernel SVM model to the training set, predict the test set results.
Making the confusion matrix.
And that's the first.
Performance evaluation method.
And there is our more advanced performance evaluation method.
K fold cross-validation.
More precisely ten fold cross-validation.
And let's see what we get.
So let's press. Command and control plus.
Enter to execute.
Here we go.
We just have a.
Data conversion warning because some integers were converted into floats.
That's fine.
Now let's see our accuracy vector and variable explorer.
So here it is accuracies.
And if we open it we get. The ten.
Accuracies of this k fold cross.
Validation process. And so what do we see here.
We can see how it was relevant to do this because the first accuracy is 80%.
But then the second accuracy is 96%.
And then 80% again.
And then 90. 3%, 80. 6%, 83. Percent.
So that clearly shows what I was telling you at the beginning of this tutorial.
That is when you test.
The performance of your model on one test. Set.
Well, you get an accuracy, but then.
When you test it again and another. Test set.
You can get a. Very. Different. Accuracy.
And so.
Evaluating the model performance on one test. Set.
Is not. Very. Relevant.
And now with.
K fold cross-validation we are.
Testing it on ten test. Sets.
And therefore we will get.
And therefore.
Now what we're going to do is take the mean.
Of all these ten accuracies.
And that will give us a much better idea of the average model.
Performance of our model.
So let's press okay.
And let's get this average.
To get this average it's actually very simple.
We just need to say our accuracy is vector.
Here we go. Then just add a. Dot.
And then just add the mean function.
And that's it.
Just running this will give.
Us the mean of the ten.
Accuracies of this accuracy vector.
So we get 90%.
That means that the mean of.
These ten accuracies here is actually 90%.
And so in conclusion, this 90%.
Accuracy is the relevant.
Evaluation of our model performance.
It is not 80%.
It is not either 96%.
It is the mean of. At least.
Ten model. Evaluations.
That is ten. Accuracies.
So perfect.
And if we want to push the analysis further, what.
We can do is.
Compute also the standard deviation of this accuracy vector.
That will tell us if there is a high variance or low variance.
So that's also very interesting.
And to do this we do the same.
We take our accuracies vector.
Then we add a dot.
And then we. Use the STD. Function.
That will give. Us the standard deviation.
Of this accuracy vector. So here we go.
Let's run it.
And we get a six.
Percent standard deviation.
So what does that mean.
That means. That the average of the differences.
Between the different.
Accuracies that we'll get.
When evaluating our model performance.
And the average.
Accuracy that is. 90%.
Is 6%.
So that's actually not too high variance.
That's okay.
Because that means that
when we evaluate our model performance, well most of the time it will be.
Around 84. Percent and 90. 6%.
So eventually that.
Means that we are in this low bias and low variance.
Category.
So that's pretty. Good.
And now congratulations you have a much more advanced way.
Of evaluating.
Your model performance in your data science toolkit.
Which you'll see. In the next tutorial.
We'll see a very powerful technique
that will help us choose the optimal hyperparameters.
Of any machine learning model we. Built.
So I look forward to doing. That in the next tutorial.
And until then, enjoy machine learning.