Hello my friends.
Welcome back to this model selection section.
In the previous tutorial we implemented the k.
Fold cross-validation.
Technique, which is a new tool you now have in order to measure
the most relevant way.
The. Performance of your model.
And now I'm giving you an extra.
Tool, which is grid search.
And which. Will allow you to.
Find the optimal.
Values of the. Hyperparameters in any model.
To therefore. Get an even higher. Accuracy.
Than, you know, the standard version of the model.
So we're. Going to work on the exact same data.
Set. Social network ad.
So I'm not going to remind this again.
And therefore let's.
Open our.
Grid.
Search implementation either in Google.
Colaboratory or Jupyter Notebook as you wish.
And actually this grid.
Search implementation.
Is exactly the.
Same as this one.
Only we're just going to implement one.
Extra cell which will be, you know, the cell applying grid.
Search to find the best model and the best parameter, the best model version.
Actually of the kernel SVM.
All right.
Are you ready for this?
I'm sure. You are.
So let's create right away. Copy.
Because this file. Is in read only mode.
And we're going to, you know, re-implement.
That cell building the grid search technique okay.
Let's now remove this because we're going to work with our copies.
And there we go.
Now let's scroll down, you know, to find.
That cell. Implementing grid. Search.
Let's remove it.
You know put it in the. Trash.
And there.
We go.
Let's apply grid search.
To find the best model and the. Best parameters.
All right. So let's create a new code cell.
And now the first thing I want to do.
Is actually go to the scikit learn API.
There we go to show you actually.
Again. The class.
That implements the kernel SVM model. Because.
I actually want to show. You the hyperparameters.
Which we can tune to find the best version of the kernel SVM model.
So here take the first link.
That's the scikit learn website.
Then please go to API.
Here you need to find all the classes and the functions.
And remember.
Kernel SVM is actually.
In the. SVM. Section.
So we have to scroll down to find the S letter.
And we should. Find it very soon. There we go.
Support vector machines sklearn dot SVM.
So that's SVM module. By the cycler library.
And remember the.
Class is this.
One SVM dot.
See. Right.
If we you know check on our implementation.
You know we import this.
SVC class to build this kernel.
SVM model.
With a radial. Basis function kernel. Okay.
All right.
So k fold cross.
Validation grid. Search.
There we go back. In the. Cycling API.
Well the thing I wanted to show you was.
That indeed the kernel SVM model.
Has a lot of hyperparameters.
I remind that hyperparameters are the parameters which are not learned
during the training process.
You know, these are different parameters than the weights or, you know, the.
Coefficients of the model.
And these hyperparameters are, well, you know, there are many of them.
You have.
The c parameter, which is the.
Regularization parameter.
That. You can tune to. Actually.
Improve the training performance.
By reducing overfitting.
I remind that overfitting is.
A situation where you have a high accuracy.
On the training set.
But a poor. One on the test. Set, which.
Means that your model. Was over.
Trained on the training set.
You know, too well.
Trained and we definitely want to avoid. This.
So that parameter is actually very important.
And we will tune it to find. The best value.
The default value. Is 1.0.
But we will try several values. We will try.
Oh point 25, 0.5, oh point 75 and one.
Because as you can see, it's. Important to notice. This.
The strength of the. Regularization is.
Inversely proportional to C, meaning that the lower is C,
the stronger will be the regularization.
So you know we're going to try the value of.
Oh point 25.
Which will be therefore a strong. Regularization.
We'll try. 0.5. As well.
Still quite a strong regularization, oh point 75 less strong regularization.
And one and we can even try, you know, oh point 25, 0.5 as you want.
Feel free. I will actually. Improvise. This a bit.
I'll see what.
I end up choosing.
As the values of C will. Experiment.
And then we have the kernel.
So of course that's a very important hyperparameter.
And we will actually tune it.
Meaning we will find the best kernel of the kernel SVM model among these one.
So we'll actually try both.
The linear one and RBF one.
But feel free to also try for example.
The poly or. Sigmoid one.
And then. I would also like to. Tune. This.
Parameter gamma.
Which is a parameter.
You only enter when you choose.
Either RBF. Kernel or polynomial kernel.
Or a. Sigmoid kernel.
And since we're going to tune the linear and RBF.
Kernels well for the RBF kernel, I would like.
To tune, you know, different.
Values of. Gamma to actually find the best one.
And actually.
We will try all the gamma values.
From 0.1 to. Zero.
Point nine, okay. And gamma is.
Of course the. Coefficient in, you know, the.
Formula of the kernel, which. Kernel.
And I have already showed you in part three classification.
All right.
So let's tune. These three parameters.
And if you want to tune more of them will. Feel totally free.
Or if you want to tune more. Values of. Them.
But you will see that already with this. Tuning.
We will get. An excellent performance.
And speaking of performance, actually it's important to.
Highlight that each. Time we.
Will try these.
You know, different combinations of.
These hyperparameter values.
Well, the accuracy of the model will be evaluated through k fold.
Cross-Validation and not on a single test. Set.
Okay.
So we will have super.
Relevant measures of. The accuracies for each.
Of the combinations of these. Parameters.
And therefore we will be. Confident.
That the best.
Set, you know, the best combination of parameters we end up.
With. Is indeed. The right one.
All right.
So let's do this.
Let's implement grid search now.
And of.
Course as usual we're going to start.
From. Scikit learn from. Scikit learn.
By the. Way let's.
Upload the notebook now so that we can, you know, benefit from Google.
Collab. Assistance. Let's click upload here.
And there we go.
We already have the right folder and we select social.
Network. Ads dot csv.
That's the data set.
With which we're going to experiment. Grid search.
And so there we go. Now we should.
Get the help of Google Colab.
So we're going to start from scikit learn.
And then same we're going to get access to the model selection.
There we go module from which we're going to import this time a.
Class which is the grid search CV class.
There we go okay.
And now already in the next step
we're going to enter the different combinations of parameters.
We want to experiment.
And so I'm going to create a new variable here which I'm going to call parameters.
And then I'm going to enter these combinations
of hyperparameters we want to experiment with in a list.
So you know this parameters variable here will simply be the list of the different.
Combinations of hyperparameters we want to.
Test.
And then within that list I'm actually going.
To create two. Dictionaries.
Why to because.
Actually I want to test. Two. Different kernels.
So in one dictionary we.
Will have the. Linear kernel.
And in a.
Separate dictionary we'll have the RBF kernel.
And the reason why.
I have to separate these two dictionaries is.
Because, you know, the gamma parameter which we want to tune, can only.
Be used with the RBF kernel.
And not with the linear kernel.
If it could be used.
With the linear kernel, then I wouldn't have to make.
Two dictionaries.
Only one would be fine and I would just enter the different.
Values of all the parameters.
Which we want to experiment.
But that's not the case.
So indeed I have to create these two dictionaries, which.
I'm going to put, you know. Below each other, just like that.
And there we go.
Now let's enter the first dictionary.
So how.
Are we going to do.
This. Well, just the way dictionaries work.
First we have to enter the key.
And the key is actually the parameter itself.
So remember we want to tune first.
See the regularization. Parameter.
Then the kernel.
And then in the case of.
The RBF kernel the degree. And so.
In my dictionary. Here.
I'm going to enter.
As the first key. In. Quotes.
Well this parameter c.
And then I'm adding here some colon.
And right after the.
Colon I have to enter the.
Value of that. Key.
And the value of.
That key is exactly the values of C.
We want to experiment, and we have to enter them in a new.
List. In.
Pair of square brackets.
And as we. Said, we want to try the.
Following different.
Values of this regularization.
Parameter.
C, which are, as we said first oh point.
20. Five.
Then oh point. Five.
Then oh point 75, and then one.
Right.
So here on the left we have strong regularization. And.
The closer. We get to one, the less strong is.
The regularization.
All right.
So that's for this parameter c.
Then we can enter a second key within this dictionary.
And we have to separate with a comma.
And that second key will.
Be of course that's other.
Parameter which we want to tune. Which is.
The kernel.
And therefore in quotes here I'm going to enter kernel.
And then for the value.
Of that key which has to be separated.
By Colin.
Well this value. Will be once again in the. List.
Therefore in the.
Pair of square brackets, the different values of the.
Kernel, we want to experiment.
However, since we have these two dictionaries to separate the case of.
The linear kernel and the RBF. Kernel.
Because, you know, if this gamma parameter.
Well, here.
In this.
Kernel list, I.
Will only enter in quotes.
Once again, the linear kernel.
And now check what I'm going to do.
I'm going to.
Copy all of these.
You know inside. This first dictionary.
I'm going to paste that. Inside this.
Second dictionary.
And then here we're going to test the same values.
Of the parameter c.
But we're going to test. Them.
With. An RBF kernel.
And therefore since now we're experimenting.
The RBF kernel, well. Then we can add this.
Extra hyper.
Parameter to tune which. Is the.
Gamma parameter which is a. New key of this.
Second dictionary.
So gamma.
And the value.
Of that key will be the list of the different values.
Of gamma. We want to.
Try it. Right.
And as we. Said these are oh point.
One than oh.
Point two than 0.3 and 0.4,
0.5, 0.6, oh point.
Seven.
0.8. And 0.9.
All right. Good. So you see how this works.
We have to enter our parameters into a list.
And then inside each list.
Well either we don't have to separate some cases.
And we can put all the hyperparameters and their values into a same dictionary.
Or we have.
To separate some cases and we enter them with.
Two separate dictionaries. This way.
All right. Excellent.
And now. That we prepared correctly our.
Parameters, you know our hyperparameters combinations.
Well time to call.
That Gridsearchcv class.
Because indeed, as you can guess, this list of parameters will be.
One of the inputs.
Of that Gridsearchcv class.
Or you know, the instance.
Of that class we will create.
And well, speaking of.
The instance of that. Class, that's.
Exactly our next step. Here.
And so I'm going to create a.
New object, a new variable.
Which will be an object. Of this Gridsearchcv class.
In which I'm going to call.
Simply. Grid underscore. Search.
All right.
Then of course. We have to call a class to create.
Such an instance.
So grid search.
There we go. I was waiting for Google Colab.
Then in parenthesis let's enter the parameters.
You can find them on the.
Cyclone API, but I'll. Just give them to you now.
So the first one is. Estimator.
Right.
And this will be equal of course to your classifier.
You know just like. The estimator argument in the crossover.
Score function.
So classifier and which is of course your kernel.
SVM. Classifier.
Then next argument.
Well next argument.
Is actually our parameters.
You know the different combinations of the hyperparameters.
We want to experiment and try to find the best value.
And so the name of that argument is param underscore grid.
And this.
Will be of course equal to or parameters variable that list.
Of all the combinations.
Of. Hyperparameter values.
Then next argument.
Well the next argument is an important one.
That's the. Scoring. And that's. Basically.
The. Metric with which you want to evaluate.
The performance of your model.
For each of these.
Combinations. Of hyperparameter values.
And well, since now we're doing classification well.
The value for this argument will be in quotes accuracy of course.
All right then next argument.
So as I told. You, you know each. Of these.
Combinations of.
Hyperparameters will be evaluated through fold cross-validation.
And not on a single. Test set.
And therefore here the next argument is just to choose the number of train test.
Fools that will. Have when applying k.
Fold cross-validation for each of.
The combinations, and that parameters name is the same.
As right here it is CV.
And once again we will choose a.
Very common value. Of ten.
Train test. Folds.
And finally a very optional parameter which.
You don't have too much to worry.
About, but.
I'm adding it here in case you run this code on your machine.
This is a parameter that will set how to run your processor.
As you know, in your machine.
And I'm just adding.
It here with the parameter.
Value minus one, which means that all.
Your processors will be used.
You know, in your machine which will optimize the.
Grid search process.
Because you're going to see that it can be a very long process, you know, to.
Test all. These combinations.
Well, it.
Can actually take some time.
Okay.
So that's it for this Gridsearchcv class.
We now have our object.
And as you might. Guess.
The next.
Step will be to connect this object to our training set.
I remind that. Grid search has to be applied.
On the training set.
Only because the test set once.
Again is something you want to have. Separate it.
Whenever you want to. Test your.
Model or you know your tuned.
Model, you know after you tune it. On a.
Set of new observations to see how.
It performs on a data set like in production.
Okay. So we will. Apply now grid.
Search to. The training set only.
And to do this well we're going to call our object grid search.
From which we're going to call once again that very famous method which.
You're so. Familiar with now.
Which is the fit. Method.
And of course, this fit method.
Takes two. Arguments as input.
Which are first, the features of the.
Training set, meaning.
X. Train, and then the dependent variable.
Vector of the training set.
That is why train just like a classic training, right?
Because actually now we're going to.
Do a lot of. Trainings, you know, training for.
Each of these.
Combinations of. Hyperparameter values.
Okay. So that's basically.
The multiple training.
And then there we go.
Once this multiple training is done well.
We'll get all. The final results.
Meaning all the different. Accuracies.
Resulting from all these combinations of hyperparameters.
And therefore.
After getting all these accuracies, well, we'll get the.
Highest of them. And we'll get.
Mostly the set of parameters that lead to that highest accuracy.
And so first let's get that best accuracy.
Which we're going to put into a new variable, which we're going to call.
Best accuracy.
And to get this well we're.
Actually going to get it from an.
Attribute of the greet search CV class which is called Best Score.
I remind that an.
Attribute is just a value.
We can get from a. Specific object.
And that value here is of course the best accuracy.
Taken from that grid. Search object.
So I'm going to copy this.
And just. Below I'm going to paste it.
From which I'm going to call. That best.
Underscore. Score.
And then we have to add an underscore.
Again that's usually. How our named. Attributes.
All right. So best.
Score meaning best. Accuracy because we.
Chose an accuracy scoring.
And now now that.
We have the best accuracy well let's see which.
Parameters. Led to that best accuracy.
And to do this well we're going to.
Put that best set of.
Parameters once again into a new.
Variable which we're going to call best.
Underscore. Parameters.
All right.
And once again to get them we're going to call another.
Attribute of our Gridsearchcv. Class.
Or you know, our. Grid search. Instance of that. Class.
And therefore I.
Just copy this and will paste that here, because this.
Time the name of the.
Attribute is best params.
Best params. Okay.
And now we're going to finish this.
With two beautiful. Prints.
And these prints are actually.
Very similar. To the ones we did.
For k fold cross validation.
So I'm just.
Copying this.
And. Right here. Pasting them.
And now instead of printing you know.
Accuracy a. String we're going to print.
Best accuracy.
Then we're going to.
Keep the same format of you know, the two decimals.
After the. Comma in a float.
And for the.
Variable that will get us this, well, of.
Course this won't be the mean.
Of the accuracies here, but simply that
best accuracy variable value right.
This gives directly the best accuracy.
Resulting from the best combination of hyperparameters.
So let's just do this. And.
Let's just replace that accuracy.
Dot mean you know the mean function by.
This best accuracy variable.
And then we have to multiply by 100 in order to get.
The accuracy. In this format.
You know, with the percent format.
Which we have. Here once again.
And then we don't want to print the standard deviation.
But the best parameters.
And then I'm just going to print these.
Best parameters the classic. Way, you know.
The simple way. Of printing some things.
So I'm just going to remove everything here.
You know that format function and that variable.
So because it will. Be a different variable.
And then look I'm just. Going to do this.
This is the string that will be printed.
And then to print the value of that string of the best parameters themselves,
I'm just going to.
Separate that.
String with a comma and then adding.
The variable that will give us these best parameters.
That's just a classic way.
Of printing something with, you know, a string indicating what we're printing.
And so here.
We just need to add best parameters.
And there we go, my friends.
We're done with this implementation.
Well we're almost done.
We just forgot to remove one parenthesis.
Which was. I think coming from this format function.
But there.
We go.
Now the implementation. Of grid search is done.
So we're going to experiment this now.
And I look. Forward to see which.
Combination of all.
These hyperparameters will lead to the best accuracy.
And of.
Course we will compare.
That best accuracy to.
The relevant accuracy.
We got with the.
Standard version of the kernel SVM model.
You know, with the.
RBF kernel.
And the. Default value of.
One for.
That regularization parameter c.
And then the same default.
Value. Of the gamma parameter.
So let's see we have our data set.
So we are ready to do this restart and run.
so there we go.
321 go.
And yes.
All right all.
Right. So now the.
Cells. Will be running.
There we go. Grid search is now running.
And we're about to get okay.
So very. Very interesting.
We get a best accuracy.
Of 90.67% which.
Is slightly.
Better than this 90.33%.
But you will see. In your.
Future machinery projects that.
Even a slight improvement of the accuracy can make a difference.
And the. Best.
Combination of parameters that.
Lead to that best accuracy is the following.
First, regularization parameter c of.
0.5, meaning that. Indeed. It was necessary.
To reduce a bit that hyperparameter C to reduce overfitting.
Then, of course, we get the best accuracy.
With an RBF kernel, meaning with the.
Kernel.
SVM model instead of the.
Classic linear. SVM.
And the best value of.
This gamma parameter, you know, when we have a.
RBF kernel. Isn't it? 0.6.
All right. So it was actually good to test.
All of them.
And that's what. We get as. The best model.
So now.
Feel free to.
Try some other values you know if you want and maybe some other hyperparameters.
And if.
You get a better accuracy.
Than 90.67%, which I'm sure you can get easily.
You know, I haven't tried all the combinations.
Well, once again, feel free to share this in the Q&A section or in private message.
But Q&A section is.
Best so that other. Students can see.
How you did it.
All right. And now we're going to move on to the.
Very. Exciting final section. Of this course of.
The bonus section, which. Is about XGBoost.
And which will. Basically give.
You an.
Extra super.
Powerful machine learning model that you can actually.
Both apply to classification.
And. Regression.
So you.
Definitely want this as the final. Tool of your.
Huge machine learning. Toolkit. So I look forward.
To implementing this final model with you.
And until then, enjoy machine learning.