Hello and welcome to this art tutorial.
So you learned a lot of stuff, but it would be a shame
to leave this course without having some introduction to one of the most popular
algorithm in machine learning that is quite recently popular.
But still,
it is definitely a very powerful model, especially if you work on large data sets.
It will offer you very high performance while being fast to execute.
And speaking of performance and execution speed,
it is important to remind that XGBoost is the most powerful implementation
of gradient boosting in terms of model performance and execution speed.
Therefore, it's very important for you to have it in your toolkit.
So let's implement XGBoost.
This is only going to be an introduction, so we will make a simple
implementation of XGBoost.
But you will have the code template on your computer and you'll be able
to try it on your problems on your data sets.
And you'll see that even with this simple implementation,
it will definitely give you some excellent performance.
And so now what we're going to do is take one of the business problem
we dealt with in this course.
This is going to be actually the problem that we solved in the deep
learning section.
Remember this is the churn modeling problem
where we need to predict the customers of the bank.
That will leave the bank.
So this was a classification problem where we classify the customers
in two classes,
those who will leave the bank and those who will not leave the bank.
And so remember for this problem we obtained an accuracy of 86%.
But that took quite a while because we trained an artificial neural network
with many epochs, and therefore it took quite some time to execute.
And so now in this section we're going to do the same.
We're going to apply XGBoost on this churn modeling problem.
This data set contains,
if I remember, 13 features, but that's not a large data set.
And what is important to highlight is that even if this was a large data set,
a very large data set, well, XGBoost would be one of the best model
in terms of performance, that is to get a good accuracy and execution speed.
So for example,
if you are working with a large data set, I strongly encourage you to test XGBoost.
All right.
So what we're going to do now is take the pre-processing phase.
So there is only part one because part two is to implement
the artificial neural network.
And so we just want to preprocess the data for this churn modeling problem
associated to this churn modeling CSV file.
But actually we're not going to take everything in this pre-processing phase.
The reason is that for the artificial
neural network, well, feature scaling was totally compulsory.
No questions asked.
Feature scaling must be applied for deep learning.
But the good news is that for XGBoost, well, since XGBoost is a gradient
boosting model with decision
trees, well, accordingly feature scaling is totally unnecessary.
And that's one of the very good thing about XGBoost.
Besides, it's high performance and it's fast execution speed.
It's that you can keep the interpretation of your problem,
of your data set, and of the results you'll get after building the model.
So we can understand now why XGBoost is so popular.
It's because it has the three qualities first quality, high performance,
second quality, first execution, speed, and third quality.
You can keep all the interpretation of your problem and your model.
So definitely a model to have in your toolkit.
All right.
So feature scaling is unnecessary here.
And therefore we will take everything from here up to the top like this.
Copy.
And we'll paste that in our XGBoost file.
Here we go.
And now we can implement XGBoost.
So first let's introduce new section fitting
XGBoost to the training set.
All right.
And first let's install XGBoost.
So as usual there's a package.
It's XGBoost package that will allow us to implement XGBoost very efficiently.
So let's type here.
As usual install dot packages
and inside the name of the extra boost package,
which is simply XGBoost.
Like this.
So then you select this line and press Command
and Control plus enter to execute.
And that's installing the XGBoost package.
All right. We can see it's processing.
And here again it's downloaded.
Binary packages are in this package folder.
All good XGBoost is installed.
So let's put that section in comment
there.
And now let's import the execute package because indeed we installed it.
But if we go down to the bottom XGBoost is installed but not imported.
And we want to make it automatic.
So as usual we use the command library and inside XGBoost
and that will import the package.
All right.
And now let's implement XGBoost.
And actually this is going to take one line
because we're just going to make the classifier the XGBoost classifier itself.
And so basically we just need to create a new variable
that we call as usual classifier and then equals.
And then we use the XGBoost function from this XGBoost package.
So XGBoost and parenthesis
and let's click here
press F1 and get some information about this execute function.
So the information is we are interested in the arguments
and what arguments do we need here okay.
So first we see that
we have this params parameter which is actually a list of parameters.
And these parameters are all the parameters that you can see here.
For example the eta parameter that controls the learning rate,
the gamma parameter which is the minimum loss reduction.
Well, you have a lot of these parameters, but
this tutorial is just an introduction of XGBoost.
So we will not do some tuning on our XGBoost model in this course.
But I'm sure in some future courses I will make some more complex
implementations of XGBoost on some more complex problems,
which in this course is just to end with a simple introduction of boost
so that you can at least have some knowledge about it
and have it in your toolkit.
So let's not focus on this now, and let's move on to the compulsory
parameters that are of course the first one is data.
So data is of course your training set
the data sets on which you want to train your XGBoost model.
And so let's import that right now.
So first argument data equals then training set.
Here we go.
And actually here we only need the features in the training set.
So we will remove the dependent variable from this training set.
Because this training set contains both the features and the dependent variable.
But what this data parameter expects is only the features.
So here we add some brackets and we remove that have been invaluable.
And what is this index.
Well to do this we need to import the data set.
But first before importing the data
set let's quickly set the right folder as working directory.
So right now we're importing then section 49 XGBoost.
That's the right folder.
Make sure that you have the churn modeling CSV file.
And then click on Sets Working Directory here.
And here we go.
Now we can import the data set.
So let's import it. And that's the data set.
But remember in this data sets we don't take all the independent
variables because we're not interested in row number customer ID and surname.
We know that these three variables have no impact on the dependent variable.
So we remove them.
And that's what we do in this line.
Data set equals data set for 14.
That means that we take all the variables from the fourth variable of the data set.
That is credit score up to the last variable, exited the dependent variable.
That's the dependent variable.
And so let's select this line and execute.
And now if we look at our data set well this contains all the relevant features.
And the dependent variable exited.
And so the challenge is with all these independent variables here
we want to predict if the customer will leave or stay in the bank.
And so that's the data set.
We consider to train the model and test its performance.
And therefore the index of the dependent variable we have to remove.
Now in the XGBoost function for the data parameter
is the last index here of the exited column.
And since we have 11 variables, well that index is 11.
So let's go back to XGBoost and let's go back to our function.
And therefore here we have to input -11.
All right.
So we have our whole training set but without the dependent variable.
So that's perfect. That's exactly what we want.
So now let's go back to help
to see if we need some more info about this first parameter.
Well indeed there is some very important information that we need to consider here.
It's that this input data set needs to be an XGBoost data matrix.
So that's basically a type of matrix.
But we can also see that in addition data
the data parameter also accepts matrix S.
But this is not a matrix.
This is a data frame.
So this won't work if we input the features this way.
So we can either convert this into an XGBoost matrix or a simple matrix.
So let's take the simple solution.
Let's convert this DataFrame features into a matrix.
And you know how to do this.
We just need to use the as dot
matrix function and put inside some parenthesis.
Because it's a function.
This dataframe of features.
Here we go.
And now this becomes a matrix.
And that's exactly what we need.
All right perfect then next argument.
So here again you have a lot of other arguments.
But these are not compulsory.
So we won't focus on them now.
But the next compulsory argument is this label argument.
Because indeed here we input the matrix of features.
But of course to train a classification model we need
not only the matrix of features but also the dependent variable.
And that's what we put in this label parameter.
And so as you might expect, since we input the features into a matrix,
well we need to input this label parameter as a vector.
And to get our dependent variable as a vector,
we need to input label equals our training set.
Then dollar.
And then we take the name of our dependent variable which is exited.
And this will give us a vector.
So training set our exited is the dependent variable but given as a vector.
So that's exactly what we need.
Because indeed, as you can see label is expected to be a vector.
The vector of response values.
The response values are of course the values of the dependent variable.
All right.
Now next argument.
What is the next argument.
Well there is a third compulsory argument that we need to input here.
And that is actually above.
But I wanted to put the label after the matrix of features that made
kind of sense.
And now there is a third argument that we need to input,
which is the in rounds argument.
And the in rounds argument is the maximum number of iterations.
So since we're not working on a two complex problem,
well, a maximum number of ten iterations will be sufficient.
So we will input here.
And rounds equals ten.
And XGBoost will be trained in maximum ten iterations.
Perfect.
And now actually this line of code is ready
to be executed to train the XGBoost classifier.
So even if XGBoost is a very advanced
machine learning problem, well, thanks to this extra boost package,
you just need a single simple line of code to implement it very efficiently.
All right, we're not going to execute this line now
because first we need to run the data preprocessing phase.
And then
I would like to add some code sections to evaluate our XGBoost model performance.
So we are going to execute the whole thing in the end.
But for now let's add the last sections to evaluate the boost performance.
And of course
we are going to take our k fold cross-validation technique to evaluate it.
And therefore here I'm going to take the k fold
cross validation section which is right here.
And we are going to use it on our XGBoost model.
So here I just need to copy this section.
Go back to my exhibitor model and paste it here.
And be careful inside of it.
We need to change the classifier
because right here that's the kernel SVM classifier.
And so basically we just need to replace this kernel SVM classifier
by our XGBoost classifier.
So I'm just copying that here.
And go back to my k fold cross-validation section
and paste the code to train the XGBoost classifier on the training set right here.
All right.
And then we need to add another line of code inside this section.
It's related to the fact that this XGBoost model
will return the predictions as probabilities.
You know it will return the probability of class one.
And therefore you know this trick to convert
the probabilities into the real predictions 0 or 1.
Well, we need to add this line of code y pred
equals and then parenthesis y pret
larger than 0.5.
So that's if the probability is larger than 0.5 then y breath will be one.
And if the probability is lower than 0.5, then y pred will be zero.
So that's where we'll get the binary outcome 0 or 1.
And that's exactly what this k fold cross-validation section expects.
And eventually before we execute the whole thing,
there are two things that we still need to change.
First, it's the fact that since the training set
is expected to be a matrix, well, that's going to be the same for the test set.
So here we also need to add as dot matrix.
And inside of the parenthesis we put our test fold.
So that's the first change.
And now the second change is of course
related to the index of the dependent variable.
Because three.
Here was the index of the dependent variable in our previous problem
where we implemented k fold cross-validation.
So we need to replace this three index by the index of the dependent
variable in our new problem which is not three but 11.
And same right here in the confusion matrix it is 11.
All right.
And now everything is ready.
We can execute the whole code.
So let's do it.
And let's see which accuracy we get.
So let's go back to the top.
We already imported the data set.
So now let's encode the categorical variables as vectors.
Here we go. Done.
Now let's split the data sets into the training set.
And the test set. Here we go. Done as well.
And now let's fit the XGBoost to the training set.
So the extra boost package was already imported.
So basically we just need to select this line and execute.
Here we go.
We get the information of the root mean squared error at each round.
So basically the root mean squared error is irrelevant.
Computation of the error.
You can picture this as the error.
And of course the lower is the error the better is your model.
And we can see that from the first iteration to the last one.
The 10th one.
Well the error decreased from oh point 41 down to oh point 29.
And besides we can see that the maximum number
of ten iterations was a good choice, because we can see that
it is more or less converging around oh point 30.
Well, feel free to try with more iterations and try to see
if it's converging to a number that is less than 30.
If you get a number close to oh point 30, then ten iterations was a good choice.
So perfect XGBoost is implemented and trained on the training set.
And now let's apply k fold cross-validation
to evaluate its performance with the accuracy metric.
And actually I'm noticing that there is still one thing to change.
It's the name of the dependent variable here.
Congratulations to those of you who noticed that we need to replace
project here by the real name of the dependent variable in our problem,
which is not purchased but exited.
So let's replace purchase here by accident.
And here we go.
Now everything should be fine.
Let's do one last check as matrix for the training set
as matrix for the test set Y pred converted into a binary outcome
0 or 1 indexes are correct for the dependent variable.
Everything looks fine.
Let's select this whole section here
and get the ultimate accuracy of our executed model.
Here we go.
All executed properly and very fast
and we get a final accuracy of 88%.
So not only that was very efficient, but also we managed to beat the accuracy
obtained with and and besides this value is the relevant accuracy of XGBoost.
So we can trust this value of 88%.
So that's very good.
Not only XGBoost was very fast, but also it gave us an amazing accuracy.
Probably the best of all the models we implemented in this course.
So that was an amazing job.
And now it is time to say goodbye,
because this was actually the last tutorial of this course.
So that's quite a feeling because this is the end of this machine
learning journey that I introduced in my very first tutorial of this course.
So yes, that's right, that's the end of the journey.
However, I am sure this is not the last machine learning journey.
This is your first machine learning journey.
I was so happy to take this adventure with you.
I really enjoyed that journey.
I hope that's the case for you too
and I'll be very happy to make some new machine
learning courses to start some new machine learning journeys.
So I hope I'll see you very soon.
And until then, enjoy machine learning.