Now we are going to discuss the third ensemble technique, which is boosting.
In boosting also we create a number of trees.
But the difference is that the trees are grown sequentially.
That does each tree is grown using information from previously grown tree.
We'll be discussing tree boosting techniques, gradient boosting.
Add a boost, an extreme boost.
First, let's understand the concept behind grading, boosting.
Gradient boosting is a slow learning procedure, that is we fed artery using current residuals rather
than the outcome as a response.
So this is what happens in a gradient boosting.
We start with a single node three, which has all the observations.
The single Node three has some predictions.
We find the difference between predictions and actuals to get did as it was.
Then we use all the variables to fit these as tools.
Using a small tree, but no duck.
We control the length of the tree in boosting, like bagging.
Very creative.
Full length trees.
So this small tree we just waited on need as it was, is multiplied with a shrinkage parameter.
And then is added to the original tree.
So the second tree is basically the sum of first tree and newly created tree multiplied by a shrinkage
parameter.
Now, using this secondary, we again find dirigibles and then using these residuals.
We again fit a small tree on these residuals.
This morti is again multiplied by LAMDA and added to this again, three to create the third three.
And so on.
And this way, we sequentially continue to create trees and add them while fitting the current Reginald's.
So instead of learning, by creating the whole tree in one go, grading, boosting.
Learn slowly by creating one small tree at a time.
This is why it is called a slow learner.
Additionally, if you keep a very small value of this shrinkage parameter lambda, it will learn even
more slowly.
So when we are learning this grading, boosting in our software, we need to provide three tuning parameters.
One of them is number of trees to be built, which will mean how many small trees will be created and
added to the main tree.
Although I like bagging boosting in overfit.
If the number of trees is too large, the second during is that shrinkage parameter lambda.
This parameter controls the rate at which model loans.
Typically it has values between zero point zero one and zero one zero zero one.
The third parameter is depth of the boosting tree.
This basically controls the growth of these small boosting trees.
Even trees of size one sometimes vote when.
Such small trees are known as stumps.
So will need to provide the value of these three tuning parameters enough for threat mandatorily to
run a gradient boosting model.
Next is adaptability or adaptive boosting?
In this first we created three.
We find out the predictions using that three.
Wherever that tree has misclassified or in case of regression, wherever the residual of that tree is
very large, we increase the importance of that particular observation.
Then we again create a tree on our observations.
Not this time, since we have increased the importance of those observations.
Our tree will try to capture or rightly classify those observations.
So this time we'll get a second tree, which will be a little bit different than the first tree.
Again, we will find out be residuals or misclassified observations.
We will increase the weightage of those observations and run the model again in this way.
We will continue to build model for some free decided number of names.
This breed decided a number of pain has to be given as a barometer.
When we run this model in our software, this is also a form of boosting because each of our three,
our new three is learning from the previously clear dead trees.
The third technique is a sea boost.
Its boost is almost similar to gradient boosting.
The only thing is in a steep boost.
We use a regularized model to control over everything.
I hope you're aware of regularisation methods in linear models matter such as Lassalle and Wrage integration.
Use these techniques if you are aware of that.
That's great.
If you are not, you can find the links to understand Agent Lessel in the description of this mutilated.
Basically when we are doing regularisation.
We are adding a cost to the number of variables.
So when we are optimizing MSE in case of regression or Jenie in case of classification, we add an additional
penalty item for the number of variables that are going to be used in that model.
And this way, we try to minimize the number of variables that come in that final model by doing regularization.
Basically our model, a white overfitting.
So a boost contains a regularization terms in the cost function otherways.
It is exactly similar as Bourdin boosting only differences.
It includes a regularization.
That is why it has better chance in preventing overfitting.
There are some other minor differences between a boost and gradient boosting, which we will not be
covering as part of discourse.