Welcome back to the Ultimate.
Data Science course and I am super.
Excited about today.
I've got an incredible tutorial prepared for you.
We're going to cover off a very important topic and that is how to build.
Models. Step by step.
I couldn't hold myself.
I had to add that extra line, step by step to the name of the tutorial
because that is exactly what we're going to be looking at.
I'm going to give you a framework
for several different methods, and it's all going to be step by step.
So let's jump straight into it.
Do you remember the good old days when we had.
One dependent variable and one independent variable?
Everything was easy
and we just had a simple linear regression that we had to build.
And everything worked great.
But now in our data we have all. These.
Columns.
Those easy days are gone.
Now all of these columns are.
Potential predictors for.
Our dependent variable.
And there's just so many of them.
And we.
Need to decide which ones we want to keep and which.
Ones we want to throw out.
And you'll ask why do we need to throw out columns?
Why do we need to get rid of data?
Why can't we just use everything.
In our model?
Well, I can think of two reasons off.
The top of my head.
Number one is garbage in, garbage out.
If you throw. In a lot. Of stuff into.
Your model, then your model will not be a good model.
It won't be reliable, it won't be doing what it's supposed to be doing.
It's going to be a, so to speak, garbage model.
And number two, at the end of the day, you're going to have to explain these.
Variables and understand
the not just the math behind them, but actually what it means that.
Certain variables predict the behavior of your.
Dependent variable.
And you will.
Have to explain that to your executives, to your boss, to,
people you're presenting to.
So if you have a.
Thousand variables, it's not going to be practical to try and explain.
That.
So you want to keep only the very important.
Ones, the ones that actually predict something.
So how do we. Construct a model?
This is the process of building the model, selecting the right variables.
So how do we construct a model.
Well there are five methods.
That we're going to discuss of building.
Models.
Number one is all in.
Number two is backward.
Elimination.
Number three is forward selection.
Number four is bidirectional elimination.
And number. Five is score. Comparison.
We're going to talk about.
Each one of them just now.
Before we do I wanted to say that.
Sometimes you'll hear. Stepwise regression.
So stepwise regression actually refers to number two three and four.
because those are like really the true step by step methods. but.
Sometimes you will hear people say stepwise stepwise.
Regression in reference. To just number four.
So they will replace B bidirectional.
Elimination. With stepwise regression.
And that's fine.
That's that's normal. That's just. Because.
That's the more as you'll see.
From what we.
Discussed, bidirectional elimination is kind of the more general approach.
And when people say stepwise regression, they kind of by default
infer imply, imply
that is bi directional elimination and you have to infer it from there.
okay.
So let's move on to our methods.
Method number one.
All in. It's not a technical term.
I just call it offline.
Basically what it means is just throw in all your variables.
Something we just discussed we shouldn't do.
When would you do that? One is if you have prior. Knowledge.
If you know that these exact variables.
Are the ones are.
Your true predictors, you don't have to build anything.
You already know that this is the case.
You might.
Know it from domain knowledge, or you might know it
because you've done this model before, or somebody just.
Gave you these variables and said, please.
Build a model. Well, then you don't really have a choice.
You just build the model.
The other one.
Is you have to perhaps, like,
I can't really think of good examples here, but maybe there's. Some
framework in your company that.
Says that you have to use these variables, right?
So it's kind of similar to.
Prior knowledge, but it's not a, it's not. A.
it's not your decision.
It's you might want. To do it differently,
but there is a framework, you know, like maybe a bank and to.
predict.
Credit, like the likelihood
of somebody defaulting on something they have to use these specific variables.
once again, I'm not sure in.
Which industries that would be the case, but perhaps that could be the case.
And number three, you would use this method if you're.
Preparing for a backward elimination type
of construction of regression, which is our next type.
So let's move on to backward elimination
okay. So here comes a step by step stuff.
You might you might want to get your pens out and write these things down.
Because we're going to have a truly step by step methanol.
All right.
Backward elimination.
First thing step one.
You have to select a significance level to.
Stay in the model.
So by default we're going to go with 5%.
So 0.05.
And we're going to use it in the next step.
So at the beginning you.
Decide on this significance level.
Step two you fit the.
Full model with all possible. Predictors.
So you kind of do that all in approach which we just talked about.
And you put all of your variables.
Into your model.
And now we're going to start.
Getting rid of them.
Step three you considered the predictor with the highest p value.
So remember those p values that we talked about.
In for instance in gretl or in any software you can see them.
So after you've fitted the model you'll see the one with the highest p value.
So if p is greater
than your significance level then you go to step four.
And step four.
Is you have to remove that predictor.
So remove basically the variable that has the highest p value.
And from step. Four you.
Fit the model without. This variable.
So there's a star here because.
I just wanted to remind myself.
To tell you that if you just remove the variable, obviously you can't just say.
That okay.
Now now I've got the new model.
You have to actually refit them all.
You have to re recreate the model, rebuild it with the.
Fewer number of. Variables. So if you had.
Maybe I don't know, 100 variables and you removed them, one of them
you have 99 now. Well you have to rebuild it.
So the coefficients are going to be different.
The constant is going to be different.
And you need to perform that step.
Because once you remove a variable it affects.
All the other variables in your whole regression.
And so after step five you go back to step three.
Once again you look for the variable with the highest p value in.
Your new model. You take it out, you remove.
So basically step four you remove it.
You fit the.
Model again with one less variable and so on.
You keep doing that until you come to a point where
though even the variable with the highest p value
that's p value is still less than your significance level.
So if that condition p is greater.
Than SL is not correct.
Then you don't go to step four anymore.
You go to Finn.
In this case, Finn is the finish.
Your model is ready.
So as soon as all of the variables that you have left in your.
Model are.
There, p values are.
Less than the significance level.
Your models prepared.
So that's how the backward elimination method works.
Let's move on to the next one.
Next method is the forward selection.
This is.
It sounds like the opposite
on the picture in the top right corner does illustrate the opposite.
But it's much more complex than just simply reversing.
the.
Procedure you will see that it's it's a much.
Larger. Procedure.
We started with step one.
Select a significance level to enter the model.
And in this case once again we're going to select 5%.
Then we go to step two.
We fit all possible simple regression models.
So we take.
The dependent variable.
And we create a.
Regression model with every single independent.
Variable that we have.
And then we select out of all those models we select the. One.
Which has the lowest p value for.
The independent variable.
So you can already see that that is by in itself.
A lot of. Work.
Then what we do is we move to step three.
We keep this variable that we've just,
chosen, and we fit all.
Other possible models with one extra.
Predictor added to the one you already have.
So what does that mean?
That means we've selected a, Simple.
Linear regression.
With one variable.
Now we need to construct all possible.
Linear regressions with two variables where one of those two
variables is the one over you selected.
So basically we.
Add on
all possible all of the other variables one by. One.
So we choose okay. Let's add on this variable.
And then let's add on the next one like but separately.
So we construct all possible two variable linear regressions.
And just keeping.
Definitely keeping the variable that. We've already selected.
So what do we do after that.
Out of all.
Of these possible two variable regressions, we consider the. One.
Where the new variable that we added had the lowest p value.
So if that p value is.
Less than our significance level meaning that you know
that variable is a good one, it's it's, it's a significant variable.
Then we move back to step three.
What does that mean?
It means that.
Now we have a regression with two variables.
And now we will add a third variable.
We'll try all possible.
Variables that we have left as.
Our third variable.
And then out of all of those models.
With three variables we will go to step four. And we'll select.
Again the one with the.
Lowest p value for that third variable.
That we added.
And we'll keep doing that.
So basically we'll keep growing the regression.
Model but not just random. And it will be. Actually selecting.
Out of the old all of the possible combinations every single time
and growing it one variable at a time.
And then we will only stop.
When the variable that we've added it has a p value that is.
Greater than our significance level.
So when. This condition P is less.
Than SL is not true.
Then we don't go to step three.
We finish the regression.
Why will.
Because that variable that we just added.
Is no longer.
Is not. Significant anymore.
And we.
Also know that we selected.
The one with. The lowest p value.
So there's no other variable that we can add that.
Its p value will be greater.
will be less than SL.
Any, any regression which is from.
Then onwards.
It will the variable, the new variable will always be insignificant.
And so here we finish the regression.
And the trick here is that you keep.
Not the current model but the previous one.
And that makes.
Sense because you've just added a variable which is insignificant.
So what's the point of that variable. Just move a step back.
So that's how forward selection works.
I know it's a bit confusing,
but just try to wrap your head around and maybe read through these instructions.
One more time.
It does make a lot of sense when you and this like kind of picture, what.
Exactly is. Going on.
And next we're moving on to the bidirectional elimination.
So this one, as you can.
Assume or perhaps guess.
It combines the two.
Step one.
Select a significance level to stay or to enter.
And a. Significance. Level to. Stay.
So we're going to select in both cases 5%.
But it's up to you what you select.
Step two.
Perform the next. Step of the.
Forward selection.
meaning that the one that.
We just discussed.
So where new variables, when they enter in order to enter they have to be.
Less than the significance.
Level two enter.
So basically add on a new variable based on the forward selection method.
Step three perform all of the steps of the backward elimination process.
So now.
If you have two variables, start getting rid of them and see if you can get rid of.
Any of them. And then
move back to step two.
So then grow it by another variable.
And every time you grow it by a variable, say let's
let's say you were at, five variables and you went to six.
Since you went to six, you have to from all of the steps of backward elimination.
So you don't just. Eliminate one variable.
If you can eliminate one, two.
Three, however many you can.
And then from there you can go back to step two.
And so this is a very iterative process.
You keep doing that.
Until at some. Point you cannot.
Add new variables.
No variables can enter or no old variables can exit.
And as soon as you get there, then you proceed to the finish because.
Your model is. Ready.
You can't you can't add anything. You can't take anything out.
That means you've, you've created the model.
So this. Is one of the more.
Tedious methods.
Of course, you would have to,
get a computer to do this for you, because otherwise you'd go go insane.
Putting variables in and taking them out.
But that's how bidirectional elimination works.
and once again, some people call it
stepwise regression and finally all possible models.
So this is.
The most probably thorough approach, but also the most, resource.
Consuming approach because you select a criterion of goodness of fit.
For instance, like chi key criterion can be the R squared.
lots of different criteria.
And then you construct all possible regression model.
So if you had. And variables then.
There will be a two.
To the power of n minus one total combinations.
Of these variables.
And that that's exactly how many models there can possibly be.
And then.
Step three you select the. One of these models.
With the best criterion. That you're looking at.
There you go.
Your model is ready.
So sounds easy, but let's have a look at an example.
Even if you have.
Ten columns in your data, you'll have 1023 models.
That's insane. That's an insane amount of models.
And we're not talking about columns that.
You've already filter. Out.
So columns that, you, you know, that's like in our example,
you might have 5 or 6 columns.
Now we're talking about when you get a data set that you.
You just it's pretty much raw.
And it has like maybe 100 columns like I've worked with
data sets were around that.
Maybe 50 to 1. Hundred, maybe more columns.
And instead of going through them, this is what this.
Method is suggesting.
Instead of going through them and picking out the ones that.
You think should be in your model, you just throw. Everything in.
Well, it's not a. Good approach because
basically it's.
it's growing exponentially.
The number of models is growing exponentially.
And, it's, It's very resource consuming to get.
A result from this approach.
And finally.
So where have we come to?
We've come to our conclusion.
We have five methods of building model models, all in backward elimination.
For selection.
Bidirectional elimination and score comparison.
So the one we're going to be looking at in these tutorials, in order to get.
Our head around.
How to build models step by step and get some practice, is the backward.
Elimination. Process because it is the.
Fastest one at all of them.
And you will still get to see exactly how the step by.
Step method works.
And plus, we'll.
Throw in a few extra tricks along the way to make sure.
Our models are very robust.
Can't wait to get started!
Lots of theory.
Let's get to the practice. I'll look forward to seeing you next time.