So once we have trained our model.
I told you that we can quantify the quality of fate of our model using the means squared error term,
which is given by this formula, is basically residual more squares divided by N.
If the predicted responses are very close to these observations, the means good at it will be small.
But if we find out, I mean, square error on these same data which we use to bring our model, it is
called training MASC.
Reigning in it is not something we are interested in.
We are interested in the accuracy of these predictions when we apply our method to previously unseen
best data.
For example, I'm predicting home price.
I don't really care how well our method predicts house price of previously completed transactions.
I get about Holwell.
Will it predict the whole place of the future transactions?
Similarly, if I want to predict the risk of a particular disease in different individuals.
I want to do it for future patients and not for the ones I already know the outcome for.
So what we're going to do is.
We are going to split our data into two parts.
One will be called training set.
This will be used to train the model.
And the other part will be called the test said.
This will be the unseen data and it will be used to assess the accuracy of our model.
So mathematically, I have these in pairs of observations, X one by one, X two way to do X and Y in
these will be part of my training data and I'll use them to train my model.
Once I have used them, I will have identified the functional form of play.
That is the ethics.
Now, I will use previously unseen set of observations exito ways to.
These observations will come from our guest said.
And I will try to find out the best error which is given by this formula, so best mean squared, it
is average of.
Squared of difference between the predicted value of life to the actual value of life on the given BASTIDA.
Two, four different types of models.
I will compare the value of this test error and then to LIGNE model with least tested.
I hope you understand the idea behind having best entering data.
Basically, we have training data and corresponding training at edit body models selected.
But there is no guarantee that the matter with Lewis reading it, it will also have law tested at.
Roughly speaking, many statistical methods specifically estimate redoes does so that we are able to
minimize prunings, Ed.
And for such methods, all the training so that there will be small, but the actual test set, it can
be quite large.
In this graph, you can see four different lamed.
This black one.
Is the true function.
Mitt, we want to predict.
This orange line is the output of a linear regression model.
And these blue and green lines are the result of some other more flexible models.
And these small circle that we are seeing are the data points which were used to train the model.
You can see as I am increasing the flexibility of the model, that is, I'm allowing it to change its
shape or its direction.
Many times it is touching more points on this graph.
So this green card, which has high flexibility, is putting the maximum number of point, whereas the
orange girl, which has least flexibility, is touching very few points.
You can see after a certain level of flexibility.
This flexibility is making the code more bigly.
That is it is following the individual data points and not the overall function.
To the effect of flexibility on training at it and tested, it can be seen on the graph, on debate.
You can see that this great plot is off Gradinger as you keep on increasing the flexibility, the training,
it keeps coming down.
That is the model will be fitting or will be getting a lot of these sample points.
But after a certain point, the test it which is given by this red, got it start increasing with the
increasing flexibility.
You can see this Orange Point is the best entrain ever for this orange glow, which is District Lanco.
This blue point is for the blue go, which is approximating the blue function very closely.
And this green point is for the green go, which is very flexible.
It has low training other than the bluegill.
Is putting these points more closely, but it has hired Estero because it is not approximating the true
function.
So we want to identify this blue point.
When we get the minimum tested it out, there are several techniques to split the data into training
and test so that we can find this minimum point.
So we are going to discuss the three most popular techniques.
First is called Vegetation's and Approach.
Second is Leverne out cross-validation.
And the third one is gatefold cross-validation.
The first technique, which is validation set approach, is the simplest approach.
In this matter, we will randomly divide the data into two parts, a training set and a tested.
The model will be fitted on the training set and once the model is trained.
The Air Force did this, it will be calculated.
Estimate detested.
We usually do a split of 80, 20.
That is, we use 80 percent of the data for training purposes.
And 20 percent of the data for testing purposes.
We'll be running this approach in our software package in a separate video.
There are basically two limitations of this approach.
One is that part of the available data will not be used for training.
And as we know, the more data we use during training, better will be the performance of the model.
So if we keep some data for testing, the green model will not be as good.
And if you have a limited number of observations, your training will be severely impacted.
Secondly, the best it can be, highly variable, depending on which observation is selected for training
and which observation is selected for testing.
So to handle these two issues, there are these two alternative approaches.
In the Leverne out cross-validation.
Suppose we have an observations.
We'll keep the first observation.
For testing purposes and random order on delimiting and minus one observations, then we will keep the
second observation for testing purposes.
And run the model on the remaining and minus one observations.
And we will run this model and things that every time we'll keep one observation or testing and the
other and minus one for training.
And will take the average of the error on each of these testing of the missions.
So since we'll need to run this model and times, this method can be computationally expensive.
So an alternative to this Libano cross-validation is the K4 cross-validation.
And this we were divided into two kasit.
And then we're trained to do it on game minus one six.
And Newsday gets it for testing purposes.
You can see that.
Leave a note cross-validation is a special case of Caple validation.
If you have.
Gays equal to N then give or validation and leave a note cross-validation are the same thing.
So we will not be covering these two techniques in this software package.
We will only be running the validations at abroad and our software package.