So once we have trained our model.
I told you that we can quantify the performance of our model using the confusion matrix.
We can see how many times we have got it right and how many times it is wrong.
But when we are growing the confusion matrix on the same data which we use to green our model, it is
called training.
It is.
Training in it is not something we are interested in.
We are interested in the accuracy of the predictions when we apply our method to a previously unseen
test data.
For example, when I'm predicting whether the House will be sold within three months or not, I don't
really care how well the method predicts whether the house will be sold or not on the previously completed
transactions, which my system I see.
I want to know how well it will predict whether it will be sold on the future transactions.
Similarly, if I want to predict the risk of a particular disease in different individuals, I want
to do it for future patients and not for the ones I already know the outcome for.
So to handle this issue, what you're going to do is you're going to split our data into two parts.
One part will be called the training set.
And the other part will be called dataset trainings that will be used to train the model and the test
will be used to test its performance.
So that test, it will be the unseen data and it will be used to assess the accuracy of our model.
So mathematically, I have these pairs of X and ways x1 y by one, X two.
I do.
And so on.
So these and build of X and ways will be my training set.
I will use them to train my model.
And once my model is trained, I have this functional form of relationship between X and Y.
Now I will use this previously unseen set of data.
And I'll feed these observations into this model to predict the value of life.
This predicted value of Y and the actual value of Y available for this test set.
That is these Y zeros.
These will be compared to create the confusion matrix and this confusion matrix will be used to assess
the accuracy of our model.
So when we have three different types of classifiers, that is logistic regression, linear discriminant,
analysis and gain nearest neighbors, we will draw the confusion matrix for the test set for all these
three classifiers and then compare their performance on this previously unseen data instead of the training
data.
The main reason why we have to separate the data and to test it and the training set is because there
is no guarantee that if a model is giving low training error, it will also have low test at it.
Roughly speaking, many statistical methods specifically estimate why values so that we are able to
minimize deep training error.
So far, such methods training it and may be very low.
But that test data will be quite large.
And this graph, you can see that the true function of this dataset is discovered, Lane.
But if I'm using a less flexible method, such as a straight line to estimate the values, it will have
a lot of error.
But on the other hand, if I increase the flexibility to too much, my line will exactly fall.
Each and every point instead of following the general trend.
So having too much flexibility makes the model overfit.
The data, which also results in increasing error.
So in this situation, if you notice the training, it it will be very low.
As each point is captured by this code.
So this particular model will give a very low training error.
But in fact, if you use this model on any unseen data, this model will be giving a very high error
rate, probably even more than the straight line.
So we need to find the balance when selecting the flexibility of our model.
And hence, we should compare our different models.
This is their performance on unseen data instead of previously seen data or the training data.
This graph tells us how the error rate changes along with flexibility.
If I keep on increasing the flexibility of a of my model, the training error which is given by this
light blue line, it keeps on decreasing continuously.
So we may be tempted to use a very flexible method so as to get low training error.
But in reality, if I present team data, this model will perform worse than this model.
So we need to see the test error rate, which first decreases and then increases as we increase the
flexibility.
And we have to identify this point with the test error rate as minimum.
Now, there are several techniques to split the data into grading and test it so that we can find this
minimum point.
We are going to discuss the three most popular techniques here.
First one is called validation set approach.
Second is leave a note cross validation and third one is gatefold cross-validation.
The first technique, which is validation.
That approach is the simplest approach.
We will randomly divide the data into two parts.
A training set and a test set model will be fitted on the training set.
And once the model is train, the error for test will be calculated on the test.
We usually split the relevant data in a ratio of eight is to 20.
That is, 80 percent of the data will be used for training purpose and 20 percent will be used for testing
bookless.
There are basically two limitations of this approach.
One is that part of the available data will not be used for training.
And as we know, the more data we use during training, better will be the performance or model.
So if we keep some data for testing, the train model will not be as good as we can get.
And if you have a limited number of observations, that is not a lot of observations.
Your training will be severely impacted.
Secondly, the testator can be highly variable, depending on which observation is selected for printing
and which observation is selected for testing.
So to handle these two issues, there are these two alternative approaches.
And the leverne out cross-validation.
We will keep the first observation and Randy Madelon remaining and minus one observations in the next
round.
We will keep the second observation for testing both of us and run the model on the remaining and minus
one observations again and each cycle.
We will use that one observation for testing, and the error calculated in each cycle will be averaged
to establish the test error in this method.
So since we need to run the model several times, this method can be computationally expensive and automated
to just leave one out.
Cross validation is gave for cross validation.
In this, we will divide the data into Kasit.
We will keep one set for testing and use the other key minus one set for training.
You can see that leave one out cross-validation is a special case of careful cross-validation.
If you have gays equal to N, that is gay is equal to the total number of observations.
This is exactly same as did leave a note cross-validation.
We will not be covering these two techniques in the software package.
We'll be only using the validation that approach.