Cap analysis.
We've talked about Cap a lot, and in fact, we've talked about cap that much
that I'm no longer even saying cumulative accuracy profile
because I am assuming that you're entirely comfortable
with this abbreviation and the whole term and what it means.
So let's see how to analyze the cap.
As we've discussed,
there are three lines that are important on the cap curve.
The blue line which is the random line.
When you select your samples at random, the red line which is our model line.
The and different models will have different red lines,
but basically it looks something like that.
And the gray line which is the perfect model.
Or when you have a crystal ball, when you can select all of the future
Turner's or purchasers or whatever action takers,
and you can select them right away on the dot
without even selecting one single person that you don't want to select.
And so these are the three main lines.
And how do we analyze this cap curve.
We already know how to build it.
But what can we derive.
What insights can we derive from here.
Well it's kind of intuitive that the closer your red line is to the gray line,
the better your model, the closer is to the blue line, the worse.
So how can we quantify this effect?
Well, there is a standard approach to calculate the accuracy ratio.
And to calculate the accuracy ratio, you need to take the area under
the perfect model or the perfect line which is colored in gray here.
And it's called the a p. And
then you need to take the area under the red line
which is colored in red here, which is a R.
And then you need to divide one by the other.
So you need to divide a r by a p.
And then this ratio that you get is obviously between 0 and 1.
And the closer this ratio is to one the better.
The further it is away from one and closer to zero, the worse.
However, it can be quite complicated to calculate this area under the curve.
Statistical tools can do it for you, but how can you assess
the cap curve by just looking at it?
So visually, it's not that easy to get this quantifiable metric
just by looking at the curve.
So there's a second approach.
And that's what we're going to discuss now.
Let's get rid of the areas.
And instead of looking at the area, what you can do is look at the 50% line
on the horizontal axis and look where it crosses your model,
and then look at where
that line, the horizontal line from there crosses the vertical axis.
So basically how many turns will you pick up or action takers
or how many positive outcomes are you going to identify?
if you take 50% of your population
and in this case, we can see it's around 90% or something like that.
And just by looking at that, there's a
like a rule of thumb, how you can assess your model based on that X number.
And here it is. Are you ready? Here we go.
So if x is less than 60% the model is rubbish.
Basically it's not useful at all.
You ha you can create a better one.
Probably you can create a better one.
And you need to try again.
If, your model, your X is between 60% and 70%,
then the model is considered to be poor, poor or average.
And by the way, these are my this is my rule of thumb.
Other people might have a different rule of thumb, but this is what I go by.
If it's between 60% and 70%, it's it's a poor model, to be honest.
Like you can you can do better than that.
if it's if X is between 70% and 80%, that's a good model.
That's already where you should be aiming for anything above 70%.
That's, can deliver
good quality insights to the business and actually deliver value.
Anything between 80% and 90% like we see here is a very good it's extremely good.
That's if you can get a model over 80%.
That is an amazing result.
And anything above 90% up to 100, that is just too good.
It is too good to believe. And they are.
There's one option that you should be very careful here with is overfitting.
If your model is showing you results like 90% or so,
if a model showing 100%, then the obvious answer there is that
one of your independent variables is actually a post facto variable,
meaning that it shouldn't be in the data because it's looking into the future.
The person who supplied you that variable forgot to take it out
or forgot to explain to you that, you know, their credit score actually,
is turned into zero after they leave the bank,
and therefore everybody with a zero credit score
obviously has left the bank, and therefore your model is picking them up.
Like, like is super easy.
So if you have 100%, that's definitely something on a few variables.
Even if you have 9,000%, you have to check that there could be some
forward looking variables.
The other thing is overfitting.
You could be overfitting your model and what that means is that you,
your model has been so well fit to that
specific data set that you supplied it, that when you true
that it's just heavily relying on the anomalies in that data set.
And when you feed it a new data set, like, you know, in a month time or
something like not not training data, not the data that you train your model on.
And we'll talk about this a bit, a lot more actually in the coming tutorials.
But so if you feed this model some data that you want to actually predict on,
then it will crash. Well, it won't crash it.
It won't perform as well.
Perform, you know, at the 60% mark or something.
So that means your model is overfitted.
And be very careful about that.
We'll talk about overfitting more.
In fact, in the coming tutorials we will learn how to avoid that problem.
And finally, if you can get an an X or this,
parameter to be between 90%, 100% and you're
not using forward looking parameters or you're not overfitting,
then give me a call because I might have a job for you.
People like that are rare, and I have,
a lot of headhunters looking for people who can, do modeling like that.
So definitely keep that in mind and look forward to seeing you then.
Until next time, happy analyzing.