In the previous tutorial we talked about the accuracy paradox.
Hopefully now you see why we need more robust methods to assess our models.
And today we talking
about the cumulative accuracy profile, which is in fact one of those methods.
Let's look at a scenario.
Let's say you're a data scientist at a store which sells clothes,
and your store has a total of 100,000 customers.
I'm placing that number on the horizontal axis
and you know that
from experience, whenever you send an offer like an email
to all your customers or to any random sample of your customers,
approximately 10% of them respond and purchase the product.
So I'm going to place 10,000, which is 10% of the total.
on the vertical axis.
And so what we're going to do is we're we've got an offer that we want to send,
and we want, to see how many customers are going to, purchase our product.
We send it off.
So if we send it to zero, customers obviously will get, zero responses.
Right.
What do you think will happen if we send it to 20,000 customers?
How many do you think will respond?
Well, because this is a random sample and we know that about 10% respond.
So we would say about 2000 would respond.
Fair enough. Right.
If 40,000 if we send to the offer to 40,000 of our customers,
then about 4000 will respond.
60,000 6000 80,000 8000 100,000.
Then 10,000 of our customers should respond.
And this, is a random selection process.
So here we can draw a line which will actually represent,
this, random selection.
The slope of the line equals to that 10% that, we know that respond
on average to our offers if we just send them out like that.
Now the question is, can we somehow improve this experience?
Can we, get more customers to respond to offers?
when we send out, our letter?
So basically, can we somehow target our customers more appropriately?
So to get a better response rate.
And how about instead of sending out these offers
randomly to, say, a random sample of 20,000 customers?
How about we pick and choose the customers we send these offers to?
And how do we pick and choose?
Well, to start off with,
let's build a model just like we did in the previous section.
Basically, a customer segmentation model.
Joe, demographic segmentation model,
but which want to predict whether or not they will leave the company.
It will actually predict whether or not they will purchase a product.
It's a very simple process actually.
In fact, it's the same thing because purchase is also a binary variable.
Yes or no.
And we can also run the same experiment.
We can take a group of customers before we send out the offer,
and then look back and see who purchased with a male or female.
Which country were they in?
what age predominantly were they?
Were they browsing on mobile or were they browsing, via computer?
And all of these factors, we can take them into account,
measure them, put them into a logistic regression
and get a model which will help us assess the likelihood
of certain types of customers purchasing based on their characteristics,
so they change demographic status and and other characteristics.
And once we've built this model, how about
we apply it to select the customers we will send the offer to.
So what the model will tell us like
just like in the example in the previous section where females
of female customers of a bank whose favorite color is red,
they're most likely to leave the bank here.
We'll have a similar result.
It'll say, perhaps male customers in this certain age group,
who browse on mobile, are most likely to purchase the product.
It will tell us something or will actually rank our customers.
It'll give them a probability of purchasing our product.
And then we can use that probability to actually contact our customers.
So of course if we contact zero, customers will get zero response rate.
But if we contact 20,000, we'll probably get a much higher response
rate than just 2000, because we're picking out the customers
that are at the highest risk of accepting this offer.
We know from their previous behavior or from the previous behavior of customers
similar to them, that they have a 90% chance
or an 80% chance of purchasing this product.
And we will go for them first.
We will put them at the top of our list of people who we contact.
Then when we contact, let's say we contact not 20,000 but 40,000.
Our response rate will be higher than 4000,
which we get in the random scenario.
If we if our model is really good, then by the time we're around at around 60,000.
So more just over half of our total customer base,
we are already getting to that 10,000 mark.
So we know that 10,000 people will respond in total.
There's no way we can get above that because that's just, the response rate.
If we contact everybody, it'll be 10,000.
But we're getting very close already.
So even at 60,000, we're already at 9500 respondents or purchases.
We we could actually stop here.
We've already pretty much contacted everyone, but if we want to contact more,
if we send it out to 80,000, we're getting even closer to 10,000 responses.
And if we contact 100,000, we will still be back at our 10,000 responses.
So now let's draw a line through these,
crosses.
So what you see, this line here
is called the cumulative accuracy profile of your model.
And as you can imagine, the better your model, the
larger will be the the area under this line.
So the area between the red
and the blue lines, it will increase as your model gets better.
And if your model is worse, then this red line
will be closer to the blue line, so it'll be closer to random.
The next step we want to do is convert these axes
from absolute values to percentages.
So so they range from 0 to 100%.
And this is how the cap curve is normally represented.
Now let's say we ran another regression model.
And this time we use less variables lesson dependent variables.
Or just because we had less access to independent variables or we didn't see
that there's a multicollinearity effect in our model or something else
that went wrong and that model, because it'll be worse.
This is what its cap curve will look like.
And therefore, by plotting the cap curves, you'll be able to compare models
to each other and understand how much gain.
This is also sometimes called the gain chart,
how much gain you get in each of these models
compared to the random scenario, or how much gain you get.
Additional gain you get
from switching from one model to the next, or from the green one to the red one.
For instance.
You're improving your hit ratio
and therefore you're improving your return on investment.
So therefore the red model is better.
And this is how we are going to be assessing models.
So let's label them.
The blue line is a random selection process.
Like a monkey could do that.
You just pick a random sample
and you send the letter or you just send it to everybody.
You get your 100% of respondents.
The green line is a poor model.
So it's it's a model
is better than random, but it's still not as good as the red one.
The red one is a good model.
As you can see here, at around the 50% mark,
we're getting just over 80% responses.
That's considered a good model.
And there's one more line that you can think of here.
And it's this line.
This line is the ideal line. And
this is what would happen
if you had a crystal ball, if you could predict
exactly who is going to purchase and contact those people,
this is what it would look like.
Why? Well, because if you look at that,
the place where that split occurs,
you will see that it's exactly 10% and 10%.
As you remember, we know that only 10% of our customers ever purchase.
So basically you're saying that on the horizontal axis, I'm going to take 10%.
And each and every single one of those customers
I pick in that 10%, they are going to be those that purchase.
That means I will go right straight to 100%.
with, this last scenario, this actually took me a while to get my head around,
when I first heard about it, because I never understood.
Why is this split at the top?
Why does it break like that?
But that's exactly the reason.
Because you you you might you can imagine that
you have a crystal ball and you contact in the first 10%
or however many in your specific,
business scenario, customers ever purchase.
You contact them right away,
and then it's just flat from there, because it doesn't matter
how many more you contact, they're not going to purchase.
That's just the reality of things.
And that is the curves that you can have on a cap curve.
If you ever see a model that goes under the blue line,
I didn't even draw one here.
But if that happens, that's a very bad
model is basically doing you a disservice
if it's if you see the curve on
the blue line and we'll talk about model deterioration further
in the course when you're talking about maintaining your models.
So that's it for the cap curve for the introduction to cap curve
will be using the cap curve very actively in this section to assess our model.
And in fact we'll actually build
two of them and one for our model and one for our test data.
So that would be very interesting to compare.
One last thing I wanted to mention is and note
that we have the cap which is a cumulative accuracy profile.
And we have a rock which is a receiver operating characteristic.
And a lot of people get these things confused.
And I myself included, I used to, get it confused.
I even tried proving, one time a, a colleague of mine
who knew this stuff really well at the time,
and I was just learning it, that, he was wrong.
But that was a funny experience.
But they're not the same thing.
So cumulative accuracy profiles.
And when we talked about receiver
operating characteristic, we won't be covering, in this course.
It'll be in my advanced course on statistics.
it's very similar. It looks similar.
And that's why a lot of people get confused and actually
I think,
the other reason is that the ROC curve is in Wikipedia,
there's an article for the ROC curve,
but there isn't one in English for the cumulative accuracy profile.
So it's quite hard to, find information on the cap curve.
just, just by searching in Google.
So, maybe you'll be the first person to write
a Wikipedia article on the cap curve.
Who knows? maybe.
Anyway,
I look forward to seeing you in next tutorial
and where we will be working with the cap curve.
And until then, happy analyzing.