In this lesson we're going to look more closely at the predictions our model got right and the predictions
our model got wrong.
Now previously we've looked at accuracy as one metric for evaluating how well our model is doing.
The accuracy modelled how many predictions were correct.
Out of all the predictions.
Now here's a question for you.
Can you think of a serious shortcoming of using this metric.
Why might we not want to select our models purely based on accuracy.
Well let me ask you this imagine you were hired by the British National Health Service to build a model
that detects cancer.
Now suppose you build a model that classifies every single patient as not having cancer.
Each new data point is just labeled as being cancer free how accurate with this model.
Well the entire population of the UK is around 65 and a half million and there are around two and a
half million people living with cancer.
So if we do the math then this model is actually 96 percent accurate and ninety six is a very high number.
So it seems like it's a very accurate model.
But if we're getting 96 percent accuracy for a model that does nothing whatsoever then there's probably
something amiss right now as you can tell from these bond charts the underlying reason is is that in
this example one category greatly outnumbers the other category and you'd be surprised.
This is actually a very very common problem in data science and machine learning.
So clearly we need to move beyond the accuracy metric and look at some other metrics as well.
Enter the concept of false positives and false negatives.
My favorite way to think about false positives and false negatives is actually through one of Aesop's
Fables.
It's the story of The Boy Who Cried Wolf.
Now I was told a story a long long time ago when I was a child and I was still shiny and new.
So I'll have to do my best to paraphrase.
But once upon a time there was a shepherd boy who liked to trick his fellow villagers every once in
a while he would run to the village and cry that a wolf had attacked his sheep alas.
When the villagers came to investigate.
No Wolf was found and this is what a false positive is.
The villagers did not like false positives and they were very very angry with the boy.
And having played this trick a few more times on the unsuspecting villagers one day our shepherd boy
actually sees a wolf.
He runs to the villagers and shouts Oi.
There is a wolf eating all the sheep.
However the villagers think it's another false positive and that there is no Wolf.
Little do they know that this time round there actually is a wolf.
And the boy was telling the truth.
And what they've got in this case is actually a true positive.
But as a consequence of not grabbing their pitchforks though Wolf eight all the sheep and the villagers
go hungry.
No more roast lamb for the villagers.
Only potatoes.
And this is where the story ends.
But we have two more cases to think about actually for the third case let's imagine a version of the
story where the boy rocks up to the village and truthfully announces everyday that there is no wealth
no wealth today.
Dear villagers all as well.
Now since there is no Wolf and the boy said that there was no Wolf.
This is called a true negative and that leaves us with one more case in this case.
The boy goes to the village and announces that there is no Wolf.
But this is in fact incorrect the case where the boy says there is no Wolf but there actually is a wolf.
It's called the false negative so what are the false positives and false negatives in our spam classification
context.
Well for the false positive are spam classifier would predict that an email is spam but it's actually
a legitimate email.
False positives are the reason why you have to go into your spam folder and occasionally fish out the
non spam emails.
And the false negatives.
Well a false negative is when the spammer manages to get around the spam filter and land in your inbox.
In other words the email is actually spam but it was incorrectly classified by the spam filter now with
these tools in hand.
We can look at some other metrics for our spam classifier to determine whether it's any good.
So let's head on back into the Jupiter note.
So let me add a little markdown cell here.
That reads false positives and false negatives.
In the next couple of cells we're going to be calculating our true positives are false positives and
our false negatives.
So let's check how often we actually predicted non spam and how many times we've predicted spam gunplay
actually has a very handy method called Unique and this takes two inputs one will be our prediction
vector and the other one well just the return counts equals true.
And here we can see that we've predicted non spam one thousand one hundred sixty three times and we've
predicted spam five hundred and sixty times.
The next thing I'll do is I'll create a number higher rate of the true positives.
So I want to make a comparison between each element in the y underscore test number higher rate and
each element in the prediction no higher rate.
And I want to store this result in a variable called true on a score.
P O S now to check whether an email is spam and r y underscore test.
We can say why does good test double equals 1.
So this will check for each element in y underscore test.
If it's equal to 1 and to check whether our predictions are equal to 1 we can do it like so right prediction
W equals 1 to make a comparison between these two conditions right.
Why underscore a test double equals 1 and prediction tablet equals 1.
What we've previously done is we've used the double Ampersand right.
We've previously made comparisons with this logical and this double ampersand is a boolean operator
but in this case we can't use it.
We won't get the results that we want.
If we use it like so if we want to make an element by element comparison we use a single ampersand.
So in this case we've got a bit wise and operator not the boolean and operator.
And this will allow us to make an element by element comparison so if I execute the cell and then come
down here and sum up my results then I can see that I've got five hundred and forty eight true positives.
In case you're wondering true underscore pause looks like this is just an umpire rate of true and false
values.
So as a challenge can you create a number higher rate that measures the false positives for each data
point call this variable false underscore pulse and then work out how many false positives that were.
And after you've done that do the same for the false negatives store those in a variable called false
underscore neck.
I'll give you a few seconds to pause the video before I show you the solution
all right.
Ready.
I want to store the false positives in a number higher rate as well.
I'm going to call it false underscore Paul's and that's going to be equal to it's gonna be equal to
well where our prediction was spam but actually we had a non spam message.
So why on a school test is equal to non spam zero but we're comparing that with where our prediction
is equal to one where our prediction was spam.
Those are false positives and in total we have about 12 of them.
So in 12 cases are naive based model thought that an email was spam when it really was just a normal
email.
What about the false negatives though false on the score.
Neg is equal to Y and a school test double equals 1.
So in this case an email was actually spam and we're going to compare that with prediction W equals
zero.
So in this case our prediction was that it is a non spam email here.
Our spam message inbox and we missed it with our spam filter so false on a school neg Dot.
Some will show us how many emails actually made it into our inbox and out of all of our test emails
40 spam messages made it into the inbox.
And you know what.
We can actually see these values very very clearly on our chart with the decision boundary.
Let me show you.
If we scroll up and we look here every time we've got one of the Red Crosses below the decision boundary
in this area here where all the non spam messages are we've misclassified this email in the next lesson
we're going to be looking at three other metrics that complement our use of the accuracy metric to help
us evaluate how good our model is looking forward to seeing you in the next lesson.
Take care.