Now that we've discussed the final step on how our Naive Bayes Classifier makes decisions, we should talk 1
about where the probabilities that feed into this decision making actually come from. 2
And this means backing up a bit and talking about statistics. Now this is the part that no one gets excited 3
about. 4
I've yet to hear somebody say "Yes, statistics! 5
My favorite topic!", but the thing is, statistics is everything in machine learning and this project will 6
give us a chance to get some exposure. 7
All right. 8
So we're interested in calculating the probability that an email is spam. 9
However, it's not like we can simply work out the probability the same way we would work out the probability 10
of say flipping heads on a coin flip or rolling a six with a die. With coins and dice, working out the 11
probabilities is fairly straightforward. 12
But let's review some of these concepts nonetheless because they're gonna come in handy later. 13
Now with coins, we know that there's two sides and a flip has a 50/50 chance of showing heads. With this 14
dice here, 15
we know that there are six sides and we've got a 1 in 6 or roughly 17% chance of rolling 16
a six or any particular number. 17
Now let me ask you a question outside of the realm of flipping coins and rolling dice, 18
how would you work out your probability of getting hit by lightning? 19
Yeah, I know, you could probably ask Google and get the answer, but if you had to calculate it yourself, 20
how would you do it? Well, 21
the simplest way to do this is by dividing two numbers, the total number of times people get hit by lightning 22
and the total number of lightning strikes. 23
Now I trawled through Wikipedia for you and I've dug out these figures. About 240000 people are injured 24
by lightning every year. 25
Now, 240000 actually sounds like quite a lot of people but there are an order of magnitude more lightning 26
strikes. Every year around 350 million lightning bolts actually strike the ground. 27
So then, just given these two numbers, what's the chance of you being hurt by lightning? 28
Well, it'll be 240000 divided by 350 million or 0.07%. 29
The point I'm trying to get across here is how to use basic probability. 30
We took some observations, like the number of times a lightning struck a person and the total number 31
of times we observed lightning in a year 32
to calculate this figure. 33
Now, suppose we had to work out the chance of an email being spam. 34
Any email that is, right, any email in the whole world. We can apply the same technique as in the lightning 35
example. The chance of an email being spam should also depend on two things, namely one, how many spam 36
emails were sent; and two, how many emails were sent in total. With these two quantities in hand, 37
we can work it out. So I trawled the internet and here's what I pulled up. In 2017, 38
there were an estimated 148 billion spam emails sent. 39
That's right, billion. And the total number of emails being sent was approximately 269 40
billion. 41
So that means, if a new email comes into your inbox, the probability of that email being spam or having 42
been spam in 2017 was 55%. 43
And this is simply based on the observation of the frequencies, namely the total number of spam emails 44
divided by the total number of all email traffic. 45
So you can think of calculating the basic probability as step one. 46
We figured out the overall probability of spam, but we can't build a classifier with this alone. 47
So what's step two?