So far we've talked about basic probability, independence, joint probability and conditional probability. 1
We're now ready to start talking about Bayes theorem. 2
Bayes theorem is where the Naive Bayes classifier gets its name. 3
You see, a long, long time ago, in the 17th hundreds in Kent, England, there lived a clergyman by the name 4
of Reverend Thomas Bayes. Thomas Bayes got very, very interested in probability 5
later on in his life and he came up with a neat little trick. Let's go back to our weather example. With 6
the conditional probability formula, 7
the hardest part was this joint probability term, namely what is the probability of it raining and being 8
cloudy. 9
Now it turns out, one can phrase the conditional probability formula in another way. 10
Instead of having this joint probability in there, we can have another conditional probability, namely, given 11
that it is raining, 12
what is the probability of it being cloudy. Now this is different from given that it is cloudy, 13
what is the probability of rain. 14
These two are not the same. 15
Have you seen rain on a sunny day? 16
Yes, but not very often, right? 17
The probability of clouds when it is raining is probably around 95%. 18
On the other hand, the probability of it raining, 19
given that it is cloudy is probably more around 40%. 20
So these two numbers are as different as the conditions. 21
But the point I'm trying to make here is that changing the top part of this fraction in this formula 22
is all Bayes theorem is. 23
Reverend Bayes basically figured out how to reverse a conditional probability in the formula to make 24
this whole thing easier to calculate. The conditional probability. 25
looks like this. 26
Here's the generic formula for Bayes Theorem in terms of A and B and you'll find this theorem very early 27
on in any statistics textbook. But let's go back to our spam example. 28
Given that the email has the word "Viagra" in it, what is the probability of this email being spam? Using 29
Bayes theorem 30
we can now calculate the same probability like this. Using this formula, 31
the calculation becomes very, very easy and that's why I'm harping on about this so much. 32
Let's tackle one term at a time starting with the probability of spam. 33
We've actually already discussed this. 34
We figured out how common spam emails were. 35
This was 55% in 2017 36
if you remember. So let's plug that number in here - 37
0.55. 38
Now, what about the probability of the word "Viagra" coming up in an email? 39
To calculate this number, we have to figure out how common the word Viagra is 40
in emails as a whole. 41
How often do people use the word "Viagra" in an email? 42
Both spam and in normal e-mails. Say we've looked at an enormous dataset of emails containing 700000 43
words across thousands of e-mails and we count how many times "Viagra" gets mentioned. If the word "Viagra" 44
gets mentioned 75 times, then the probability of "Viagra" coming up in an email is 75 divided 45
by 700000. 46
In this case, we're just applying basic probability, just like with our calculation of the probability 47
of getting hit by lightning. We take the total number of times "Viagra" gets mentioned and we divid it 48
by the total number of words that we looked at in our dataset. 49
Next let's tackle the probability of "Viagra" being in an email 50
given that the email is spam. What does this mean? 51
This is a conditional probability. Given that the email is spam, 52
what is the probability that the email contains the word "Viagra"? 53
How would we calculate this? 54
Again we look at the frequency of the word "Viagra", 55
but this time just within the spam emails. This allows us to figure out what is the probability that 56
an email contains the word "Viagra" 57
given that it is spam. And looking at our dataset of emails, we count the number of times the word "Viagra" 58
occurs and we look at the total number of words in our dataset that are in spam emails, say 59
370000. 60
If we find that the word "Viagra" appears 65 times, the conditional probability is simply 61
65 divided by 370000. And 62
that's it. 63
We've got all our numbers. 64
We can work out the probability that an email is spam given that it contains the word "Viagra". 65
And based on our data here, the probability is around 90%. 66
What this example shows is that the frequencies of a word really are key. By looking at the frequencies 67
of a word in a spam messages vs. all messages, 68
our algorithm learns which words are spammy. 69
The reason we think words like "Online Pharmacy", "Double your cash", etc. are spammy is because these words 70
often appear in spam emails. 71
But the story doesn't stop there. 72
If we look at an email message, then we're going to have more than one word in the email body, right? 73
Well that depends on your friends, actually. 74
But my point is that we can calculate the conditional probability for every single word in the emails 75
in the entire dataset. 76
Not only will we have worked out the conditional probability of an email being spam if it contains the 77
word "Viagra" but we will have worked out the probabilities of all the other words as well, like "Expert" 78
"Free", "Cash" and so on. 79
So now when an email comes in, what if it has both the words "Free" and the word "Viagra" in it? 80
What's the probability of an email being spam in that case? Let me rephrase that. 81
Given that the email has the word "Free" and the word "Viagra" in it, what is the probability that the email 82
is spam? At this point we've come full circle all the way back to joint probability. 83
Remember how with the coin flipping example, we simply multiplied the probability of heads 84
times the probability of heads to figure out the chances of getting two heads in a row? 85
The reason we could do that was because the two events were independent. 86
And this brings us to the Naive part of the Naive Bayes classifier. 87
The reason our algorithm is naive is because it assumes independence between the words in the email. 88
In other words, if our email has both the words "Viagra" and the word "Free" in it, we can multiply the two probabilities 89
together and we can do this for every single word in the email. 90
The more and more spammy the words are, the higher the final number. 91
Now let's continue with this line of thinking. 92
If an email has lots and lots of spammy words in it, it's probably spam. 93
And if an email has very, very few spammy words in it, it's probably a normal email. 94
So this brings us back to the final decision where we compare two probabilities. 95
Say we have an email with the words "Hello friend, 96
want free viagra?". 97
Well using Bayes rule we can calculate the conditional probabilities for all these words and figure 98
out what the probability is that that email is spam. 99
Then using that independence assumption, we multiply all these probabilities together to get a final 100
number, namely the probability that the email is spam. And then we can compare this number to the probability 101
that this email is a normal email. 102
So for that first term, we would have: Given that this email contains the word "Hello", 103
what is the probability that the email is a normal email? Next, 104
given that this word contains the word "Want", what is the probability that we have a normal email? 105
And so on. 106
We then multiply all these probabilities together and then we can do our comparison. 107
This is the classification step that we talked about in the very, very beginning. 108
We will classify this email simply based on which final number is higher. 109
What I've just described to you is called the Bag of Words approach for classifying documents. Each word 110
is looked in isolation and the frequency of each word becomes a feature in our machine learning model 111
and the fact that we've looked at each word in isolation is why this model is called Naive. 112
We're treating each word separately. 113
We're ignoring grammar. 114
We're ignoring sarcasm. 115
We treat the city name New York as two separate words, 116
New and York. We're treating the phrase Not Bad as two words Not and Bad. 117
The context is lost with the Bag of Words approach. The dependencies between the words are ignored. 118
We are assuming that the words are independent, like well, words in a bag. At this point you're probably 119
skeptical. 120
Will this really work? 121
This whole approach seems super strange and the assumptions also seem really crude. 122
Well I don't want to spoil it for you. 123
You will find out just how well this works 124
in the coming lessons. And now that we've covered the theory behind the Naive Bayes model, our path for 125
the Python code is clear. 126
The first step will involve extracting all the text from the email bodies and this means moving on to 127
the next parts of our project workflow - 128
namely cleaning and exploring the data. 129
For example, we will need to figure out just how common each word is 130
in an email. We will need to do this for every single word, not just spammy words like Viagra but every word. 131
How often do people use the word Japan in emails? 132
How often do people use the word weekend, apple, Android, computer, lawyer or friend in emails? 133
We will need to count how often these words appear in all the emails in our dataset and we have over 134
5000 different emails in this dataset that we're working with and we will store all this text inside 135
a pandas dataframe in order to manipulate it and we're gonna do all that in the upcoming programming 136
lessons. 137
I'll see you there. 138
You know, I vividly remember learning about probability in school and all my textbook exercises were 139
constantly about urns, always, balls and urns, red balls from an urn, picking out a sequence of balls from 140
an urn, picking out a particular ball out of an urn. 141
Oh man, the memories.