Okay, so step one was calculating the basic probabilities. It was gathering information on the frequencies 1
of an event and figuring out the probability of that event happening 2
based on that. In step two we're going to talk about a concept called the joint probability. 3
Say you flip a coin twice. What are the chances that you get heads both times? 4
What's the probability that you get heads two times in a row? 5
The way you see this written in mathematical syntax is with this little upside down "u" symbol. This 6
symbol stands for 7
"and". Also every textbook out there will refer to an event like the result of a coin flip with a letter. 8
So you'll see this written as "what's the probability of A and B?". A is getting heads on the first flip 9
and B is getting heads on the second flip 10
in our case. If probability is a new topic for you then maybe just quickly pause the video and figure 11
out what the probability is of getting heads two times in a row. 12
Did you have a go? 13
If so, how did you figure this out? 14
What was your approach? 15
One way that you can solve this kind of problem is to draw a matrix. 16
This matrix has all the possible combinations in it and there you can see that there's a total of four 17
possible outcomes with the coin flipping problem. From our little chart here 18
we can see that both coins showing heads only happens 1 in 4 times so therefore the probability of getting 19
heads on two flips in a row is 25 percent. 20
But, you know what? 21
There's a better way to calculate a joint probability. 22
All we have to do in our coin flipping example is to multiply the probability of getting heads times 23
the probability of getting heads. Since we have a 50/50 chance of getting heads, it's 0.5* 24
0.5 which is 0.25 or 25%. 25
In other words we're multiplying the probability of A times the probability of B 26
and that's how we can get the probability of A and B. 27
Now I think this formula is really, really handy when you don't want to draw a huge matrix, because say 28
you want to calculate the probability of getting heads on a coin flip 4 times in a row. 29
Well simply by applying this formula, you know it's 0.5*0.5*0.5* 30
0.5, which is equal to about 6.25%. 31
So now let's use this technique with a dice. Pause the video and as a challenge work out the probability 32
of rolling a 6 on a dice like this three times in a row. 33
Did you have a go? With a six sided die 34
you essentially have a 1 in 6 chance to roll any particular number. 35
So rolling a 6 three times in a row has a joint probability of 1/6 * 1/6 * 1/6 and that's equal 36
to approximately 0.46%. 37
So less than one half of a percent. 38
Very, very unlikely. 39
Now I know these were some very, very simple examples just now but I hope you can see how rolling more 40
and more sixes in a row gets less and less likely and the same with flipping coins, right. 41
It would be strange to see somebody get heads after heads after heads after heads on a fair coin that 42
is. Now I think personally 43
this is like the very friendly and simple side of probability but it's still interesting, because let 44
me ask you a question. 45
Imagine you and I are hanging out on a Friday night chilling, flipping coins and talking about probability 46
as we do. 47
And I've just flipped my coin twice. Both times 48
I got heads. 49
What's the probability that I get 50
heads on my third flip as well? 51
Is it 1/2 * 1/2 * 1/2 or 0.125? 52
No no no no no. 53
It is not. 54
Most definitely not. 55
And the reason is is that the past coin flips don't affect the next coin flip. Each coin flip is independent. 56
The probability of me getting heads on that third flip is still 50%. 57
Now this might seem obvious to you, but intuitively I've seen a lot of people just sort of accept this 58
idea of independence. 59
Intelligent people too. 60
It's like, you know, people don't want to accept this. 61
And if you don't believe me you can actually observe this behavior for yourself. 62
All you need to do is go to a roulette table in a crowded casino and just wait and watch, just watch 63
people. 64
Now if you've never played roulette or you're not familiar with this game, in the casino people spin 65
this wheel and people win when they've predicted where the ball will land. People can bet on a number 66
of things but they can also bet on the colors, right, the color that the ball will land on. 67
So in this game a lot of people love betting on a particular color. 68
Right. 69
Red or black. 70
Will the ball end up on the red or on the black, for example. 71
Now when the ball has been landing on black for a few times in a row you will see that people increasingly 72
start betting on red because they think it's the more likely outcome for the next spin. 73
Intuitively many people think: It's been black four times in a row, on the next spin the ball just has 74
to land on red. 75
Right. 76
But the truth is: Red is not more likely. 77
Why? 78
Because each spin of the roulette wheel is independent. 79
The wheel itself doesn't have any memory of the previous outcomes. 80
OK. 81
So that's a real world example of this concept of independence in action. 82
And we've actually covered quite a few important concepts in probability so far 83
already. We've covered basic probability, which was very much based on observing the frequency of a particular 84
event. We've covered this idea of independence where two events have nothing to do with one another. 85
And finally we've also covered this idea of joint probability, and what we saw was that for independent 86
events like rolls of dice and coin tosses, the formula for both A and B happening was simply by multiplying 87
the probability of A times the probability of B. 88
Now, let's take this discussion back to spam classification. In order to work out the probability of an 89
email being spam we should look at something other than the frequency of spam emails out there in the 90
wild. 91
Right? 92
We should look at the contents of each individual email. 93
This means looking at the message itself which is, well, where it gets interesting. 94
Now if you fire up Hotmail or GMail right now and you look through your email spam folder, what kind 95
of words do you tend to see in the message bodies for these e-mails? 96
I bet you can kind of spot a pattern. 97
You can spot certain themes coming up over and over again in your spam emails. 98
Certain words are just much more likely to come up in spam email than in regular legitimate email. 99
I'm thinking of words like "free", "access", "viagra", "loan", "online pharmacy", "adult", "great offer", "winner", "casino" "Bitcoin", 100
what have you. Like when was the last time a friend of yours send you a legitimate email about an online 101
pharmacy or free bitcoins for example? 102
So clearly there's going to be a clue in the message body whether an email is going to be a spam email 103
or not, just purely based on what the email is about. 104
So imagine a new email arrives in your inbox. 105
The e-mail body contains the word "Viagra". 106
Should this e-mail be classified as spam? 107
Let me rephrase that question - given that this e-mail contains the word "Viagra", 108
what's the probability of this email being spam? This question that I just posed 109
is all about conditional probability. 110
And this is where things are going to start getting pretty wild. 111
Conditional Probability is in my personal opinion both something unintuitive and difficult and we're 112
about to get our hands dirty with some of these meaty statistics. 113
Oh yes. 114
You see this is really, really worthwhile doing actually. Many of the basic statistical concepts like 115
probability independence, joint probability and conditional probability are used everywhere in machine 116
learning. 117
Even if you don't see it right away. 118
Even our box standard linear regression models come straight out of a statistics textbook. 119
The nice thing about the Naive Bayes classifier is that we don't use statistics and probability under 120
the hood, we're gonna use probability explicitly. Our Naive Bayes algorithm uses statistics in its raw 121
and pure format to classify email spam. And conditional probability is at the very heart of it. Conditional 122
probability measures the probability of some event given that another event has occurred. 123
The mathematical syntax looks like this. 124
What we have here reads the "Probability of A given B". For example - what's the probability of me getting 125
heads on my next coin flip given that my last coin flip was heads? Well, we just talked about this, the first 126
coin flip doesn't affect the second coin flip, 127
so the two events are conditionally independent. 128
The probability of me getting heads on that next flip is still 50%. 129
But, but, but, what if the two events are in fact dependent? 130
Going back to our email example, what's the probability of an email being spam given that it contains 131
the word "Viagra"? In this case the probability of the email being spam actually depends on the probability 132
that the email contains the word "Viagra". 133
What? 134
That sounds weird, right? Now, 135
I don't expect you to accept this at face value. 136
Let me try and explain this with a different example where the intuition is is less murky, 137
it's much more clear. 138
Say you look out the window and it's cloudy outside. 139
What's the probability of it raining today? 140
In other words: Given that it is cloudy, 141
what is the probability of rain? Now, in this picture, 142
it looks like it's about to rain any minute, but why rely on eyesight when you can have a mathematical 143
formula? 144
You see, like all good things, 145
conditional Probability has a mathematical formula associated with it and here it is: The probability 146
of rain 147
given that it is cloudy is equal to the probability of rain and it being cloudy divided by the probability 148
of it being cloudy. Hmm. 149
OK, this looks like a very, very strange formula indeed, but the important learning point here is that the 150
probability of it raining depends on the probability of it being cloudy and that's the bottom part of 151
the fraction here. 152
The bottom part of the fraction is the probability of it being cloudy. 153
If it was cloudy for 250 days a year last year, then this part of the fraction is 250/365. 154
Now what about the top part of the fraction? 155
Well, guess what. 156
This is our old friend joint probability. 157
It's the amount of overlap between the days that were cloudy and the days that it rained. So in London, 158
it rains on approximately 107 days per year. 159
But, you know what, two of those rainy days were actually fairly sunny and I saw a rainbow. 160
So the number of days where it was both raining and cloudy was only one 105. 161
So the top part of that fraction is going gonna be 105/365 now doing this calculation gives us 162
a conditional probability of approximately 42%. The probability of it raining given that it is 163
cloudy outside is 42% according to this formula. 164
I know this may sound a little bit trivial, but the really, really cool thing is that what we've just 165
done is calculate the probability of something knowing something else and that sounds a lot like fundamental 166
machine learning, right. 167
Knowing the movie budget we can calculate expected movie revenue. 168
Knowing the number of rooms we can calculate the expected property price. 169
Think of it being cloudy outside as a feature and think of it raining today as your target, your y. 170
So I hope this example shows how conceptually conditional probability is actually widely applicable 171
to machine learning and if you open any statistics textbook out there, you'll see that conditional probability 172
is expressed in very general terms just like this. 173
The probability of A given B is equal to the probability of A and B over the probability of B. What would 174
this formula look like in our spam example? Substituting for A and B, we get something like this, the probability 175
of an email being spam given that it contains the word "Viagra" is equal to the probability of an email 176
being spam and the probability of the email containing the word "Viagra" over the probability of an email 177
containing the word "Viagra". 178
Looking at this formula, there's one problem though, and that's the top term in this fraction - the joint 179
probability. Calculating the joint probability was super easy when we were dealing with dice and with 180
coins because each event was independent. 181
In that case, we could just multiply the probabilities of the events together but with dependent events 182
like clouds and rain or email and the word "Viagra", this calculation is no longer so simple and that's 183
why to calculate the conditional probability and create our spam classifier, we have to take it to the 184
next level. 185
We have to use Bayes theorem.