Next let's talk about a cornerstone of patient statistics namely the prior.
But what exactly is a prior a Prior is a guess or a belief about some quantity in our case.
We're going to express a belief about the percentage of spam messages in the test dataset for clarity.
Let's add the Bayes formula in latex markdown right here in our notebook.
So that changes cell to markdown.
And I'll add set the prior as a subheading and our latex code will look as follows.
Dollar dollar P parentheses spam space backslash comma pipe backslash comma X parentheses is equal to
backslash F R A C curly braces P parentheses x backslash comma pipe backslash comma spam backslash comma
parentheses backslash comma the parentheses spam closing parentheses closing curly brace open curly
brace P open parentheses x closing parenthesis closing curly brace and the closing dollar signs you
might be wondering what these backslash commas are.
And these are just the latex symbols for spacing this will become clear when I had shift enter because
what you can see now is that there's a nice little gap between the word spam and the X and that pipe
symbol in the middle and you can also see there's a gap here between the M in the spam and that parenthesis
here.
If I take out this backslash call my hip which I've inserted then that gap will disappear.
So as I was saying about the prior the thing about the Bayesian approach to statistics is that we're
allowed to have a first guess about some quantity we're allowed to have a guess before any evidence
is examined and this case can be based on prior knowledge.
For example from previous experiments or it can even be entirely subjective.
In fact different people will actually have different priors the prior that we care about is our probability
of spam.
In fact we calculated this probability in our previous notebook.
The probability of spam that we found was thirty one point one six percent APR.
So let's use that value here as our prior up put it in all caps prob on a school spam is equal to zero
point 3 1 1 6.
How it looking at that formula above hidden latex markdown you can see that this is expressed in kind
of like raw probabilities but we're gonna make a small transformation we're going to work with log probabilities
instead and this will make our calculations a lot simpler because we're gonna be dealing with addition
and subtraction instead of multiplication and division.
Also since we're working with very very small numbers that are very close together in fact taking the
logs will spread at our values a bit and this will make our graphs and plots a lot prettier and easier
to interpret when we create them later on as part of this project now as a challenge do you recall how
to take the log of a name pi array.
I'd like you to calculate the log probabilities of the tokens given that the email was spam.
This was stored in our variable prob on a score token underscore spam.
I'll give you a few seconds to pause the video and give this a go.
Before I show you the solution so as discussed we're gonna use the token probability here probably underscore
a token on this score spam which we've retrieved from a file using num PIs load text function to convert
this to log probabilities we're simply going to use num pi again with n pitot log and then feed in as
an argument prob just token and score spam.
These are our log probabilities.
All right.
So now the rubber is going to meet the road as my old boss used to say we're going to calculate our
joint probability in log format so I'll add a mark down sell here and I'll just write joint problem
ability and a log format in this calculation.
We're going to combine the joint probability and the conditional probability we're going to calculate
the probability that an email is spam given tokens and we'll do this for all the emails in our testing
dataset.
We'll store the results in a variable called joint on the score log underscore spam and that will be
equal to the top product.
So X on a score test.
Dot dot open parentheses of the law.
Probability of a token given that the email is spam.
This will be MP dot log open parentheses prob and it's got token on this call.
Spam minus the probability that a token occurs in any email B spam or non spam.
So that's MP dot log open parentheses prob on a score all on his core tokens and to this we add the
log probability of a spam email occurring so that's N.P. to log open parentheses problem on a score
spam and that's it.
This is what we've been building up to.
There's a lot going on in this single line of code let me pull up the first five values in this name
PIRA sells say joint on a score log on the school spam square brackets colon five and there we see our
probabilities that an email is spam given that it has the tokens that it has now all we need is the
probability that an email is non spam given its tokens and when I leave that to you as a challenge can
you calculate the law probability that the emails are non spam given their tokens store the result of
your calculation in a variable called joint just go log on to score how I'll give you a few seconds
to pause the video and give this a go
all right.
So here's the solution.
Let me scroll up here real quick and copy this formula right here.
This was the calculation for the spam side of things right.
So if I come down here changed the cell to markdown paste design that I can modify this formula for
our non spam emails so I'll replace the word spam with ham but I'm also gonna have to make a change
to this term right here.
The probability that an email is non spam will be 1 minus.
So parentheses one minus the probability that an email is spam.
So this is the formula that we're gonna be working with to calculate the joint conditional probabilities.
We're gonna create that variable as we said joint and it's got a log in the score.
Hmm.
And that's gonna be the result of the dot product X on the score test.
Dot dot parentheses and p dot log probability on a score token on a score ham minus end p dot log parentheses
probably underscore all those square tokens plus end p dot log parentheses 1 minus and then our prior
prob on a score spam and there we go.
This is the solution.
Once again we can take a look at the first five entries so bring that up with the square brackets and
colon and then 5 and they look like this.
We can also verify that our dot product and our calculation has given us a value for all one thousand
seven hundred and twenty three emails in our testing dataset so joined on a square log on a square ham
dot size will confirm that this is indeed the case in the next lesson we're gonna get to one of the
most exciting parts of the project we're going to be making predictions as to whether an email is spam
or not spam.
I'll see you there and if I've been sounding a little bit silly throughout this recording I do apologize.
I was sitting across from some children the other day and they've been doing their best to cough straight
out to me and I think now I've gotten whatever they've had.
These are the times when I trust in my ginger lemon garlic and to make tea that I'm sitting on right
now.