Over the next couple of lessons we will be training our naive bayes classifier.
But what does this involve the training step primarily involves calculating the token probabilities
we need to work out the probabilities for an individual word or token.
How if you haven't watched the lessons on probability theory and the base theorem from the previous
module pause here and watch the now before you continue you'll recall that the formula for conditional
probability under Bayes Theorem looks like this.
This formula gives us the probability of an email being spam.
Given that it contains the word diagram to work out the final numbers for this formula we need a couple
of things.
First off we need the overall probability of an email being spam.
Second we need the probability of Agora occurring in any email.
And third we need the probability of an email containing the would diagram given that the email is spam.
This calculation boils down to five numbers.
So what are these numbers that we need to work this out.
The first number at the top is how often the token question in this case the word diagram appears and
spam emails.
The second number is the total number of words in spam emails.
The third number is the probability of spam e-mails occurring in the first place.
On this slide I've put this number at 55 percent but will actually work this out from our training data.
In the denominator of this fraction we have our fourth number the total number of times that the wood
Viagra occurs in all emails both in spam emails and in non spam e-mails.
And finally we've got the total number of words that we looked at across all emails in the first place.
Both spam messages and non spam messages across the entire dataset.
The python code that we're gonna write next will tease out a couple of these numbers with these numbers
in hand.
We can calculate the relevant probabilities for our tokens.
This is effectively the training step for our need based classifier I'll add a quick markdown cell here
that reads training the naive bayes model and for subbing here will read calculating the problem ability
of spam in the slide.
I had the set at 55 percent but I think we should work this out from our data.
This actually makes for a very good challenge.
I'd like you to quickly calculate the overall probability of spam will assume that this is the percentage
of spam messages in the training dataset store this value in a variable called prob on a score spam.
I'll give you a few seconds to pause the video and tackle this challenge
Java go.
The total number of messages that we have in our dataset is full.
On the score train on the score data thought category dot size.
We've got about 4000 and 14 messages.
The number of spam messages is full.
On a score train and a score of data but category dot some.
The reason I can do this is because I've labeled the category of spam as one and the category of non
spam at zero.
So simply summing up all the ones in my category column will give me the number of spam messages of
which I've got one thousand two hundred and forty nine.
Therefore the probability of spam problem on a score spam is in fact equal to the training data category.
Some divided by full on is called train on the score data adult category don't size and that we can
print out as probability prob Bill T of spam is come up.
Problem is score spam.
There we go.
It's around thirty one percent.
This is the first of the numbers that we're looking for in our Bayes theorem the next number that we
are looking for is the total number of words or tokens we need to count the total number of tokens that
we have in our dataset.
And we also need to calculate a number of tokens that belong to the spam category and the number of
tokens that belong to the non spam category.
But first let's create the total number.
What I'm gonna do is I'm going to create a subset from our full on a school train and a score.
Data data frame.
I'll call the subset full on a school train on a school features.
So this will be just the tokens.
I'll set this equal to full unescorted train.
And as for data dot allow C ATL C's for selecting a number of rows square brackets colon.
This is for selecting all the rows.
But I'm not going to select all the columns I'll just select the columns with our word at ease.
In other words I leave out the category column ellipses perfect for this to leave out the category column.
I'll just say well take all the columns that you've gone full on as Katrina on the score.
Data
columns.
But don't pick the category so I'll say not equal to single quotes category that exclamation mark that
you see here is Python syntax for the logical not together with that equals sign.
This logical condition here reads not equal to category so I'll pick all 2500 columns except the category
column.
Now mind you this string here is of course case sensitive.
Let's look at the head of full on this quatrain on this go features and see what this data frame looks
like.
There you go.
We've just excluded a particular column from the subset using logic.
Now how do we get the total number of words.
Well let's take a two step approach.
Let's sum up all the tokens per email.
So if you think about it we should sum up all the numbers in this row.
Then we should also sum up all the numbers in this row and all the numbers in this row.
That would be the total number of tokens per email.
We can do this really really easily with the sum function.
So I'm going to store this thing in a variable called email to school lengths so that we'll hold on
to the total number of tokens per email and to use that some functionality.
I'll grab my features data frame full on escort train and escort features and I'll use some function
but I'll provide an argument here.
I have to specify how this function should some things right left the right bottom to top up to down.
We have to be more specific and we can do this with this access parameter him.
Access is equal to one This will sum up our columns.
Let's take a look at what email and scroll lengths.
Shape gives us so we've done the summation and we've got four thousand and 14 emails exactly what we
want the first five of these look like this email I just call lengths square brackets.
Colon five and we see that the first email has 50 tokens.
The second email 76 tokens the third e-mail 87 tokens and so on.
So what we've got here is a handy little panda series of all the tokens to get the total number of tokens
our total would count if you will.
All we need to do is sum up the values in the entire series.
Right.
I'll store this number in a variable for later use.
So I'll say.
Total on a score WC for would count and I'll set that equal to email.
I just call lengths not some to see what that value is.
I'll just put total unescorted w see below and hit shift enter and what we see is that we've got approximately
four hundred and forty six thousand tokens.
Looking back at our formula here We've now worked out this white number here and this blue number here.
Now let's work out this green one.
Let's work out the total number of tokens within spam and also within the non spam messages which is
not shown on this slide here a lot of this as a markdown cell number of tokens and Spam and Ham emails.
I reckon this actually makes a good many challenge.
Can you create a subset of the email length series that only contains the spam messages called a subset
are spam unescorted lengths and then count the total number of tokens that occur in this subset also
do the same for the non spam emails create a subset called Ham lengths from the email on the score length
series and then also count the number of words that occur in that non spam or ham emails I'll give you
a few seconds to pause the video and give this a go.
You ready.
Here's the solution.
What I'm going to do is I'll create this variable spam links and I'll subset this whole thing from email
underscore lengths using the square bracket notation where my full on the school train and it's got
data dot category is equal to one double equal to one that is that's the logical condition to create
the subset from email and a scroll lengths the number of emails that are spam are of course spam underscore
lengths dot shape.
Here you can see we've got one thousand two hundred and forty nine to get the number of words or the
number of tokens that are in the spam emails.
I'll store this whole thing under spam on a school W C for word count and I'll set that equal to spam.
On just go lengths don't
and if I print this out as an output we can see that we've got approximately one hundred ninety five
thousand different tokens that are part of the spam emails.
This is the first part of the exercise.
The second part of the exercise.
Doing the same thing for the ham emails the non spam emails we can write ham on the school lengths set
that equal to email that lengths and subset this whole thing based on the condition that full on a school
train and a data not category is equal to zero in this case ham underscore lengths dot shape will show
us that we've got two thousand seven hundred and sixty five.
How many emails are good thing to do at this point maybe is check your work right email dot lengths
dot shape square brackets zero minus spam to underscore lengths dot shape square brackets zero minus
hem underscore the lengths that shape square brackets 0 should be equal to zero because the two subsets
together should have the same number of emails as our dataset as a whole.
Right.
This is a quick check that you can do to make sure you've subset of things correctly the total non spam
word count the word count for the ham messages should be equal to ham on a scroll lengths.
Dot some and this value is equal to two hundred and fifty thousand approximately Hennigan.
I think this is a good time to also do a quick check spam on a school word count plus non spam on a
score word count minus the total word count should be equal to zero.
There we go.
I've actually got an add on challenge for this.
I just thought of an interesting question to ask.
Do you think that the spam emails or the non spam emails tend to be longer.
Which category do you think has longer emails.
On average what I'd like you to do real quick is to take a quick guess and then verify it in the data.
How would you go about doing that
here's the solution.
I want to print out the average number of words in spam emails.
This is gonna be spam on a school W C divided by spam lengths.
Don't shape square brackets.
0 The average number of words in the spam emails in our dataset is approximate one hundred and fifty
six hundred fifty seven.
If this number is a little tough to read we can actually format this have much much more nicely with
curly braces.
Colon daunt 0 F stands for float.
I'll get rid of this comma and I'll write dot format and I'll put the calculation inside the method
call here to format here I'm running to the nearest poll number.
This is a nice little trick if you want your own print statements to look pretty and not give you a
lot of numbers after the decimal point.
Let's compare this to that how many emails the non spam emails will take the ham would count or non
spam would count and will divided by ham on the lengths don't shape and just to show you how this string
formatting works.
I'm going to have a three here instead of a zero so we'll get three decimal points in our output.
There you go.
So we can clearly see that spam emails actually tend to be worse here which I think is actually quite
interesting as well.
Now that we've got a big headline numbers in our formulas sorted we need to take a closer look at the
tokens themselves.
We need to sum up the tokens that occur in the spam messages and the tokens that occur in the normal
emails.
And we need to do this for each token for each single word I.D. and we'll get right on that in the next
lesson.
I'll see you there.