Now I think this is the perfect time to kind of check our understanding of all the work that we've done 1
and maybe also look at some of these pre-processing data munching subtleties that we might have missed. 2
I'll add a quick section heading here just for that. 3
Now the way I'd like to tackle this is actually kind of with a challenge. 4
You see, we started with about 5796 emails. 5
We split these emails into four 4057 emails for training and about 6
1739 emails for testing. 7
But how many of these individual emails were actually included in the text files that we saved to our 8
disk? 9
Let's just look at the testing emails for now. 10
Can you figure out how many individual testing emails were included in the txt file that we saved 11
to the disk using the "train_grouped" dataframe and then compare that to the amount of emails 12
in X_test? After splitting and shuffling our data, how many individual emails were actually 13
included in the X_test dataframe? Is the number the same? And if not, which emails were excluded 14
and why? I recommend comparing the doc ID values to find out. Did you have a go at this? 15
This might have been a little bit of a tricky challenge actually. 16
So let me show you the solution. The way I'm going to tackle this is using Python sets once again. 17
Now there's other ways for sure, but I think this code is terse and it's also clear. 18
I'll create two variables which will hold on to these sets. The first one will be "train_doc 19
_ids" and that's gonna be our set of all the document IDs which were under "train_ 20
grouped". Then I'll create a second set called "test_doc_ids" and set that 21
equal to a set that I create from "test_grouped.DOC_ID". 22
This will give me a comparison. 23
The reason I'm doing this is because I can easily print out the number of individual emails using these 24
sets, "len(train_doc_ids)" will give me the number of individual emails in our training 25
set. 26
So that's 4014 compared to the 4057. "len(test_ 27
doc_ids)" will give me the number of individual emails that we actually saved under the testing txt 28
file, so that's 1723 and we can compare this number to the length of X_test which was 29
1739 emails. 30
So 16 emails have gone missing or have been excluded, but which ones? 31
Which emails were excluded? 32
Well, remember how we said that Python sets are very, very useful for checking membership to find out 33
which emails are included in one set but not in the other set? 34
We can use, well, Python sets again, only this time we will take the difference between two sets. 35
Check it out. If you recall "X_test.index" actually stored our document IDs. 36
If I take this whole thing and grab the values, then I can transform my index values into a Python set. 37
To see which values are included in this one but missing from this one, all I have to do is take the 38
difference between the two and that will give me the answer. 39
Here are the 16 emails that are not included in our txt file, but that still doesn't answer the question 40
why. We've only identified the specific document IDs that were problematic. 41
Let's dig in and take a closer look at these messages. 42
The first one I'm going to pull up through "data.MESSAGE" is message number 14. 43
So "data.MESSAGE[ 44
14]" 45
will show us the message text of the first email that didn't make it. 46
And this email looks like this. 47
It's, well, it's.. 48
It looks like a private key or a public key of some sort. 49
It's complete gibberish. 50
What about some of the other ones? 51
Let's check 52
325, 416 and 445. 325 53
looks like this; 416 54
looks like this and 445 looks like this. 55
Now I'm starting to spot a pattern here. 56
All of these e-mails seem to look like a complete mess, but maybe the reason they look like this is because 57
we had a problem reading this file, right. 58
Maybe this is related to our encoding that we used and this is why we're getting some sort of like 59
Mojibake here or something, right. 60
Maybe that's the source of the problem. 61
So let me actually pull up one of these files in my text editor. 62
I'm going to go with file number, say 14. 63
The way I can find out which file this actually was is by going to "data.loc" and then feeding in the 64
number 14. 65
This will give me the actual file name, so that starts with 00095 and then something. 66
I've located this file under my "spam_1" folder in my Spam Assassin Corpus. 67
Opening this file in my Atom text editor gives me the following. 68
I got my email header up top and then down here I've got my email body. 69
So what we saw in Jupyter notebook is actually the email body, right, 70
1-to-1, we didn't screw something up on the encoding. 71
Now would you like to venture a guess as to what we get when we stick this email through our message 72
cleaning function? So "clean_ 73
msg_no_html(data.at[14, 'MESSAGE'])". 74
Would you like to venture a guess of what we get when we stick our message body from this particular 75
email through our cleaning function? We get a string like this. Now this particular string, as you can 76
probably guess is not part of our vocabulary. 77
It's not part of the top 2500 words that we're using for our classifier and hence this email would not 78
have been included when we created our sparse matrix. 79
This condition here "if word in word_set" filters out if any of the words in the email are not part of 80
our vocabulary. Now the story is quite similar with some of these other emails. 81
But one exception is this one that I found here, 1096. This document "data.MESSAGE[ 82
1096]" looks like so. And looking at this text, we see that it's 83
not all gibberish, right. 84
This is mostly HTML actually, right? 85
We can see a whole bunch of HTML email tags in the body of this email. 86
And what happens when we clean this message and remove the HTML? So "clean_msg_ 87
no_html( 88
data.at[1096, ' 89
MESSAGE']" is that we get an empty list. 90
This particular message contains so much HTML all that Beautiful Soup, 91
the tool that we use to remove all the HTML tags and strip those from our email bodies actually leaves 92
nothing behind. 93
There is no single word that actually makes it into our list. 94
The reason that I know it's Beautiful Soup and not our word stemmer or something else is if I call our 95
other function "clean_message()" where the HTML tags are left in and our feed in the very same 96
code, "data.at[1096, 'MESSAGE']", ugh typo, then I get the following. I get all my 97
HTML tags as individual words on this list. So I think that was actually quite subtle, 98
there was something that was happening in the background that we might not have noticed while we were 99
writing our code, but it's always good to check if your code is doing what you expect it to do 100
and I think this also gave us a chance to have another go at practicing using Python sets and really 101
seeing their use in checking membership, 102
especially if you're looking for differences and trying to see which values are included in one but 103
not on the other one. 104
And that's it. 105
This wraps up our pre-processing. 106
We've actually done an incredible amount of work in the past couple of lessons. 107
We've extensively cleaned our data, explored our data, visualized our data and the text files and the 108
JSONs and all these files that we've saved to our disk create checkpoints that mean that we don't have 109
to rerun all the work and all the code in our Jupyter notebooks. 110
In fact, we've worked really hard to create the files that contain the data that we will feed into our 111
naive Bayes' classifier algorithm. 112
And if you take another look at this training data and this testing data that we've created, you'll notice 113
that all the strings, all the text has disappeared from them. 114
We're exclusively working with numbers now. All the stemmed words are replaced just by word IDs, by 115
integers. 116
It's much more abstract, right? 117
But what we've effectively done is we've transformed our words into tokens, but not only that, we've also 118
counted how often each token appears in every email message. 119
With this in hand, it's time to train our naive Bayes' classifier. 120
We can move on to the next step in our project. Now our Jupyter notebook, 121
is getting very, very long and as such it's getting also a little bit unwieldy. 122
So I'm quickly gonna go up here and rename this notebook. 123
I'll call it "06 Bayes Classifier - Pre-Processong", click "Rename" and then for the next part where we're 124
training our algorithm, we're gonna be doing it in a new fresh notebook. 125
That way we've separated out all the pre-processing that we're doing in one Jupyter notebook and we're 126
going to separate the training that we're gonna do for this project into a separate notebook. 127
And on that note, I'll see you in the next lessons. 128
Take care.