Welcome back. 1
Having just completed some very beautiful visualizations let's get back to pre-processing our data for 2
our Bayes' classifier. 3
Now there's lots of individual words among the 5800 odd emails that constitute 4
our dataset. 5
We won't actually use every single word that came up in these email bodies. 6
We're just going to use the 2500 most frequent words. I'm going to add a markdown cell 7
here to commemorate this and it's going to read "Generate Vocabulary & Dictionary" The 8
2500 most frequent words in our dataset are going to form our vocabulary and we will generate 9
this vocabulary from our stemmed list of words. To get our stemmed list of words we're once again going 10
to call our "clean_msg_no_html" function that we've created earlier. 11
Now I know that for the word cloud we commented out this line. 12
So if you have this commented out, comment it back in. because we actually do understand the words this 13
time round and if you made this change make sure that you do two things. 14
If you comment this line back in, make sure you comment this one back out and also press Shift+Enter 15
on this cell. 16
Otherwise you're going to get some very unexpected results later on. 17
All right. 18
So let's use our apply method and call this function right here. 19
I'll create a variable called "stemmed_nested_list", set that equal to "data.MESSAGE. 20
apply()" and then I'll feed in the name of our function "clean_msg_no_ 21
html". 22
And because this is a nested list I'm going to flatten this list and store it under "flat_stemmed_list" and 23
set that equal to the result of some Python list comprehension, 24
"[item for sublist in stemmed_nested_list for item in sublist]". 25
So far nothing new. 26
Let's run this cell and move on. The next step 27
will be getting a unique set of words. 28
This is gonna make up our vocabulary. 29
The easiest way to do this I think is to generate a pandas series and then use the "value_counts" method. 30
Once again you can ignore this warning here. 31
This is aimed at people who are trying to use Beautiful Soup to open a URL. Now to create that series 32
of unique words, 33
I'll create a variable really quickly that's called "unique_words" and I'll set that equal to "pd.Series()", 34
I'll provide the "flat_stemmed_list" that we created a minute ago and then I'm going to call a 35
method by the name of "value_counts". 36
Let me show you what we've just done here. 37
The number of unique words I can print out using the shape of my variable, so I'll say "Nr of unique 38
words" is going to be in a print statement, then a comma, and I'll say "unique_words.shape[ 39
0]" 40
This will print out the number of unique words in this series. And to look at the first five rows, the 41
first five entries in this series, 42
I'll say "unique_words.head()". 43
See what we get. And what we see here is that after cleaning and stemming, we are left with 44
27320 words in our dataset, 27000 unique words across all 45
our email bodies. 46
Now this is an absolutely huge number and we're actually only going to train our classifier with a subset 47
of this number, namely to 2500 most frequent words. Now you might be wondering why "http" 48
is up here. 49
Well, "http" pretty much precedes every single URL, 50
so this goes to show how many hyperlinks people have included in their emails. Now to get the 51
2500 most frequent words I wanna throw this over to you. As a challenge, 52
and the reason is this is another good opportunity to practice subsetting and working with these series. 53
Can you create a subset of this unique words series and store it in a variable called "frequent_ 54
words which will only contain the most frequent 2500 words out of the total? 55
And then afterwards print out the top 10 words. 56
These, of course, are going to overlap with the top 5 words that you see above. 57
I'll give you a few seconds to pause the video and give this a go. 58
Here's the solution. 59
"frequent_words = unique_words" and then we're going to use that square bracket notation, [0: 60
2500]". 61
That's how we create a subset. And to print out the top 10, 62
we'll say "print('Most common words') and then I'm even going to use a escape character and a new line, comma 63
"frequent_words" 64
and once again I'm going to create a subset, this time from the beginning, so I'm even gonna leave out the zero 65
to 10. 66
These are the top 10 words and here they are. 67
With this notation, when you're creating a subset you're setting a starting point and an ending point. 68
And with this notation of creating a subset you're going from the beginning to an end point. 69
So I hope this was a useful review. 70
One thing that we can do to improve our code though is removing some of these magic numbers that we 71
see here. 72
So what I'd like is instead of having 2500 float around in my code here, I'd like 73
to define a constant 74
at the very top called "VOCAB_SIZE" and set that equal to the size of the vocabulary that 75
I'm gonna use in my code later on. That way if I ever want to make a change all I have to do is change 76
this number here and it will filter through as long as I replace this number here with my constant "VOCAB_ 77
SIZE". 78
Now you've got to remember, if you've changed a cell up top, you've gotta press Shift+Enter. 79
Otherwise you're gonna get an error. 80
Now with frequent words, we're working with a series, right. 81
so "type(frequent_words)", we can see that we have a pandas series, and not only that; if we 82
look at frequent words, we see that this bit here, the actual words form our index and the number of occurrences 83
are actually the values in this series. 84
Let's practice how we would go between a series and a dataframe and how to work with these indices. 85
We're also going to take this opportunity to assign a word ID to each word, similar to how we assigned 86
a doc ID in an earlier lesson. I'll add a markdown cell here real quick that's gonna read "Create Vocabulary 87
DataFrame with a WORD_ID". Now our word IDs are just going to be integers, they're going to be ranging 88
from zero to 2499, meaning we're gonna work again with this range 89
object and we can create a range very easily with "range(0,)" and then going up to 90
our "VOCAB_SIZE", right. 91
This is how we create our range. Now what we can do is store all these numbers in a list. 92
So I'll wrap this call to the range into a list and I'm actually gonna store this in a variable called 93
"word_ids". 94
So "word_ids = list(range(0, VOCAB_SIZE))" and then closing 95
the parentheses. 96
Now let's create our dataframe with "pd.DataFrame()" and then what I'm going to do is 97
I'm going to provide a dictionary, so I'll have those two curly braces, our dictionary is gonna consist of 98
a key and a value. 99
So the key will be whatever I want that column heading to read, "VOCAB_WORD" 100
sounds good to me. 101
Now the values, I want those to be the actual words. Scrolling up a little bit, 102
the words are here in our series. 103
So this means that if I use "frequent_words" like so, I'm actually accessing the frequencies, the numbers. 104
What I need to do to get the words is work with our index, right. 105
So "index.values" will be the way I can store all these different strings in a column for our 106
dataframe. 107
So far so good. 108
Let's see what we've got. 109
Let me hit Shift+Enter on this cell. At the moment, our dataframe looks like this. Fair enough. 110
Let's add our word IDs explicitly to this dataframe. So we can do that with setting the dataframe's 111
index, 112
right. 113
So "index = word_ids" and then we can also give that index a name. 114
But first let me give our dataframe a name as well, so I'll say "vocab = pd.DataFrame" and on 115
the line below, 116
I'll say "vocab.index.name = 'WORD_ID'". Let's look at the first 117
five rows in our dataframe with "vocab.head()" and then Shift+Enter. 118
There we go. 119
Fantastic. 120
We've generated our vocabulary that we're going to train our classifier with and previously we've had 121
a pandas dataframe and we've used this "to_json" functionality to save a file in the JSON format 122
to our disk. 123
Now "to_json" is very well and good but of course pandas can save many different file types. 124
A common one that you're gonna be working with a lot is a CSV file, a comma separated value. 125
This is a file format that can easily be opened and nicely formatted with Microsoft Excel or Google 126
Sheets. 127
Now you probably already surmised that you're going to need a file path for this "to_csv" function that 128
we're gonna call. 129
So let's go back up to our constants and create a constant that will hold on to our file path and our file 130
name for this CSV file that we're gonna create. I'm going to copy 131
this constant right here, paste it below 132
and then make a few changes to it of course. 133
So I'll call this one, in all caps, "WORD_ID_FILE" and we're still gonna save 134
it in our Processing folder but we're going to call it "word-by-id" and then add the 135
".csv" extension to it. 136
So that's our file path and I'll hit Shift+Enter to make sure this is saved and then down here gonna add a 137
quick section heading. It's gonna read "Save the Vocabulary as a CSV File". As you can guess we'll access 138
our "to_csv" method from our vocab object, so "vocab.to_csv()" and then we're gonna 139
pass in our "WORD_ID_FILE" path and name. 140
Now if I hit Shift+Tab on this, I can see some of these other parameters that I can specify and that 141
includes a header and an index. 142
Now what does this mean? Coming down here, 143
I can see that our header can be a list of strings and our index label can also be a string. By default, 144
no index label is provided. 145
So let's provide these two additional inputs to our "to_csv" method call. So I'll add a comma here and the 146
first thing I'll do is provide the index label. 147
Now I can provide it as a string with single quotes and say 'WORD_ID', but what I could also 148
do is instead of typing this out and risk making a typo. 149
If I waned to make sure it matches what I've got in my dataframe I can access this directly with "vocab. 150
index.name' and for our header it's actually the very same thing. 151
I could either write 'VOCAB_WORD' which is our header right here, but alternatively 152
I can also grab our column, 153
so with "vocab.VOCAB_WORD. 154
name". This will accomplish the very, very same thing. 155
Now I'm not gonna hit Shift+Enter on this right now. 156
What I want to do instead is bring up my folder here on the right hand side and then hit Shift+Enter, 157
so you can see the file appearing. 158
Here you go. 159
There it is. 160
Now you can open this in a text editor, say like Atom, and the CSV file will be formatted something like 161
this. 162
It's not particularly impressive. 163
But if you have a spreadsheet program like Microsoft Excel or Google Sheets or in my case, this Numbers 164
program that comes with Mac then you'll see the values formatted nicely in these columns. 165
So as you can see the CSV format is very, very handy. 166
Now we're covering quite a lot of stuff in these lessons and with programming 167
the best way of learning it is by doing it. 168
So the next two lessons will consist of some very quick exercises to review some of the concepts that 169
we've talked about. 170
I'm off to grab some more coffee and then I'll see you there.