Throughout the next lessons I want to do the final steps of the data preparation for our Bayes' classifier 1
and also show you a common format for data sets that you'll often encounter out there in the wild. 2
Over the next lessons we'll be writing some Python code to create our feature vectors, but not only that, 3
we'll create our features as a sparse matrix. 4
What do I mean by that? 5
Well, let's take it one step ahead of time. 6
Let's consider a full matrix first. 7
In that case, we have our document IDs in one column and then we'll have our words in another column. 8
Now of course these will be the word IDs, but for illustration, I've written an actual word here on this 9
slide. 10
Then, in that third column, we'll have our Label. Our Label will be equal to 1 if the e-mail is spam 11
and it will be equal to 0 12
if our email is non-spam. 13
So in this case, email number 5795 is a non-spam email and the 14
label is equal to 0. 15
In that last column, where it says Occurrence, we will capture how often the word in the Word column 16
appears in the email. 17
So if the word "free" appears 3 times, then occurrence will be equal to 3. 18
Makes sense, right? 19
The reason that this kind of table is called a full matrix is because for each email, there's an entry 20
for each word, 21
even if that word doesn't occur in the email. The full matrix has an entry for each word in the vocabulary 22
for each and every single email. 23
Now we set our vocabulary size at 2500, 24
remember? Therefore for each document ID, there will be 2500 entries, many of 25
which will be 0. 26
And this is where the sparse matrix comes in. With a sparse matrix, 27
we will remove the entries which are 0. 28
And that means we actually only include the rows which have a word that occurs in the email. 29
In this example, the word "mortgage" and the word "pay" does not appear in email number 30
5795, 31
therefore those rows will not be present in the sparse matrix. So you can see how the sparse matrix is 32
just a compressed version of the full matrix with fewer rows. 33
Now let's head over to Jupyter notebook and write some Python code. 34
First, let me add a markdown cell to commemorate what we're gonna do. 35
We're going to generate features and a sparse matrix and the first part of that is going to be creating 36
a dataframe with one word per column. 37
Now I say creating a data frame with one word per column, 38
but what are we creating the dataframe from? Where are we at in terms of our data. 39
We're going to be working with our stemmed nested list. Our stemmed nested list looks like this, 40
so we've got a list of words for each document, for each email. 41
This thing about this is that our stemmed nested list is a series, right. 42
It's a pandas series that holds on to individual lists. 43
So if I want to access the email at position 2 then I would see that even though our stemmed nested 44
list as a whole is a pandas series, it contains individual lists. 45
So it's actually a series of lists, each entry is a list. That makes it a little bit of an unwieldy data 46
structure, but it's one we're gonna work with, 47
and what we're gonna do is we're going to convert it from a series containing lists to one of a list 48
containing lists. 49
And the reason for doing that is that there exists a very, very handy method to convert list of lists 50
into a dataframe. 51
So let me quickly copy and paste this cell and show you the method we're gonna use to convert our series 52
to a list. 53
And it's simply called "to_list". Our pandas series has a method called "to_list" which converts the whole 54
thing to a Python list. 55
Fair enough. 56
The way this looks is what you might expect, namely two square brackets, 57
and then in each pair of square brackets you have an individual email, 58
so the first e-mail ends here, 59
the second one starts here. 60
It's... 61
It's kind of messy actually, but we don't have to worry about this so much, because we're going to create 62
a pandas dataframe using this code here. 63
And the way we're gonna do that is with "pd.DataFrame" and instead of the parentheses and enclosing 64
it like so, what we're gonna do instead is we're gonna put a dot and call a method called "from_records", 65
"from_records" that is. And this is where we're going to feed in our "stemmed_nested_ 66
list.to_list()". Now before I hit Shift+Enter on this, let's store this whole thing, this whole 67
dataframe, in a variable called "word_columns_df", "df" for dataframe and we'll 68
set that equal to "pd.DataFrame.from_records" and then below, we're going to print out the head 69
of this dataframe, so "word_columns_df.head()". 70
Now let's take a look at the first five rows, and what we see here is as the index, we've got our document 71
IDs, we'll add a label for this shortly, and then we have each word split up as an individual data point 72
in a column and it looks like we have a total of 7661 columns. 73
The overall shape of our dataframe looks like this. 74
We've got 5796 rows and 7661 75
columns. 76
Now here's a question to you. 77
Why does the dataframe have this shape? Well, 5796 is the total 78
number of emails that we have and 7661 is the number of words, stemmed 79
words that is, in the longest email. And we know this to be true because we've worked this out in a previous 80
exercise. Now it's time to take the next step and that is splitting the data into a training and testing 81
dataset. 82
It's time to shuffle and split the data, 83
now that we've got our "word_columns_df". 84
Now, we've actually done this before. 85
So I want to throw this over to you as a challenge. 86
Can you split the data into a training and testing dataset using scikit-learn? As you're doing this 87
set the test size at 30%. 88
That means that the training data should include around 4057 emails and also as you're shuffling 89
set the seed value to 42. 90
And as you pause this video and try to solve this challenge, have a think about what the target values 91
should be as you're doing this. 92
all right. 93
So here's the solution. 94
We're gonna be using scikit learn's "train_test_split" function to accomplish this. 95
So we have to import this whole thing into our notebook, so we'll say "from sklearn.model_ 96
selection"; 97
this is where the whole thing lives; 98
"import train_test_split". Hitting Tab on your keyboard after typing a few of the letters should help you 99
avoid any typos on this relatively long import statement. 100
Now hit Shift+Enter on this cell and let's continue where we left off at the bottom of the notebook. The 101
"train_test_split" 102
function will give us four outputs. So let's store them in four separate variables. 103
The first one I'll call "X_train", the next one I'll call "X_test", then lowercase 104
"y_train", 105
comma 106
"y_test". 107
And that's gonna be equal to "train_test_split(word_columns_df)". 108
This is going to be our first argument. 109
Then we have to supply our y-values. The dataframe that we just created after all was just the features, 110
right, the different words. The y-values that we're trying to predict are actually our categories and 111
I'm going to grab those from our "data" dataframe. It has a column called CATEGORY and this is what I'll 112
supply here. 113
Next I'll set the test size to 0.3, so 30%, and then I'll set my "random_ 114
state" to 42. 115
This is where I'm specifying a seed value. 116
And if you and I specify the same seed value, we'll get the same results, we'll get exactly the same shuffle. 117
So let me run this and then we'll look at our analytics. First, let me print out the number of training 118
samples and I'll print out "X_train.shape[ 119
0]". 120
These are the number of training samples and we've got 30% of the total which is 121
4057. Next, 122
let's just verify the fraction of the training set. So the fraction of the training set 123
is gonna be this number, 4057 divided by the total number of entries, say in 124
our features dataframe. 125
so this was "word_columns_df.shape[0]" and that's 70% or very close 126
to that, meaning the test set is going to be one minus this number, which is 30%, which we've specified 127
here. 128
Now let's take a look at what we've actually got. 129
So I'll take "X_train.head()". 130
These are the first five rows of our shuffled "word_columns_df". 131
Now I said that these numbers here would refer to the index. 132
Right. 133
Our document IDs, and we can add this label very, very easily. 134
So we'll say "X_train.index.name = 'DOC_ 135
ID'", Shift+Enter, 136
we'll have our index name show up in the output right here. But say we wanted to add this index name 137
to both the training dataset and the testing dataset. 138
So "X_test" as well. 139
We can actually do that in the very same line of code by inserting another equal sign before we assign 140
this string value here and write "X_test.index.name". In this case we're setting 141
both of these indices equal to "DOC_ID". Let me hit Shift+Enter and show you that document 142
IDs actually match up after shuffling. Let's pull up "y_train.head()" and there we see 143
the first five rows of our target values. You can see here that the document IDs match up with what 144
we see in the training dataset. Now of course, that should be true regardless of whether this says 145
X_train or X_test, the order of the features and the target values will be the same, 146
but proof is in the pudding, right? 147
"X_test" looks like this and "y_test" looks like this. In the next lesson 148
we're going to create a sparse matrix from our training dataset and we're gonna do that by transforming 149
the values in our dataframe. I'll see you there.