We're going to group our words by email and to do that, 1
we'll use the pandas groupby method. 2
Let me add a markdown cell here that reads "Combine", can't spell, 3
"Combine Occurrences with the Pandas groupby method". 4
Now I'm guessing that you will have used Excel, Microsoft Excel at some point in the past and in Excel 5
there is a very powerful functionality called a pivot table. 6
And this groupby method works in a very, very similar way. It will allow us to group the occurrences 7
by document ID, word ID and label. And then we can sum up our occurrences. 8
Let me show you. 9
"train_grouped = sparse_train_df.groupby()" 10
and I will provide a list 11
square brackets 12
single quotes, 13
"['DOC_ID', 'WORD_ID', 14
'LABEL']" and at the very end 15
we're going to chain another method called, namely the summation, so ".sum()" at the end will sum up our 16
occurrences after they've been grouped, but I think 17
seeing is believing. 18
So let me show you what this looks like. 19
So "train_grouped.head()" 20
will show us the result. 21
Here we go. 22
What we see here is that for our document with ID 0, our first document, we've got a bunch of words 23
in here that are grouped together by IDs. 24
The word with ID number 0 occurs twice in this first email. 25
This is all that this table is showing us right here. 26
Now you might say "All right, well, what's.. what's.. what's that word ID number 0? 27
What's... what's word 0?". 28
So we can pull that up, right? We can go to our vocabulary and we can say ".at[0, 29
'VOCAB_ 30
WORD". This if you recall was the column name in our dataframe 31
And this here is the index name which corresponded to our word ID. The actual word that occurs twice 32
in our vocabulary is "http". 33
Why does this occur twice? 34
Well it's because there's two hyperlinks in our email. The original email, 35
right, 36
with document ID 0. 37
We can pull up with "data.MESSAGE[0]" and this e-mail reads "Dear homeowner... Interest 38
rates are at their lowest level... blah blah". 39
If I look further down in the text, I see the first hyperlink here and I see the second hyperlink here. 40
This is why the word "http" appears twice in this document. 41
So our groupby function combined with the summation method seems to have worked really well. 42
The thing I would quite like though is to have less of this pivot table feel to it and repeat the document 43
ID on every single row. We can do that with "train_grouped = ", 44
so we're just gonna overwrite it, right, "train_grouped.reset_index()", "reset_index" will make our document 45
ID appear on every single row. "train_grouped.head()" 46
will show you exactly that. 47
Fantastic. 48
Let's take a quick look at the tail of this dataframe as well. 49
"train_grouped.tail()" gives us this result. And we're going to to the same very quick 50
sense check on this result as well. In particular, let's take a look at what word corresponds to 51
1923. 52
It appears to occur twice in this e-mail. 53
Now you can either work ahead or follow along with me. 54
But we've done this already, "vocab.at[1923, 55
'VOCAB_WORD']". 56
This gives us the result, 57
"welch", which is a very odd word, right, and doesn't quite seem like a like a real word, but it could be 58
a stem word, right? 59
So maybe that's why it's a bit strange. 60
Let's pull up the actual message and see why this word appears twice, "data.MESSAGE[ 61
5795]". 62
This brings up quite a short e-mail and it appears that "welch" is actually a name. 63
It's the last name of this guy Brent Welch, a software architect. 64
And the word appears again in his email address which is at the very end of this email. 65
So this is why it's here. 66
So I'm quite happy about this. 67
I think this is this is good. 68
This seems to pass the sense check. 69
The only thing I'd be quite curious to find out now is how big of a reduction we've achieved in the 70
number of rows from before? 71
So if I say "train_grouped.shape", then I can see that we've reduced the size of our data 72
frame quite a bit. 73
We've gone from 450000 to approximate 265000 74
rows. 75
That's still a lot but it's about a 40% reduction 76
and I think this puts us in a really good place to save our work, so let's do that now. I'll add a very quick 77
markdown cell here, 78
call this one "Save 79
Training Data as .txt file". In the previous lessons we've saved our files once before as a 80
.json File and as a .csv file. This is how we saved our files to our disk previously. 81
Now let's save a file as a plain text file. 82
And for this we're going to use numpy's functionality. 83
But before we do that, we're gonna need a relative file path at the top of our notebook. 84
That way it sits nicely with its friends. 85
Now I'm planning to save this file in a slightly different folder right. 86
But first let's give it a name. 87
I'm going to call it "TRAINING_DATA_FILE" and I'll set that equal to "SpamData/ 88
02_Training/train-data.txt". I'll be adding our 89
text file to this folder right here. Now, 90
be sure to hit Shift+Enter on your cell with your constants and then join me down here at the bottom 91
of the notebook. With "np.savetxt(TRAINING_DATA_FILE, train_grouped, fmt = 92
'%d')". 93
If I hit Shift+Tab on my keyboard to bring up the quick documentation, we see that this is the file 94
name including the relative file path. 95
This here, the second argument is the data and the third argument is "fmt" which stands for format. 96
If I hit the plus sign here and scroll down, then I can see that our format argument requires a string 97
or a sequence of strings. 98
This essentially allows us to specify the number format. 99
Lucky for us we're just dealing with integers. 100
If I bring up my folder here side by side and hit Shift+Enter now, then I should see my text file appear 101
right here. Before I open this text file and peek inside, let me show you what the columns are called 102
in Jupyter notebook. 103
So "train_grouped.columns" 104
will bring up our column names, so it's "DOC_ID", "WORD_ID", "LABEL" and "OCCURENCE". 105
Now let's look at this text file. 106
If I open it up in my text editor here, then I can see the four columns clearly outlined. 107
The first one is my document ID. The second number here is the word ID. The third number is the category 108
or label. 109
So the first message is in fact a spam email. 110
And that fourth number here is the occurrence of the word with this ID. 111
Looking at line 16 here, the word with ID 105 occurs twice in this spam email. 112
So I think that almost wraps it up, except we haven't done this for our test data yet. 113
And that's where I want to throw it over to you. As a challenge, 114
can you create a sparse matrix for the test data and then group all the occurrences of the same word 115
in the same email together just like we did with the training data? After you've done all that, save the 116
data as a txt file. 117
Now I realize, you're gonna have to save that data to some place and give it a file name. 118
So let's do that right now so that you and I have the same file names going forward. Scrolling up to 119
our constants, 120
I'm just quickly going to copy this relative path, paste it in and then change the file name here to "test" 121
and change the constant name from "training" to "test" as well. So "TEST_DATA_FILE" is 122
equal to this relative path file name and extension right here. 123
Now I don't have to ask you to pause the video because I'm going to show you the solution in the next lesson. 124
I'll see you there.