Okay, ready? 1
Hope you had a go at doing all this. 2
Here's the solution. Let's very quickly just recap where we are in terms of the data that we're working 3
with. "X_test.head()" will show us the first five rows of our feature matrix. The first 4
five rows of our target values "y_test" look like this. So you've got document 5
4675 as the very first row in both. "X_test.shape" will show us the 6
size of our dataframe that we're working with. This is gonna be much smaller than "X_train" 7
and as such the function call that we're gonna make to our "make_sparse_matrix" function 8
will take a lot less time to run. Let's check it out. 9
"%%time" will benchmark this for us and I'm going to call a variable "sparse_test_ 10
df" to hold on to result from our function call, so "sparse_test_df = make_ 11
sparse_matrix(X_test, word_index)". The word index 12
the same for both our training data and our test data and then comma, third argument y_test. 13
Let's run this and see what we get. 14
Scroll down a bit, add a few rows in the meantime. There we go. 15
Now we play the waiting game. 16
I could try and yodel for you to help pass the time but I think neither us would enjoy that very much. 17
Oh man, come on, come on. 18
These are the times when you start feeling a bit of the pain of working on a 4 year old laptop, but my 19
patience has paid off, 2 minutes 45 seconds to complete this calculation. 20
Let's take a look at how many rows we've got here. 21
"sparse_test_df.shape" reveals to us that we've got about 190000 22
individual rows. 23
Let me create another variable called "test_grouped" and set that equal to "sparse_ 24
test_df.groupby([])" and I've got to have those column names, 25
right. 26
The first one was "DOC_ID". 27
The second one was "WORD_ID", everything's case sensitive of course and spelling really 28
matters which makes this extra difficult for me. 29
Third one is "LABEL". 30
All of these are in a list and at the end we'll sum it up and I'm going to chain "reset_index()" 31
straight onto this. Finally, I'm going look at the first five rows, "test_grouped.head()" will give 32
me the following. 33
As you can see, it summed up the occurrence of word number 19 in email number 8 quite nicely. "test_ 34
grouped.shape" will also show me the has far fewer rows, only 110000 35
as opposed to 190000. To save this as a txt file we'll use numpy again, 36
"np.savetxt()" and now we supply that constant we created, "TEST_DATA_FILE, test_grouped, 37
fmt = '%d')". 38
I'll move my browser over a little bit, keep that Finder window in view, hit Shift+Enter and there it is, 39
there's my "test-data.txt" file. If I open it in Atom, then I can see that the first five rows 40
that we print out in Jupyter here and are showing as an output mirror what we see in the text file exactly. 41
The document ID is the first column, word ID is the second one, label is the third and occurrence is 42
the fourth column.