Now it's time to apply the cleaning and tokenization to all the 5800 messages. 1
But before we jump right into that, we have to review and practice a few key skills and that involves 2
slicing dataframes and creating subsets, 3
but not only that. We have to find a way to call our clean messages function on all the e-mail bodies 4
stored in our dataframe. 5
I'm going to change this cell to markdown and quickly commemorate what we're doing - we're going to "Apply Cleaning 6
and Tokenisation to all messages", but before we dive into cleaning 5800 email bodies, it'll be handy 7
to review these techniques for slicing pandas dataframes and series. 8
So I'm going to add a little subheading here that reads 9
"Slicing Dataframes and Series & Creating Subsets". 10
Okay. 11
So recently we've used the "at" attribute to select a particular cell or a scalar from our dataframe. 12
This was a very efficient way of accessing a particular entry in the dataframe and the way this "at" 13
attribute worked was that we had to specify both an index name and a column name. 14
Now I realize the column name here is a string and the index name here is a number, but that's because we've 15
given our index a numerical value here. 16
The important thing to remember is that the "at" attribute works off names. If you'd like to work off a 17
position, say at position 1, position 2, position 3, then there is an alternative attribute that 18
you can use instead. 19
Let me show you what I mean. 20
If we say "data.iat[2,1]", we get the entry at position number 21
2 and column number 1. Column number 1 22
in this case is our "Messages" column. Column number 0 was our "Category" column, 23
Column number 2 is our "Filename" column. 24
So with "at" you're working off a name to get a particular entry 25
and with "iat" you're working off a location or a position. 26
This is very, very useful for selecting a single message or a single entry in the dataframe, especially 27
if you're working in a loop or your code is iterating over a certain part of your dataframe with a 28
numerical value, 29
this "iat" attribute will make your life a lot easier than the "at" attribute which works off the names. 30
Now what if you wanted to select a subset of the dataframe instead? 31
What if you wanted to select more than one value, more than one row 32
for example? In this case, you can use the "iloc", iloc attribute, 33
so this is "data.iloc[0:2]". 34
This code here will select the first two entries in our dataframe, the first two rows if you will. If 35
I hit Shift+Enter we can see what they are. 36
If we wanted to select the first five rows, then we would simply substitute the five for a two and here's 37
the challenge. 38
What would you put in between the square brackets if you wanted to select the entries with "DOC_ID" 39
5, 6, 7, 8, 9 and 10? 40
What would you supply here? 41
In that case, you would go for "5:11" and that's because the first entry is at index 0. 42
Now the nice thing about this "iloc" attribute is 43
it works both on dataframes as well as series. 44
So if I had only a single column selected, say "data.MESSAGE", I could still use this attribute "iloc 45
[0:3]" and select only the first three entries in this series. 46
This is very, very handy for creating or selecting a subset. 47
So now that we've covered this let's move on to the next question. 48
Say we want to try out our "clean_message" function on the first three emails out of the 49
5800 odd emails. 50
How would we do it? 51
Well step 1 is selecting these email bodies of course 52
and we've done that right here. 53
Step two is using the pandas 54
"apply" function. 55
Now I absolutely love this function. 56
This is this genius. Say we store these three entries in a series call it "first_emails", set that equal to 57
"data.MESSAGE. 58
iloc[0:3]" and then we can take this series, 59
put a dot after it, use "apply()" and then supply the name of our function which was "clean_ 60
message". Note there is no parentheses after "clean_message". We're supplying just 61
the name of our function to the "apply" method which we're calling on our series object here. 62
Let's see what happens. What we get back 63
is kind of a list of lists. We've got all our words in the list split out for our first e-mail. 64
Same thing with our second e-mail. 65
And again we've cut all the words split out for our third e-mail as well. 66
If you're curious what kind of type of object we're dealing with here, you can take a look by just wrapping 67
this in the type function and you see that you still are working with a series. 68
So that's good. 69
That's really, really interesting. 70
Now I'm gonna give this thing a name. 71
I'm going to call it "nested_list" and I'm going to set it equal to the output from this "apply" 72
function. 73
I'm calling it a nested list because effectively I've almost got like a list of lists, right. Each individual 74
entry in my pandas series here is a list, but here's a question: What if I wanted just one list of words? What 75
if I didn't want a list of lists? Let me hit Shift+Enter on this. 76
I'm going to create an empty list here and then just write a for loop, right. A nested one probably. I have to go over 77
all the items in the list first. So I'll say "sublist in nested_list", that's my first part of my loop 78
and then my second part of my loop, the inner loop, I'll say "for item in sublist:". The outer loop goes 79
over all the lists and the inner loop goes over all the individual words in each sublist. Put a colon there 80
and then say "flat_list.append(item)". That will pretty much do the job. So if I wanted to 81
check what the length of this flat_list is, then I can use "len(flat_list)". Let's see what 82
we get - 390, 83
right. So that's 390 words now in a single list. If we wanted to take a peek what it 84
looks like and it looks like this, it's just all the email words in one big list. And you know this is 85
certainly one way of doing it, but let me show you an alternative. 86
Let me show you how this would look like with something called Python list comprehension. 87
So I'm going to comment all of this out. If you select the text like me and then press Control + /, 88
this is the shortcut for commenting out an entire block of code. If you're on a Mac of course, it'll 89
be Command + /. But for everybody else Control + /. The Python list comprehension 90
syntax for accomplishing exactly the same thing as with these two nested loops looks like this. 91
So I'll still use flat_list as my variable and I'll have square brackets and then I'm basically going to 92
move my loop inside these two square brackets. I'll say "item for 93
sublist in nested_list for item in sublist". 94
This here you'll recognize as the outer loop "for sublist in nested_list" 95
and this here you'll recognize as the inner loop "for item in sublist". 96
This first bit of code is what appends the item to the list. 97
Let's see what happens when I've run this again. 98
I in fact get the exact same result. 99
So now you've got another application of this Python list comprehension syntax. 100
You can often use it instead of nesting these for loops. Now you've probably put two and two together 101
on how to call our "clean_messages" function on all the 5800 odd items. We're gonna 102
do something very, very similar to what we did a minute ago. 103
We'll create a nested list and we'll set it equal to "data.MESSAGE.apply(clean_message 104
no_html)". This here will use "apply" on all the messages in the dataframe. Now before I hit Shift+Enter and 105
execute this code, I'll use a little bit of Jupyter magic. I'll put 2 percent signs here and then use 106
that keyword "time" and what this will do is it'll spit out how long this computation will take. 107
So this is kind of like a little bit of a benchmarking tool for looking at the performance of your Python 108
code. 109
It'll also help you see just how long it takes for my code to execute on my computer and you can compare 110
this to your own. 111
The first thing I'm seeing here is that our Beautiful Soup package seems to think that we're trying 112
to open a Web site. 113
Now this is definitely not the case. 114
So this user warning is a little bit unnecessary. 115
Our code still working, nothing's breaking, but this is something that Beautiful Soup has included in 116
their code as a suggestion for people who are looking to open Web sites. 117
The other thing that you can see is that in order to apply this function to a all my 5800 118
odd messages is that my computer ran for like a good minute. 119
It got there in the end, but it definitely did some work. 120
So let's check out the head of our nested_list. 121
Let's look at the first few lines. Looks good. 122
What about the tail? 123
The last few lines. 124
Brilliant. 125
That also looks good. 126
Nothing unexpected.