Welcome back. 1
In this video we're going to talk about word stems and word stemming, as well as removing punctuation. 2
These are gonna be the next two steps in our pre-processing stage. Again, we'll work with an example 3
sentence before applying the all of this to our email dataset. 4
I'm going to add a quick markdown cell here that reads "Word Stems and Stemming". Now, what do I mean when 5
I say word stems and word stemming? You see, stemming is the process of reducing words to their base 6
or their root form. 7
The idea behind word stemming is to treat inflected or derived words in the same way. 8
So for example, the words "fishing", "fished", "fisher" and "fishlike" are all reduced by the stemmer to the word 9
"fish". Endings like "ing" in "fishing" or "ed" in "fished" are removed by the stemming software. 10
Now the thing to note about stemming is that the stemmer might not produce "proper words". 11
That is to say, you might not end up with a real word 12
after removing the stem. 13
So for example, the words "argue", "argued", "argues" and "arguing" are all stemmed to the word "argu". Now this 14
is not an error. 15
The purpose of the stemming algorithm is to bring the variant forms of the word together and not to 16
map a word to its paradigm form if you will. 17
The stemmer that I'd like to introduce to you is the de facto standard stemmer for the English language, 18
the Porter Stemmer. 19
This algorithm was written by Martin Porter all the way back in the 1980s at the University of Cambridge. 20
In our previous lesson, 21
we've already imported the PorterStemmer functionality from NLTK at the top of our Jupyter notebook. 22
Now it's time to put it to use. 23
I'm going to copy this cell here and then I'm going to paste it below my markup. What I can do now is 24
simply save the PorterStemmer to a variable, so I'll just say "stemmer = PorterStemmer()" 25
and then to use this stemmer, 26
I'm going to go inside my for loop, just before appending the words, I'll create another variable called 27
"stemmed_word" and this will be equal to the result of the "stem" method from the stemmer. 28
So I'm going to use the stemmer, put a dot after it, call the stem method and then here between the parentheses 29
I'm going to supply the word that our loop is looping over, stem the word, store it inside this variable 30
here and then, instead of appending the original word, I will simply append the stemmed word. 31
Now looking at our example sentence here, the word "makes" is a clearm clear candidate for stemming. Let 32
me hit Shift+Enter and see what it will be stemmed to. "Makes" is stemmed to "make". 33
Now it turns out this was only one word in our example sentence that was stemmed. 34
Perhaps we should add another word that's a stemming candidate at the very end just to try out how the 35
stemmer works. 36
I'm going to expand the example sentence with a few more words, so I'm gonna wrap my line across two 37
lines in Python so I don't have a very, very long sentence all in the same line. 38
So I'm going to use that backslash, that escape character which escapes me pressing Enter on my keyboard. 39
Now I'm going to add a few more words to my example sentence - 40
"Nobody expects the Spanish Inquisition". 41
There we go. 42
Let's see how our Porter stemmer handles this. 43
So quite interesting, "nobody" gets stemmed to "nobodi" with an "i". With "expects", the stemmer drops the "s" and 44
with the word "inquisition" the stemmer drops the letters "ion" at the end. Now one thing I'll say is 45
that you're not actually limited to using the PorterStemmer from the NLTK tool box. There's quite a few 46
to choose from, there's almost like a menu. The reason you might want to use a different stemmer other 47
than the Porter stemmer for example is if you're stemming a different language. Scrolling up to the top, to our 48
imports, a popular choice for other stemmers is the Snowball stemmer, so "nltk.stem" can also import 49
the "SnowballStemmer" and the nice thing with the Snowball stemmer is that if I come down here, copy this 50
line, comment this out, paste it in and substitute the "SnowballStemmer" here I can choose a language, for 51
example, yeah, English obviously, we can use the Snowball steamer with English, but if we go to the documentation 52
here from NLTK and we scroll down a bit to the Snowball stemmer, then what you'll see is that there's 53
other options too, right, there's Arabic, there's Finnish, there's French, there's German, there's Hungarian, 54
Swedish, Norwegian, quite a few, Romanian, like the list goes on, right, Russian, so you can have a look, I'll 55
put the link in the lesson resources, so yeah if you ever want to stem words and use this tool on text 56
that is not English, 57
the Snowball stemmer is your friend. Okay, so that pretty much covers stemming. The next thing that we'll 58
do to clean up the email text and the words is to remove the punctuation. Our spam classifier is not 59
gonna be very interested in the punctuation for the sentences. We can see at the moment, we still have 60
these full stops in our output and a exclamation mark and if we add question marks or anything else 61
it'll show up as well. 62
To remove the punctuation I'm going to copy this cell, paste it below and then also just quickly add a markdown 63
cell here to commemorate what we're doing. 64
So I'll say "Removing Punctuation" and then I'll delete a few of these comments here, format this slightly 65
differently. Maybe I add the odd question mark here, hit Shift+Enter and now we're ready to go. 66
Removing punctuation is, well I think there's like an easy way and there's a hard way and I'll show you 67
the easiest way you can do this. You see, Python strings have a fantastic method called "isalpha", so if 68
you've got a string say a single character, the letter "p" and you put a dot after it and then type 69
"isalpha()" just like so, then this will check if you've got a character or punctuation. In this case, the method 70
returns True, but if we had say a question mark and wrote "isalpha()", then this would return False. I'm going to move 71
these cells up slightly, so we've got them up here and I want to maybe pose a mini challenge to you. Can 72
you modify our code in this cell so that all these special characters here, all these punctuation characters, 73
full stops, question marks, exclamation marks get excluded from the output. What would you change in our 74
code here to accomplish this? I'll give you a few seconds to pause the video and then I'll show you the 75
solution. Did you have a go? What I would do is to modify this condition here. Not only would I check if 76
the word is part of the stop words, but I would also say that punctuation should not be included in our 77
list. So we can take the word, put a dot after it and say "isalpha()". This bit of code will only 78
return True if it hits an actual word, like "boy" or "adult". It will not return True for the full stops or 79
the question marks. Let's check it out if it works. 80
Surprise, surprise, I've planned out this tutorial and it, and it does work. 81
So there you go. 82
This is how you can use a built-in method from the Python strings to check for punctuation and exclude it 83
if necessary. In the next lesson, 84
I'm going to show you how to tackle the HTML tags in the emails. 85
I'll see you there. 86
Stay tuned.