Welcome back. 1
In the next couple of lessons I've got a really exciting topic for you. 2
We're gonna be talking about NLP, natural language processing. Natural language processing is a huge 3
field. 4
It used to be a subfield of artificial intelligence actually, but it moved more and more into the domain 5
of machine learning and NLP is big business too, all sorts of things fall into the domain of natural 6
language processing. 7
For example, it covers things like search, sentiment analysis of tweets or reviews, Google AdWords, automatic 8
translation, spellcheck, auto-correct, Siri, Alexa, you name it. As you can guess, NLP is what most of Google's 9
earnings actually depend on. 10
Now how are we going to use NLP for our naive Bayes' classifier? Well, we're gonna use it to prepare 11
a piece of text for our learning algorithm. 12
We have to convert our email bodies into a form that the algorithm can understand and this means pre- 13
processing our text. 14
Now what kind of things do I mean by pre-processing? 15
Here's the high level overview. 16
First off, we're gonna start converting all our text to lower case. Second, 17
we're gonna tokenize our text, meaning we're gonna split up the individual words in a sentence. 18
Third, we're gonna be removing the stop words. By stop words I mean very common English words, like the 19
word "the", which is there to convey grammar rather than meaning. Next we're also going to strip out the 20
HTML tags that are in the emails. 21
A lot of the emails are not written in plain text, but contain a lot of HTML formatting which we're 22
not going to feed into our algorithm. 23
Next, we're gonna do some word stemming and that means converting the individual words to their stem 24
word. 25
So for example, if you have the words "going", "goes" and "go", then all of these words actually share the same 26
word stem, it's only really the grammar that changes their spelling. By stemming the words we're able 27
to treat them all as the same word. 28
And lastly, we're also going to remove the punctuation and that is because as you can tell our Naive 29
Bayes' Classifier will ignore the grammar. 30
Now without further ado, let's get started. 31
All right. 32
So I'm going to add a few markdown cells once again in Jupyter, so that we can find this section really 33
easily when we're scrolling through it. 34
So I'll call the first heading "Natural Language Processing", with two s's not three and then I'll add 35
a subheading that reads "Text Pre-Processing". 36
Now the first step is normalizing the casing of the letters. 37
Very often the case of the words should not matter. If I search for "what is the airspeed velocity of 38
an unladen swallow?", then if I were to type "wHaT iS thE AirSPEed VeLocITy of An UnLaDen SWaLloW?", 39
now, now even though that is horrible to read, the answer to this vitally important question should not 40
depend on the upper or lower casing of my letters. 41
And you can also verify at home that when you type in a search query, Google completely ignores the upper 42
casing of your letters. The casing of the letters doesn't affect our search results. 43
And similarly for our spam classifier we will treat the words "loan" or "Viagra" the same way regardless 44
whether they're spelled with uppercase letters or lowercase letters. So coming back to our Python code, 45
suppose we have a message. 46
We have some sort of string that reads "All work and no play makes Jack a dull boy.". 47
How can we convert all of these letters to lowercase? 48
How can we ignore the casing of the words in this string? Well, Python strings have a handy little function 49
called "lower()", so "msg.lower()" will actually convert all the letters in the string to lower 50
case, so you can see that "Jack" becomes lowercase and the word "All" also becomes lowercase. So converting 51
to lowercase is one kind of text pre-processing that you can do. Now for a lot of other pre-processing 52
that we're going to do, we're going to use a Python package called The Natural Language Toolkit or 53
NLTK. The Web site for this module looks like this, it's on nltk.org and this is actually 54
a package that almost every professional in the NLP field will use at some point for their natural language 55
processing needs. 56
The NLTK package can do a huge number of things and we're gonna start using it with some of the fundamentals, 57
namely pre-processing our text, so that our machine learning algorithm can use it. 58
Now since we're gonna be using the NLTK resources, I'm going to add a very quick section heading here, 59
to "Download the NLTK Resources" and those include something called a Tokenizer and a list of stop 60
words amongst other things. But before I do that, I'm going to import the package itself along with a 61
couple of the tools. So I'm going to come up here to my notebook imports and then I'm going to say "import nltk" and 62
then from nltk.stem we're going to import the "PorterStemmer", 63
from nltk.corpus 64
we're going to import "stopwords" 65
and from nltk.tokenize we're going to import "word_tokenize". 66
I think this will do for now. I'm going to import the package as a whole 67
and then we're going to import three additional pieces of functionality - the PorterStemmer, stopwords 68
and a word tokenizer. So I'm going to hit Shift+Enter on this cell and then scroll down here to our section where 69
I'm going to show you how to download the NLTK resources. And this is what we're going to do in 70
the next lesson to tokenize our words. 71
I'll see you there.