All right.
So in this lesson, we're going to write the pipeline functions for e-mail processing.
Let me add a markdown cell.
Just add a quick section heading here and save functions for e-mail processing.
I'm going to create a function called clean up a school message.
And I want this function to eventually return to me.
A list of filtered words.
So I'll create an empty list filtered on the school words.
And then we're gonna write some code.
And then at the end, we want this function to return.
This list of filtered words for us.
Now, what sort of inputs should this function have?
Well, the first one should be a.
E-mail message.
E-mail body.
We can then use this message.
And first of all, converted to lower case.
So I can say message dot lower.
This will convert all the contents in the message that are being passed to this function to lower case.
And we can also tokenized the message.
So word on the school tokenized parentheses message dot lower.
Well, tokenized all the words in our message.
So this is kind of a review of a lot of the previous steps that we've taken.
So let me create a variable called words and set that equal to the result of all this work that's taking
place.
And this line of code words will, in fact, be a list of all the individual words in the email body.
So this means that we can iterate over this list of words, right, so we can see for word in words,
semicolon.
And then inside the loop, we can do some similar kind of work that we did before.
We can remove the stop words and we can remove the punctuation.
If the word is not in stop words and the word.
It's not punctuation.
In other words, word dot is alpha parentheses, then stem the word.
So we'll use our Stemmer and stem the word.
Once we're happy with that, we can take our our list of filtered words.
And upend the word from the stemmer.
Now we'll actually want to make this function a little bit more independent from the previous cells
so we can say the stemmer is going to be equal to the Porta Stemmer.
What we're doing here is making Stemmer an optional argument and then our stop words are going to be
equal to.
The set of stop words, words.
From Inglish.
Now, we've created a function where we can swap out the Stemmer and swap out the list of stop words
if we wanted to.
This bit of code here converts to lowercase and splits up the individual words.
This bit here removes the stop words and punctuation.
Now, let me hit shift, enter and try out this function a little bit earlier on in the project when
we were learning about reading files.
We had this example e-mail and we saved this e-mail text in a variable called e-mail body.
Let's try it.
Our clean message function on this e-mail body right here.
So coming down, I've got a clean message, parentheses and then email underscore Bonnie.
Plum hit shift, enter and see what we get.
We get an error.
And that's because I've got a typo right here.
Here we're dealing with the filtered words variable.
But here I've left out the S.
Let's try again.
So I'm going to shift enter on this hand, hit shift and her on this.
And here's our output.
The entire contents of the e-mail are tokenized and also stemmed.
So let me quickly copy this cell.
Come down here, pasted in and modify the name of our function.
I'm going to call it clean message underscore no, underscore h t Amelle.
I want to pose a challenge to you.
I'd like you to modify the function that we've just written to also remove the H.
Timal tags.
And then I'd like you to test this function on the email with document I.D. number two.
Namely, this e-mail right here.
So pause the video and give this a go.
Did you figure it out?
Here's the solution, quickly add a comment, remove HD him all tags, and then I'm going to use beautiful
soup.
So I'll say soup is equal to.
Beautiful soup parentheses.
And then what?
I have to provide two arguments, right?
The first one will have to be the e-mail body, which will be our message.
And then I'm going to select the team, El Paso, as my default Partha.
To remove all the tags, I'll say soup.
Don't get an escort.
Text parentheses, but I'm going to store the output in a variable as well.
I'll say maybe.
Cleaned and a school text is equal to soup, don't get taxed.
And then what I'll do is instead of saying message to Laura case, I'll see cleaned text to lower case.
And if I hit shift, enter now.
I can come down here quick in this cell and hit tab on my keyboard, bring up clean message.
No, in Tamil and then in the parentheses provide data, dots and square brackets to karma and single
quotes message.
And if I hit shift enter now, we should see just a list of stemmed words and no H.M.S. tags.
Wonderful.
Now, we've successfully cleaned and tokenized a single email from our dataset.
Now it's time to apply the cleaning and tokenization to all the 5800 messages.
And that's what we're going to be working up to in the next lesson.
I'll see you there.