Hello my friends.
Welcome back.
Are you ready.
To finish and complete that cleaning of.
All our reviews?
Let's do this.
So just to recap on what we did so far.
We made this for loop to, you know.
Treat each review separately.
And clean each of them. Separately.
And the first thing we.
Did for each of these reviews was to replace all these. Non.
Letters by spaces and basically all.
The punctuations by. Spaces.
Then we turned all the letters.
Into. Lowercase.
So all these capital letters were. Turned into.
Lowercase. And finally.
We splitted each review.
Into its different words. Because now.
We were about to proceed with.
Stemming, which consists.
Of. Simplifying.
Each word by the root of the word.
So for.
Example. Loved will be replaced by love.
And I remind that it's very important to.
Do this to.
Optimize the dimensionality of the sparse matrix.
Which will be created through. The next.
Step with the. Bag of words model.
Indeed, we will create that sparse matrix where each.
Column is. A. Word among all the.
Different words of our different reviews after.
The cleaning, and therefore we need to simplify it to the maximum.
Well, the different. Words that are going to be.
In this sparse matrix.
So it's very. Very important to apply stemming.
And that's. What we're about to do. Right now.
All right.
So remember we imported this.
Porter Stemmer class.
That's exactly the class that will apply stemming.
And so now the first thing we'll do is.
Actually. Create an object.
Of this class, which we will call p s for, you know, for a steamer. And.
Forest Emma is a particular type. Of stem here.
You can check it out online. But that's. Basically the classic way.
Of applying stemming.
So and now.
We will just call the class Porter.
Stem nerd. Perfect.
Then adding some parentheses.
And we don't have to enter any parameters.
So let me actually. Scroll down a bit.
All right.
This will be easier for you.
So Porter. Stemmer now we. Have our object.
So basically we have our tool to apply the stemming.
And therefore well.
The next. Step will be exactly to apply.
Stemming to. This particular review.
We're dealing with right now inside.
This for loop iterating all the reviews.
So the way we're going to do.
This because you know, we have to apply stemming to.
Each of the words. Inside. This review.
And the words have already been gathered separately by splitting this review.
And therefore what we're going to do now.
Is first, we'll update our review.
Again to become this time.
A list of all these separate words.
Coming from the. Split.
But stepped, you know.
After we applied. Stemming on them.
And so now the question is, how are we going to apply stemming to.
Each of the words of the review?
Well, we're going to do that, of course, with a single row for loop,
as I like to call it, you know, a full loop.
Inside a same row.
Which is, you know, usually applied to lists.
And since now.
We actually have a list of. The words in the reviews.
Thanks to this.
Steps. And here, well, we can totally apply.
This single row for loop to iterate through all the words of the review.
And apply stemming to each of them.
So let's actually start with this for loop.
We're going to start with four which will iterate through all the word you know.
So word is just a iterated variable that will be equal to the.
Different words of. The review.
So for word.
Then in.
Review right.
Because review is now a. List of different words.
So it will iterate through all the different words of this review.
And then we will add something which.
Is probably. Something. You will see for the.
First time if you're. New to Python.
But here, after a single row for loop.
We can. Actually.
Add a condition to omit.
Some of the words we are. Iterating through.
And according. To you, what. Do we want to emit?
You know, what do we want not to include in the full loop?
Well, that's of. Course remember them.
The stopwords.
Right. Remember, we want to get rid.
Of all the words that.
Won't help us or, you know, that won't help the machine learning model
to understand whether.
A review is positive or negative.
And these include.
All the articles like the, Unlike an.
Apple, we would just keep Apple, for.
Example, and even all the pronouns, you know.
Like I, you, he she.
We, you know, then.
They all these words give us.
Absolutely no hint to.
Figure out whether review is positive. Or negative.
And therefore we want. To get rid of all the stopwords.
And that's again for the same.
Purpose as. Stemming.
It is to simplify the final sparse matrix, which will contain
all the different words of.
All our. Reviews into columns.
So we not only want to simplify the different versions of words
or, you know, branches of words by only keeping their root. But.
Also we want to get with.
All the stopwords.
In order not to include them in the sparse matrix.
And a way to do this, you know, the way to omit.
These words.
When iterating through the different words in the review.
Is by just. Adding. If, here.
And then and not, you know you will understand after I finished this.
So let me just write it.
If not, word in.
The set up.
Then I'm going to.
Call the Stopwords module here.
You know this stopwords here.
From which I'm going to get all the English.
Stopwords because. You know, our reviews are in English.
And the way to. Do this is to add here words.
And then in parentheses because that's a function.
And then in quotes English okay.
So we clearly. Understand what this means.
It means that. If the word of the review.
You know we're dealing with right now in this.
For loop, if this word is.
Not in the set, you know, the symbol.
Of all the English stopwords. Well, we.
Will consider it in this for loop.
And then we will. Apply the. Stemming.
However, if this word is. In all.
These stopwords list, well.
We won't include it in this for loop and therefore we won't apply the stemming
to. It.
Therefore it won't. Be in the future. Sparse matrix.
So you see the trick. Pretty cool.
But so.
Far we only have the for loop and we don't know what we're going to.
Do for each of these words. So we have to specify it now.
And remember we.
Have to specify this before the for loop because that's a single row for loop.
And so.
What we're going to specify is of course.
That we want. To apply. The stemming.
To each of these. Words in the review, which are.
Not just upwards.
And so now the question is.
How do we apply. The stemming.
Well remember we. Just created our.
Border stemmer object. So we're going to start.
With this calling our object.
From which we're going to call specific method.
Which is exactly the.
Stem method. That will apply. The stemming.
And since it is method I'm adding some parentheses.
And now I'm sure you. Guessed perfectly.
What's inside these parentheses.
You know what must be inside.
And that of course our word right.
Our word on which we want to.
Apply. Stemming.
And now there you go, my friends.
All this line of. Code apply.
The stemming to.
All the words in your review except the stopwords, except all.
The non relevant words that are not.
Helpful to predict whether a review is positive.
Or negative. Great.
So that's a good step. Done.
Now you know how. To apply.
Stemming to. A review.
And now since.
We actually have our.
Different words in the review into a. List, we will just.
Join them back.
Together because, you know, we actually had to do. This in order to.
Easily apply. The stemming to all. The words.
Thanks to this for loop, but now that it's done, we will.
Just join these words back together to get the.
Original format of the review, you know, as a string.
And therefore the.
Way to do this is to.
Well, you know, update.
Our review variable. Again.
By setting it equal to, well.
The junction.
Of these different words after we applied.
Stemming to them.
And the way to. Do this is.
By calling the join function, which takes as input.
Exactly. Well our list. You know, the. Review.
Which contains a list of the different words after stemming was applied to them.
And however, if we. Join them, this.
Way, this. Will, you know.
Stick the words together.
And we will end up with one word that makes no sense.
So in order.
To separate these words by a space.
You know, as in a classic string or text, well, we just need to add here in quotes.
And then inside the quotes we add. A. Space.
And then separating this.
With the join with you by a dot.
All right. So this will. Not only join your.
Review but also adding space between each word of your review.
All right.
So good. Now you know how to. Do this as well.
And finally we have. A final step here.
I'm sure you would guess what it is. Otherwise.
Please press pause to just.
Finish. This. Cell implementation.
But that's pretty obvious.
All we did here.
Was a transformation on a single review.
And thanks to this for loop.
Now all.
The different reviews of our data set will.
Be transformed and will be cleaned and essentially will.
Be. Simplified.
But now remember.
We have this corpus here that.
Was initialized as an empty list.
And we want to add. All.
These cleaned reviews into.
The corpus, because that's what will be expected.
By the method of the.
Class that will create the.
Bag of words model.
And therefore we must absolutely now.
Add each review to the corpus.
And the.
Way to do this is by taking our corpus list.
And then.
Call. The append function to append.
Indeed the review. And that's it my friends.
We are done with the cleaning of the texts.
So now we're going to.
Execute. This. You know I think we deserve it.
That was quite a. Long cell implementation.
So let's do this.
However, my run time.
Actually stopped so I have to restart it.
By clicking, you know, runtime and then. Restart.
Runtime. I will execute the two first cells.
And then this one.
And we'll make sure that everything works correctly.
All right.
All good.
Here let's do this.
Let's first re-import the. Libraries.
Done. Import the. Data set.
Done as well.
And now let's see.
And let's hope we. Didn't make any mistake.
Let's play the cell. Cleaning.
All the texts, all the reviews and.
It seems to be all good. And yes.
This is what we were supposed to get. Perfect.
So here it downloaded the, you know.
Stopwords package from this.
Path containing all the Nltk data.
And also it's unzipped.
This zip file. Containing.
All the stopwords and perfect.
And then of. Course, it applied all. This cleaning process.
To each of the reviews.
And adding them to.
The final corpus. Perfect.
So now now it is time for the next essential step.
When doing sentiment.
Analysis,
it will be to create the bag of words model, which will consist basically of.
Creating this sparse matrix containing now all.
The words of the reviews after they. Were cleaned.
So we will take all the different words of all the reviews.
We will put them in different columns of the sparse matrix, and then that will.
Be actually our future matrix of features, which.
We will combine to the dependent variable vector containing all the binary outcomes
to train our future machine learning model, which will be the Naive Bayes
model, to learn the text and understand whether the.
Reviews are positive or negative.
So I can't wait to proceed to this next step.
Creating the.
Bag of Words model with.
You and until then, enjoy machine learning.