Hello and welcome to this art tutorial.
So we did the main steps.
We cleaned all the texts, all the reviews.
We created our backwards model and now we have to do one more thing,
which is of course to build our machine learning classification model.
And we can do that because we have all our independent variables
in this sparse matrix DTM here builds thanks to this function Documenter matrix.
And besides, we applied a filter here to remove the non frequent words.
Well a few of them.
But still this considerably reduced the number of words in the matrix.
So that's always good for our model to run faster.
All right. So now let's build the model.
So what we'll do is go to our files to get back to the part three classification.
Because what we'll do of course is take a classification model
that we already built.
And we will apply it on our text here.
Because out of this text, we managed to create a matrix of features
containing the independent variables.
And of course we have
one dependent variable which is the second column of our data
set the liked column, which tells if yes or no.
The review is positive.
So we have
everything and therefore we only need to take our model now.
And therefore we go two by three classification.
And here we can find all our classification models.
All right so which one to pick.
Which one to choose for natural language processing.
Well in general based on experience, the most common classification models used
for natural language processing or Naive
Bayes decision tree or random forest.
You also have the cart model, which is another type of decision tree model,
and you also have the maximum entropy model, which is based on entropy as well.
Like for decision trees.
So these models work very well for natural language processing.
And therefore he will pick one that is related to entropy.
And that's the case for decision tree classification model
as well as our random forest classification model.
Because of course
a random forest is a combination of trees making the same predictions together.
And keep in mind that you can also use Naive Bayes,
which is commonly used as well for natural language processing.
But here in this tutorial we will choose random Forest classification.
So let's go into this section and here are all the files.
You know the data set the classification templates and our model in Python and R.
So let's take the one in R.
So I'm just clicking on the file. Here we go.
Model open here.
So what do we need here.
Well first of all let's notice that you know when we are using the random
forest classification model we are starting with a data set
which is a data frame.
And that contains both the independent variables and the dependent variable.
So what do we have to do right now is go back to our natural language
processing file and create exactly the same.
That is, create a data
set containing independent variables and one dependent variable.
And that will be the input of this model here.
Because you know we have our data set here.
And then we use our data set in each of the code sections here.
And then you know, we split the data sets into a training set and a test set.
And we train our machine learning classification model on the training
set here.
So what we only have to do is just to create this data
set containing the independent variables and the dependent variable.
So that's very simple.
We already have our independent variables.
But the problem is that our independent variables right now are in a matrix.
Because you know this document term matrix function returns a matrix.
So DTM is a matrix right now.
And as you remember in our classification
models on R well this data set is a dataframe.
It's not a matrix.
So we need to make sure that here for the inputs of the model
that we are going to apply on this bag of words model that we just created
in the previous tutorials.
Well we need to make sure that we have a dataframe,
but that's actually very simple.
We just need to take our matrix and use the function
as that data dot frame.
And we will input our sparse matrix DTM inside.
And that will transform our DTM sparse matrix into a data frame.
So let's do this.
And since you know we are just going to copy paste
our random forest classification
here, well, since the input of this model is basically this data set,
well here we will use the same name to create this data frame.
And so we will call it data set.
Data set equals.
And then that's when we use the as dot data dot frame.
Here it is. That's the first one.
Here we go.
And so now we need to input the matrix.
We want to transform into a data frame.
And that's of course DTM.
And just to make sure we have the matrix type
expected by this as dot data frame function.
Well we need to use here the function as dot matrix
and put DTM as input of this as matrix function.
Because you know this sparse matrix DTM here is definitely a matrix,
but it doesn't have the type expected by this as the data frame function.
And to make sure we have the right matrix type,
well we need to use this add that matrix function.
All right.
And now let's be careful.
We lost one parenthesis.
So I'm just adding it. All right. Now we're good.
We are ready to transform our sparse matrix of features into a data frame.
So let's do it I'm going to select this line and execute.
All right.
And now what's interesting to see is that we have the real data set.
You know with all the reviews and the rows and all the words that we took
from the corpus.
And then we're filtered thanks to this remove sparse terms function.
Well, we can see the full data set here with this 1000 rows
and all these 691 columns,
each one corresponding to a word that comes from the reviews in the corpus.
And that was not filtered by the remove sparse terms function.
All right.
So here you can have a look at this huge table.
And we can clearly see here
that this is a sparse matrix because basically we can only see zeros.
Well we have very few ones.
We have one here one here.
But all the rest is zeros.
And so for example if I take this one here,
well this one belongs to the also column and to the 23rd row.
That is the 23rd review.
And so this one here means that the word
also appears in the review 23.
All right. So that's the sparse matrix.
And now you can really see what it is with your own eyes.
All right.
So let's go back to our natural language processing file.
So we have our data set which is now a data
frame as we wanted but still incomplete.
You know why.
It's because the data set we start with in this random forest classification model.
And in general we say classification models is a data frame.
So we're good on that.
But a data frame containing both the independent variables
and the dependent variable.
So what we need to do right now
is add the dependent variable to this data frame data set.
Because right now it only contains the independent variables.
All right.
So you might remember how to add the dependent variable column to a data set.
That is a data frame.
Remember we need to take our data set.
Then add a dollar sign here.
And then after this dollar sign we can either take one of
the existing column here if we want to update the column,
or create a new column to add to this data set.
And that's exactly what we want to do.
We want to create a new column to this data set.
Well that's an existing column.
That's the light column.
But we created for this data set because it is new column.
And so we'll give to this column the same name as the real dependent
variable column. That is light.
All right.
So by doing this we are adding this new column that we call liked
and then equals.
And then after this equal
we need to specify what we want to add in this new column.
And what we want to add is nothing else than the existing
liked column of our data set.
But be careful, because our data set was just a data to this new data frame,
and therefore we no longer have the data set that we imported originally.
So what we'll do is very simple.
We'll just rename this data set by adding an underscore and then original.
Here we go.
And we will select this line again and execute.
All right. So now we have our original data set.
And therefore we can have access to the liked column of this
original data set which is going to be our dependent variable.
So let's add this dependent variable right now to our data set.
And so to take this dependent variable we need to take our data set original.
Here it is
because that's the original data set containing the dependent variable liked.
And so to take this dependent variable vector
we need to add a dollar sign here same.
And then take the column we want which is the liked column.
All right. So that's good.
By selecting this line and executing it
we add the light dependent variable vector column
to our data set already containing the independent variables
that are all the filtered words of our cleaned reviews in the corpus.
All right. So now we have everything we need.
And we are ready to take our machine learning classification model
because we have our data set that not only is a data frame,
but also contains both the independent variables and the dependent variable.
So we have everything.
What is expecting a random forest classification model here?
So what we only need to do here is take everything from here and not from here.
You know, because this section is to import the data set.
But we already have our data set that is ready for classification model.
So we just need to take everything from here because this is where
the data set starts to be processed.
And so we take everything from here to
here.
And we can not take this because this is to plot the results in 2D.
That is two independent variables.
And here since of course we have a lot more than two independent variables.
Well, we cannot use this to plot the results,
but we will definitely have a look at the confusion matrix
to see the number of correct predictions,
as well as the number of incorrect predictions,
so that we can evaluate the model performance.
All right.
So let's get back to our natural language processing file.
And we will paste our random forest classification model right here.
All right.
So now we just need to modify
a very few things because everything is basically ready.
But let's see what we can modify.
Well first here in the section that encodes the target feature as vector.
Well of course we need to replace this purchased dependent variable
which was the dependent variable in part three.
Well we need to replace it with our new dependent variable which is liked.
All right.
And same here we replace purchased by like
all right good for this section.
Then in the next section we split the data sets into the training set
and the test set.
Well that's very important to do this.
Unless you want to create a new review.
But you know we will train our random forest classification
models on say for example, 800 reviews.
And we will test the predictive power of random forests on 200
new reviews on which our random forest classification model was not trained.
And therefore these 200 reviews
and the test set will be new reviews for a random forest classification model.
And so we will see how it manages to predict
whether each of these 200 reviews is positive or negative.
And then that's in the confusion matrix that will see
the number of correct predictions and the number of incorrect
predictions in this 200 new reviews.
All right. So that's what is done in this section.
And since I just gave as an example 800 reviews to train the model
and 200 reviews to test it, well let's go with this choice of numbers.
And so we need to change the split ratio here to 0.8 because that's 80%.
And we have 1000 reviews.
So 80% of 1000 reviews is 800 reviews to go to the training set,
and therefore 200 reviews to go to the test set.
All right. So that's good.
And of course, let's not forget to replace
the purchased variable here by our new dependent variable.
That is light.
All right.
So I think we're good with this section.
So now let's move on to the next one.
The next one is about feature scaling.
And so here do we need to apply feature scaling.
Well not really
because we only have zeros and ones in the sparse matrix of features.
And therefore we don't have one
independent variable dominating another independent variable.
So we don't need to apply feature scaling.
So we will remove this section.
All right.
And so what about this one.
Yes of course we keep this one because this is the section where we build
a random forest classification model that will classify the reviews.
And that's
where we train the random forest classification model on the training set.
And therefore here we need to change two things.
First, the index here that you know is the index of the dependent variable
that we need to remove from x because x is supposed
to be the training set without the dependent variable.
So we need to remove it with the index of our new dependent variable
like it is not three but is 692.
We can see that here very easily.
So let's replace three by 692.
All right. Good.
And now the second thing that we need to change is of course this purchased here
that we still need to replace by light
this way.
And then if we want we can train our random forest
classification with more trees.
Right now we have ten trees.
So we will keep ten trees.
That might be enough for our 1000 reviews,
which is quite a small number of reviews, and especially our 692
words columns that we have in our sparse matrix of features.
Ten trees might be enough, but of course, you're welcome
to try more random forest classification models with more trees.
So we're good with this section.
And now let's move on to the next one.
The next one is about predicting the test results.
So making the predictions on 200 new reviews
that our model won't know anything about.
And therefore for this new reviews, our model is going to try to predict
if those reviews are positive or negative and therefore it will be very interesting
to see if it's making some correct predictions on new reviews.
So right now it's the same.
We have to replace this index here that corresponds to the index
of the dependent variable.
And so we need to replace three by of course 692.
That's exactly the same as we did for the training set here.
And so now we're good for this section.
We're finally getting to the last section that is making the confusion matrix.
That's the interesting section that will tell us the number of correct
predictions and the number of incorrect prediction for these 200 new reviews.
So we will see that right now.
But of course we need to replace this three index that corresponds
to the index of the dependent variable still the same.
And replace it by 692.
All right.
So now everything is good.
We are ready to train our random forest classification model
on our 800 reviews of the training set.
And then evaluate the predictive power of our model
on our 200 new reviews in the test set.
So let's do it.
Since we already executed everything up to here, what we need to do now
is just select everything from here to the bottom.
And now we're good.
We just need to press command or control plus enter to execute to train the model
and test it on the test set,
and eventually have a look at the number of correct predictions and the number
of incorrect predictions on 200 new reviews.
So let's do it.
I'm going to press Command Plus Enter to execute.
And here we go.
Everything worked properly. Great.
So let's have a look.
We will have a look at the confusion matrix.
Of course by typing here c m in the console.
Here we go. So let's see what we have.
We have 79 correct predictions of negative reviews, 70 correct
predictions of positive reviews, 21 incorrect predictions of negative reviews,
and 30 incorrect predictions of positive reviews.
All right, so that's actually not too bad.
You know, because we only had 800 reviews to train the model.
That's not much when you're working with text.
And therefore 30 plus 21 equals 51.
Incorrect prediction is not bad out of 200 new reviews.
When you know that you train your classification model on only 800 reviews.
And actually, let's have a look at the accuracy.
The accuracy is the number of correct predictions that is 79
plus 70 divided by the total number
of observations in the test set, and that is 200.
So let's have a look at the accuracy.
Pressing enter here.
And the accuracy is 74.5%.
So again
that's not bad considering the fact that we trained our model on only 800 reviews.
And you'll clearly see that if you had a lot more reviews to train
your classification model, you will get a much better accuracy.
All right, so that's the end of natural language processing
and are congratulations for having completed all this.
Creating the Bag of Words model training and classification model.
And this data set.
But that's not the end of your natural language processing journey.
Because right after this video you'll get a little challenge.
So we'll let you find out about that.
And until then, enjoy machine learning.