In this lesson we're gonna go from a sparse matrix to a full matrix.
First I'll show you how to create an empty data frame that we can populate with values later on and
then we will write the python function that will do this transformation from sparse to full matrix.
Let me add a quick markdown cell here.
That reads how to create an M T data frame.
Now why might you want to do this.
Well sometimes you want to create the data frame first and then populated with values later on the key
things that you need of course with this is you need to specify what sort of column names you want how
many columns you want to have how many rows you want what the index names should be and perhaps also
provide some sort of dummy value for the cells right.
Do you want the cells to be equal to zero or none or 1 or something else.
Let's tackle one thing at a time.
Let's start with the column names.
So I'm going to create a variable that will hold onto all of these column and the school names.
It's going to be equal to square brackets.
Single quotes doc on the school I.D. When have our document I.D. for example as a column then we'll
have a plus and let's add the category as well.
So square brackets.
Single quotes category.
And then let's add our word ideas to go 0 1 2 3 4 all the way up to two thousand four hundred ninety
nine.
I'll pass this in as a range.
So list parentheses range parentheses zero comma vocab size our vocab size is saved as a constant up
top if you want to take a look at what the first five columns will look like they're gonna look like
this.
Column names square brackets colon five.
There you go.
So as you can see we've simply created a list that starts with two strings and then some numbers from
our range object.
Now of course the list is quite long right.
So alien parentheses column names shows us that we've got two thousand five hundred and two columns.
So two additional columns plus the number of words in our vocabulary.
Now for our index names.
So index underscore names we'll use maybe the unique values of the different document I.D. in our sparse
training data.
So I want to do this with NUM pi so NDP and I'm going to use the unique function from num Pi and I can
specify my sparse training data here and select one particular column of this name pie array with square
brackets colon comma and then zero for the first column index on the code names.
Now looks like this 0 1 2 and then the last three values are 5 7 9 1 5 7 9 4 and 5 7 9 5.
Now mind you these are not in numerical order of course because some of the documentaries of course
belong to our training data and other documents have been excluded as we've seen in the last lesson
of the previous module.
All right.
So with these two things in hand we can say create a data frame like so full on a school train on the
school data is equal to PD for pandas don't data frame parentheses and then we'll specify some arguments
here.
The index is gonna be equal to our index names and the columns of this data frame are gonna be equal
to our column names.
Fair enough.
Easy right.
So let me shift enter on this so I'll run for a little while and we can look at our head of this data
frame and see what it looks like.
So this is interesting right.
We've got any n not a number.
Nan values for all the cells.
What if we wanted this data frame instead to be filled with zero values as the default instead of these
Nan values we can do that with a handy little function called Phil and a so we can take our data frame
put a dollar for it and write fill in a parentheses.
Value equals zero.
This here is a method from the panel's data frame that allows you to facilitate a frame using a value
of your choice.
We will fill this data frame with the value 0.
And we're also gonna use this in-place argument here.
So instead of writing something like full on a train on it's got data is equal to full on a square train
underscore data don't fill in a we will use in place to just replace our existing data frame in place
is equal to true.
There we go.
So let's shift enter on this refresh this cell and then let's take a look at the head of our data frame.
What are instead of the nine values we have our default values equal to zero now.
Now let's create a full matrix from a sparse matrix.
So if imported a sparse matrix earlier from the text file and let's write a function that will create
our full matrix for us.
Now remember the structure of our sparse matrix was document 80 word i.e. label and then occurrence
of this particular word.
That the structure that we're working with.
These are columns 0 1 2 and 3.
So let's define a function make underscore full on a school matrix and this function shall have quite
a few arguments.
The first one of course will be the sparse matrix that serves as a starting point.
The second argument shall be the number of words an R on a score words in the vocabulary.
The third one maybe the document index.
So our document I.D. art index 0.
So I'll say doc on the school index is equal to zero.
So I'll just give that one a default value then I'll specify as well what the index is for the word
right.
So that way we don't forget the word index it's one that category index was two capped the score Ida
X is equal to 2.
And lastly are number of occurrences are frequencies frequency on the school index is equal to three.
This is gonna be the signature of our function.
Here's how I'd like it to work.
Melody Doc string with three pairs of single quotes and this will read um form a full matrix from a
sparse matrix and it's going to uh return a panda's data frame.
The keyword arguments that we've specified above are as follows.
So the first one was the sparse matrix right.
So this will be a non pi array then we're gonna have the number of words which was the size of the vocabulary
in other words the total number of tokens right.
The document index.
It's just the position of the document I.D. in the sparse matrix and by default.
It's the first column.
Now a lot of the description for the other keyword arguments as well.
And by then I'll have a brief description of what this function should do and how one should use it.
This is handy for ourselves who might be looking at this code in the future and not remembering how
it works exactly.
And it's also handy for anybody else working with our code and in Jupiter notebook when you press shift
tab the dock string is what you see pop up.
Now how we're gonna do this.
Well let's first start out with an empty data frame.
We can reuse some of this code that we've had above.
So let's just copy this line here.
Pasted in for the column names a copy.
This one here pasted in for the index names but I'm going to rename this instead to document I.D. and
it's got names.
I think that's a lot more helpful.
And we also gonna make sure that we replace spots on a train and a squat ditto with our parameter spouse
on a square matrix.
Otherwise we're going to have a big problem later on.
So there you go.
Then I'll copy these two lines of code and I'll come down and I'll add those in here.
Now one of the things that you always want to check in Python is the indentation here.
I'm working within the body of my function but here I am not.
So when I move this over a bit so that I'm within the body of the function and then I'm going to have
to change my index names to document I.D. names.
After all we've renamed this on the line above.
And then at the end I'll return this full matrix right here in between these two lines of code.
We're gonna be doing all our work we're gonna be populating this full matrix.
Here's what I've got in mind our sparse matrix kind of looks like this right.
We've got four columns and they're labeled document I.D. word Daddy category and occurrence.
Now what I'm after for the full matrix is something like this.
I want the document is in one column.
I want the categories in one column but then I want the word I.D. as separate columns and I want zero
values for all the words that don't occur.
In other words the frequencies or the occurrences of the individual tokens will of course be in the
rows below the word I.D..
Here's a fuller picture of the data frame that we're going to populate every single token has a separate
column in this data frame.
These columns go from zero to two thousand four hundred and ninety nine because we've got two thousand
five hundred words in our vocabulary.
Now of course many many of these columns will be equal to zero.
But what we're gonna do is going to go through our sparse matrix row by row by row by row and we're
going to populate the entries of our full matrix which are non-zero with the correct occurrence.
So without further ado let's get started the way we're gonna go through the sparse matrix is with a
loop.
Now I'm quite partial to the for loop so for I in range.
It's gonna go from zero to the number of rows in the sparse matrix so that's going to be sparse matrix
dot shape square brackets zero colon at the end.
Now let's grab the data that's in each individual row our document I.D. or our document number if you
will.
It's gonna be equal to sparse matrix square brackets.
I so the eighth row and then it'll be the first value so it'll be zero.
The word I.D. is gonna be sparse matrix square brackets.
I square brackets one right.
Our label.
The category will be the sparse matrix square brackets.
I square brackets too.
Now what you can see is that for these hardcoded values we can actually use the parameters that we specified
above.
So far our document 90 I'll replace the 0 with Doc on a school I.D. X for our word I.D. I'll replace
this one with word on the school I.D. X and for our label of replace this with Cat on a school I.D.
X the occurrence is of course equal to spots on the school matrix.
Square brackets I square brackets frequency I.D. x these four lines of code here.
Store all the values in a particular row in four separate variables.
If you'd like to see this square bracket notation in action I can show you what I'm planning to do up
here.
So if I take our response training data and say we look at the rows between I don't know 10 and 13.
In that case we'll see these values printed out here.
What we're doing with our sparse training data our no higher rate and the square bracket notation is
we're going into a particular row with say 10.
So this will be the value of AI.
And then for that second pair of square brackets we're selecting a particular value.
So if that value is zero we're getting the document.
If that value is equal to one we're getting the word I.D. for it to work getting the category and for
3 we're getting the frequency.
This is all we're doing inside of our loop makes sense now that we've extracted this data from a particular
row we can come down here and we can populate our full matrix with data at a particular cell.
So full matrix dot at square brackets document no comma single quotes document I.D. and single quotes
and square brackets.
It's gonna be equal to our document
fool on his call matrix at square brackets.
Document no comma single quotes.
Category is gonna be equal to our label and full on a school matrix at square brackets.
Document no comma word I.D..
It's gonna be equal to the occurrence.
Now if these three lines look a little bit mysterious what we're doing here with dot at is we're selecting
a particular cell in this data frame that we've created which cell that we select.
Well second value here is the column name.
So this one here will always go to the document I.D. column.
This one here will always go to the category column.
Right.
And this first part here is the row number.
So the row number will correspond to the document i.e. from the sparse matrix.
So we're selecting a single cell here and then we're setting its value.
The document I.D. of course has to go the document and column the category of course has to go in the
category column and the frequency of a particular token it's gonna go under the word I.D. of a particular
document in this data frame.
And this pretty much completes the body of our loop we're grabbing the data from our sparse matrix and
we're populating our previously empty data frame inside our loop.
The only thing left to do is maybe set the index right.
So full matrix don't set underscore index it's gonna be equal to the document that column single quotes
doc on a school I.D. and as a second argument I will provide in place is equal to true.
Okay.
So let's shift enter on this and try this out I'll include our benchmarking code here with.
Percent percent time in the cell below and then I'll create a variable which we'll hold on to our full
matrix full honest quatrain on a score data.
This is gonna be all our training data.
And when I make a function call to make full matrix and in the parentheses we'll provide our sparse
training data and our number of words.
So our vocabulary size vocab on a school science.
Let me hit shift enter on this.
This should take 10 to 15 seconds to run.
There we go.
Now let's take a look at the head of the state of frame.
See what we've got full on as Katrina was called data don't hit.
So the good news is that this is kind of what I've expected to see.
I've expected to see far more occurrences for the frequent words with word idea 0 1 2 3 4.
Then I see for the less frequent words with the word that he's close to two thousand five hundred and
this is the pattern that we should also see in the tail of our data frame brilliant in the next lessons
we're gonna start training our naive pays model in particular this means calculating the probabilities
for the individual tokens.
This is where all the probability theory that we covered in the previous module comes into play.
I'm pretty excited about this so I'll see you there.