All right.
Welcome to a new module.
Over the course of this module we'll be training our base classifier.
But the first step is setting up our notebook and loading our data in your projects folder.
Click on new Python 3 notebook and let's give this a notebook a name let's call it 0 7 bayes classifier
hyphen training to set up this notebook.
Let's add a few markdown cells.
The first one will be notebook imports and the second one will read constants under the notebook imports.
Let's import pandas as PD and let's import num pi as NDP we always need these two guys under the constants.
You can probably copy over some of them from the previous notebook so in particular I'm thinking of
our two file paths and our vocabulary size.
So when a copy this over and pasted in here and also add our vocab underscore size which was two thousand
and five hundred.
There you go the very first thing we're gonna do is load our features from the two text files into a
num pi array.
I'm going to add a markdown cell here to commemorate this read and load features from t x t files into
num pi array.
The way we're gonna do this is we're gonna use the load P T function from num Pi.
Our training data comes as a sparse matrix so I'll store this in a variable called sparse on a school
train on a squat data and that's equal to NDP dot load t x t parentheses training data file comma delimiter
single quotes and a space comma data type or D type equals.
And so what I'm doing here is full I'm giving everybody a chance who hasn't done the data pre processing
to come in at the training part of the project.
And I'm doing this by calling a function from num pi called load text and I'm supplying three arguments
to this function.
The first one is my relative path to my data file.
We created this data file in the previous module but you can also download it separately the second
thing is I'm adding a delimiter.
Now if you haven't heard this word before a delimiter is a character that's used to specify the boundary
between independent regions of plain text or data.
We are using a single whitespace as our delimiter and you can see this if you open your text file in
a text editor.
You can see that the single whitespace It's what's used to separate the different values.
The first value here is our document.
The second value here is our word I.D. the third value here is the category or label.
So 1 for spam 2 for non spam and the third value here is the occurrence the number of times a particular
word with say I.D. 0 occurs in the email with document I.D. 0 the whitespace separates these different
kinds of data.
Now we can open our word by I.D. dot CSP file to give you an example of a different kind of delimiter
CSP of course stands for camera separated values.
So there you can see that the delimiter is well surprise surprise a comma in this file.
We have the word Eddy as the very first value in every single line and we have the string.
What the site stands for has the second kind of value.
If you were to open this kind of file and Microsoft Excel or some other spreadsheet program you'd see
that all of these values get put into separate columns.
So that's how it delimiter works.
Lastly I supplied some information about the kind of data that we're importing him.
We're exclusively dealing with whole numbers.
Right.
Our document adds our whole numbers are words our whole numbers our categories our whole numbers and
the number of times a particular would cause is also a whole number.
So I've set the data type as an integer.
Let me shift enter on the cell.
You'll see that it runs a little while and let's import our test data as well so sparse on the squat
test on the score data is equal to N p dot load t t and then it'll be test data file which is our relative
path to our test hyphen data that t XY and you also have the same delimiter single space and it also
has the same data type an integer.
So now we have to num pi arrays.
Let's see if we loaded them successfully by we're just looking at the first five and the last five rows.
So I'll pick spots train data and have maybe a square bracket call on a five to look at the first five
rows and to just verify that the last five rows are OK I can use the square bracket notation but this
time I'll put minus five and then a colon at the end.
There we go.
And if you do this what you should see is that there's a match between the values in the text file and
the values that are shown here as an output and Jupiter notebook.
Now both of these arrays are actually quite large and give you an idea of the kind of data that we're
working with him.
Let's print out the number of rows in both the training and the test file and let's print out the number
of e-mails that we're working with in our training file and our testing files managed to cram this kind
of data a couple of times before with the data frames.
But now that we've got a num pi array here let me show you the methods that you'd use to accomplish
the same thing.
So I'll drop everything in a print statement print parentheses single quotes number of rows in training
file karma and LP are sparse train data don't shape square brackets.
Zero.
Copy this line pasted below.
And then I'll just swap training for test
and hit shift enter.
There you can see that we've got quite a few entries in both of these files our training file has about
two hundred sixty five thousand rows and our testing file has one hundred and ten thousand rows.
The number of unique emails though will be determined by the number of unique document ideas that are
contained in these matrices so I can print this out with print single quotes number of emails in training
file come after it and then I'll use none PI's unique method.
So NDP don't unique parentheses spots.
Train data and then I'll put a square brackets afterwards and maybe select everything right in the first
column.
That way I'm only selecting the document ideas semicolon comma zero.
This is the notation for selecting all the rows in the first column and then because this will actually
return another array I have to change on an attribute here namely the size attribute.
Let's see if this works.
There we go.
The number of emails in the training file is four thousand and fourteen.
If we check out the number of unique emails in the testing file then we just swap this over for a test
and shift enter and we see that we're working with a thousand seven hundred and twenty three different
emails in the testing file.
So there we go.
I think that completes our setup and importing the data.
We've also talked about the format that our data is in.
We've got four columns.
The first one is a document I.D. which identifies a particular email.
The second one is a word I.D. which identifies a particular token or word.
The third one here is our label or category.
So zero for non spam and one for spam.
And the fourth one here is the number of times this particular word occurs in the email in the next
lessons we'll be transforming this name pie array into a pan this data frame.
And we're also going to be we structuring it so that it's no longer a sparse matrix but instead it's
a full matrix.
If you remember the full matrix included the zero values this past matrix did not looking forward to
seeing you in the next lessons.