All right. 1
So in this video I want to show you how to drop certain undesirable rows from a dataframe 2
and I also want to show you how we can change up the index on our dataframe to add document IDs 3
to track these emails, 4
so numbering these emails sequentially rather than with these unintelligible file names that we've got 5
by default. 6
So our first goal is dropping these three rows here with the index by the name of "commands" - "cmds" and 7
dropping the index with the name ".DS_Store". 8
So I'll add a little markdown cell here and that's going to read "Remove System File Entries from Dataframe" 9
and the way we're gonna do this is using the dataframe's drop function. 10
So this is a method that takes a couple of arguments, 11
right. 12
It needs to know which file names to drop. 13
So I'm going to supply that has a list, 14
square brackets, 15
single quotes and then "'cmds', '.DS_Store'". 16
Now remember, no typos here and you'll succeed. 17
So as it is, this will drop the three rows with this index and it will drop the one row with this index. 18
What we could do is we could overwrite our data dataframe with this modified version here, so we can 19
drop the rows and we can overwrite the data that's stored in our dataframe. 20
But the alternative of doing it this way is supplying another argument to this method and that's called 21
"inplace", 'inplace" is set to False by default. 22
And if we set it to True then we don't have to do this, we just can write the method like so and it will 23
update our dataframe. 24
Now before I hit Shift+Enter on this cell, let me copy this line of code here and paste it here, just to 25
make sure that this row does indeed disappear. 26
Let's take a look. 27
All right. 28
So the entire thing shifted up so we're not seeing the same emails right here. 29
We're having a different number of rows. 30
So it seems to have worked. 31
Let's take a look at what the shape is of our dataframe now, 32
"data.shape" gives us 5796, 33
so 4 entries have been dropped and this is how we've done it. 34
Brilliant. 35
Now let's replace these index names. 36
Let's change these index names to something else. 37
Maybe just some numbers right. 38
So let's just number our rows from 1 to 5796. 39
I'll quickly add a markdown cell and then I;ll say "Add Document IDs to Track Emails in Dataset". 40
We're going to be doing some manipulation of these emails, 41
so it's going to be quite nice to be able to have a specific ID associated with each specific email 42
so we can pull it up and refer to it later on. 43
Let's generate our document IDs first, so I'll create a variable called "document_ids" and 44
this will be equal to the values, say 0 to 5796. 45
So we can use the in-built range function from Python starting from zero and going through the length 46
of our dataframe, 47
so "data.index" will give us the length of our dataframe. 48
Let's take a look at what this looks like, "document_ids" is now a range from zero to 49
5796. 50
We're not printing out the individual numbers here, but that actually doesn't change how we can use this 51
object. 52
So we can create a new column say, "data['DOC_ID']", so doc ID 53
and set that equal to our document IDs and if we take a look at what this actually looks like, then 54
we would get something like so, we'd still have our file names as the index, but now we have a column 55
with all the document IDs from zero to 5795. 56
Right. 57
Ninety five, 58
why? 59
Because it's this length minus one, right? 60
There's 5796 entries, 61
but since we start counting from zero, the last entry is 5795. 62
All right. 63
So this is what our new column looks like. 64
Fair enough. 65
Now let's shift all these file names into another column. 66
I'll say "data['FILE_NAME']", all in caps, is equal to "data. 67
index", the index being these filenames right here. 68
This will create a new column with all these file names. So if I say "data.head()" now what we see is 69
we've got our index, we've got our category here, we've got the message column, we've got the document 70
ID column and we've got the filename column which at the moment is the same as our index. 71
However, what I'm going to do now is I'm going to set my index to be equal to my document IDs and the 72
way we can do this is simply by saying "data = data.set_index()", 73
so this is a method on our dataframe and we simply specify 'DOC_ID' in single quotes. 74
If I hit Shift+Enter now this will update, but similar to our drop method which had this inplace parameter 75
here that we can set to 76
True, we can do the very, very same thing with set_index; Shift+Tab on my keyboard brings up the quick 77
documentation and I can see here that we can change this here to True as well and then get rid of this 78
bit of code and write a comma here 79
and then "inplace = True". 80
This will accomplish the exact same thing. Here's what it looks like. 81
Now we've got our document IDs as our index. 82
We've got our category, 1 for spam, 0 for non-spam; our email bodies in the message column and our file 83
names 84
we've preserved as a separate column in our dataframe. Fantastic. 85
Now let's quickly check what the end of our dataframe looks like, 86
"data.tail()" we can see the last five rows. The last row has the document ID 87
5795 88
that holds a message body starting with the words 89
"If you run". All right, 90
so we've done a lot of data cleaning now. 91
We've extracted our relevant data from the raw text files, namely the email bodies. 92
We've converted them into a dataframe. 93
We've checked for empty emails and we've checked for null or missing values as well. 94
And then we've dropped all the rows that didn't contain an email body from our pandas dataframe.