Alright! So in the last lesson we've talked about file paths. In this lesson we're gonna put those to use. 1
First stop, 2
let's put some headings into our notebook. So this one here, 3
I'm going to change to Markup and we're gonna keep all our notebook imports in the cell below. The next 4
section heading I'm going to add is gonna be called "Constants". This is where we're going to add all our 5
file paths. 6
We're gonna have a single cell where we're gonna put all the pieces of information that don't change. 7
Now I want to work more in the middle of the screen for you guys, but one of the handy things about Jupyter 8
notebook is that it's got all these handy keyboard shortcuts and under the Help menu, the keyboard 9
shortcuts for Jupyter are outlined right here. 10
So I can see that scrolling down, my keyboard shortcut for inserting a new cell it's just pressing the 11
letter B when I'm in command mode. 12
Yeah, there's two modes. 13
There's the Edit mode and there's the Command mode. 14
You can tell that you're in the Edit mode when the cell is green because your cursor is blinking there. 15
And you can tell that you're in the Command mode when the cell is blue. 16
So if I press B on my keyboard right now when I'm in command mode, then I get new cells appearing like 17
so. 18
Now what I'll do is I'm going to create a constant and call it "EXAMPLE_FILE" and this constant 19
will have my file path to my "practice_emal.txt" file and this is gonna be my relative 20
path. A good naming convention by the way for constants, 21
in other words for values that you don't want to update is to write them like so with capital letters 22
separated by an underscore. The next thing I'm going to do is I'm going to copy the file path to this 23
practice email. 24
So I'm going to go to "Get Info" and I'm going to copy this file path here, close this, go here and then 25
in my single quotes, I'm going to paste it in here. Because I'm just interested in the relative file path 26
not the absolute file path, I'm going to delete this bit here. 27
So it's just going to read "SpamData/ 28
01_Processing", 29
and then what I'm going to do at the end is I'm going to tack on the file name. 30
So "practice_ 31
email.txt". 32
I'm going to include the extension as well. So I'll have my relative path from my MLProjects folder 33
where this notebook lives, going into the 01_Processing folder and then I'll have my file name at the 34
end. 35
Now if you're working alongside with me, make sure you don't have any typos here. 36
Make sure you are using the forward slashes instead of the back slashes. 37
And if your text matches mine and your "practice_email.txt" file is in exactly the same 38
location as my file then we're not going to have any issues. 39
Now let's practice reading a single file. So I'll add a new section heading here, 40
call it "Reading Files" and in here we're going to start talking to our operating system. 41
The kind of object, the kind of thing that we need to read a file from the disk is called a stream or 42
a file object. In Python, 43
you can open a particular file with the built in "open()" function and this open function of course needs 44
to know which file it's meant to open. In our case, 45
we're just going to use the constant that we defined above. 46
I'm going to type "EX" and hit Tab on my keyboard to insert the rest of this code. Because we've hit Shift+ 47
Enter on this cell, Jupyter notebook actually recognizes this name here and it will autocomplete the 48
name for us 49
which really speeds up our typing. Now that we've called the open function, 50
it will return to us a file object or a stream. 51
Now I want to store this object in a variable, so I'm going to say "stream = open( 52
EXAMPLE_FILE)". Once I've got my stream I can read the individual lines in this file. So I'm going to say "stream. 53
read()". This method will go through the lines of the file one by one and I'm going to 54
store the output of this method in a variable called "message". 55
When I'm done reading my file, I'm going to tell Python to stop looking at this file, because we've reached 56
the end and we're not planning to do anything further with it. 57
So I'm going to say "stream.close()". 58
All right, so we've opened a file, we've read the contents, we've stored those contents in a variable and 59
then we've closed our stream because we're done reading the file. 60
Let's print out the contents of this file so "print(message)", Shift+Enter and here's what we have. 61
This is the structure of an email. 62
The first bit is called the email header, so it has all this information about who sent the email, who 63
it was to, who was cc'd, what was the subject and so on. 64
And then after the header there's a blank line 65
what follows is the email body. 66
Now this text here is an email body that I've copied from a book called "The Timewaster Letters" by 67
Robin Cooper. 68
And if you're ever looking for a humorous read, then check out this book. Coming back up to our Python code, 69
let me press Shift+Tab on my open function and take a closer look at the quick documentation. We can 70
see here that in the parameters there's a couple of inputs. 71
The first one of course is the file. 72
The second input has a default value of 'r'. 73
So this means that by default we are only reading the file, not writing one. 74
The next parameter that I want to talk about is this thing called "encoding", which by default is set to 75
None. 76
An Encoding is how the computer handles letters and text. After all, 77
every character needs to be translated into ones and zeros at the end of the day. 78
In the days of early computers, this was fairly simple because computers really only supported 127 American 79
English characters. 80
This was called ASCII. 81
If you were French for example and had an accent above a letter then you were out of luck. 82
No accents for you Frédéric Chopin. 83
If you are Russian or Chinese or Thai or Indian or German or, well, from anywhere really, then you are 84
also screwed. 85
ASCII doesn't have a character for the letters in your alphabet. 86
On the upside people more creative than I have used ASCII characters to create some really neat art 87
to make the Internet a much more interesting place. 88
Back in our Python code. 89
If we leave the encoding blank as we have done here, we will use the default system encoding. What's the 90
default encoding on our system? 91
Well let's make Jupyter actually tell us explicitly. If we import the system library and then write "sys. 92
getfilesystemencoding()" and hit Shift+Enter, 93
we can see what the default file system encoding is on our own machines. 94
So in my case it's utf-8. UTF-8 is Unicode which is the Python 3 standard as well. 95
If you have a different default encoding set on your machine, I'd actually be quite curious to know, so 96
please do share the comments section for this lesson. 97
Now, since your default and my default might be different and I also know that the spam messages in 98
our dataset are written in English, you and I should specify the same encoding. That way we will both 99
get the same results and for that purpose, we will use an encoding called "Latin-1". Coming back 100
up here to our open function and we add an argument for our encoding. 101
I can type "en", Tab on my keyboard then Enter to insert 'latin-1' and if I hit Shift+ 102
Enter now on this cell, we can see that we read the exact same file and we can still retrieve our message 103
body. 104
Now one thing that you might be quite curious about is what is this message variable here? 105
In other words, what does this read method actually return from the stream? 106
We can have a look at this by printing out the type of the message variable. 107
So if I wrap my type function into a print statement, we can do just that. So "print(type( 108
message))", and then closing the parentheses and hitting Shift+Enter, we can see that we are dealing with 109
a string. 110
In other words, we open the file, we read the contents of the file. That leaves us with a string or a piece 111
of text, then we close the file and printing the file out like this shows it to us below the cell. 112
Brilliant. 113
Now even though this is an example email that I've included as part of the lesson resources, 114
this example email still shows us the general format for all the emails in the dataset and they all 115
look very, very similar right. 116
They have a header at the very top which includes all sorts of information, namely where the email was 117
sent from, who it was sent to, the subject of the email, 118
the timestamps when the email was sent and even the timestamps of the email transfer agents and the 119
rooting information of the email. 120
Now typically as a user if you fire up Gmail or Hotmail or Outlook or what have you, 121
all of this is hidden but it's still there even though it's usually rendered invisible. 122
Now for our purposes of training the spam classifier, what we will do is we will look exclusively at 123
the body of our email files. 124
We're actually going to ignore the header and just focus our attention on extracting the email body 125
and I'll show you how to do just that in the next lessons. 126
Now before I move on, 127
I just wanna leave you with a nice tidbit. Encodings are actually a pretty interesting niche topic 128
on their own and the XKCD webcomic has actually done a lovely strip on Unicode and, you know what, 129
I've even seen the ASCII table get its own scene and be a major plot point in the movie The Martian 130
which was, as far as I remember, all about saving Matt Damon. 131
Anyhow check it out and I'll see you in the next lesson.