All right. 1
So in this lesson I'm going to talk a little bit more about the data that we're working with 2
and I'm also going to show you how you can get hold of it. In terms of where we are in the project, 3
well, we've got our objective sorted and we've clarified what it is that we're going to do. 4
In other words, we've nailed down the question that we're looking to answer that is: Is an e-mail spam 5
or is an email legitimate? Now we're gonna go on to the next step, which is gathering the data to answer 6
this question. 7
Remember, the premise is that a company has provided the data for you. 8
And I'm going to play the role of the company here. 9
So in this lesson's resources there is a zip file that you can download with everything that you need 10
for this module. 11
Now I highly encourage you to pause the video and do that right now, just download the lesson resources. 12
But this isn't what this video is about, right. 13
What I really want to show you is where I got the data from in the first place and that way you'll know 14
how to source the data and how you can pull up detailed information about the data yourself. 15
You never know, this kind of stuff might really come in handy with future projects of your own. 16
Right. 17
OK. 18
So let's dive right in. 19
So where does the data actually come from? 20
Well, all the e-mail data that we're gonna be working with actually comes from an open source anti spam 21
platform called SpamAssassin. 22
Now I know that in terms of design, this website doesn't look like much but I think this is a really 23
really great project. 24
And the other thing is that Spam Assassin actually have a huge data set of spam emails and legitimate 25
e-mails that they're making available to the public to do things like train a spam classifier. 26
The easiest way that you can find this data set is to google the words Spam Assassin public corpus. 27
And if you do that the top results should look something like this. 28
If I click on this link I could take you to this page here. 29
Mind you the euro up here might change right. 30
That's completely up to the people who are behind Spam Assassin. 31
So hence googling Spam Assassin public Corpus is a surefire way to find the right Web site. 32
Again the design here can be a little bit off putting, it looks a little bit dodge, but what you've essentially 33
got here are a list of archives, you've got a list of collections of spam e-mails that you can download, 34
the file naming here gives a little bit of a clue to the recency of the archive itself. 35
Right. 36
So sometimes you'll see like a duplicate because the archive was actually updated - for example this file 37
here, 38
it's probably just an updated version of this one here. 39
I'm just going by the file name here. 40
I don't think the last modified column is actually that useful in this case. 41
Now a moment ago you saw me use the word corpus in my google search query and you see that word again 42
here. 43
Corpus. 44
Right? 45
This is actually quite an important piece of jargon to know, because it'll come up again and again when 46
you're working with text datasets. A corpus is defined as a large and structured set of texts. 47
You'll hear the word corpus also used when people refer to say all the written works of a particular 48
author like Shakespeare, 49
right? Now a related term that you'll come across as well when gathering text data is the word "Document". 50
Now this actually means something quite specific in this context. A document refers to a particular email 51
in our corpus, so the corpus refers to a set of all documents, 52
a. document refers to a particular item in the corpus. 53
So you'll see both of these two pieces of jargon come up again and again in this module. 54
And here's a bonus question for you. 55
Can you guess the plural of the word corpus? 56
I'm not gonna leave you hanging there. 57
It's a "Corpora" - one corpus, three corpora, not corpuses or corepi or what have you. 58
I found this out the hard way by the way, with people giving me quite funny looks at a conference and 59
I don't want the same thing to happen to you. 60
So there you go, 61
corpora. Now you know. Back on the Spam Assassin website, let me point you towards the detailed 62
description of the data set that we're working with. 63
It's right here under this unassuming "readme.html". 64
This is where you can see the revision history of the dataset and a brief explanation of what's actually 65
in the archive files. 66
Now, speaking of archives, you might be wondering how to actually download and open these files to look 67
at an individual email. 68
It's actually not that straightforward, because you don't tend to see these ".tar.bz2" file 69
types very, very often and it's not an extension you come across every day, right. 70
So let me show you how you can work with these files on a Mac 71
first and after that I'll show you the workflow on a Windows machine too. 72
Obviously, the first thing you have to do is download one of these files. 73
So I'm gonna click on this spam archive here, 74
the most recent one from 2005 and I'm going to save it to my downloads folder. 75
If I pull up my downloads folder I can see it sitting right here, 76
it's got the icon of a zip archive and it's 2.1 megabytes. On Mac, 77
all you have to do is pretty much double click on it, it expands it meaning it unarchives or unzips 78
it and it'll create a new folder for you like so. 79
So let me open this folder. 80
You can already see here it contains 1397 items. 81
So there's quite a few things in here and they all look like this. 82
That's pretty intimidating 83
and looks like complete junk when you see it for like the first time. 84
So let me show you what these files actually are and how you would look at them. To open one of these 85
files, 86
you need to use a text editor. On Windows, Notepad comes preinstalled. On Mac, 87
it's TextEdit, but the thing is if you right click on this and you go to "Open With", my operating system 88
isn't even suggesting a text editor here to open it with, 89
so I manually have to choose it. 90
One of my favorite text editors these days is actually called Atom, so you can download and install this 91
thing if you like. 92
Let me show you what it looks like when I open this file with this text editor. This is pretty much what 93
you'll see. 94
You'll see the top of the e-mail here followed by the body text of the email, 95
like so. 96
So we've successfully opened one of these emails in a text editor after unarchiving the thing that we 97
downloaded from the Spam Assassin web site. 98
This is the workflow on Mac 99
in any case. Everybody who's on Windows, 100
I apologize for making you wait this long. 101
I'm going to get to that right now. 102
All right. 103
So this is a Windows 8 machine, but Windows 10 isn't so different and all the same principles apply. 104
We first have to extract the zip file and then we can open a file in a text editor like Notepad or, yes, 105
Atom. 106
So I'm going to download this "tar.bz2" file and save it to my Downloads folder which I've got 107
open for you right here and then I can open it with an unarchiver of my choice. 108
So I've got a couple. 109
There's quite a few options here. 110
I'm gonna use a program called "7-Zip" and you can follow the two step process here with me. 111
So this is the "7-Zip" program with the archive file inside and I'm going to say "Extract" and I'll 112
extract it to my Downloads folder and you should see the "tar" file pop up here. 113
So this is actually another archive, so I'm going to have to repeat this step one more time. So I'm going 114
to open this again with "7-Zip" and then I'm going to extract my spam email folder from inside this 115
one and put it into, say, my Downloads folder. 116
So this is the process that I have to follow on my Windows machine to get to my spam emails - extracting 117
them twice. Now let me show you how you can pull up one of these emails in Notepad. If I double click on 118
this thing, I'm going to have to choose what program I want to use to open this file. The file extension 119
of this particular file isn't helping Windows figure out which program is appropriate to open this file 120
with by default. 121
So again we have to choose it manually. If I go to "More Options" I can scroll down here and I will find 122
the Notepad program listed here. 123
So this is Notepad. 124
Uh good ole, good ole Notepad. 125
And it does open the file perfectly. 126
It's not formatting it in a very friendly way, but it is opening it. 127
So this is how you can look at a particular email with a default program not having to install anything 128
on your machine. 129
Right. 130
So you shouldn't have a problem opening any of these files and looking at the emails individually if 131
you want to. Now I think you'll tire of Notepad pretty quickly, 132
so you know, if you have a decent text editor like Atom or say Notepad++ or what have you then 133
it formats it for you a lot more user friendly. 134
And this is what I would encourage you to use rather than the default program that's installed.