Hi. In this video we will get a little bit familiar with our data set,
which we will use in this module for analysis.
You can find it in the data folder. Uh, but I downloaded it from,
uh, the website called U C I Machine Learning Repository.
What is interesting on this website is that it contains a lot of other, uh,
publicly available data sets, which you can use for your own projects.
This data set is a small database of emails with it attributes
and a plus variable, which tells us, uh,
if the email was marked by the user as spam or not.
We have three ref files.
The data file contains the data documentation file,
contains the description of variables and some other useful information.
And the names file and this contains the column names.
Now let's explore each of them.
We see that the data file is only a text file,
basically with comm limited values,
and that is very good for us because we can open this file very easily using
pandas
in the documentation file. Uh,
we can find some useful information like, uh,
what is the number of instances, what is the number of attributes,
and what is the most important information is the description of the variables.
We see that 48 of the variables are continuous variables,
and they tell us what is the frequency ratio of selected
word occurrence in the text of the email. For example,
if we hit an email with 10 words and we will use word
make, uh, two times the value will be 20 for this word.
And here's the calculation.
Then we have six continuous variables,
which cover a few special characters in the email.
And the calculation is, uh, basically similar as per words.
And then we have three more variables,
which tells us a bit information about the sequences of capital letters.
Then we see the class distribution that almost 40%
of the emails are marked as spam and 60%
as not spam. And this is, uh, quite a balanced data set.
And we also see some attribute statistics.
Now let's open the name file.
Here are the variable names and we see that we'll need a little bit of
reprocessing, uh, to get only the pure variable names.
This is this part.
In our next video, we will load the data set.
We will pre-process the column names. We will also make, uh,
correlation analysis,
and during that we will learn how to work with sales and how to execute code.