Hi everyone, and welcome to our seventh module,
which will be focused on data visualization in Jupyter.
The goal of this module is to teach you not how to plot the most beautiful
charts,
but how to effectively and visually via charts and descriptive
statistics, find patterns in your dataset.
We'll learn how to perform uni variate analysis,
and this is an analysis of one variable,
how to perform b variate analysis.
And this is the analysis of, uh, relationship between two variables.
How to do multivariate analysis.
This is the analysis of relationship between more than two
variables. The main library will be pandas and its,
uh, plot method,
but we will use also seaboard library and just a little for
few chart plotly.
We will also check the Jupiter lab extensions,
which may help us do our exploratory analysis a little bit, uh,
quicker.
And we will also learn the basics of the descriptive statistics.
Exploratory analysis is a very important skill.
I think it's a little bit underappreciated in my opinion, but,
uh,
you see that most questions can be answered just based on proper visualizations
and descriptive statistics.
Now let's go to our Jupyter Notebook and let's explore first
our dataset.
Open the module seven notebook.
The dataset I choose for this module is a dataset of connections,
including normal connections and attacks in a simulated land
network military environment. This dataset was introduced in,
uh, 99 in K D D intrusion detection contest.
It has 42 variables and around, uh,
500 K observations where each observation is a connection
marked as normal connection or as an attack.
It is just 10% of the original full dataset,
which contains around 5 million observations.
There are total of 24 attack types which fall together into,
uh, four categories
here in this H T M L file. Uh, you have, uh, more information about this,
the denial of service attacks.
Then there is a group of attacks regarding unauthorized access
from r o machine,
then unauthorized access to local super user privileges
and then probing.
There are variables which were derived based on the domain knowledge,
which would help us, uh, better distinguish normal corrections and attacks.
We have, uh, one data file. This is this one,
one file with the variable names and this H TM L file with the information.
Okay, here we loaded the packages, which we will use.
We loaded the text file into the data frame. Here.
We process the column names and appended them to the data frame.
Regarding a little bit of the initial data update. Uh,
when we do the info method, uh,
we see that most of the variables are float or integer.
Just few are objects and, uh, that we have no new values in any role.
When we check for a number of unique values in each column, uh,
we have in a lot of columns, which are integer or flow type. Uh,
there are just few distinct values. Uh, so we will make sense to change them,
uh, into a category, but we will not do this, uh,
because for the purpose of this module,
we will use just few selected variables to look for interesting patterns.
We can also draw variables which have only one distinct value. For example,
these two.
Now let's examine the attack variable. Uh,
we see that we have the mentioned, uh, 24 attacks, uh,
plus a normal connection,
and that they are highly disproportional. Basically,
most of the attacks are MPH attack and Neptune attack,
and the normal connections they take, uh,
only about 20% of the data.
Other attacks are in, uh, very low numbers.
Now let's do the describe method.
Here we see a new parameter include all,
and that means that all variables, including categorical or object variables,
uh, we will see, uh, in the statistics
we see that most of the variables have very little variance.
For example, the destination buys variable. Uh, we see that, uh,
at least 75% of the data are zeros. Uh, same applies for,
uh, duration and a lot of other variables.
Basically in the whole dataset, uh,
the most value is zero,
and the variables which are more interesting is this one source
bytes and count and service count,
and also maybe these three variables and plus the categorical ones
attack protocol type service and flag.
In our next video,
we will look into the uni variate analysis of the numeric variables.