Hi. In this video we will see two Jupiter lab extensions, which help us,
uh, in our exploratory data analysis. We
can think of them like Jupiter lab extensions,
but they are a regular Python packages. Uh, the first one is, uh,
data prep, and the second one is pig voler. Uh,
please install them using people. I prepare two notebooks,
the data prep and pig voler notebook, uh, where I illustrated few, uh,
functionalities of these two libraries. Uh,
let's go first to data prep notebook.
With data prep, we can see in a nice, uh, a way the initial data,
uh, information about, uh,
the distributions and their similarity to other variables. For example,
uh, if the distribution is skewed, uh,
if there are zeros and how much, uh,
percentage of the zeros are there
and many other here. Now we can see, uh,
each of the variable including,
uh, statistics and also details.
Here are all of the 42 variables, which we have in the dataset.
We see the distributions, of course,
and there are different plot types. And you have categorical variable,
numeric variable, uh, so you have to properly set, uh, the data type, uh,
in the data frame.
We can explore two numeric variables in an interactive scatter plot,
but unfortunately we cannot set here any other parameters like the alpha
parameter or to add the regression line and this kind of stuff.
We see here. Also the correlation matrix, uh, where we can, uh,
check the personal correlations spear or Kendall Tal correlation.
The last section, section, uh, regarding statistics of missing values, but, uh,
we do not have any in our dataset.
Now let's open the second notebook. Uh, with pig volcker,
with pig volker, we can explore the variables, uh, via drag and drop. Uh,
very similar like Tableau works. Uh,
we see here the data top where we can set, uh,
the data type of the variable, the main category. It's, uh,
the option of dimension. This is a categorical variable or a measure.
These are numeric variables. Uh, each of them has, uh,
its own subtypes.
And here in the visualization tab we can make our own visualization.
Now,
let's make a scatter plot of the count variable and service count variable
color by attack. For example, like we did, uh, before, we
can put count on the x axis A. We count on the Y axis,
but we, we cut,
but we see here only one point because we have aggregation turned on.
Let's turn it off. Okay, now it's better.
We can change, uh, dimensions of the chart.
And now let's add the attack variable a color.
So you see, you can nicely drag and drop the variables inside these fields.
You can apply also filters. For example,
here you can map other variables to opacity of the
points, uh, size of the points shape.
Okay. Now let's make a other chart.
For example, a bar chart of, uh, flag variable.
With it count,
you can put the flag on the x axis and
row count here on the axxis.
Okay,
now let's just for fun, put, uh, the protocol type,
uh, as the color.
Oh, nice.
We see now we have a stacked bar chart.
We can sort the data. We can, um,
change the side of the charts. We can export the charts and so on.
Here we can change the chart type
and change if it's stuck bar chart or not.
It is a nice extension for those of you which want to explore the data more
interactively. Uh, but it has limitations, uh,
compared to when you are coding and using the ploting libraries, which we want.
Okay, that was it in this module. Uh, we came to an end,
I hope. Uh, you liked it. Uh, so now let's summarize it.
We have learned what this ate analysis and what are the types of the charge
used.
We have learned what is the b variate analysis and how to analyze
relationship of two variables and how to effectively look for patterns.
Also, when doing the multivariate analysis,
we saw that most of the plots for exploratory analysis, uh,
we can do using the pandas plot method,
but for some specific charts, uh,
when we want to dig deeper inside data or to better view the underlying
patterns, we have to use other libraries like Seaborn or plotly.
We also made an introduction to Pig Walker and data prep, which are packages,
uh, which, uh,
make the exploratory analysis more automated and interactive.
And, uh, in our next video, we will start our new module, uh,
focused on analyzing much bigger data sets using dusk.