Hi.
In this video we will learn the most common methods for initial data audit.
Firstly, after loading the data to Panda's data frame,
usually check the contents with the head method,
which brings the specific number of, uh, rows, uh, the top rows.
And with the dev, uh, tail method,
which brings you the last few rows,
it is also sometime useful just to sample a few rows from the dataset
with the sample method.
And now regarding the other useful methods that we can use, uh,
with the shape attribute of the data frame,
we can print the number of rows and columns in the data frame.
We can get more detailed view about our data frame with the inform method.
We see here the number of rows and columns, uh,
but also column names, uh, number of values in each column,
which are not new, and their data type. Uh,
we see also the summary of how many columns we have per data type,
and also the memory usage of the whole data frame.
We can specifically get only the data type with the D type attribute
of the data frame.
We see here that most of our columns are object and the port
are integer.
With the size attribute,
we can print the number of nono elements in the whole data frame
with memory usage method. Uh,
we can see the memory usage of each column in bytes.
And we also see that the resulting type of this method is the panda
series, and this also applies for all methods, uh,
used on the data frame, which return your multiple elements.
Now we can compare the sum of, uh, memory usage of each column, uh,
to the memory usage of whole data frame.
We see that is approximately the same 15 megabytes.
Next method is the count method.
With this, we can get the counts of non-no elements in each column.
Other values useful method is the s n A method, uh, this,
uh, method by itself. I'll return you, uh, mask. Then I think if, uh,
some particular value in each column is, uh, null or not.
When we change this method with some,
we can get the counts of null values per column because it sums the true values.
Uh, like once. Basically we see that in our data frame,
only two columns have some null values.
All other values for method is, uh,
to check number of distinct values for each column. For example,
we see that we have only 14 distinct values of the attack category.
Another very useful method is the describe method,
which summarizes numeric variables. It calculates you the,
the count average standard deviation, minimum, maximum, uh,
median, 25th percentile, and, uh, fifth percentile.
In our case,
it doesn't make much sense because we have set only the ports as the numeric
variables, but they, uh, do not measure anything. Uh,
we can think of them more like as a categorical variables. We
also see that the return value of this described method is
upon US data frame. And the last method,
uh, is the duplicated method. Uh,
this method tells us how many, uh,
duplicated roles we have in our data frame.
We see that we have zero duplicated growth.
Now let's check only one column, for example,
the source port to see what are interesting methods,
which will tell us more about just one panda series.
We can use the same methods like on the data frame,
like the head sale temple.
Describe info size, number of byte
shape as well is an A two.
But, uh, in addition, we can use the value count method,
which will return as a series which counts for each of the value.
For example, we see that we have in our dataset 90,000 rows with or zero.
Sometimes you do not need the value counts,
just the distinct values of some column or just number of distinct values.
In our next video, we will learn more about pandas data types.