Hi. In this video we will learn more about pandas data types.
We see that our data frame after loading has just two data types,
object and integer, but pandas supports, uh, more of them,
so, uh, which are supported
objects. These might be strings or other types like, uh, lists, dictionaries,
two pole or even another nested panda series or data frames,
integers, loads, bullying data,
time delta.
These are differences between days and a category.
The category corresponds to categorical variables in statistics
with a limited and fixed number of categories.
There might be also ordered categories as well.
It is useful if you have some string or even integer variable with
only few distinct values.
It can significantly reduce the memory usage of this column.
Here I created a small data frame,
and here we can see here I created a small data frame,
and here we can see that each of the value in column one is a list in
column two, dictionary in column three,
apple in column four, a series in column five, A data frame.
And the last column six has mixed data integer, a string and float. And
we see that there are all identified by pandas as object.
Now let's change few variables.
Data type we have seen in our initial data audit that, uh,
at category subcategory,
protocol source and destination IP have just few Disney values
and let's change them to categories.
We can do this very easily using the asai method.
And when we see now the information about our data frame, uh,
we can also see that is memory usage is now around 10
megabytes when compared to the previous data frame, which had, uh,
15 megabytes
in practice.
This conversion to category type is useful when you have limited amount
of memory available. In this case,
you can also downgrade the numeric variables from integer 64 bits to
32 bits, or 16 bits, for example. Same applies to floats,
but be aware that they differ in their supported range and the similar points.
For example, if we downgrade flow 64 bits to flows, 32 bits,
it'll use half the memory.
Now let's do also other reprocessing steps like, uh, for example,
the.com. We can drop it, uh, because it's useless,
it contains only dot values.
Then, uh, we can process the time variable.
We see that it is basically a string, uh, with the starting and ending,
uh, unique timestamp created by dash.
And the number is basically, uh, the number of seconds.
Since the first January 970,
we can split the string into two variables, start time and and time,
and we can drop the time variable afterwards.
Now, when it's done, we can calculate a new variable. Uh,
what is the difference in seconds between the start and end date of the attack
here in this cell. We change the data type of these two columns,
uh, to daytime.
Then we can change the start and end date columns to type datetime.
We can do this with pandas helper method pandas dot daytime.
We want to do it on two columns, uh,
so we have to use it with the apply method and, uh,
we'll talk about the apply method a little bit later.
But basically it applies this one method Pandas Today time,
column by column on a filtered subset of the data frame containing
only these two columns.
In our next video, we will learn more about PANDAS index and multi-index.