Hi. In this video we will learn about the most common data frame operations.
Let's see how to create new columns. It's, uh, very easy.
You just put a new column name into a square brackets and assign the values.
During this cell,
I created new variables accessing the daytime accessor object dt,
and the day, month, a year, weekday, hour,
minute and seconds of each start time day and put it into the
new columns. We
can create new columns by passing a list or MPI arrays of the same length
as the number of rows in the data frame,
or we can pass just, uh, one value. In this case,
pandas will create the column with just, uh, this one value in each of the rows.
We can see it here
if we would pass something different, it'll throw an error. For example,
if the length of the past list will not be same as the row numbers.
Here I created an MPI array with just 400
elements and try to assign it to the new variable, but it didn't work.
We
can create, uh, new columns, uh,
using existing columns in the calculations. For example,
here I created two new columns,
which are the number of hours and minutes between the start and end time of the
attack.
All the operation, which you will do frequently is to rename the columns.
We can rename them with the rename method. Uh,
we just passed the current columns, uh,
skis to a dictionary,
and the values will be the new column names.
I recommend column names in a lowercase without, uh, any special characters,
uh, space and so on.
Filtering in the data frame can be done in multiple ways. Uh,
first is to just, uh, use the data frame name
and you can put your column, uh, inside the curly brackets like this.
And if you want to filter multiple columns, you put them, uh, into a list.
Here. I created new data frame with just, uh,
first and values of the original data frame, and we just, uh, these, uh,
three columns a month, year and day. Uh,
but I also replaced the default index, uh, with these labels. Uh,
to show you some examples on how to filter own, uh, index labels.
Uh, this method is, uh, very limited. Basically, you can filter only the column.
Another method, uh, probably the most flexible one I use it, uh,
most of the time is via the lock method. Uh,
this way we can filter rows and columns advance, uh,
where the conditions or labels of rows and columns are separated by
comma.
Important is the concept of slicing. Uh, you can, uh,
very easily return the rows and columns. Uh,
what do you want using slices, for example,
here I want to return all the rows.
Notice the column and columns. Uh,
starting the month column to the end.
Here, I did the same,
but I want to see just the columns between months and hour.
We can do the same on, uh, row index, uh, but it must have labels. Uh,
that's why I replaced the index in the df filtery data frame.
Another method, which you will use from time to time is the I log method.
Each row and column has its interposition, for example, first column,
second column, first row, second row, and so on.
Doesn't matter if you use the labels or not.
For example, here I filter the value in the first column and first row.
Uh, and we can use the slicing here the same way as in the log method.
Now, let's filter some data where we put some condition. For example,
let's filter our roles where the destination port is 80.
And we want to see just the time this column,
here's how we do it. And just to remind you that the first part
before the comma, uh, filters the roles. And the second part, after the comma,
select the columns.
The thing which I would like to remind you, uh,
that the result of this condition is a,
upon the series with bullying values denoting if the role satisfies the
condition or not.
Now let's change some values of some column where some condition of
another column is met.
Let's create a new column,
which tells us if the start date of the attack is weakened or not.
We can do it this way. Let's create, uh, this new column,
uh, where all values are false.
That means we first set all rows to non weakened.
Then the second step is to check which roles satisfies the condition
for the day of five or six in the big day column, uh,
this is the Saturday or Sunday.
And then to change these roles in our newly created column to through.
That's it
from time to time. You will also need to filter columns, uh,
based on their data type, for example, for uh, specific, uh, processing,
uh, where you make different set of operations on category curve variables,
numeric or volume variables.
And you can do it with the select D types method, uh,
where you pass the data type you want.
Now, how to delete columns, deleting columns is very easy.
You just use the drop method where you specify the columns in a list.
Uh, just to mention the in place parameter,
I am sure that you already noticed that I used it already on many methods.
And, uh, it basically means that the data frame is modified in place.
And, uh, this method will not return the modified data frame,
which you then may assign to the original or error data frame.
Now, to filter on multi-index, uh,
this is also very easy. Uh,
remember the data frame which we created before, which tells us, uh,
what is the count, average,
median of the time diff column and minimum and maximum of the
destination port, uh, per attack category and protocol.
Here we see that, uh,
the row index is uh,
labeled multi-index with two levels, same as columns.
Attack category is the first level,
protocol is the second level in columns.
The first level is the column,
and the second level is the aggregate statistic.
We can use the log method for this and to let know the log
method that we want to filter on the index,
we have to use the round brackets.
So basically at apple with the number of elements,
same as the number of levels in the index,
where in, uh, currently brackets, uh,
we can filter just the values for this specific index level.
For example, here I filtered the atta category. Uh,
so the first level to be warm,
and the second level sold the protocol to be three PC
and T c p.
The same applies to colon malting index. For example.
Here I filtered all rows, uh, but just the time this column,
uh, so this is the first level and uh,
from the second level, uh, I just filtered the mean and the median.
And this is the aggregate statistic.
In our next video, we will learn more about the apply method.