Hi. In this video we'll continue on analyzing, uh, our data.
Now let's make some time series chart. Uh, for example,
let's plot the overall length of bite,
which were recorded during the whole time period per second.
And let's do it first on the smaller dust data frame.
We'll not use the dust dashboard anymore because it slows down, uh,
lot the calculations. I also experimented a lot to solve the lagging,
but basically my processor is to slow.
Uh, here. Uh, first, let's change the timestamp column,
which is currently an object data type. Uh, today time, uh, we can do this, uh,
like in pandas. Uh, with the today time method. Uh,
we just have to specify also the format, uh, what is the, uh,
timestamp shrinking.
And now when we call our data frame,
we see that the data type of the timestamp column changed to daytime.
We can get the sum of bytes, uh, per second in multiple ways. Uh,
the most straightforward way is to first set the timestamp as index
and then resample the data on the second. And then we calculate the sum,
uh, when we call the compute method here, uh, by this v gda, uh,
panda series into the ram here,
I divided the data by 1 million, uh, to see the data in, uh,
megabytes and to plot, uh, the chart, uh,
for these are used exponential weighted, uh,
moving average and with a span of 50 seconds, uh,
to eliminate the jumps. And we see that, uh,
the data covers around, uh, four minutes.
Now, uh, let's do it on the whole data frame. Uh,
we have to use another approach to do this, uh,
because the set index method needs to sort the whole data,
and this is a very time consuming operation. Uh,
we can use the regular group I for this,
where we group the data based on year, month, day, hour,
minute and second. And we calculate the sum of length.
And of course, with the compute method, uh, we get the Panda series back to ram.
During this cell, I changed the, uh, index back to Pandas daytime,
and now we can plot it. Uh, I also use the exponential weight moving, uh,
average for this. Uh, but this time with a span of, uh, 500 seconds.
We can also make a scatter plot here. Uh, it's, uh,
better to see the periods where I post my, uh, wire capture.
And now we also nicely see how many hours I did the capture. Uh,
we see I captured, uh, around 25 hours.
Okay, now let's try a few other methods, uh, like we are, uh,
used to do in pandas. Uh, for example, let's, uh, uh, try, uh,
to calculate the describe method, uh, per each protocol. Uh,
we can do it via, uh, group by and, uh,
in combination with apply and compute, of course,
uh, this approach, uh,
it's a little bit more of ram. Uh, so, uh,
we can do it a little bit different way, uh, by, uh,
calculating the statistics, like mean and standard deviation minimum,
maximum median, uh, separately.
And then we can call dusk compute. And all the series, uh,
uh, in our list,
we see here the results.
Now, uh, let's calculate the value count in the expert info.
Uh, we can do this, uh, like in pandas, uh, with the value kinds method.
Now let's calculate the sum of received bytes, uh, in, uh,
local IP addresses. Uh,
we can create a new column on our dust data frame, which tells, uh,
us if the destination IP starts, uh, with, uh,
1, 9 2.
Then we create a new data frame where the value of this column is true.
Uh, then we, uh, group the data, uh,
by the destination and calculate the sum off length column.
We also divided by the 1 million to see the data in, uh, megabytes.
And lastly, we have sorted the values testing.
In our next video,
we will learn how to make visualizations on this data frame via data shader.