Hi. In this video, we will start to analyze the whole data.
Uh, let's read all Target files into, uh, this data frame.
Uh, we see that,
uh, our data frame, uh, has, uh, these columns, uh,
with their data types, and also that it has, uh, 500 partitions.
Uh,
we can check the number of partitions with the end partitions attribute as well.
You know, that, uh, that data frame supports most of the methods like pandas,
but not all of them. Um, but, uh,
dusk has a very good documentation with a lot of examples. Now,
we can check, uh, uh,
how many rows has our data frame by calling the length method,
and let's check our dashboard. What is happening there? Uh,
we see here now nicely the process of ongoing calculations.
Uh, for example, we see that dask, uh,
needs to calculate 500 times the length, uh,
on each partition. And then one task, uh,
is to aggregate all 500 results.
When we hoover over the chart, we see how long it took to perform the task.
Uh, here we also see, uh, how many tasks are there in total,
how many are waiting queue, how many are in processing and in memory?
And also, uh, how many encounters? Some error.
Now, after the calculations have finished, uh, we see here, uh, this one, uh,
red part at the end. Uh,
this is time when workers interchange data between them. Uh,
good dust graphs are designed with the minimum of interwork communication,
but sometimes you can't avoid it, uh, when sorting the whole data, for example,
when we click on the profile tab, uh, now we can also see,
uh, how much, uh, c p u time each function took in our calculations.
Now let's go back into our notebook. Now we can check, uh,
the head of our data frame, the tail of our data frame.
Uh, when we print the info method on our data frame, uh, we get only,
uh, very little information. We cut.
We see here now, uh,
the number of columns and their data types.
Now, let's try, uh, method like we did before in pandas, uh,
the describe method just on few numeric variables, uh,
the delta time length, and T c p length.
And let's check the dashboard again.
Now we see, uh, here a lot of various, uh, tasks, uh,
which need to be calculated, uh, together, uh,
6,184.
Uh, you see that dask works, uh,
by doing necessary calculations on a partition, which is loaded into the ram.
After it calculated, which was needed, it, uh,
will drop the data of this one partition from the memory and continues on
another partition. Uh,
now we also see that the memory consumption is a little bit higher than before.
Uh, remember that our dataset has more than 13 gigabytes.
If you would like to load the whole data into a pandas data frame, uh,
but we are now using only around, uh, one gigabyte.
It may also happen to you that, uh, in the notebook,
the calculation will finish a little bit earlier, uh, than, uh,
what is visualized in the dashboard,
because sometimes it's, uh, lagging.
Now we see it, uh, that our calculation has already finished. But, uh,
let's wait, uh, until, uh, the dashboard will be clear.
Okay, it's done. Now we can check the memory consumption in the next cell.
Okay, it's done. Uh,
we see that our dataset would have, uh, dimension, uh,
13 gigabytes.
Now let's check for number of unique values in, uh, selected columns, uh,
which are basically categorical. And
now we can also calculate the number of, uh,
new values in all of the columns.
And now we can calculate the number of, uh, low values in each of the columns.
Okay, it's done and we see the results.
In our next video. We'll continue, uh, with analytics on, uh,
our desk data frame.