Hi. In this video we will start to work with dusk and explain the dusk graph.
Let's go to a notebook.
After importing the packages, the first thing,
what we have to do is to start our local dust cluster.
First, we define the parameters, and with this command, we can start it.
Let's explain the parameters and workers.
This is the number of workers we want to use. In our case,
this will be the separate Python processing. Uh, working on our tasks,
uh, with parameter processes equals, uh,
true we ensure they are run as separate processes.
Memory limit is the memory limit for one worker.
I set it to eight gigabytes,
but I have only 20 gigabytes physical ram available on my computer.
So when you sum it up, it's total of, uh, 32 gigabytes. But, uh,
this is okay. Nothing will happen unless your workers eat all the memory.
I prefer to set the cluster this way when I have a limited amount of RAM
available, uh, because when one worker gets close to his memory limit, uh,
it start to spill data onto disc, and then it'll be, uh,
very slow
risk per worker. This is, as the name suggests,
the number of threats per one worker. Uh, one word to parallelization.
It is not topic of this course, uh,
but you can paralyze your code using DK and also using other Python
library, like multi-processing via processes or via threats.
Uh, there are differences between them and one or the,
another option is better to use in specific cases,
depending on your calculation. I encourage you, uh,
to learn more about this topic.
And the last parameter is the dashboard address to our, uh, second map port
here. Uh, when we, uh, create the client and print it, uh,
we see the information nicely structured.
We see the information about our cluster,
about the scheduler,
and also about our workers.
Okay, now we can create our dust data frame. Uh,
first let's create a smaller dust data frame just from, uh, four,
uh, pocket files, uh, from these ones.
I highly recommend, uh, when you are learning to work with dusk,
do not use the whole data set right away because, uh,
depending on your data size, uh,
the calculations may take minutes or tens of minutes, or even more,
create a smaller dusk data frame,
create your calculations and check if they're working as intended,
and then apply them to the big D data frame. And with the whole data,
it is also better to examine the dust graph on smaller DK data frame as well.
We can create the D data frame, uh, using the red parking method, uh,
with just these, uh, four target files in the list.
Now, uh, when we call a D data frame, uh,
we'll not see the raw data, just the metadata. Uh,
no data was loaded into Ram yet we see just the columns, uh,
the types and the number of partitions as well.
So what is, uh, dusk data frame? Uh, dusk data frame consists of,
uh, multiple partitions, and each partition is a pandas data frame.
In our case, we have dusk data frame, which has four partitions,
and each partition is a pandas data frame from the par files. We loaded.
Remember that, uh, no data was loaded into memory yet. Uh, we can do this, uh,
with the compute method. Uh, with this we execute our dust graph, uh,
which in our case consists just from loading the packet files,
and we get the results of the compute method into memory in form of a pandas
data frame.
We can create the dask graph, uh, by calling, uh,
visualize on our operations with the dask, uh, data frame,
or better to say we can visualize our graph when we call
it, uh, on a data frame. Uh, we see just four times. Uh,
the re parking method,
thus graph is basically a graph of all the computations or tasks
which need to be performed in order to get to the result. Uh,
it is very nicely demonstrated, uh, on the calculation of the average.
We can now create a new variable and assign the, uh,
the mean of the length column,
and we can visualize our dust graph.
Now we see how the dust graph looks like, and we see the detailed operations,
which must be performed on each partition. Uh,
here dask, uh, needs to calculate, uh,
the sum and count on, uh, each partition
here. After that aggregates, uh, sum and count,
it calculates the overall mean.
And now, uh, when we call the compute method,
we trigger the calculations and get the result. This is the lazy evaluation.
In practice,
most of the operations on that data frame are lazy. Uh,
but some trigger the calculation, uh, uh, right away.
For example,
when we want to get the length of the D data frame,
we do not have to call the compute methods because, uh,
the calculation is triggered. Uh, right away.
In our next video, we will start to explore the D dashboard.