Hi. In this video we will introduce dusk.
Okay, so what is dusk? Uh,
dusk is an open source python library for parallel computing in Python.
Uh, it fits very nicely into the Nampa Pandas, uh,
stick learn ecosystem because it allows you to use familiar interfaces, uh,
classes and methods. Uh, it is a library for, uh,
scalable distributed computing.
You can create a local dust cluster on your laptop,
but you can scale it into the hundreds of machines and terabytes of memory
available. Dusk uses a lazy evaluation.
That means that first you create your computation sequence without executing
the operations,
and dusk makes the computation only when explicitly called.
This allows dusk to analyze and optimize the computations within whole context.
Dusk provides excellence, the agnostic tool called, uh, dusk dashboard,
where you can analyze your computations and progress.
And to get more insights about the state of the workers
dusk, uh, main collections are dusk array, which is based on MPI array,
DK data frame, which is based on pandas data frame and a bag,
uh, which is, um,
suitable for working with unstructured or semi-structured data in
systems for distributed computing like dusk.
You will frequently see the 10 of scheduler and workers
scheduler is the process, uh,
which coordinates the execution of tasks and it, uh, manages, uh,
resources deciding where and when to run these tasks.
Dusk has a single machine scheduler and, uh, distributed scheduler,
uh, but do not be confused by the name distributor scheduler.
It can be launched also on a single machine,
and this is exactly what we'll use, uh,
a local dusk cluster with distributor scheduler.
Dusk has also available multiple interesting extensions, for example,
dusk ml, which allows you to work with, uh,
many machine learning algorithms in a distributed fashion.
D XG boost, which allows you to train xg boost models on d data frames,
dusk, uh, Q D E, uh, which integrates, uh,
cuda data frames into dusk.
These are data frames which operate on GPUs and are there for faster,
and also there are other extensions as well.
Now, let's launch a new container.
We have to do it because we need to map two ports.
We also need to have the ability to perform pseudo commands. Uh,
we launch it, uh, with this command to run,
but please replace the path of the host folder to yours. Also,
please notice that here we map the two ports.
One port will be used for the Jupiter lab,
and the second port will be used for a dust dashboard. Also,
notice that, uh, here with this option, user route and grant pseudo yes,
we'll ensure that we'll use the route user and we
will have the permit to perform pseudo command Also,
if needed, change the name of the container as well.
After you create the container platform, you need these commands.
Uh, pseudo applicate update, uh, then install graph piece, uh,
library with pseudo applicate install graph piece and install
these three packages, uh, graph that they share and hollow views, uh,
via peep into the anacondas main environment.
In our next video, we will introduce our dataset.