Hi everyone, and welcome to our eighth module,
which will be focused on working with big data using dusk.
Let's introduce this module first. Uh, what you will learn,
you'll learn about the strategies on how to deal with the big data.
We'll learn how to do basic analytics on D data frame,
what are the differences between task and pandas data frame and so on.
We'll learn how to start our local D cluster with distributed scheduler.
We'll learn also how to make the diagnostics with DK dashboard on our
D graph,
and we will see how to plot big data using dusk and data shader.
Okay, let's start, uh, with, uh,
what is big data and what are the strategies to use when you have to work with,
uh, big dataset? So what is big data?
As big data, we can consider a dataset, uh,
which is too big to work with and to analyze within available memory.
You see that, uh,
it is recommended to have available memory at least five times more
than the size of the dataset. But this depends highly on task.
What you want to do, if you want to calculate just simple average,
you do not have to have that much additional memory available.
If you want to calculate just simple average, you do not have to have, uh,
that much additional memory available compared to when you want to do
sorting, for example. Also,
depending on your analysis and analytic task,
you may consider to sample the data. For example,
calculating the average doesn't require full dataset.
So a big enough representative sample is enough. Also,
when training machine learning algorithms, uh,
good sample is also in many cases enough.
You may also consider to work with databases,
but if you prefer to use your favorite tools, uh,
and need the whole data into memory,
you can also rent a single virtual machine instance on cloud,
which can have memory even in terabytes.
I frequently work with VMs,
which have usually 250 or 500 gigabytes you have to pay for
using them. So good practice is to prepare whole analysis on sample data,
then use the instance with already prepared code on a full data set.
You can also use libraries for parallel and distributed processing,
like probably the most famous spark or dusk, which we will learn to work with.
And lastly,
I also mentioned in one of our videos with pandas that you can significantly
reduce the size of the dataset by downgrading data types,
change data types to categorical,
or you can log out into memory just needed columns.
In our next video, we will introduce dusk.