In this session, you're going to see how to handle Outlash, just like non-value, we will have outliers
in our data set.
In fact, the chances for outliers in the data set are much, much higher compared to normal values.
So it is important that as machine learning data scientists, you learn how to identify outliers and
manage your class.
Right.
So let's start this session with an understanding of what is an outlier.
Our client is an observation that is quite different from the rest of the data points in your dataset.
It appears far away from other data points and is significantly different from the other data points
significantly diverging from the war on poverty.
If you see the image that is there on your screen, this is an outlier.
You see all the other fishes.
They're going from right to left.
They are either blue, light blue, dark blue color, whereas the red fish, which is the odd man out,
is red in color, is going from left to right.
So definitely this red fish is an all.
In fact, it's an outlier on two counts, the direction in which it is going and the color height.
So Outlash can be a data point, can be an outlier on a single.
Variable, not a single parameter, or it can be an outlier when you combine multiple parameters together,
as you're seeing in this fish example, like.
Even if that red fish had to be blue in color, it will still be an outlier because it is going from
left to right.
Are you getting it?
So how do you understand what is an outlier?
Let's understand what will be the impact of our clients?
Our clients typically swing the results.
OK, one way or the other, like they can change the results significantly.
It basically means there can be buyers.
The variation can be high.
The accuracy and classification will take a hit.
In addition to that, the mere presence of outliers actually impact the basic assumption behind many
statistical tax tests and machine learning models.
Right.
So if the outliers are randomly distributed, the decrease in non-magnetic in fact, normality is a
requirement.
Again, many of the statistical tests and machine learning models.
Right.
So it is important that we identify outliers and statistically manage the outliers, which is what we
are going to see in the rest of the.
Right.
So how do we identify outliers?
Do you can identify outliers?
You know, using Histogram Scatterplot Walks Block is the best visualization method to identify outliers.
It clearly shows visually how many outliers are there and what is the extent of the outliers compared
to the rest of the population.
Right, so backplane is the best way to identify outliers, and that's what we going to be using.
In fact, Biden comes with a pretty loaded package, right, using which he can generate box block.
Very easy.
OK.
In addition to box plug, there are other terminals.
These can be used to identify outliers like one any value, which is beyond the point.
One point five times the average.
Right.
What is it?
Port and rail you're going to see very shortly.
You can also use capping letters at any value that is outside of the 90th percentile.
I will consider as an outline like any data points that is beyond three or more standard deviations
on both sides of the mean.
OK, that is also considered as an outlier, right?
In addition to this, if you have multiple factor outliers, that is a variable identified as an outlier
based on more than one factor, right?
You can use cukes, distance and other distance matters to identify a place.
Right.
So these rules are also used in addition to Boxleitner.
Right.
We want to identify outliers because outlines are very, very important aspect in exploratory.
Did I want to either remove outliers or transform the data?
We want to see that shortly.
OK, so that outliers are reduced and hence you manage the fact of outliers, right?
So let's see, what is this box like, so box plot, right, helps us to identify outlines how our own
layers identify any observation outside this maximum and minimum right is normally considered as an
outlier, as per box.
So what is this maximum on minimum?
It is 75 percent plus or minus plus one point five times the difference between 70 percent and 24 percent.
Does this plus or minus one point five times the difference between 70, 15, 25 percent is called as
an airport average.
In fact, I just mentioned right as per the tunnel, one of the tunnels, any point outside, plus or
minus one point five times the difference between 70 and 24 percent by itself can be considered as an
outlier.
But one box plot does is it adds one more data point, right.
For maximum, it adds to 75 percent.
And for minimum, it adds the.
Twenty percent, percent, right, so any point outside of this maximum or minimum is calling this an
outlier, right?
So this is what we are going to be primarily using.
As I said, there are other dumb rules which you can use depending on the appropriateness of the situation.
Right.
So what is this person like, because I use the term 24 seven and on 58 percent, right?
So this person is for computing percentile.
First we arrange the data in an ascending order.
Small, too big by law to hide, you arrange the data in an ascending order.
Right here I am trying to find what will be the 80th percentile height for this dataset.
So this dataset contains the heights of of 20 people like so suppose let's say the 17 person rate has
a height of five feet, 11 inches.
So if you take the 17th person.
Right, I arrange the data in an ascending order.
So 16 people are less than that particular individuals like like 16 people.
How heights lower than the height of the 75 percent.
Right.
80 percent of 2060.
And so I'm saying 16 individuals are below this height.
So if this individual has got five feet, 11 inches, all these people like 80 percent of the population
in this, 80 percent of the people in this set.
Right.
Have a height less than five feet, 11 inches.
So five feet, 11 inches is the 88 percent height.
That means 80 percent of the people are below this particular.
That's the concept of percentile, percentile is a ranking mechanism, it is used in many competitive
exams like the Gyari Gmod and in India Gate and cat examinations.
So in those examinations, the ranking is determined based on percentile calculations.
If someone says, you know, I have secured the 98 percent, then that means his score is more than
90 percent of people that go to that particular examination.
Right.
So it's a ranking.
Are you getting it?
So it is a personal concept that I used in Boxtel.
Right now that we have understood.
What is an outlier?
How do I identify an outlier?
We are going to see how to manage our lives, right?
So because, as I said, outliers can swing the results one way or the other.
One of the strategies that is adopted is to remove outliers like many people doing.
The other way is to transform the event values, right?
I can convert the data for that particular factor into a logarithmic.
Transformation and transforming the variable rate from numerical to a logarithmic one, and then I identified
outliers, I invariably find that the number of outliers has come.
So converting data is not a numeric data into a lottery.
To make data is one of the ways I manage outliers like the other use what is called as meaning.
Meaning is the process of transforming numerical variables into categorical in logarithmic.
I convert a numeric into values writing letters in meaning I can work numerical into categories.
I suppose, let's say the age of people.
Is that right?
If we want to convert that into a categorical you know, I can say that young, old, middle aged,
you know, very young and stuff like this helps us to manage our clients very well.
What did.
Now, let's see some examples, if you see this example that is there enough for the kids study that
we are looking at?
I have now developed the box block for applicants, so you can see that.
So you can see that there are many outliers here, right?
And Koplik and I see outliers beyond the seven thousand five hundred here it is beyond 10000 and beyond
20000 fewer outliers, like most of the data ones, are clustered around this.
Right.
The same for KOPLIK and income.
And a similar trend is seen following the models beyond this loan amount value outliers and beyond this
point, some outliers of the.
So as you can see, this plot helps us to get a very good view of how close the box produced, really
useful and very powerful.
So now, having seen the outlines of the dataset, I want to transform.
Right.
So, as I indicated, here I am converting the applicant, including.
OK, into a logarithmic transmission, right, and I am doing the same thing for the lunar models because
there are outliers.
Right, and if you really see the number of our clients have come to.
Right.
The number of our lives have come down after eight transformed the data and redid the box from.
So I recreated the box plot for the Transformed leader and obviously the number of our players have
come down.
So I use this transform data in my machine learning.
So even here you have outlets finally we will live with it.
Like if you find the forecast accuracy is not up to the mark, maybe remove these outliers in the transform
data, see a on the model, see if there is any significant improvement.
If that is a significant improvement, use it.
But I go with the removal of the outliers and he transformed into.
So this completes the session on managing outlets.
Right, so outlines are very, very important concept, you must first identify them and either remove
or transform the data so that the number of outliers come down and you can go ahead and continue with
your model building activities.
OK.