In our data, we need to do our prior treatment also.
What is an outlier?
First of all.
Our values are the values that appear to be or diverge from overall pattern of a variable.
Outlays are usually because some occurred in the measurement or data entry or sampling process.
For example, the error was during data entry and you wanted to enter one point three, but instead
entered 13 because you missed the decimal point.
Or it was measurement error, and while measuring one of these sensors malfunctioned and given erroneous
value.
Reasons may be any, but it must be handled carefully before feeding the data to our model, because
these outlying values will otherwise lead to high error variants and lesser prediction accuracy.
To find out liars, we should use editing and visualization tools such as box plot scatterplot and histograms.
We will see this when we work on the software tools.
Once we have identified the outliers, we replace these values with some other values.
There are several options of what value we should use to impute outliers.
First, let me show you how one single outlet value impacted the data and how can we identify these
outliers?
These two datasets are same, except the one on the right has one outlier value of hundred.
This lead dimino, right, want to be shifted towards this outlier?
But median is not doing much.
So, Wendy, Data has outliers, there is a larger difference between mean and median.
Also, the standard deviation in such a case is higher.
So once we have identified the outliers, we have the following options we can do capping and floating.
What this means is we find the upper limit and lower limit beyond which we will change the values.
So suppose we select the 99 percentile as upper limit and all values larger than this value will be
assigned a particular value?
I hope you remember what percentile values.
If you order all the values that are available with you in ascending order, then 99 percentile value
is the value, which is larger than 99 percent of other values.
So only one percent of values which are larger than this value will be considered as outliers and they
will be assigned a different value.
Usually we assign a value such as three times the 99 percentile value.
This multiplier of three is a personal choice.
Similarly, for lower limit, we can use first percentile value and a multiplier of point three.
Second method is exponential smoldering.
In this matter, we extrapolate Politicker of 1950, percentile to 99 percentile beyond the 99 percentile
point.
And to the values which are going beyond 99 percent, a point to fall on this good.
This must seem difficult, but software packages do it straightaway.
However, I always preferred Method one.
Third is the Sigma approach, which is popular in the manufacturing industry, in this approach, all
values beyond some sigma, for example, all values which are far from mean by more than three times
standard deviation.
Are replaced with the value meaning plus three standard deviation.
This gives results similar to the first one.
So one and three are actually equivalent.
With this, we now know how to identify outliers and how to treat them.