So let's identify and treat outliers in Python.
If you remember, we saved the word data as B.F..
So let's first try another function that is in full function will be, if not in full.
You can see using this function.
I can get that Nembutal KONE's.
And the data type of that attack that we missed this while discussing you really?
That's why we discussed earlier.
Now let's move on to outlier treatment and identification.
So from our EDT, we identify three variables.
That we want to look at first, one is the crime rate.
Second one is the hot Groomes and the third one is the rainfall.
If you remember, we slaughtered Scatterplot for rainfall.
And and ah, Groomes, we confirmed that this book contains outlier.
However.
We were not able to judge whether it was Eau Claire or Skewness in the case of crime rate.
Since we definitely know that rainfall and and hot grooms gun ban all play it.
We will directly write a function to read them.
Before that, if you remember, in order to re lecture, we discuss about capping and loading.
So first, we need to identify them 99 and the one, but until value of these two variables to do that.
That is a function in numbat called percentile.
We'll first look at the function, we'll write and be good.
But Sunday.
Then first, the argument, we have to pass the column name.
So we read B.F. Dot and Groomes.
And then in the square of ahead, we will write the percentile value.
We want to see.
So we want the 99 percent.
All right, 99.
Then on this score.
You can see this is an array and the 99 percentile of and hot grooms is fifteen point three nine nine
five two.
But the output here is an array.
So remember, if we want to fed the first number, often today, we have to specify the location of
that value and the square record.
So we'll write and be that person day.
B, if.
And our brooms.
99 for 99 percent percentile.
And then after that, and they squared record, we'll read zero.
So we are actually vetting the first element of this.
We will save the value of this 99 percentile.
And another variable which we call upper limit.
So we will write you.
We.
So we are just saving this fifteen point three nine nine value in another way, Raybuck, you we.
Now, how to identify the rules where the hard growing value is more than this number, who do that
will right B.F. and squid record will right.
The condition we're.
B.F. Dorte and her groom.
It's more then you'll be.
So we'll ride it.
Then you'll be.
Then you we when we done this, go on.
You can see we are getting all the values we're about and half growing value is greater than this 99
percentile value and courtroom's.
You remember the you we well, you, us, we've been born three nine nine.
So if you see we are getting all the rules we have, this value is greater than this SUV.
Well, you know, we want to limit this value.
If you remember, in our capping and loading, we discuss that we can multiply this value by any BGA
or any value.
We can replace those values.
Now we know how to identify the outliers in our data.
Now let's just cap this well loose.
Now for our case, we are taking an inquiry to the multiplication of a local two three.
Since we only want to be the genuine outliers, which is a hundred and one and eighty one, and we don't
want to touch these three outliers.
That is fifteen point four zero, which is very close to a what do we value.
That's why we are taking an inquiry to three will write the F thought and hard Groomes.
We want to change the values of this table.
That's why we are selecting this variable only and in record will specify the condition where B of dot
and heart Groomes.
Is greater than three, we.
So 3U is approximately 46.
So you can see for these two values.
Four hundred and one and eighty six.
We want to limit this and cap this well loose.
And we want to limit this by a value equal to three.
We.
Feed on this.
This is just a warning, not an.
So we can continue now if we rerun this a statement.
We'll see.
We have limited value for basics.
This was our own hundred.
And this was it on it.
Now we are getting a constant value of what, BASIX?
This is how we treat outliers using Python.
Similarly, in the rainfall, there are values which are outlier on the lower side.
Let's identify the outliers in rainfall and treat them as a.
We will write and we don't, but Sunday.
The if not green for.
And we won the first percentile value.
Since the outlier on Lordan and we won the first value of this update, that's why we have put Zettl.
The newest trendy.
Now, we will save this value in another variable called Elvi.
That is the lower value.
Let me quote put this on.
Our Elvie is a variable which is containing this first percentile value.
Now we compare this Elvie with our rate and we'll try to identify all the values which are lower than
this and we value.
Right.
Beer.
We're being screened for is less than an.
Run this.
You can see we are only getting one single value.
We are the rainfall value is three.
So this is definitely an outlier and we should treat it.
As mentioned in the teary lecture for the lower values will multiply by a decimal point.
So in our case, we will write zero point three.
We will select all the values.
Where we have no rainfall is less than zero point three and two, Elvie.
And we will equate it to zero one three times.
And we.
We've done this and we'll.
We've done this statement again.
You can see that the.
Rainfall value is now six and sort of three.
That is how we treat our players using Biton.
Now, let's look at the next variable, which is the crime rate even.
Since we don't know exactly whether climate contains Eau Claire.
Or their distribution is skewed.
Well, first, to join a lot of crime rate.
What says our dependent video?
But.
Right.
Giant plot where X is crime rate.
And why is our price would even.
And data is being read on this.
Plus, if you see that histogram of crime rate.
There is a large concentration of values at the lower respecter of crime rate, but as we move along,
as we move along to the higher end, the crime rate, the density of distribution reduces.
So more so four points are concentrated towards low crime rate, whereas there are only a few bad which
have high crime rate.
And if you see the relationship with Y, you can see that is somewhat while the normal relationship
here with Y.
For low crime rate, the price is high.
But as the crime rate is increasing, the price of it is decreasing.
And since when you view this as a scatterplot, there is no linear relationship between price and crime
rate.
There is somewhat of polynomial relationship.
So there is a way to treat this relationship.
There is a way to take log or exponential or a square root of crime rate to make it more linear.
And when we do that, our old players will automatically will be gone.
So, for example, if I have value one, ten and a hundred.
If we apply our normal rules, we treat hundred as an outlier.
But if we take log.
With the base 10 of all this value of what one will become zero, over ten will become one and over
hundred will become two.
So there are Wiess.
To remove outliers without actually removing it, just by transforming the function or taking lall or
bigging are maitake digging exponential or a square root of those values.
Since we are getting the kind of relationship here between our X and Y, we first want to transform
our variable crime rate.
And then after transforming, we'll look at whether the outliers are present or not.
Here for.
Will not greet outliers here.
First, we'll transform this variable and after after that, we'll look out for outliers.
We will look at valuably transformation while creating dummy variables.
So now, since we have treated our outlets, let's take a look at our UDD once more will write the if
not risque.
Let's look at the hotel rooms and rainfall.
As you can see now, the maximum value is 46.
And the mean and median values are a lot closer than before.
Similarly for rainfall, the lower value is now six instead of three.
And again, the median value is closer to the mean value.
That's all for Eau Claire treatment and identification and buy them.