All right.
Let's fill some of those missing values shall we.
So make a little heading here.
Fill missing values.
Beautiful.
And we've got a fair few missing values here.
So what we might do.
I think we'll do the numeric missing values first.
So let's put it here.
So another little heading and we'll go.
Let's change this to number two.
Just because I like the stick I'll go fill numeric missing values first.
Wonderful.
Now how do we check what columns and numeric and what columns aren't numeric so what we might do remember
there is string day type.
We did that before we loop through our data frame right up here.
You can play press shift into their Where is it.
So is string daytime.
So there's another one as you might have guessed.
There's also is numeric data type is it up here.
No it might be.
Here we go.
Here is numeric go.
Check with the provided array or daytime is of numeric detail because you want to fill numeric data
we want to find out which columns are numeric.
Let's loop through for label content in the F temp dot item so we're gonna loop through the column names
as well as what each column contains.
If API dot types dot is numeric data that makes sense where we need to put a name.
If the content yes if the content in the column is numeric we want to print out the labels that's gonna
be the name of the column.
Wonderful.
So let's check out one of these.
We made these ones down here so we know that they're numeric but let's check out just for fun model
i.e.
there's gonna be a model a day or actually we can check what that is model day identifier for your unique
machine model i.e. if I model description.
Beautiful.
OK.
So these are our numeric types.
And now what we might do is find out which ones of these have missing values.
We're more focus on it might we want to fill the missing numeric columns with some sort of value check
for which numeric columns have now.
Now you know how might we do this.
Well first of all let's start out with four label content or just write exactly what we did before the
attempt the items.
And now we want to go if PD dot API dot types dot is numeric tried to do tab but there's too many options
is numeric day type content.
Wonderful.
And then what we might need to do is check there's a feature or a function in Panda's called is no.
So if we go PDA is no
pandas is no.
Here we go.
So this detects missing values for an array like object so we can do this by pandas is now content.
That's what we want.
So we'll do another segment here.
So PDA is now so this is going to go.
Is there no values in the content a.k.a. the column in the whole column.
Is there no values.
But we need to make sure there's a sum so total.
So if there's any null values so if this is higher than zero we want to print the label.
This should tell us which columns have no values.
All right.
Now that we know which columns or which numeric columns have no values we can fill them with something.
So what we might do is fill the numeric rows with the median so how might we do that we can just take
this we'll copy this down here.
Right.
We're gonna go fill numeric rows.
If the median.
Wonderful.
But there's one other thing that we have to actually do go here.
Well you don't have to do it but we're gonna do it anyway.
And I'm going to write it out first because we're going to add a binary column which tells us if the
data was missing or not.
That comment is a bit long but that doesn't matter.
What do you think we might do this.
I just that you have a think about it while we do it.
We'll go here is missing equals PD is now content.
Wonderful.
And then what we're going to do here is fill missing numeric values with medium DFT temp label equals
content.
Fill in a content dot medium to we only build upon what we've done before but we added two new steps
in here before we hit shift and enter on this.
We're going against our rule F and outrun the code I want you to just think why we might put this column
here.
You might be asking as well why are we filling it with the median.
Why not the main well the reason why we're adding this column here is because if we just fill up all
of our missing data with the median then there might be a reason for that missing value that was there.
We're keeping that information the fact that that variable was missing by adding a column which is true
or false.
If it was missing or not.
So that means if we find a row that has a missing auctioneer I.D. value and then we fill it with the
median of the auction idea column there's gonna be a column on the very end of our data frame which
says true which is basically 0 0 1 which tells us that hey we filled up this auctioneer idea row but
originally that caller all this row was missing this value.
We're keeping that date and we're keeping that information at the fact that that particular sample had
a missing auctioneer Audie in the dataset.
So we're keeping that information there but then we're filling it up with the media and you might be
asking why the median right.
Why of the main.
Well the thing here the reason being is because median is more robust than the mean and you might be
wondering what do you mean the median is more robust in the mean.
Well the thing is when you use the mean when we have a lot of different values so we have four hundred
twelve thousand values the mean of a lot of different values can be very sensitive to outlaw.
So let me demonstrate it with a kind of a rough example.
So demonstrate how median is more robust than Maine.
And we've just run this sell so we'll check out what happened with that.
Second I just want to give you an example of why we might use the median you might go in our search
for this go why use median over mean that could be some sort of extension you could do some research
there but we go here you go tens equals NDP for what this is going to do.
Creating an array a thousand long way a value of one hundred.
So say there's thousand people why we're going in this tens.
So one hundred hundreds there's a thousand people each with 100 hundred dollars each.
OK.
And then we'll make one we hundreds and a billion let's say out of those thousand people.
Bill Gates just walked in and he's got a billion dollars in his pocket.
So we'll go.
Not tens or hundreds.
So there's a thousand people.
One hundred dollars.
But now Bill Gates has walked in and he has a billion.
That's nine zeros.
My goodness.
That's a lot of zeros.
One two three.
One two three one two three.
Well done OK.
So what we're going to do here.
He's going to empty main hundreds and then we're going to empty main hundreds billion and then we'll
give me your MP maybe in hundreds and then we're going to go empty median hundreds billion.
So what we're doing is we're taking the main and the main of each of these and the median meaning of
each of these let's say Woo.
So you see what I mean here in our array hundreds let's have a look at it.
This is just a thousand examples of the number one hundred.
The main is one hundred.
That makes sense and the median is also one hundred but when we add one billion to the end it should
be right down here somewhere.
Well maybe it's not.
It's in the hundreds billion.
Now we have one example of a billion been right at the end.
Look how much that influences our main.
And now this is an extreme example right.
We're using a billion versus one hundred.
So we've got a couple of orders of magnitude here.
Look how different it is to the median in its purest form.
This is why we're using the median and so you might come across that in your research of why using median
to the main but the main reason is is because the median is more robust to outliers.
But with that being covered let's have a look at what we just did.
What did we just do with this.
So to figure that out what we can do now is check if there is any no numeric values and how did we do
that before.
Well we did for one label content endeared tend to items.
That's what we're after.
Yep.
If PD on API dot is on a type that isn't getting ahead of myself here numeric daytime content wonderful.
If payday is now content that some print label so this should tell us if there is still any numeric
values or numeric columns that have no values and it should print out those column names and it prints
out nothing.
The reason being is because we just filled the content the ones which were missing with the median.
Now let's check to see what our binary column did.
OK.
Check to see if this worked we should have Let's go up here.
How many auctioneer idea values were missing.
Can we find that out from our previous one auctioneer I.D..
OK.
So we should have the fact that we filled up false should be the total number of rows minus the so that
true should be this value of our binary column.
Right because it wasn't missing.
So let's go here on a true should be this number because that's how many missing values there were.
See there.
So what we've done here before we filled our values we've checked the sounding missing values there
are.
So our binary column should be true for twenty thousand one hundred thirty six examples because that's
how many examples we've filled.
So let's check that out.
So dear on the score 10 dot because we've created a column auctioneer I.D. is missing.
See here we've got label plus is missing.
Now this should be true and false beautiful.
What that means is that we filled up twenty thousand one hundred and thirty six values in the auctioneer
I.D. column.
Not that is missing column because maybe we just made this with the median of the original auction idea
column for now.
That was pretty full on but we've still got some missing values in our data front much checking out.
What else do we have.
We filled up all the numeric missing values but we've still got some of the eight we still got a fair
few all these ones are zero.
They don't have any missing value so that's good.
They're getting ready to use with our machine learning model but we still have to fill all of these
before we can use our data with a machine learning model.
So let's do that in the next video.