OK.
Now we've made a copy of our original data frame we can start to edit DFT temp without any worries of
damaging our original data.
So if we ever needed to revert back to it we can with no troubles at all.
So what we're going to do in this video is we're gonna take these dates.
We're going to use pandas day time as well as some pandas date time attributes to enrich our dataset.
And if you're wondering what is Daniel talking about enrich our data set while there's little practice
that we're about to dive into it's called feature engineering.
If you remember right back up in step four.
Well we've covered a lot already.
We've got a whole bunch of features and these are all listed here.
What feature engineering means is it creating a new feature.
So something like usage band may be completely different to usage man but we were just added on here
and then that could be a description or we would dive into one of these and go sale date and we change
it around a little bit.
So that's the process of feature engineering so let's do that.
But with the dates.
Now how can we do that.
Well first of all we're going to see it in action then we're going to discuss what's happening.
This is something you might want to do for almost any time series project you're working on.
So let's make a little heading here and date time parameters for sale date column.
Wonderful and pandas has a beautiful little pandas date time.
Well I just did pandas data.
I'm going to trigger happy date time attributes this is what we're going to be using right.
We've turned our sale date column into a date time index.
See there immutable in the array of data on data what we can do now is access some attributes of that
date.
This is what I mean here.
So we can go year the year of that date time the month the day the hour the minute the second the microsecond
nano second.
Far out.
That's that's getting pretty deep.
Is it the start of the month.
Is it the end of the month.
Is it the start of a quarter.
Now I want you to think about why might this value these features be viable just without even seeing
it without even coding it up if you're making a sale.
Why would it be a good idea to maybe know whether or not it was the end of a quarter or the start of
a quarter or the start a month or something like that.
Why would it be now.
It's okay if you're unsure but this is part of being a data scientist or a machine learning engineer.
Right it's about knowing a little bit about everything.
So if you're working on time series sales maybe it's a good idea to know whether it's the start of a
quarter because sales might be increasing because it's the start of quarter or maybe it's the end of
a quarter and sales are going down because everyone's tied on budget.
And maybe if it's the start of the year no one wants to buy anything but it's the end of the year and
people are like oh yes let's let's use the rest of our budget let's buy heaps of things you don't have
to know in-depth knowledge this is just a one thing you can do to enrich your date time objects.
So let's go in here how can we do this.
We can add a column what we can do is add a date time parameter for the sale date column on the end
of our DFT help.
And now again talking about it isn't as good as having a look.
So we're going to say oh yea and how would we do this.
So what we're doing here is we're making a column DFT hemp sale year Eagles def Tim sale date.
So this means we're accessing the sale date column and then we can access the date time part of Saturday
by using D.T..
So DTV date time and then we can access the year attribute by putting diet year.
So let's have a look at this.
What does this tell us.
Daytime indexed or what year the year of the date time.
Mm hmm.
Before we even reassign a column let's just take the first row and see what this does.
Nineteen eighty nine.
Okay.
Now let's do the same again but let's do it with Day okay 17.
All right.
And now if we just did it with nothing.
So if we just did the sale day.
There we go.
So we see that that's access the year that's access today.
So let's enrich our data frame with a few columns pertaining to these different values and if you're
not sure what I mean.
Let's let's see in an action.
If in doubt run the code sale year equals that and then what we might do is we might just do month day
day of week day of year.
And of course we could do more but in the essence of time really we can make a function that does basically
all of these.
But what we're going to do is just do five or six to demonstrate.
So let's do that.
We could copy it but we're going to practice coding it out.
So we go here say all month equals DL temp dot sale date tab order complete a month and DL temp sale
day equals the Tampa sale date DTA day DL temp and we're going to go sale day of week.
And that was when we said day of week Can you read that.
I mean we did camel case beautiful acres DLF temp dot say sale date dot EDT dot day of week the temp
sale day of year equals def temp.
Now as you can see there's a lot of redundant code here.
There might be a function in there somewhere.
That's a good challenge for you is to see if you can function like this and maybe even go to the panda's
documentation and add a few more different parameters to it to enrich our data set.
And now what do you think that this might do well let's have a look.
We'll run that and then we'll go here def 10.
We're going to view the head and we'll transpose it so we can see all the columns.
Let's go right down the bottom.
There we go.
So what we've done is we've created so see here.
So a year seven months our day Soledad we sell today of year we've created these five columns.
And what it's done is it just taken the sale date and extracted some of those values and added them
as more features on the back end of our data frame.
Now of course these features won't be listed in our data dictionary but if we were to go in and share
this with someone else maybe that would be something we add to the data dictionary but we won't do that
for now.
And so with this information here we probably don't need the sale date column anymore this one here
probably a bit excess now.
So what we might do is remove that.
So let's drop it.
So we'll put it in here a little comment.
Now we've enriched our data frame and remember because we're working on 10 if we ever needed to sell
date column back.
We could just go back to our ADF our original data frame that we we imported now have enriched our data
frame with date time features.
We can remove Saturday.
Let's go DFT Tam dot drop and we're going to go south right on axis equals 1 in place Eagles true.
Remember in-place happens in place so we don't have to go DFT equals
wonderful.
Now if we tried to do this we're going to get an error because SDI doesn't exist anymore but that is
okay because we've got the information it's just now across five different columns then and then then
boom.
And if we ever wanted to back we'd go back to our original data frame now I also want to check the values
of different columns.
Now this is all about HDI Rod and exploratory data analysis is just really checking out what's happening
and what we might do.
So let's pick one
what do we got here.
How about state.
US state in which sale occurred wondered which state has the most sales.
If you're from the US whatever state you're in direct in a direction you've made the most sense.
Let's have a look.
Value counts maybe it's gonna be Wyoming.
No Florida Texas a lot of a lot of sales are happening Wow.
OK.
So Washington does why Wyoming even have a single sale
I can't see it and at least maybe you can.
There we go.
Wyoming.
One thousand six hundred seventy two.
Okay so this another part of.
You could check more columns if you want.
You could go through and check different value accounts.
Remember value Kansas is really handy.
This is you're just looking at the spread of data.
That's another way of looking at the distribution only tired of of exploring the data let's do modelling.
Yeah let's do that and that's why more fun building a machine learning model.
So I've done enough.
I could always do more.
But let's start to do some model driven 88 now.
What is model driven 88.
Well because there's so many features here we could spend all day just going through each of these columns
finding the value counts doing a distribution floor making more plots but at the end of the day we kind
of know what our valuation metric is.
So why don't we try and just head towards that see and if we can.
Well how would we build a model.
Let's build a machine learning model.
But member what are we trying to do.
Well if we come back let's check data frame.
We're trying to use all of the independent variables to predict sale price.
All right.
So that means if we just drop all the columns drop Sal Price that should be out x and then if we use
Sal Price the column itself that should be our y.
But what kind of machine learning model do we use.
Well remember we've got psychic line machine learning math if we go here choosing the right estimate.
And this is our all faithful.
So if we were to start here if we go through this do we have above 50 samples.
Yes.
Are we predicting category.
No predicting a quantity.
Yes we are predicting a number.
We are doing a regression problem.
Now if we were to follow this through we've got a bunch of different model options here in the green.
But because we already had some experience with random forest a random forest classifier that is what
we might do is checking out its counterpart the Random Forests regress.
So if we were to go through here forests to randomize trees there's a classifier and there's random
forest regress.
All right.
Well let's see what we can do.
We've seen this before with psychic line of how to import a model.
So if we go from S line dot ensemble and remember if you want to see exactly how to do this you could
just go into here random forest regress.
And then I'll tell you what to do.
There might even be an example down here.
There we go from SCA learned on ensemble import random forest regress.
That's what we're gonna do.
Import random forest regress.
Paul this is exciting we're getting into modeling already.
Well instantiate a model so model equals random forest regress.
And we're going to set the number of jobs to equal negative 1.
What does this mean.
Well that means it's going to use as many cause as possible on my computer because we're dealing with
how many rows are we dealing with DFT.
We want the len we're dealing a four hundred and twelve thousand rows so that means our machine learning
model has to go through four hundred twelve thousand plus rows of this try to work out all the patterns
between these columns and then try and calculate Sal Price.
That's a lot.
And what we might also do from now on is remember how we used to set we could do this any random seed
42.
We could set that in all of our cells.
But another way to do it is with our models there's a there's a parameter called random state.
We can go equals 42 and remember to check the documentation of what's going on oh have we not a I can't
press shift tab for some reason.
I'll just go to the documentation here.
We'll see what random state means.
Here we go.
Controls both the randomness of the bootstrapping of the samples when used when building trees.
If bootstrapping was true and the sampling of the features to consider okay.
So long story short this is going to.
It acts like any random seed and if we go here and jobs there we go.
None means one unless job lab negative one means using all processes.
So these are the two that we're setting right.
Wonderful.
And so to fit it we would go for it and we need to drop the sale price columns.
This is our ex SO sale price and access equals one.
So we're dropping Sal Price and then we're going to pass it just the sale price column as the target.
So this is Model not fit.
This is ex and this is why so DFT lamp sale price.
Let's say if it works.
Oh no.
We got a value error.
What's it saying.
Could not convert string to float low.
You know why this is is because if we go DFA dot info all of our data is not in numeric format are int
float int object usage band.
Okay.
What what type is that.
Let's have a look.
This is part of model driven EDTA right where our machine learning model just told us that thanks to
the beautiful people of psychic line they've put in these errors to help us out of what's going on.
So usage.
No not use a warning.
Usage band dot D type.
What's gone on it.
D type O object.
Okay so maybe it's a string or something like that and the other thing is I just remembered we've got
a whole bunch of missing values.
So from what we know about our machine learning models which we're again modeling but we've got data
that's not numeric and a whole bunch of data that's missing.
So before we build a machine learning model we'll have to take care of that.
So that's really what we'll do over the next couple of videos we'll do some model driven PDA.
And the beautiful thing is is we just tried to run that machine learning model and it told us hey we
can't it's not going to work because you've got non numbers in your data.
You've got some strings in your data.
And if we were to go again it might tell us eventually that we've got missing data.
So that's what we need to handle first.