All right.
So now we've got our own custom evaluation functions.
We can evaluate how our model is doing.
So the next logical step would be to train another one on the training data set and then evaluate it
on the validation data set or just using this beautiful function that we've crafted not so fast.
Let's see why.
So testing our model this section is gonna be we're gonna do it on a subset.
So testing our model on a subset.
Now this is mostly to tune the hybrid parameters because remember what we've been stressing we want
to decrease the amount of time it takes us between experiments and if we come back up.
Remember how we time traveled in a previous video that took six minutes 58 seconds to fit a model on
all those data.
So on 400000 Rose we don't have enough time to do that again.
We can only time travel every so often.
And so if we were to just ride it out again we would instantiate our models and model equals random
forest regress our end jobs equals negative one and a random state of 42.
Wonderful.
And then if we just did modeled outfit X train I need to be careful here not to be trigger happy if
we did this right.
If we put a little time up here time that's gonna take far too long.
Right.
So I might put a note here.
This takes far too long for experimenting because that's going to take about five minutes.
That's not what we're after.
And again if you have run this cell before actually I really should've told you this earlier if you
did run this cell on your computer and it's taking far too long for it to load.
Remember you can always press the stop button up here.
So if we run this Oh well that's too fast that runs too fast.
But if your cell is running remember how there's a little star here I'll show you.
There'll be a star while that that happens too quick for this.
That you got to trust me on this one when you run a cell.
This will turn into a star if it's running.
You can stop it by pressing this button with this cell highlighted and that'll stop that cell so come
back.
Let's not run that.
We'll make sure we can't so we commented out beautiful.
Because why.
Because our data frame is 400000 rows there and my little MacBook Pro is already calculate a lot of
patents in 400000 rise and it's like well I'm going to take a while so you need to be patient with me.
Well that's fair enough because I would take far longer than my MacBook Pro does to calculate three
four hundred thousand rows.
So what our solution is is to speed up our time between experiments and that's what we're doing testing
our model on a subset.
We could do something like this could just copy this model of it.
Maybe we want to only do it on ten thousand rows
but there's an even better way.
And it's right built into we could do that right.
That's going to test it on 10000 rows.
But now if there's a better way to do it that is one option.
So keep that in mind that you can just slice your you're training Denison but the beautiful thing about
random forest and this is why we're focusing on it so much is that they have this little feature in
it called Max samples.
So change Max sample's value.
All right.
And if we wanted to figure out what Max samples does where would you go.
Well one option is if we go model equals random forest regress.
We're gonna have heaps of practice instantiating these models.
It's what it's about though Ryan is just about almost making this second nature of what it's like to
just get a model instantiate and get it ready.
Random state equals 42.
There's our model so that could be one option but if we press shift tab here we'll see the doc string
Max features not Max samples Max samples.
Boom.
Now if we come down here let's find it.
Max samples there we go.
If bootstrap is true the number of samples to draw from X to train each base estimate.
Mm hmm.
What does this mean if a guy model like bootstrap equals true.
Yes.
And now each estimate as sees Max samples that way as Max samples none.
So what does it mean if none.
It's gonna be all isn't it let's have a look this is what I want you to have some practice doing right
is if you're not sure on something it's looking it up and reading it in the docs.
If bootstrap is true the number of samples to draw from x.
Okay so X is what we pass a model.
If none then draw X dot shape samples so it's going to draw them all.
Now if and then draw Max sample samples.
So what this means is that if we go X train does shape zero so if we leave Max samples as none every
estimate are in our case there's 100 because the default end estimate is in random forest regressive
is 100.
So you can imagine an estimated as a small model.
So each small model.
So one hundred small models are going to see every single one of these.
So that's that's a large number right.
So this is all the calculations that we have to make probably more than this.
But this is what you can imagine.
But instead if we set Max samples to something else to say for example maybe if we only wanted to to
try 10000 examples instead of four hundred thousand.
Right.
So closer to the shape of our validation set we could change Max samples set to ten thousand.
So this means that instead of looking at 400000 rows every estimate up to 100 estimates for an estimate
as we could also change this to be honest but we'll leave it as default just for this example.
So that means because we change it to ten thousand times 100 well.
So let's have a look here.
Let's divide these two.
How many times less what do we got here.
What numbers and how many zeros a million.
That's a million I can count 40 times less.
So 40 times less data to compute on.
So that means we should see a speed increase of 40 x or about that.
Right.
Because it's not going to be perfect.
So let's go here change what we might do.
Yeah.
That looks good.
And then we'll create a new selling year.
We'll need our little time here because this is what we want to practice.
So cutting down on the max number of samples each estimate I can see improves.
This is our hypothesis improved training time.
Now let's have a look model dot fit we'll do it on our training data now.
So this shouldn't take too long maybe we might even have time to wait for it still might take a few
seconds
this might be another opportunity the time travel Hall didn't need own child will march.
Look at that wall time 16 seconds instead of seven minutes.
Now that's a bit better right now.
That's something that we can potentially just sit here and wait for if you're doing seven minute experiments
all the time or even longer if you had more data you're going to lose patience you're gonna get distracted.
But if we can keep our experiments like somewhere around 10 seconds so that's that's pretty good right.
I can wait 10 seconds.
And now the beauty comes in here when we can call our show scores function.
So let's see what happens when we go show scores hopefully this works because we've coded up this function
ourselves.
Of course it will show scores.
What are we going to find.
Oh no.
We've messed it up.
No I has no spirit.
Instead of square root this is where we've messed up.
None pi.
There we go.
Typo classic couldn't get it right the first go could you Daniel.
Nope.
So we'll see how our model does.
We've just trained it here on a subset of the data be you to a fall.
Oh right.
Look at that.
So training mean absolute error.
So valid.
Yeah.
See here how I said the valid error will be slightly higher than the training.
That's good.
That means we're not over feeling and now are squared off a training invalid or are far worse than the
point nine eight seven we saw before.
But that's all right because we're only training on a subset of 10000 examples not all 400000 so we'd
expect these metrics to be worse than if we were to set Max samples to be none or train on all of the
training data.
Now what we might do is seen this is how valid I am s Ellie of point to nine I wonder where if we were
to compare that figure with the leaderboard where would we end up.
So point these is about 425 teams in this competition so if we were to submit our sample as it is now
with our valid arm SLA score of point 2 9 3 only on 10000 samples where do we end up.
Point 2 9 3.
Well we're in the top 100.
Looking at we're in the top 100 with just a simple model on ten thousand examples so somewhere in here.
All right now we've got an idea of how to train our model a bit faster what we might do is try some
different hyper parameters and I'll what you think about it before we get into the next video.
What's a way to find different hyper parameters rather than us having to manually adjust these hyper
parameters about random forest aggressor if you're not sure don't worry we'll cover in the next video
but have a think.
However we found better hyper parameters in the past because that's what we're gonna use for the next
video.