Hello and
welcome back to your Machine Learning A-To-Z course.
Are you super excited?
Because I am super excited about this tutorial.
We're talking about nonlinear support vector regression.
This is going to be quite an advanced tutorial, but don't worry,
we will break it down.
And as per our mission will make the complex simple.
So and before we start, I just wanted to make sure we're on the same page.
So this is one of the most advanced
tutorials in the Support Vector series of tutorials.
Please make sure that you've completed all the previous tutorials,
on SVR and SVM and kernels before proceeding with this tutorial,
because you'll need all that knowledge to be able to tackle this one.
All right. And off we go.
Here it is.
So imagine you have a data set like this.
and you're trying to fit
a support support vector regression onto this, plot.
You want to see what kind of, trend line is there
or what kind of, equation is behind this?
How do you model?
What kind of model can we fit to this?
A support vector regression model
so that you can predict what the next value is going to be for.
and you point that is added to the plot.
So if you know x, y is going to be y,
well you might want to try and fit a linear support vector regression.
And it'll look something like this. You'll have these support vectors.
But as you can see like intuitively you can tell it doesn't fit right.
Like you can tell that something is going on here beyond linear.
And like while this it might model may be okay there somewhere at some points.
Clearly if you get a point with X over here,
you're going to get the incorrect value.
Or if X is over here you're going to get an incorrect value which doesn't fit.
Looks like something else is going on.
You might try something like that, then that won't fit as well.
That doesn't fit.
So it doesn't look like a linear model fits here.
And it looks like something more complex is going on.
So how do we build a support vector regression
that will fit, our data here.
Well, this is where nonlinear SVR comes in.
It's going to be, quite interesting because we need to go to a new dimension.
And that's what that's why we're going to add
this box is just for visualization purposes.
Like our data hasn't changed. Our axes are the same.
But just for us, once we're transitioning to a third dimension,
to keep track of things, we're going to add this box
and we're going to add these diagonals as well as a point, a red,
a circle in the middle.
Just so visually we can see what's going on.
it'll help us.
It's just like a visual aid has nothing to do with the algorithm itself.
It's for illustrative purposes.
So now we're going to actually take this and,
look at it from a different perspective at an angles.
So at an angle it looks like this.
and that will allow us to build that dimension Z.
Now let's get rid of the, previous view.
And we're going to make a copy of it, because there's
going to be quite a lot of things going on.
And in order to keep track of everything,
we'll have two charts that are developing in parallel.
So now on the right, what we're going to do is, just as we did
previously with supposed to support Vector Machine,
and that's why it's important to have seen those tutorials.
we're going to add a kernel.
And in this case we're going to add a new function
to take us into a third dimension.
We're going to add the, a radial basis function or the RBF.
So here we go. That's what it looks like.
That's with the RBF.
You can see our data vaguely underneath.
But here you can see it more clearly.
And just as a reminder, the, formula
for the RBA function is over there.
Now, if we project our data
into, onto this RBF, you'll see how it works.
So first of all, the center is zero or this .0000 on x,
the y axis, projects straight into the very top.
And if you, we, we discuss this before in the, tutorials,
if you put 000L is the distance from the center,
you'll get, at the very top, you'll get a one.
Now let's see how these other points,
the other points in our visualization will be projected onto the view.
So we're going to start with this point over here.
So that point will be projected into this point in the z axis somewhere here.
By the way, some of these things might not be absolutely 100% accurate.
as I'm, drawing them here, but they deliver.
Hopefully they deliver the point home. Okay.
So now this point would be projected to over here somewhere
next point, over there and this point, this point.
And then the following points would be almost at zero.
So all this blue stuff over here, the bottom is very, very close to zero.
So it would be very close to zero.
So you can see we've been keeping track on on the left.
We've done all of these points over here.
Now the other points the remaining points these ones
they're actually we can't see them are not in order not to clutter things.
Anything in this quadrant, in this triangle
here, we're not going to plot it because it's on the back.
It's on the opposite side of this radial basis function or this, view.
And we just are going to plot it, but just we can just imagine
that they are also projecting onto this mountain on the other side.
Now, what happens next?
Well, next we need to in,
these three dimensions, we need
to, create run a linear model.
So now we're in three dimensions.
Now we can run a linear model and a linear model in three dimensions is not a line.
So in two dimensions it's a straight line in a three them.
In three dimensions a linear model is a hyperplane.
So now if we run a linear model you'll see that it will.
How does it fit our data would fit our data.
Something like this.
and what do we want from here.
So once we fit the hyperplane
to our data in these three dimensions, what do we want from here?
we want to see where does it actually cross,
where does it intersect our RBF, our radial basis function.
So basically all the surfaces we see here, including the floor, this whole surface
where does it intersect will intersect it over across this yellow line.
So that's the point of intersect or the line of intersect.
Now if we get rid of the plane, we're left of this line.
And if we projected back down onto the two dimensions x and y.
Look watch what happens.
So we're not going to draw it on this plot. We can draw it on this plot.
So watch what happens.
There we go. That's our two dimensional line.
So so that's our intersect line projected on the two dimensional plot.
And that effectively is our model.
So if we now look at it in the perspective that we started with, you'll see that
that is what it looks like.
That's the, non-linear SVR.
Now, I know you probably have a few questions.
Let's, try to dig into some of the,
comments or some of the concepts behind that.
So first of all, here we have this two dimensional plot.
Here is our, three dimensional three dimensions.
So why is this SVR right.
So where are the support vectors? Well, you know, three dimensions.
You know, three dimensional space.
SVR would run very similarly to how it runs in a two dimensional space
instead of having but instead of having like a, epsilon instead sort of cube,
we would have an epsilon insensitive space between hyperplanes.
So we would add these two new hyperplanes.
One would be epsilon above, one would be epsilon below.
Remember how we had for linear SVR.
We had just a tube.
but now here we have this space in between them.
And so basically what that means is any, points
that are in between, these two x like the most outmost,
hyperplanes, any points in between them, they won't be considered towards,
like, the error for those points will not be considered.
What we want to do is we want to minimize the error, from this epsilon
insensitive space to the remaining points that are outside.
And what this looks like over here is that's our bottom, hyperplane.
That's its projection away, or that's a projection of, the part
where it intersects with our,
radial basis function, and that's a top, hyperplane.
That's its intersection or the projection of its intersection.
So this line over here is projected into that one.
And, basically anything
in between was, the epsilon
insensitive space that has become this epsilon sensitive tube.
And these are the support vectors.
The crucial point here is that this is the outcome originally.
This is these hyperplanes are identified
or built from this view from the three dimensional view.
So that's where these points come in place.
So this point is actually underneath.
We can't see it. It's over there.
And there's also these two points that are on the left.
So from here they're built.
And then this is just a projection just to put it into a perspective.
So that's why it's still a support
vector regression because we have these points now.
hopefully that explains why it's a support vector regression.
There's just one last thing I wanted to mention.
The final thing is that everything
we've been talking about here is for illustrative purposes.
In reality, the way this works is very similar
to the how the support vector machine works,
how what we discussed about the kernel trick that in reality
we don't have to go into a third dimension.
Do all these projections and then, find our hyperplanes,
find this epsilon sensitive space,
and then predict
everything back and get this, it would be too computationally expensive.
So what we actually discussed here is, a more complex approach.
This would be like the full fledged approach with going into a third
dimension, doing the hyperplanes coming back, getting our end result.
That would be, like the The Full Monty in a way.
But effectively what happens or in reality is that we use the kernel trick.
So just like with the support vector machine,
we don't have to go into a third dimension.
Everything happens in the same space simply by how we use the kernel trick.
Now, we won't dive into more detail on that.
I'll let you, think about that, in your own time on how the kernel trick
would be used here, because simply it's a it's an easier concept than what
we discussed by looking at the example of the kernel trick in the SVM,
we can extrapolate how that would work or how that would benefit us here as well.
But in a nutshell, this is what the non-linear support vector
regression is all about.
And, the best part is, of course, the result that it allows us to model,
non-linear relationships like this
and get, insights from our data.
I hope you enjoyed this tutorial.
This was quite an advanced concept, so congratulations on making it to the end,
and I look forward to seeing you on the future tutorials in the course.
And until then, happy analyzing!