Welcome back to your machine.
Learning A to Z course is super excited to have you back on board.
And today we're kicking off the support vector regression intuition.
Super pumped about these tutorials.
Got some very exciting slides coming up for you.
So what is support vector regression?
Well, support vector regression was invented back in the 90s, by Vladimir
Vapnik and his colleagues who were working at the Bell Labs at the time.
That was that AT&T Bell Labs down there and knocked Nokia Bell Labs
and, a lot of, support vector
machine and support vector regression are discussed in, Vladimir.
Up next book, the nature of Statistical Learning, 1992.
In this course, we will be covering both a support vector
machine, in the part of the course to do of classification
and support vector regression.
We'll be talking about it here.
And in addition to that, we'll also be talking about,
kernel support vector machine and kernel, a support
vector regression and the kernel trick and many other things like that.
So there's a lot of exciting tutorials on these topics,
but for now we're just going to limit ourselves to support vector
regression, specifically linear support vector regression.
So here we go. here we've got two plots.
And we need that in order to compare SVR to,
the simple linear regression that'll really help us understand things.
So here on the left we've got some random plots, random dots.
I'm going to copy them over to the right.
So we know that these are identical.
There's no tricks involved.
There's an absolutely identical.
this is an absolutely identical set of data.
And, let's start with the one on the left.
We're going to apply a simple linear regression.
We've already discussed what it's like, but let's quickly refresh.
So basically we're going to have this line go through the data.
And how is this line derived.
Well a method called the ordinary least
squares method will be applied to find this line.
Basically we want to minimize, the distance between the
this value y, the actual value in the data
and y hat basically what it would have been on the trend line.
We take the difference here or difference here.
We square it and we want to minimize that.
That's ordinary least squares method.
If essentially what we're doing is minimize
the error, we want to have a line with the minimum error, possible.
So that's that's the, intuition behind
a simple linear regression, something we already talked about.
Now, how does this, support vector regression work?
Well, let's have a look on the right
with SVR instead of a simple line, you'll see a tube.
And here you have the regression line in the middle.
But then there's this tube around. And what does this tube do.
Well this tube has a width of epsilon and the width is measured vertically.
This important along this axis not perpendicular to the tube but vertically.
And this tube itself is called the epsilon insensitive tube.
And what does that mean.
Well, that means that any, points in our data
set that fall inside the tube, they won't be.
We'll be disregarding the error.
So basically this tube, think of it as a a margin of error
that we are allowing our model to have
and not care about any error inside here.
So any discrepancy between or any like distance between this,
point over here and the line as in like for instance here we could see
let's look at which point is that that's,
one, two, three, this third point, you can see the line is even different, right?
The results can be different and probably will be different.
So this third point here, there's a distance between the line here.
And we care about this error here.
We don't care about this error because it falls within this epsilon
insensitive tube.
So we're disregarding any kind of errors in here.
And that's kind of the key behind support vector regression.
And it gives a little bit of movement
or a little bit of buffer to our model.
And at the same time we have points that are outside
the epsilon insensitive tube.
there they are.
And for them we do care about the error.
And here will be measured as the distance between the, the point
and the tube itself.
So not the trend line, but the tube itself.
these distances have names.
They're either C star.
If, the point is below the tube, or see
if the point is above the tube and, they're called.
So these values are called slack variables.
So it's either C star if it's below C farthest star or if it's above.
And we do care about the error.
So we care about these distances and the way we care about it.
So we're going to try to avoid formulas that will give additional
reading something you can look into further down at the end of this tutorial.
But just for for completeness sake, here is the formula.
So in the case of OLS, it was a simple,
ordinary least squared like that here it's a bit more complex.
we're not going to talk about this part over here.
but what we're focusing on is this.
And we can see that we're minimizing.
We want these, distances, the sum of the sum of these distances
to be minimal, once again, will be additional
reading at the end if you'd like to go into it.
But effectively, these are points.
The ones that are outside our tube are dictated
what the tube will look like, how the tube will be positioned
so the error within the tube is completely disregarded.
We don't care about the error unlike in the ordinary squares,
so we're giving some kind of buffer or flexibility to our tube.
or like allowing it to, accounting
for some kind of error that, we might expect in the data.
It's normal
sometimes for there to be error, but these ones, they are important to us.
And, also one final thing.
Why is this a method called support vector regression?
Well, because effectively these points, all of these points outside.
But at any point, actually any point on this plot is a vector, right.
Can be represented
as a vector in this two dimensional space or a multi dimensional space.
If you have more features.
So in this case it's kind of presented by a two dimensional vector.
So they are all these points are vectors.
But the ones that we've highlighted
in red, the ones outside the tube, they're the support vectors
because they are dictating how this tube is created.
So basically they're supporting the structure or formation of this tube.
And that's why they're called support vectors.
And that's why this is a support vector regression.
And so there we go.
That's what it's all about.
that just important to remember the epsilon insensitive tube and that,
support vector regression
just cares about the errors of anything that's lying outside this tube.
And, to finish off, as promised, here's the additional reading.
so if you'd like to learn a bit more, have a look at chapter four Support Vector
Regression in a book called Efficient Learning Machines theories, concepts,
and Applications for Engineers and System Design by Marietta Ward and Rahul Khanna.
and here's a link here, where it's aggregated on this,
portal for, published work.
something you'll note that it might be a little bit confusing here.
They say these are potential support vectors.
They're referring only to the ones close.
I've had a look at different literature, and, the literature I prefer is
the one that says in this respect is the one that says that the support
vectors are the ones that are outside and they are, outside.
Any, any, basically any point outside the tube is a support vector.
So that's how we talked discussed it inside this tutorial.
But have a look at this different nomenclature.
Maybe that'll be a good perspective to have a different view.
But overall the first couple of paragraphs describing the whole problem, they're
very well written. I liked how they were written.
And I think, can be a valuable addition if you're looking for additional reading.
So there we go. That's, support vector regression.
Hope you enjoyed this tutorial and look forward to seeing you next time.
Until then, happy analyzing.