Hello and welcome back to the course on Machine Learning.
Today we will finally find out about the kernel trick.
So let's get started.
So here we've got the Gaussian or the radial basis function kernel.
And those are two interchangeable terms.
And now let's have a look at this function.
So K stands for kernel.
And it's the function f function applied to two vectors the x vector.
So this is a just a some sort of point in our data set.
And L stands for landmark I means there might be several landmarks,
but we're going to we're not going to worry about
I for now we're just going to look at this as a landmark.
And then that equals to an exponent in the power of minus the
the double vertical lines mean the distance between x
and the landmark squared divided by two sigma squared.
So I know this might all seem very confusing right now.
And, you're probably wondering, Carol, this makes no sense whatsoever.
What is this, even mean?
Well, let's explore this through a visual example.
So here I've got, an image which represents this particular function
for a specific sigma, for a specific landmark.
but this is what it looks like,
in a, when you visualize it.
And so what's happening here is, we've got l
the landmark is actually in the middle of this, plane.
So in the middle of this two dimensional space,
let's imagine this is the X coordinate.
This is the y coordinate in the middle. We've got zero zero.
And that's where the landmark is actually located.
And then, the vertical here, the vertical axis
represents the result that we get when we calculate this.
for every other point on this,
z axis, on this XY field or plane,
if we take any other point, then, this the results of this calculation.
So let's say,
we put that point in here and then we calculate the distance to the landmark.
then we square divided by two sigma squared
where sigma is some fixed parameter that we decided upon earlier.
And then we take the negative of that, and then we put the exponent
into that power.
Then we get this result.
That's what it will look like.
So let's, go through this step by step.
Let's, look at the tip of this,
is right in the middle of this whole,
XY, plane.
And so if you project that into back onto the plane
and the bottom, you'll have the kernel that's, that or not the kernel.
That's the, landmark. We're gonna call it the landmark.
That's where, the middle of this bottom square is.
And that's from where we're measuring, the distance.
So, k, this distance x minus l I this distance
that we're you squaring here, it will be measured from that landmark.
So let's, take a point and look at it.
So there is a point, it's somewhere on a plane.
It's quite far away from the landmark.
There's a distance.
So we're taking that distance.
We're squaring that distance, dividing it by two sigma squared.
Take it negative.
And then we want to see what the result will be.
And so, how can we confirm that this visualization
is actually indeed, aligned with this formula.
So it's pretty simple.
the distance here, let's assume it's quite large.
Right.
So it's quite a large distance
compared to some other points that are closer to the landmarks.
So basically the distance here is a large number.
And if we take a large number and we square it right,
we get an even larger number.
And then we divide by two sigma squared.
It's still assuming it's still a large number.
Again depends on the sigma.
And we'll find out the role of sigma further done in this tutorial.
But assuming this is still a very large number.
So you've got a very large number here, and then you say my negative,
you're making a negative a very large but negative number.
So if you take in an exponent
and you put it into a power of very negative, a very large
negative number, so e to the power of one, let's say -1,000,000.
Right.
Some, you know, just for argument's sake or -1000, what does that give us.
That gives us a value very close to zero.
So it's it's basically equivalent to saying one divided
by e to the power of a thousand.
And that is a very, very small number.
So that basically means when you're far away from the,
landmark or from the center, you get
pretty much zero on the vertical axis and which aligns with our image here.
Now let's have a look at another, example.
So this point is actually closer to the landmark.
And here if we measure the distance is quite small.
So now if you take a small number, you square it and you still have a small
number, you divide by two sigma square, you still have a small number.
So you look at e to the power of minus a small number.
So let's say e to the power of minus, I don't know, like,
point or E to the power of minus.
one, for example,
or E to the power power of -0, 0.1.
so that basically means.
So this number is close to zero is it's,
as you get closer to the landmark, the this number gets closer to zero.
So and we know that e to the power of minus
zero, -0.01, 0.0001 and so on.
Basically as you get close to zero e to a power,
we get closer to E, to the power of zero.
And each of the power of zero is one.
So basically, as you get closer to your landmark,
this number over here, this number here gets smaller and smaller
and smaller, and this, the whole right part over here converges to one.
So it becomes bigger, bigger. We get bigger, bigger.
And you climb this hill up to the top where you get to one
in, the very landmark itself.
So when you exactly hit on the latter, you hit the landmark, you get to the top.
And so that is just a quick way of checking that
this image is indeed, the kernel function that we're looking at.
And, what why is this all useful?
Why do we need this?
Well, because we're going to use this kernel function to,
separate our data set to build that decision boundary.
So let's have a look.
there is our two dimensional space, right?
And there is our X1, x2, just like we had here x1, x2.
And now what are we going to do is we're going to take the landmark
and put it somewhere in our,
in our, among them, our, data set.
And there is a whole methodology on how the machine learning algorithm,
when you implement it in R or Python or any other language, how it does it.
And we're not going to go into detail on that
because we just focus on the intuition.
But basically there's a way to find an optimal placement for these landmarks.
And so landmark is placed.
And next what happens is the distance,
as you can see here, the circle, the circumference around
this, kernel function
is actually projected here onto our visualization.
So, what this circumference allows us to do is it allows us to,
take all of the points that are within that circumference
and have them, like,
assign them a value of above zero.
So anything outside the circumference, all of this blue stuff.
So basically all of these red points, they'll get a value of zero right.
If you apply this function or a valid value very very close to zero.
If on the other hand, any point falls within the circumference
based on this function, it will get a value of above zero.
And that is how we can separate the two classes the green from the red.
Just if we pick the right sigma.
So here we know that, sigma actually, well, we don't know that yet,
but what's a Sigma role is that it
defines how wide this circumference is.
So if you increase sigma,
the circumference will increase like this picture and change.
But it just it should change it.
the circumference here would increase and it would take more, the space up.
or if you decrease sigma, the circumference will decrease
and therefore, you'll take less points.
And so basically, by finding the right sigma, you can set up
the correct, kernel function to
assign
zero values to all of the points that you don't want in your classification
and values above zero to the points that you do want in your classifier.
And that will allow you to, separate
the two that will allow you to classify, each one.
And that, in essence, is the kernel trick we have created
a decision boundary
without actually going into a higher dimensional space,
without having to, project all of our or create a mapping function.
that's going to, you know, take us two dimensional space
and do all the computation. The point is, we're not doing
the computations in the higher dimensional space.
We're still doing the computations in the low dimensional space. Yes.
We have this, visual representation that involves a higher dimensional space.
But at the same time, if you look at the, computational part,
we were just calculating this formula and then we're saying, if this is greater,
if this is equal to zero, then assign, you know, class red, if this is greater
or equal to greater than zero, then assign clause green.
If we if you look at the computation, it's actually happening
in still in the two dimensional space.
And that's called the kernel trick.
So all of a sudden you can adjust your,
decision boundary and, and it's non-linear.
And moreover you find yourself being able to solve much harder,
much more complex problems like this, for example.
So here this is a much it's a very simplified formula.
But if you take two kernel function
and you just add them up, in reality they have to be coefficients to,
to here, to, to here and then another to to before.
so they have to be coefficients with these with the kernel formula.
It's a bit more complex than that.
But in simple terms, if you take two kernel functions
and you just add them up, then because the,
the value of this function, if, let's say this is kernel,
or this is your a landmark one, because the value of the function
when you get further away it becomes zero. Right.
They don't really interfere.
So as you move away from this landmark, so this landmark
will only, encapsulate all of these points around here.
And then as you move away, this will be zero everywhere, right?
Everywhere else, including in these points.
But then this one will be non-zero.
Close to this landmark.
And so if you just add them up,
you will get non-zero values for exactly all of those points.
And so therefore you can draw
a non-linear decision boundary which even looks like this.
And the formula here would be the point is assigned to the green class
when this equation is greater than zero
and the point is assigned to red class when this, equation is equal to zero.
Now again, this is a very simplified example.
In reality, this is a bit different.
It's greater or equal to zero. This is less than zero.
And that's because we have the coefficients.
And it is a bit more of a, complex, more of complex mathematics behind it.
But we don't really need to go into those steps.
The point is that we understand here that we can create this nonlinear,
very complex decision boundary
without having to go into a higher dimensional space.
Everything is still happening
in those same dimensions simply because we're applying the kernel functions.
And that is why this method is called the kernel trick.
I hope you enjoyed this explanation, and I look forward to seeing you next time.
Until then, happy analyzing.