OK.
In the last lecture we learn what gradient descent is.
In this lecture we are going to see how to use this mathematical technique to find the optimum W's and
B's for this.
We first need to understand the error function which we discussed in the last lecture.
Here are the five steps that we use to implement gradient descent.
The first step is to give random ideas to all WS and B's in the system.
Then we take one training example and put its x values as input to our system.
We process through the entire network to get one predicted value.
Now on this third step I told you that we measure the distance between predicted and the actual value
using an error function.
Let's see what this means.
Suppose we predicted an output of zero point three.
Whereas the actual value is zero.
One way of calculating error of prediction could be just to subtract these two.
That is finding out actual minus predicted which will be zero minus zero point three giving us minus
zero point three to remove this negative sign in the and focus only on the magnitude of this error.
We can simply put an absolute function or a squared function on top of it.
Basically meaning minus zero point three would become point three.
Or it would be squared and it will become zero point zero nine
these two are good measures of error.
But they do not work well when we are doing classification with neural networks.
For this purpose we use a different function
this function is called Cross entropy loss function.
It is represented by this formula.
He is equal to minus of Y into log y dash minus 1 minus Y log of 1 minus Y life here.
Y represent actual value and Y dice represents the predicted output value.
I know this looks complex much complex tended to edit functions that we saw in the last slide.
But the reason for using this is that this function does not have local minimums that is the graph of
this function.
Looks like this one on the left and not like this one on the right.
If a function has local minimums are gradient descent won't work properly and it might stop here.
Instead of finding the global minima which is hit
if you don't understand the last comment.
Don't worry about it.
The simple takeaway is for classification problems.
The added function to be used is this cross entropy at a function.
We can take a look at this edit function to build some intuition around it.
As you know in classification problems the output value is either 0 or 1.
So if the output value is 1 the second part of this function that is 1 minus Y.
This entire part will become 0 because 1 minus 1 would be 0.
If the actual output is 0 then the first item of this equation will become zero and only the second
time will remain.
So let's see if the actual output is 1 for this edit function to be minimum the function should be as
close to zero as possible.
Let's see if y is equal to 1.
Edit is minus of 1 and 2 log y that plus 1 minus 1 log 1 minus Y that the second term becomes zero.
So we are left with only minus of log y dash so for this error to be small minus log y that should be
small.
This implies that log y life should be large this well implies that why that should be large since our
predicted output is between 0 and 1.
Why does being large simply means that y dash should be as close to 1 as possible.
Similarly if actual value of output is 0.
The first term of this equation will be 0 so the added function remaining would be minus log of 1 minus
Y dash for this error to be small log of 1 minus Y Dash has to be logged implying that 1 minus Y.
This has to be large implying that Y should be as small as possible although I have not given you the
mathematical justification for using this function but I guess with these particular examples you are
getting the feel of how minimizing the error or loss function is trying to match the predicted output
value to the actual value.
So now you may have guessed the job of gradient descent is to find the minimum of this error function
that is we will make small changes to the values of weights and biases in that direction where we get
maximum decrease in.
Edit We will continue changing WS and B's they'll know for their degrees in it it is possible this is
all the process looks graphically
for ease of understanding.
I have represented all of DVDs on one actors and all biases on another axis and on the vertical axis
we have the corresponding value of error these values of error are calculated using the error function
okay.
So now let's revisit our steps to implement gradient descent.
Again the first step is setting random initial values of W and B.
Then we go forward to get predicted output value.
Then we put this predicted output value in our lost function to get the error prediction.
Now we have the error W's and B's say we have a W value between 1 and 2 a biased value between 0 and
minus 1 and added value near 1.
So we are nearly here on this graph.
Now in the fourth step we do backward propagation to finding direction of movement on this graph.
Which means we find Delta W and Delta B.
That is the change in WS and B's that will take us to the minimum point if you look at this graph.
You can probably see that by decreasing the weight and increasingly biased values we are will be moving
closer to the lowest point.
So basically we have initial WS and B's.
We will be updating our W W minus all four times delta W and will be updating or b to b minus alpha
times delta B head Alpha is called the learning rate.
Basically Delta W and Delta B are unit steps that we calculate using calculus.
Alpha is controlling the number of those tapes we take in that direction.
You can imagine the impact of large vs. small values of alpha if alpha is large we are taking multiple
steps in the direction of gradient descent.
This means that we can reach t bottom faster but problem with large alpha is that we can overshoot from
the minimum.
Imagine you're very near to the bottom but on the next time you take 50 steps instead of just one in
such a situation you will climb the curve on the other side.
So a large learning rate can help in faster descent but can face issue in the final stages of convergence.
Therefore a moderate value of learning rate is to be used.
You will see what value of learning rate is to be used in practical section of the schools.
Very well.
So the step to be taken in the direction of the descent is alpha times.
Reader W and alpha times there to be now.
How do we find the letter W ending the B here.
Delta W is the change in wait and Delta B is the change in bias.
Basically we will change the initially said W's and B's in the effort to reduce error.
Now let us see how to find Delta W and Delta B.
These values are formed by doing backward propagation which means we will look back in the network to
find out the instantaneous slope with respect to eat W and b
let me take an example with a single neuron to show you how this happens.
Otherwise the mathematics and calculus in more can get quite messy and is often overwhelming for some
student if you are comfortable with calculus you can look at the complete back propagation theory in
the link shared in the description of this lecture.
However I think with this simple example you will get a solid intuition of how back propagation works
here's a single neuron with two inputs X1 and X2 it first calculate linearly.
That is it will calculate the value of Z which is equal to w 1 X1 plus W2 x2 plus B1.
It then applies a sigmoid on this value of Z
this sigmoid of the is the predicted output of this neuron.
We used this predicted output with the actual output to get the error of this particular training example
so let's start with step 1.
Step 1 is we have to randomly initialize the values of weight and bias.
We have to wait and one bias.
We randomly initialize w W1 2 with 2 W2 is equal to 3.
And bias.
Value is equal to minus 4.
Now the second step is forward propagation.
That is we will take one training example and put the input values of that training example to get a
predicted output.
We have taken this training example in which X1 value is 10 x2 value is minus 4 and the output is 1.
This way is the actual output and it is equal to 1.
So we have the w 1 value.
We have X1 we have W2 x2 and B1.
So we can calculate z.
We put all these values to get a Z value of 4.
We apply the activation function that is the sigmoid function on this value of z.
Kennedy predicted output of this neuron to sigmoid of Z that is sigmoid of 4 gives a predicted output
of zero point nine two.
This predicted output value is divided x value that we will use in the edit function.
You can see that this values already very close to the actual output which is 1.
But let's see how we can improve this value.
Now the third step is error calculation.
We have the error function with us.
We have predicted output value.
That is why dash it as 0 1 9 8 2.
And we have the actual output value for the training example as one we put these two values and this
error function to get a final added value of zero point zero 0 7 9
now country fourth step which is back propagation the next few minutes are going to be a little heavy
on mathematics.
We will cover some basics of calculus here.
If you're not comfortable with this part it is still okay.
This is happening in the background and your software is handling this but if you have some understanding
of calculus looking at this example will tell you how a neuron is doing back propagation so do not worry
if you do not understand this because this is happening in the background and your software is handling
this.
It is good to have this infusion if you know a little bit of mathematics so let's see how to do backward
propagation.
We are at the end.
We have calculated it.
The first step is finding out the slope of error.
With that predicted output that is by dash this symbol here.
The Ebi data very nice simply means that we are finding the instantaneous slope of error with respect
to wide eyes keeping everything else constant.
So if you know calculus you can find a derivative of this function with respect to wide eyes.
Then is equal to 1.
This gives us an output of minus 1 by white ash.
We go further back in our network and we find out the slope of our output function with respect to Z
the output function is a sigmoid function.
The slope of sigmoid function with respect to Z is this value it is to but minus the upon value plus
it is about minus C the whole squared if you know differentiation you can differentiate this function
with respect to the and you will get this value of slope.
Lastly we find a differential of Z with respect to W1 W2 and B so Z was equal to w 1 times X1 plus W2
times x2 plus B1.
So when we find out the differential respect to w when we get X1 which is equal to then at this current
point for W2 we get x2 which is equal to minus 4 and 4 B we get a slope of 1.
Next comes the process of combining all of this.
We moved back in our network to find all these slopes but the slope we are actually interested in is
how does the edit function change with respect to w 1.
How does it change with respect to W2 and how does it change was meant to be to find the differential
of E respect to w one we applied chain rule which means that if you want to find differential of E respect
to w 1 you can instead find differential of E respect to by Dash multiplied with differential Avinash
with respect to the multiplied with differential of the respect to differential of w 1 we have calculated
all these 3 values in our last light you can see on the top here we know the value of wildlife we know
the value of Z for this particular training example we can put all these values and calculate this differential
and it comes out to be minus zero point 1 8 6
we can do the similar exercise for W2 and Fort B also for their differential of E respect to W2 comes
out to be zero point 0 7 4 6 and the differential of even spread to be comes out to be minus zero point
zero 1 8 6.
Now these three differentials are the unit steps that we are going to take in the direction of our descent.
These are the data w ones the land w twos and Delta beats we are going to use these delta values to
update our weights and biases.
So that we move in the direction where the final loss would be less than the loss that we had earlier.
This brings us to the last step last step is we have to update W and b the new W one would be previous
w one minus alpha times delta w one previous w one was to Alpha we have taken as five we have taken
a learning rate of five year and we calculated Delta w one as minus zero point 186 this updates R W
and value to two point ninety similarly we calculate W2 value and it comes out to be two point six and
we update b value and it is now minus three point nine you can compare the previous and new W on W to
be values earlier w one was to notice two point ninety three earlier w two was three nowadays two point
six earlier B was minus four.
Now it is minus three point nine
now since we have updated our WS and B values we have to go back to our step two we have to reiterate
we have to do forward propagation again and we will calculate the predicted output again so this is
the r training example x1 is 10 x2 is minus 4 y is 1 we put these values without updated rate and bias
this time the z values come out to be fourteen point seven when we apply our activation function on
this the value we get the predicted output value that is why Dash has zero point nine nine nine if you
remember last time we got a predicted value of zero point nine it too so clearly this is an improvement
over the last values of WS and B's this process is repeated several times till we get minimum error
if we have a lot of neurons in our network the same process is followed in forward propagation we go
to the end to find a predicted output value we use that predicted output value to find the loss then
we stepwise come back do the differentials find the individual differential values with Ed function
and then we update R wait and biases so that the final edit is reduced again I will repeat that I understand
that this lecture was a little mathematics heavy but if you have some background calculus I am sure
you would have understood but if you do not have any background in calculus I understand that you would
be facing some difficulty in following all the things that I said try to listen to this lecture again
if you are facing difficulty if you are still unable to follow the concept here Do not worry you can
still implement a neural network in a software tool all this mathematical calculation will be done by
this software tool and you do not have to do anything on your own.
That is the beauty of neural networks.
If you have to do it with hand it will take a lot of pain but with computers you can have millions of
neurons and millions of features and your computer will still be able to solve it
so do focus on the practical lecture that is where you will learn how to implement these neural networks
in this software to.