OK.
In the last lecture, we learn what gradient descent is.
In this lecture, we are going to see how to use this mathematical technique to find the optimum W's
and B's.
For this, we first need to understand the air function, which we discussed in the last lecture.
Here are the five steps that we use to implement gradient descent.
The first step is to give random readings to all W's and B's in the system.
Then we take one training example and put its X values as input to our system.
We process through the entire network to get one predicted value.
Now, on this third step, I told you that we measure the distance between predicted and the actual
value using an error function.
Let's see what this means.
Suppose we predicted an output of zero point three.
What is the actual value is zero.
One way of calculating error of prediction could be just to subtract these two.
That is finding out actual minus predicted, which will be zero minus zero point three, giving us minus
zero point three.
To remove this negative sign in the air and focus only on the magnitude of this error, we can simply
put an absolute function or a square function on top of it, basically meaning minus zero point three
would become point three or it would be squared.
And it will become zero point zero nine.
These two are good measures of error, but they do not work well when we are doing classification with
neural networks.
For this purpose, we use a different function.
This one is called Cross Entropy Lost Function.
It is represented by this formula.
He is equal to minus of white into long white ash, minus one minus Y logoff one minus my life here.
Why represents actual value and why Dyche represents deep predicted output value.
I know this looks complex, much complex tended to edit functions that we saw in the last late.
But the reason for using this is that this function does not have local minimums.
That is the graph of this function looks like this one on the left and not like this one on the right.
If a function has local minimums of gradient, descent won't work properly and it might stop here instead
of finding the global minima which is here.
If you don't understand the last comment, don't worry about it.
The simple takeaway is for classification problems.
The added function to be used is this cross entropy added function.
We can take a look at this edit function to build some intuition around it.
As you know, in classification problems, the output value is either zero or one.
So if the output value is one, the second part of this function that is one minus Y, this entire part
will become zero because one minus one would be zero if the actual output is zero.
Then the first term of this equation will become zero.
And only the second time will remain.
So let's see if the actual output is one for this error function to be minimum.
The function should be as close to zero as possible.
Let's see if Y is equal to one that is minus of one and two.
Log where that plus one minus one log, one minus Y that the second time becomes zero.
So we are left with only minus of log white x.
So for this error to be small minus log, why that should be small.
This implies that blog eyes should be large.
This further implies that Vidas should be large.
Since our predicted output is between zero and one, why does being large simply means that white ash
should be as close to one as possible?
Similarly, if actual value of output is zero, the first item of this equation will be zero.
So the edited function, the meaning would be minus logoff one minus y dash.
For this error to be small, logoff one minus Y dash has to be large, implying that one minus white
ice has to be large, implying that Y should be as small as possible.
Although I have not given you the mathematical justification for using this function, but I guess with
these particular examples, you are getting the feel of how minimizing the error or loss function is
trying to match the predicted output value to the actual value.
So now you may have guessed the job of gradient descent is to find the minimum of this error function,
that is, we will make small changes to the values of weights and biases in that direction where we
get maximum degrees in edit.
We will continue changing WS and BS.
They'll know for their degrees and it is possible.
This is how the process looks graphically.
For ease of understanding, I have represented all of DVDs on monitors and all biases on another axes
and on the vertical axis.
We have the corresponding value of it.
These values of error are calculated using the error function.
OK.
So now let's revisit our steps to implement gradient descent.
Again, the first step is setting a random initial values of W and B, then we go forward to get predicted
output value.
Then we put this predicted output value in our lost function to get the error prediction.
Now we have the error.
WS and B's say we have a W value between one and two.
A biased value between zero and minus one and added, well, you need one.
So we're nearly here on this graph.
Now, in the fourth step, we do backward propagation to finding direction of movement on this graph.
Which means we find Delta W and Delta B..
That is the change in W's and B's that will take us to the minimum point.
If you look at this graph, you can probably see that by decreasing the weight and increasing the bias
values.
We are we'll be moving closer to the lowest point.
So basically, we have initial W's and B's.
We will be updating our W W minus Alpha Times data W.
And we'll be updating our B to B minus Alpha Times, Delta B.
Head Al-Fayez Gauldin Learning Rate.
Basically, Delta W and Delta B are unit steps that we calculate using calculus.
Alpha is controlling the number of those steps we take in that direction.
You can imagine the impact of large versus small values of Alpha if Al-Fayez large.
We are taking multiple steps in the direction of gradient descent.
This means that we can reach the bottom fostered.
But problem at large, alpha is that we can overshoot from the minimum.
Imagine you're very near to the bottom.
But on the next time you take 50 steps.
Instead of just one.
In such a situation, you will plainly go on the other side.
So a large learning rate can help in foster dissent, but can face issue in the final stages of convergence.
Therefore, a moderate value of learning rate is to be used.
You will see what value of learning it is to be used in practical section of the schools.
Very well.
So the steps to be taken in the direction of the descent is four times there are W and Alpha teams there
that be.
Now, how do we find their W and Dubie here?
Delta W. is the change in weight and Delta B is the change in bias.
Basically, we will change.
They initially said W's and B's in the effort to reduce error.
Now, let us see how to find Delta W. and Delta B. These values are formed by doing backward propagation,
which means we will look back in the network to find out the instantaneous slope with respect to eat
W and B..
Let me take an example with a single neuron to show you how this happens.
Otherwise, the mathematics and calculus in more can get quite messy and is often overwhelming for some
student.
If you're comfortable with calculus, you can look at the complete back propagation theory in the link
shared in the description of this lecture.
However, I think with this simple example, you will get a solid intuition of how back propagation
works.
Here's a single neuron with two inputs, X1 and x2.
It first calculate linearly.
That is, it will calculate the value of Z.
Which is equal to W one, X one plus W2 x2 plus B1.
It then plays a sigmoid on this value of ze.
This sigmoid of the is the predicted output of this neuron.
We used this predicted output with the actual output to get the error of this particular training example.
So let's start with step one.
Step one is we have to randomly initialize Devalues, offbeat and byas.
We have to wait.
And one byas we randomly initialize w one to be two, W2 is equal to three and bias value is equal to
minus four.
Now, the second step is forward propagation.
That is, we will take one training example and put the input values of that training example to get
a predicted output.
We have taken this training example in which X1 value is 10 x two, value is minus four and the output
is one.
This Y is the actual output.
And it is equal to one.
So we have the W one value.
We have X1.
We have W2, x2 and B1.
So we can calculate Z.
We put all these values to get a Z value of food.
We apply the activation function.
That is the sigmoid function on this WilliamsI.
To get predicted output of this neuron to sigmoidal zie that a sigmoid of food gives a predicted output
of zero point nine two.
This predicted output value is the widest value that we will use in the edited function.
You can see that this rally was already very close to the actual output, which is one.
But let's see how we can improve this value.
Now, the third step is a calculation.
We have the error function with us.
We have predicted output value.
That is Bardash as zero point nine eight two.
And we have the actual output value for the training example.
As one, we put these two values.
And this error function to get a final added value of zero point zero zero seven nine.
Now, county foot step, which is back propagation.
The next few minutes are going to be a little heavy on mathematics.
We will cover some basics of calculus here.
If you're not comfortable with this part, it is still OK.
This is happening in the background and your software is handling this.
But if you have some understanding of calculus, looking at this example, we'll tell you how a neuron
is doing.
Back propagation.
Do not worry if you do not understand this, because this is happening in the background and your software
is handling this.
It is good to have this intuition if you know a little bit of mathematics.
So let's see how to do backward propagation.
We are at the end.
We have calculated it at the first step is finding out the slope of error with up predicted output.
That is by dash.
This symbol here.
Delta Ebi Delta whiteknight simply means that we are finding the instantaneous slope of error.
With respect to wildlife, keeping everything else constant.
So if you know calculus, you can find a derivative of this function with respect to white ash, then
lay is equal to one.
This gives us an output of minus one by white ash.
We go further back in that network and we find out the slope of our output function with respect to
Z.
The output function is a sigmoid function.
The slope of sigmoid function with respect to Z is this value.
It is to burn minus Zi upon bone plus it is about minus C, the whole square.
If you no differentiation, you can differentiate this function with respect to see and you will get
this value of Sloup.
Lastly, we find a differential of G with respect to W1, W2 and B. So Z was equal to W one times X1
plus W two times X2 plus B1.
So when we find out the differential this back to W when we get X1, which is equal to 10 at this current
point for W2, we get X2, which is equal to minus four and four B. B we get a slope of one.
Next comes the process of combining all of this.
We moved back in our network to find all these slopes.
But the slope we are actually interested in is how does the edit function change with respect to W one?
How does it change with respect to W2?
And how does it change what is meant to be to find the differential of E respect to W one?
We apply gene pool, which means that if you want to find differential of E respect to W one, you can
instead find differential of E restricted by Desh multiplied with the eventual abiders, respect to
the multiplied with differential of ze respect to differential of W one.
We have calculated all these three values in our last slide.
You can see on the top here we know the value of white ash.
We know the value of Z for this particular training example.
We can put all these values and calculate this differential and it comes out to be minus zero point
one eight six.
We can do the similar exercise for W2 am for.
Also for the differential of E respect to W2 comes out to be zero point zero seven four six and the
differential of evils to be comes out to be minus zero point zero one eight six.
Now, these three differentials are the unit steps that we are going to take in the direction of our
descent.
These are the Delta W ones, the line W 2s and Delta beats.
We are going to use these Delta values to update our rates and biases.
So that we move in the direction we are defined, the loss would be less than the loss that we had earlier.
This brings us to the last step.
Last step is we have to update W and B, the new W one would be previous W one minus Alpha Times, Delta
W one previous W one was two Alpha.
We have taken as five.
We have taken a learning rate of five year and we calculated Delta W one as minus zero point one eight
six.
This updates are w in value to two point ninety.
Similarly, we calculate W2 value and it comes out to be two point six.
And we update B value and it is now minus three point nine.
You can compare the previous and new W1 W2 B values earlier W one was two.
Nowadays, two point ninety earlier, W two was three.
Nowadays, two point six.
Earlier B was minus four.
Nowadays, minus three point nine.
Now, since we have updated our W's and B values, we have to go back to step two.
We have to reiterate, we have to do forward propagation again and we will calculate the predicted output
again.
So this is the training example, X one is ten, x two is minus four, Y is one, we put these values
with our updated weights and byas.
This time, Disney values come out to be fourteen point seven when we apply our activation function
on this evalu we get the predicted output value.
That is why dash as zero point nine nine nine.
If you remember last time we got a predicted value of zero point nine eight two.
So clearly this is an improvement over the last values of abusing these.
This process is repeated several times.
They'll be get minimum error.
If we have a lot of neurons in our network, the same processes followed in forward propagation, we
go to the end to find our predicted output value.
We use that predicted output value to find the loss.
Then we step wise.
Come back, do these differentials, find the individual differential values with ed function, and
then we update our VATE and biases so that the final edit is reduced.
Again, I will repeat that I understand that this lecture was a little mathematics heavy, but if you
have some background calculus, I'm sure you would have understood it.
But if you do not have any background in calculus, I understand that you would be facing some difficulty
in following all the things that I said.
Try to listen to this lecture again.
If you are facing difficulty, if you're still unable to follow the concept here.
Do not worry.
You can still implement a neural network in a software tool.
All this mathematical calculation will be done by this software tool.
And you do not have to do anything on your own.
That is the beauty of neural networks.
If you have to do it with hand, it will take a lot of pain.
But with computers, you can have millions of neurons and millions of features and your computer will
still be able to solnik.
So do focus on the practical lecture.
That is where you will learn how to implement these neural networks in this offer to.