Hey everyone.
Congratulations first of all.
If you understood whatever I taught you in the last few lectures you understand neural networks in the
way they are used for prediction purposes.
In this lecture I'm going to talk about several subtropics these sub topics came up in our lectures
previously also but to maintain the flow.
I did not discuss them in detail at that time.
Also this extra knowledge is often what is asked in interview questions.
In fact I'm going to cover this lecture in a question on the format so keeping your attention as this
is also very important.
The first off topic is of activation functions and the first question that we are going to discuss is
why do we use activation functions.
Let's see if we do not have any activation function.
What is the output of a neuron.
The output would be given by this equation W1 X1 plus W2 x2 plus B is equal to Z which is the output.
This means that the output could be any real number with no boundaries if you are solving for a regression
problem then this may be acceptable.
But when we are doing classification that is we want an output of yes no type or 1 0 type.
We need to read the output d to get a 0 1 paper output.
Also if there are only linear neurons in the whole network you can only predict a linear relationship
between input and output variables.
So the answer is we use an activation function for two reasons.
First to put boundary conditions on our output for classification.
The boundaries are obvious for regression also.
If you have some boundary you can use an activation function.
The second reason is to introduce nonlinear entity so that we can find complex nonlinear patterns.
Also
next question is what are the different types of activation functions earlier.
We had discussed two activation functions.
One was the step function which is zero below threshold value.
And then it suddenly jumps to one at differential value.
Then we discussed the sigmoid function which is a continuous S shape go with zero as the lower boundary
and one as the upper boundary for most practical business purposes sigmoid function is good enough but
for rare scenarios where computation is an issue we sometimes use these two other activation functions
also because of their convergence efficiency first is the hyperbolic tangent function or the PAN edge
the graph of this function is almost similar to the sigmoid in shape but it has different boundaries.
It has upper boundary of one at a lower boundary of minus one and because it is centered at zero it
almost always has better convergence efficiency than sigmoid.
The second is the loop which is short for identified linear unit.
It is very widely used function especially in the inner layers of regression neural networks.
This is how this function looks like till 0 the function also outputs 0 but after 0 function outputs
the same as input that is if X is equal to x.
So the lower one is 0 but there is no upper bound.
This function performs well because it is very easy to execute.
The reason for using this function and it in layers is that this function introduces nonlinear entity
different layers.
However on the output layer it is rarely used because for classification the right side of the function
is not bounded and therefore it cannot be used on the other hand for regression.
The left side of this function is bone and therefore this function cannot be used.
So this function is good for activating it in layers but not for activating output layers.
You can find a summary of all these activation functions in the next light.
This is for your reference.
This brings us to the next question which we have already answered can hit in layers and output layers
have different activation functions.
Answer is yes.
As I told you we can implement redo in the layers and the sigmoid in the output layer.
Any such combination is allowed by our software to
next question is what is multi class classification and is there any special activation function for
multi class classification.
So first of all what is multi class classification.
Suppose you are classifying into yes or no or 1 0 0.
This is binary classification because there are two classes only.
But if we have more than two classes.
So if we want to classify images into shirts trousers ties and socks.
Now the output cannot be 0 or 1.
We cannot do like this that we gave 0 for shirts 1 for trousers 2 4 days 3 for socks that would not
give us the right answer.
So we have to handle the situation in a little different manner for such a situation.
We have an activation function called soft Max
this activation function works similar to sigmoid.
Mark has an additional step.
So what we do in multi class classification is we usually keep as many output neurons as we have classes.
So if we have three classes like shirts trousers and socks we keep three neurons at the output layer.
You can see in this image these three output neurons correspond to three output classes these three
output neurons have these sigmoid activation function only.
So the output of first neuron would also lay between 0 and 1 and we can see that the output value is
corresponding to the probability of whether it is a shirt or not.
The second output would also be between 0 and 1 and we can see that it is the probability of whether
it is a browser or not.
And the third output will also be between 0 and 1 and we can see that it is the probability of whether
it is socks or not.
But the thing is that the item can be only one that is the sum of probabilities should come out to be
one.
Either it is shirt or it is trouser or it is socks.
To implement this we put an additional layer of soft Max and we input the desert of these three neurons
into this sort Max layer.
This soft Max layer just divides each of the value by the sum of all the values.
Not this output can be considered as the probability of that class occurring and the sum of all these
probabilities will also be equal to one
so for multi class classification a soft Max activation is often used on the output layer.
That's all about the activation functions.
The next topic is gradient descent.
The question I want to discuss here is what is the difference between gradient descent and stochastic
gradient descent.
This is a very common question in the mind of students because they find stochastic written at some
places and it is not written in some texts.
Let me clarify this for you.
What we discussed in our previous lectures was actually stochastic gradient descent because I told you
that we take each individual training record and update our weights and biases with each training record
when we run the whole forward and backward propagation for each individual training record that is stochastic
gradient descent.
But if you run forward propagation for entire training set at one go and find out the average error
on the entire set and then a blade DVDs and biases during backward propagation that is gradient descent.
There is another variation in which we make small batches out of the training set and use these batches
instead of completed.
This one is called mini bad gradient descent.
The point is stochastic gradient descent starts updating weights and biases rapidly but it finds difficulty
in converging whereas gradient descent is slow because in each pass it has to go through the entire
training set.
But the good thing about gradient descent is it converges very well so we have to accordingly select
which optimization technique is to be used.
We will see this further in the practical part of this course
lasting I want to discuss in this lecture is epoch in neural networks.
Epoch is one cycle through the full training data.
So when we say epoch is equal to five it means we want the full training data to be fed five things.
Note that epoch is different from iterations.
Suppose you have 1000 training records now.
If you start reading the records one by one like we do doing stochastic gradient descent you will have
to I created 1000 times
so I iterations are the number of times you execute any process within one full training set.
On the other hand if you have said epoch to do then this means that the 1000 training examples will
be fed two times either one by one or all at the same time or in many batches.
The idea of using epoch is to allow the network to rejig its performance on the same data.
We will see how we can specify and use epoch in our party collectives.
That's all in this lecture in the upcoming video.
We will summarize the key parameters that you must know while implementing neural networks in the software.