Hey, everyone, congratulations, first of all.
If you understood whatever I taught you in the last few lectures, you understand neural networks in
the way they are used for prediction purposes in this lecture.
I'm going to talk about several subtopics.
These topics came up in the lectures previously, also to maintain the flow.
I did not discuss them in detail at that time.
Also, this extra knowledge is often what is asked in interview questions.
In fact, I'm going to cover this lecture in a question on the former.
So keeping attention as this is also very important.
The first topic is of activation functions.
And the first question that we are going to discuss is, why do we use activation functions?
Let's see if we do not have any activation function.
What does the output of a neuron, the output would be given by this equation, W on X1 plus W two X
two plus B is equal to Z, which is the output.
This means that the output could be any real number with no boundaries.
If you're solving for a regression problem, then this may be acceptable.
But when we are doing classification, that is, we want an output of yes no type or one zero type.
We need to read the output deals to get a zero one type of output.
Also, if there are only linear neurons in the whole network, you can only predict a linear relationship
between important output variables.
So the answer is we use an activation function for two reasons.
First, to put boundary conditions on our output for classification.
The boundaries are obvious.
For regression, also, if you have some boundary, you can use an activation function.
The second reason is to introduce non-linearity so that we can find complex, nonlinear patterns.
Also.
Next question is, what are the different types of activation functions?
Earlier, we had discussed two acquisition functions.
One was the step function, which is zero below threshold value.
And then it totally jumps to one at its actual value.
Then we discussed the sigmoid function, which is a continuous s shape goal with zero as the lowered
boundary.
And one as the upper boundary for most practical business purposes.
Sigmoid function is good enough, but for rare scenarios where competition is an issue, we sometimes
use these two other activation functions also because of their convergence efficiency.
First is the hyperbolic tangent function or the Panitch.
The graph of this function is almost similar to the sigmoid in shape, but it has different boundaries.
It has upper boundary of one at a lower boundary of minus one.
And because it is centered at zero, it almost always has better converges efficiency than sigmoid.
The second is Relu, which is Shorthold electrified linear unit.
It is very widely used function, especially in the inner layers of regression neural networks.
This is all this function looks like below zero.
The function also outputs zero after zero function outputs the same as input.
That is, if X is equal to X.
So the lawman is zero, but there is no upper one.
This function performs well because it is very easy to execute.
The reason for using this function and hidden layers is that this function introduces non-linearity
in different layers.
However, on the output layer, it is rarely used because for classification, the right side of the
function is not bounded and therefore it cannot be used.
On the other hand, for regression.
The left side of this function is bone and therefore this function cannot be used.
So this function is good for activating hidden layers.
But Northfork activating output leaves.
You can find a summary of all these activation functions in the next slide.
This is for your reference.
This brings us to the next question, which we have already answered.
Can hit a lears and output layers have different activation functions.
Answer is yes.
As I told you, we can implement RELU in different layers and a sigmoid in the output layer.
Any such combination is allowed by our software tool.
Next question is what is multiclass classification and is there any special activation function for
multiclass classification?
So first of all, what does multiclass classification.
Suppose you are classifying in2.
Yes or no.
Or one.
Or zero.
This is binary classification because there are two classes only.
But if we have more than two classes.
So if we want to classify images into shirts, trousers, ties and socks.
Now the output cannot be zero or one.
We cannot do like this that we give zero for shirts, one for trousers, two fortes, three for socks.
That would not give us the right answer.
So we have to handle the situation in a little different manner for such a situation.
We have an activation function called softmax.
This activation function works similar to sigmoid.
Mark has an additional step.
So what do we do in multiclass classification is we usually keep as many output neurons as we have classes.
So if we have three classes like shirts, trousers and socks, we keep three neurons at the output layer.
You can see in this image these three output neurons correspond to three output plus's.
These three output neurons have these sigmoid activation function only.
So the output of first neuron would also lay between zero and one.
And we can see that the output value is corresponding to the probability of whether it is a shirt or
not.
The second output would also be between zero and one.
And we can say that it is the probability of whether it is a trouser or not.
And the third output will also be between zero and one.
And we can see that it is the probability of whether it is Salk's or not.
But the thing is that the item can be only one that is the sum of probabilities should come out to be
one.
Either it is shirt ordered his trouser or the socks to implement this.
We put an additional layer of softmax and we input the dessert of these three neurons into this sort
matched layer.
This softmax layer just divides each of the value by the sum of all the values.
Not this output can be considered as the probability of that class occurring.
And the sum of all these probabilities will also be equal to one.
So for multiclass classification of softmax, Activision is often used on the output layer.
That's all about the activation functions.
The next topic is gradient descent.
The question I want to discuss here is what is the difference between gradient descent and stochastic
gradient descent?
This is a very common question in the mind of students because they find stochastic written at some
places.
And it is not written in some texts.
Let me clarify this for you.
What we discussed in our previous lectures was actually stochastic gradient descent, because I told
you that we take each individual training record and update our viewers and biases.
With each training record, when we run the whole forward and backward propagation for each individual
training record, that is stochastic gradient descent.
But if you run forward propagation for entire training set at one, go and find out the average error
on the entire set.
And then nobody reads and biases during backward propagation.
That is gradient descent.
There is another variation in which we make small batches out of the training set and use these batches
instead of completer.
This one is called mini bat, greed inducing.
The point is, stochastic gradient descent starts updating weights and biases rapidly, but it finds
difficulty in converting.
Whereas gradient descent is slow because in each pass, it has to go through the entire training set.
But the good thing about gradient descent is it converges very well.
So we have to accordingly select which optimization technique is to be used.
We will see this further in the practical part of this course.
Last thing I want to discuss in this lecture is epoch in neural networks, epoch is one cycle to the
full trading data.
So when we say epoch is equal to five, it means we want the full training data to be fed five times.
No epoch is different from iterations.
Suppose you have 1000 training records.
Now, if you start weighting the records one by one, like we do doing stochastic gradient descent,
you will have to, I predict, 1000 times.
So ideations are the number of times you executed the process within one full training set.
On the other hand, if you have said Epoch to do, then this means that the 1000 training examples will
be fed two times, either one by one or all at the same time or in many batches.
The idea of using Époque is to allow the network to rejig its performance on the same data.
We will see how we can specify and use Epoch in a particular lichter's.
That's all in this lecture, in the upcoming video.
We will summarize the key parameters that you must know while implementing neural networks in the software.