Hello and welcome back to the course on Deep Learning.
All right.
Today we're talking about the activation function.
Let's get straight into it.
So this is where we left off.
Previously we talked about the structure of a one neuron.
So there it is in the middle.
We know that it has some inputs values coming in.
It's got some weights.
Then it adds up the weighted.
It calculates a weighted sum of those inputs
and then applies the activation function.
In step three it passes on the signal to the next round.
And that's what we're talking about today.
We're talking about the value that is going to be passed over.
So we're talking about the activation function that's being applied.
So what options do we have for the activation function.
Well we're going to look at four different types of activation functions
that you could choose from.
Of course there are more different types of activation function.
But these are the predominant ones
that you'll be hearing about and that we'll be using in this course.
So here is the threshold function.
This is what it looks like.
So on the x axis you have the weighted sum of inputs.
On the y axis you have just you know the values from 0 to 1.
And basically the threshold function is a very simple type of function where
if the value is less than zero,
then the threshold function passes on zero.
If the value is more than zero or equal
to zero, then threshold function passes on a one.
So it's basically kind of like yes, no type of function.
very, very straightforward, very kind of like rigid type of function either.
Yes or no, no other options.
So there you go. That's how it works.
Very simple function.
Let's move on to something a bit more complex.
Now the sigmoid function.
very interesting formula.
that we have here.
You'll see just now there is one divide by one plus e to the power of minus x.
Whereas in this case, of course x is the value of the summed,
of the weighted sums.
And so yeah.
So this is what the sigmoid looks like.
It's function which is used in the logistic regression,
if you recall, from the machine learning course.
So what is good about this function is that it is smooth, unlike the,
threshold function, this one doesn't have those kinks in its curve.
And therefore, it's just nice and smooth, gradual, progression.
So, anything below zero, it just like, drops off
above zero, it approximates towards one.
And this, sigmoid function is very useful, in the final layer, in the output
layer, especially when you're trying to predict probabilities.
And we'll see that throughout this course.
And then we've got the rectifier function.
Rectifier function, even though it has a kink,
is one of the most popular functions for artificial neural networks.
So it goes all the way to zero.
It's is zero.
And then from there it gradually progresses
as the input value, increases as well.
And we'll see that throughout the course.
We'll see that in other intuition tutorials.
And we'll also see that,
how we use this function in the practical, side of the course.
And I will comment on this a bit more in a few slides from now.
So just remember that rectifier function is
one of the most used functions in artificial neural networks.
And finally we've got one more function, that you will probably hear about.
It's the hyperbolic tangent function.
It's very similar to the sigmoid function.
But here the the hyperbolic tangent function goes below zero.
So the values go from 0 to 1 or approximately
to one and go from 0 to -1 on the other side.
And that can be useful in some applications.
So we're not going to go into too much depth on each one of these functions.
I just wanted to, acquaint you of them so that you know what they look like
and what they're called.
if you'd like to get some additional reading,
then check out this paper by Heavier Glory.
What have you got?
called deep sparse rectified Neural Networks 2011 paper.
And there you will find out exactly why the, rectifier function
is such a, valuable function, why it's so popular to use.
But nevertheless, for now, you don't really need to know all of those things.
For now, we're just going to start applying them.
We just start using them more and more and more.
And so when you feel comfortable with the practical side of things,
then you can go and refer to this paper and then you will be able to soak in
that knowledge much quicker, and it'll make much more sense.
So but just keep this in mind that when you're ready, when you feel that
you're ready, then you can go and refer to this paper
and get some valuable knowledge from there.
So, just to quickly recap,
we have the threshold activation function, which looks like this,
the sigmoid activation function, which looks like this.
We have the rectifier function and we have the hyperbolic tangent function.
And now to finish off this tutorial, let's quickly do a few exercises.
So we'll just do two quick exercises to, help that knowledge sink in.
So first one is we've got an example here of a neural network with just one neuron.
And then right away the output layer.
And the question is assuming that your dependent variable is binary.
So it's either zero and one, which threshold function would you use.
So out of the ones that we've discussed we have the threshold function,
the sigmoid function, the rectifier function.
And we've got the hyperbolic tangent function.
in it's in their role forms.
Which ones would you be able to use?
for a binary variable.
Okay.
So the answers here are there's two options that we can approach this with.
So number one is the threshold activation function.
Because we know that it's between 0 and 1.
And it gives you a zero under certain values.
And then otherwise it gives you a one. So it only can give you two values.
It fits perfectly fits.
this requirement perfectly.
And therefore you can just say y equals, the,
threshold function of your, so weighted sum and that's it.
And then the second case which you could use is the sigmoid activation function.
It is actually also between 0 and 1, just what we need.
But at the same time you want is just zero one.
Right.
So you it's not exactly the what we need.
But in this case what you could use it as is the probability of y being yes or no.
So we want y to be zero one.
But instead we'll say that the sigmoid function, sigmoid activation
function tells us whether,
it tells us the probability of y being equal to one.
So basically, the closer you get to the top, the more likely
it is that, this is indeed a one or a yes rather than a no.
And, yeah.
So that's, very similar to the logistic regression approach.
And that's those are just two examples.
And if you have a binary variable.
And now let's have a look at another practical application.
Let's have a look at
how all this would play out if we had a neural network like this.
So in the first input layer we have some inputs.
they are sent off to our first hidden layer.
And then an activation function is applied.
And usually what you would apply here and what you'll see throughout this course
is we would apply a rectifier activation function.
So it would look something like that.
We apply the rectifier activation function.
And then from there the signals would be passed on to
the output layer where the sigmoid activation function would be applied.
And that would be our final output.
And that could predict a probability for instance.
So this combination is going to be
quite common where in the hidden layers we apply the rectifier function.
And then in the output layer we apply the sigmoid function.
So there we go.
I hope you enjoyed today's tutorial.
Now you are quite well versed in the four different types of activation functions,
and you will get some hands on practical experience with them.
Throughout this course we will be using them all over the place,
so you'll get to know them quite intimately
and you should be quite comfortable with them.
But for now, this is the knowledge that you need to progress and understand
what is going to be happening further down in this course.
And on that note, I will look forward to seeing you next time.
Until then, enjoy deep learning.