Hello and welcome back to the course on Deep Learning.
This is an additional tutorial.
to talk about the softmax and cross entropy functions.
It is not 100.
Percent necessary in order for you to.
Go through.
All of the parts that we've been through in the, main
part of this section where we're talking about the, convolutional neural networks.
But at the same time,
I thought it would be a good addition to your bag of knowledge and skill sets.
So, let's go ahead and.
Dig. Into these functions.
So to start off with.
What we have here
is the convolutional neural network that we built in the main part of the.
Section.
And then at the end it pops out some probabilities
for 0.95 for a dog and 0.055% for a cat.
Given that photo in the left as an input.
This is after the training has been conducted.
This is actually it's running and it's
classifying a certain image.
And so the question here is how.
Come these two values add up to one?
Because as far as we know, from everything that we've learned
about artificial neural networks, there is nothing to say that these two,
final neurons are connected between each other.
So how would they know what the value of that hold?
Each one of them know what the value of the other one is,
and how would they know to add their values up to one?
Well, the answer is they wouldn't.
in the classic, version of artificial neural network,
and the only way that they. Do is because we.
Introduce a special function
called the softmax function in order to help us out of the situation.
So normally what would happen is, the dog and cat
neurons would have any kind of real values that,
they don't have to be, they don't have to add up to one.
But then we would apply
the softmax function, which is written up over there at the top.
And that would bring these values to be between 0 and 1.
And it would make them add up to one.
And to. Quote Wikipedia.
the softmax
function or the normalized exponential function is a generalization
of the logistic function that, quote unquote, squashes
a k dimensional vector of arbitrary real values to a k dimensional
vector of real values in the range of 0 to 1 that add up to one.
So basically it does exactly what we want.
It brings these values
to be between 0 and 1, and makes sure that they add up to one.
And the way it works is that the way that is this is possible.
Is that because at the bottom over here, you can see that there's a summation.
So it takes the exponent,
and puts it in, the power of z and adds it up.
So z one that two across all of your classes, all of these values
and so that's your normalization happening right there.
So that's, how the softmax function works.
And it makes sense to.
Introduce the softmax function into convolutional neural networks.
Because how strange would it.
Be if you had a, possible classes of a dog and a cat?
And for the dog class, you had, probability of 80%.
And for the cat class, you had a probability of 45%.
Right. It just doesn't make sense like that.
And therefore it's much better when you introduce a softmax function.
And that's what you will find happening
most of the time in convolutional neural networks.
Now the other thing is that the.
Softmax function,
comes hand in hand with something called the cross entropy function.
And it's a very handy thing for us.
So let's first look at the formula.
This is what the cross entropy function looks like.
We're actually going to be using a different calculation.
I'm going to be using this representation of the cross entropy.
But the results are basically the same.
This is just easier to calculate.
And what I know this.
Might sound very, unrelated to anything right now.
Just formulas on your screen, but, there will be some additional recommended
reading at the end of this section, so don't worry if you're not, picking up
on the math, like, if I, if we haven't explained the math right now. But.
the point here is that what is the cross entropy?
Well, a cross entropy function.
Remember how we previously.
In artificial neural networks, we had a function
called the mean squared error function.
Which we used as the cost function for.
Assessing our network performance.
And our goal was to minimize the MSE in order to optimize our network performance.
Well that was our cost function.
Then there, there.
And in, convolutional neural networks, you can.
Still.
Use MSE, but a better option in convolutional
neural networks after you apply the softmax function.
Turns out to be the. Cross entropy function.
And in, convolutional neural networks, when you apply the cross
entropy function, it's not.
Cost called the cost. Function anymore.
It's called the loss function.
And they're very similar.
They're just little terminological differences.
And like little, a bit different in what they mean.
But for our purposes, it's pretty much.
The same thing. And.
what happens is the loss. Function.
Is, again.
something that we want to minimize in order
to maximize the performance of our network.
So let's have a look.
look at a quick example.
On how of how this, function can be applied.
So let's say we've.
Put an image of a dog into our network.
the predicted value for dog is 0.9, and this is during the training.
So we know that we know the label that is a dog.
So the predicted value is 0.9.
The predicted value for cat is 0.1.
Then here we have the label. So we know
it's a dog because this is training and zero one for dog, zero for cat.
And so in. This case.
You need to use
you need to plug
these numbers into your formula for the cross entropy.
So how you do it is, the values on the left
go into the variable
Q, the one that is under the logarithm in the, on the right side.
And the values from the right would go into p.
And so it's important to remember.
Which one goes there where.
Because if you get. Them wrong, you don't want to be taking a logarithm
from a, from a zero value and or a logarithm from a one.
So you just want to plug them in.
make sure you plug them in to the correct.
places.
And then you basically add that up.
So that's how the cross entropy works.
And we'll look at it actually, right now we're just going to look
at a specific step by step example of applying this function in real life.
And it'll kind of make make more sense what cross-entropy is.
And it'll be less like.
My goal in this tutorial is to make you more comfortable with cross-entropy,
because it can sound very convoluted and no pun intended.
it can like.
Convolutional neural networks.
It it can sound very complex, right?
Scary. But it's not.
That's that's the point.
So let's go ahead and apply it just so we know that it's not scary.
So Here's neural net. And also this will.
Explain why we're doing this.
Why we're looking at different cost functions.
So neural network one.
Neural network two let's say we have two neural networks.
And then we pass an image of a dog.
and we know that this is a dog and not a cat.
And then we have another image of a cat, this time an animal.
And it's a cat, not a dog.
And here we have a weird looking animal, which is in fact a dog,
not a cat, if you look very closely.
so we want to see what our neural networks will predict in the first case.
Neural network 190% dog.
10% cats. Correct.
Neural network number 260% dog, 40% cat.
Still correct. Worse, but correct.
second option.
first neural network 10% cat.
Dog 90%. Cats. Correct.
neural network number 230%.
Dog 70%. Cat.
Worse, but still correct.
And then finally neural network one.
And, you know, in image three, neural Network one, 40% dog, 60% cat.
Incorrect.
Neural network number. 210%.
Dog, 90% cat incorrect.
And worse.
So the key here is that even though.
both networks got it wrong in the last one.
Throughout all three images, neural.
Network one was outperforming neural network two.
So even in the last case, it was very,
it had a it gave dog like a 40% chance as opposed to neural network.
Two only gave dog a 10% chance.
So neural network one is outperforming across the board,
when compared to neural network two.
And so now we're going to look at.
The functions
that they can measure performance that we've kind of talked about already.
So let's put. These into a table.
So this is neural network one.
you have the row number.
So that's the image number.
And then for image one you have what it predicted 90% dog 10% cat.
So those are the hat variables.
And then you have the actual values.
So dog correct. Cat incorrect.
Same thing for image number two and same thing for image number three.
And same for neural network number two.
So, dogs 60%, cats 40% in the first image.
That's what it predicted.
Correct answer is dog, not a cat. And so on.
And so now let's see what errors we.
Can actually get.
So what errors we can calculate to estimate
the performance and monitor the performance of our networks.
So one type of error is.
Called the classification. Error.
And that is basically just.
Asking it did you get it right or not.
Regardless of the probabilities it's just did you get. It right or did
you not get it right.
So in. Both cases and for both.
Neural networks,
each of them, they got one or so this is how many they got wrong.
So they got one out of three wrong.
So 33% error rate, for neural network
one and 33% error rate for neural network two.
And so basically from this standpoint.
Both neural networks perform at the same level.
But we know that's not true.
We know that neural network one is outperforming neural network two.
That's why a classification error is not a good,
measure, especially for the purposes of backpropagation.
mean squared error.
different.
And by the way, I did these calculations, in Excel.
I just didn't want to bore you with them,
but you can totally just sit down and do them on a paper or an Excel.
These are very straightforward calculations.
Just basically take the, sum of squared errors
and then just take the average across your, across your observations.
And that's pretty much it.
so for the, for neural net network one, you get 25%
for neural network, two you get 71% error rate.
So as you can see this one is more accurate.
It's telling us that neural network one has a much lower error
rate than neural network two.
And then cross entropy again.
We've seen the formula. You can also calculate this.
This is actually even easier to calculate.
The mean squared error.
Cross error cross-entropy gives you 38% for neural network one
and 1.06 for neural network two.
So you can see the results are a bit different.
when you look at them like that, when you look at,
you know, the mean squared error and cross entropy, The question of
why would you use cross entropy over,
mean squared error isn't just about.
The kind of like the numbers that they spit. Out. This these calculations.
Were just to show you that this.
Is all it's all. Doable. You can just do it on a paper.
It's it's not these are not very intense mathematics.
These are.
The pretty. Simple, straightforward things.
But the question of why would you use mean, cross
entropy over mean squared error?
It's a very, very good question to ask.
I'm glad you asked it.
the the answer to that is like
there's several advantages of,
cross entropy over mean squared error, which are not obvious.
And so I'll, I'll mention a couple, but then.
I'll, I'll let you know where you can find out more.
So One of them is that if, if, for instance, you're at the very start.
Of your, Backpropagation,
your output value is very, very, very, very tiny.
Very. Tiny.
So it's much smaller.
Than the actual value that you. Want.
Then at the very start, the gradient
in your gradient descent will be very, very low.
And you it won't be enough, it'll be very hard for.
The neural network to actually. Start doing.
Something and start moving around and start adjusting those weights.
And So you start actually.
Moving in the right direction.
Whereas when you use something. Like the.
Cross entropy, because it's got that logarithm in it, it
actually, helps the network assess even.
A small error like that and just do something about it.
Here's how to. Think about it.
So let's say, in again, this is very and in very intuitive approach.
There's this, there's going to be
a link to the mathematics, and you can derive these things
through the mathematics in more detail.
But a very intuitive approach. Let's say.
your,
like your.
Outcome that you want is. Is one.
And right now you are at, one, one millionth of one.
Right. So 0.000001.
And then you improve next.
Time you improve your outcome from, from one millionth to, 1,000th.
And in terms of if you calculate this squared error, you just.
Subtracting one from the other, or basically in each case
you're calculating the square. Error and you'll see that the squared errors,
when you compare one case versus the other, it didn't change that much.
You didn't improve your.
Network that much when you're looking at the mean squared error.
But if you're looking at cross entropy
because you're taking a logarithm and then you're comparing the.
Two dividing one by the other, You will see
that you have actually improved your network significantly.
So you that that jump from, one millionth to 1,000th in mean
squared error terms will be very low, it will be insignificant, and it won't,
it won't guide your gradient,
boosting process or your backpropagation in the right direction.
It will, it will it will guide it in the right direction.
But it'll be like a very slow guidance.
It won't have enough power.
whereas.
If you do it through cross entropy, cross entropy will.
Understand that. Oh, even though these are very small.
Adjustments that are just, you know, making.
A tiny change in absolute terms in relative terms, it's a huge improvement.
And we we are definitely going in the right direction.
Let's keep going that way.
So cross entropy will help your neural network.
Get to the right gets to the optimal state.
it's a better way for the neural network to get to get it to an optimal state.
But, bear in mind.
That this only works when, the cross entropy is only the preferred.
Method, only for classification.
So, if.
You're talking about things like regression.
Like which we had in artificial neural networks.
then you would rather.
Go with mean squared error.
Whereas cross.
Entropy is better for classification.
And again, it has to do with the fact that we're using softmax function.
So that's a kind of intuitive explanation of that.
a good place to learn a bit more about that if you're.
Really interested in, you know, why are we using, cross entropy versus
mean squared error?
Google a video by Geoffrey Hinton called the.
Softmax output function.
And, he explains it very well.
And, you know, being
the godfather of deep learning, who can explain it better anyway?
and by. The way, any video.
By Geoffrey Hinton is golden.
He's just got a huge talent for explaining things.
Anyway, so that's, that's.
Softmax versus cross. Entropy.
I hope that gives you kind of like an intuitive.
Understanding of what's going on here, but.
More importantly, that you're not put off by the term cross entropy,
because Hudlin. Will mention it in the practical tutorials.
And I wanted to make sure that you're prepared for that.
And it's it's just another.
Way of calculating your loss function and another way
of optimizing your network, which is specifically tailored to,
classification problems and therefore convolutional neural
networks and comes in hand, hand in hand with the softmax function.
So additional reading if you'd like a light introduction
into, cross entropy if you're interested.
In the cross entropy of a bit. More.
Of course, a good article to check out is called A Friendly Introduction
to Cross Entropy Loss by Rob deep, 2016.
Here's the link below.
very, very nice.
Very soft.
Nothing. No.
Super complex math.
good analogies, good examples.
Use analogies of cars.
And you look at cars and talks about information and bits
and restrictions, and, you know, how would you encode this?
How do you code that? It's it's a it's a good article to.
Have a look at.
And we'll give you a, a good overview of, cross entropy.
like from an introductory standpoint.
If you want to dig.
Into the heavy math, like what you see here, then check out an article
by or a blog by How to Implement a Neural Network Intermezzo two.
So Intermezzo is like is like an intermediate thing, like a.
interim intermittent.
In, you know, like when you go to a theater and you have like a break,
between, the first part and the second part.
So because he's like, going through all these steps and then he's like.
And then he says, I got to explain this first.
and yeah. So that's why it's called Intermezzo.
No, the reason as far as I understand, the.
Article is by Peter Rowlands.
20. 16 as well.
So both are quite recent and Yeah.
Check out this if you would like to dig into the mathematics behind,
cross entropy behind softmax and cross entropy in this article actually.
So there we go.
That's, all there is to these two.
Hopefully I was able to add some additional clarity and, good luck
with that.
It's, It's going to be fun and, enjoy the practical tutorials.
I'll see you next time. Until then, enjoy deep learning.