Hello and welcome back to the course on Deep Learning.
Today we're talking about stochastic gradient descent.
Previously we learned about gradient descent.
And we found out that it is a very efficient method to solve
our optimization problem where we are trying to minimize the cost function.
It basically
takes us from ten to the power of 57 years to solving a problem
within, minutes or hours or within a day or so.
And it really helps speed things up, because we can see which way is downhill,
and we can just go in that direction and take steps and get to the minimum faster.
But the thing with the stick with, gradient descent
is that this method requires for, the cost function to be convex.
And as you can see here, we've specifically chosen a convex cost
function.
Basically, convex means, that the function looks similar
to what we are seeing now that, it's just convex
into, one direction and that it,
in essence has one global minimum.
And that's the one that we're going to find.
but what if our function is not convex?
What if our cost function is not convex?
what if it looks something like this?
Well, first of all, how could that happen?
Well, that could happen,
because if we first of all choose a cost function, which is not,
the squared difference between y hat and y,
or if, we do choose the cost function, which is like that,
but then in a dimensional space, it can actually turn into something
that is not convex.
And so what would happen in this case if we just tried
to apply our normal gradient descent method.
something like this could happen.
We could find a local minimum of the cost function rather than the global one.
So this one was the best one.
And we found the wrong one.
And therefore we don't have the correct weights.
We don't have an optimized neural network.
we have a subpar neural network.
And so what do we do in this case?
Well, the answer here is, stochastic gradient descent.
And it turns out the stochastic gradient descent doesn't require
for the cost function to be convex.
So let's have a look at the two differences between the normal gradient
descent that we talked about and the stochastic gradient descent.
So normal gradient descent is when we take all of our rows
we plug them into our neural network.
And once again here we've got the neural network copied over several times.
But the rows are being plugged into that same neural network every time.
So there's only one neural network.
This is just for visualization purposes.
And then once we've plugged the main, we've calculated our cost function
based on the formula on the right.
And looking at the chart on the at the bottom.
And then we adjust the weights.
Then this is called the gradient descent method.
Or it's also the proper term is the batch gradient descent method.
So we take the whole batch of from our sample, we apply it and then we run
that the stochastic gradient descent method is a bit different.
Here. We take the rows one by one.
So we take this row.
We run our neural network and then we adjust the weights.
Then we move on to the second row.
We take a second row. We run our neural network.
We look at the cost function.
And then we adjust the weights again.
And then we take another row.
Take row three.
we run our neural network.
We look at the cost function, we adjust the weights.
So basically, we're looking at
we're adjusting the weights after every single row
rather than doing everything together and then adjusting weights.
two different approaches.
And now we're going to just compare the two side by side.
So here they are.
This is how to visually remember them.
So you've got the batch gradient descent where you adjusting.
the weights after you've run them.
After you've run all of the rows in your neural network.
And then, basically you adjust the weights
and you run the whole thing again. Iteration, iteration, iteration.
In the stochastic gradient descent method, you run one row at a time.
And, you adjust the weights, you adjust the way to adjust the weights.
And then you do everything again and again.
And that is called this casting gradient descent method.
The main two differences are that the stochastic gradient
descent method helps you avoid, the problem
where you find those local, extrema or local minimums
rather than the overall, overall global minimum.
And the reason for that, in simple terms, is that the SGD
or the stochastic gradient descent method, has much higher fluctuations
because it can afford them.
It's doing one iteration or one row at a time, and therefore
the fluctuations are much higher.
And it it's much more likely to find the global,
minimum, rather than just the local minimum.
And the other thing about the stochastic gradient descent,
nothing compared to the batch gradient is the it's faster.
Like the first impression that you might have is
because it's doing every single row one at a time.
It is slower, but actually in fact, it is faster because it is.
It doesn't have to, load up all the, data into memory
and run and wait until all of those rows are on all together.
You can just row, run them one by one so it's much lighter.
Algorithm is much faster in that sense.
So, though it has way more and that's in those senses,
it has more advantages over the, batch gradient descent method.
The main advantage of or the main kind of like pro for the batch
gradient descent method is that it is a deterministic algorithm
rather than, stochastic gradient descent, being a stochastic algorithm,
meaning it's random and with the batch gradient descent method,
as long as you have the same starting weights.
for your neural network,
every time you run the batch gradient descent method, you will get,
the same, iterations, the same results for your, for the way
your weights are being updated, for us to have for the stochastic gradient
descent method, you won't get that because it is a stochastic method.
You are picking your rows, possibly at random.
And, you are updating your neural network in a stochastic manner and therefore,
you're just going to every single time you run the stochastic gradient
descent method, even
if you have the same weights at the start, you're going to have a different,
process at different, different iterations to get there.
So that's in a nutshell.
What's stochastic gradient descent is also there's a method in between
the two called the mini batch gradient descent method,
where you combine the two and you basically, run
rather than running a whole batch or running one at a time.
You run batches of rows maybe five, ten, 100.
However many rows you decide to set, you run those that number of rows at a time.
Then you update your weights, synaptic weights, and so on.
And that's called the mini batch gradient descent method.
If you'd like to learn more about gradient descent,
there's a great article which you can have a look at.
It's called, a Neural Network in 13 Layers of Python
part two Gradient Descent by Andrew Trask.
and the links below.
It's on GitHub.
12 to 15 article.
very well written, very in very simple terms.
it's got some interesting, philosophical or just interesting
thoughts on, how to apply gradient descent.
What, you know, the advantages and disadvantages and how to be
how to do things in certain situations.
So he's got some very cool tips, tricks and hacks.
very easy to read.
So definitely check that out.
And another one a bit more, heavier read.
For those of you who are into mathematics, who want to get to the bottom
of the mathematics, why gradient descent in that specific, what are the formulas
that are driving gradients and how is it calculate and so on.
check out the article or actually the book.
it's a free online
book called Neural Networks and Deep Learning by Michael Nielsen.
2015 book.
It's, just basically it's all online.
You can go ahead and, check it out there and there.
Again, very soft introduction to the mathematics.
But then for mother estimate.
But the mathematics are pretty heavy
as you go along as you read through the article.
but at the same time it gets you into, into that mood.
I think he has, like, a warm up, chapter.
We used first warm up with the math and then you jump into him.
So interested in math.
Then this is the article to go to.
And there we go. So that's in a nutshell.
The difference between gradient descent and stochastic gradient descent.
And how the two work.
And on that note, we're going to wrap up today's tutorial.
I look forward to seeing you on the next one.
And until then, enjoy deep learning.