Hello and welcome back to the course on Deep Learning.
Now that we've seen neural networks in action, it's time for us to find out
how they learn.
So let's dive straight into it.
There are two fundamentally different
approaches to getting a program to do what you want it to do.
One is hard coded coding where you actually tell
the program specific rules and what outcomes you want,
and you just guide it throughout the whole way, and you account
for all the possible options that the program has to deal with.
On the other hand, you have neural networks where you create a facility
for the program to be able to understand what it needs to do on its own,
to basically create this neural network where you provided inputs.
You tell it what you want as outputs, and then you let it figure everything
out on its own.
Two fundamentally different approaches.
And that is something to keep in mind as we go through these tutorials.
Our goal is to create this network which then learns on its own.
We are going to avoid it trying to put in the rules.
And a good example that I can give you right now is,
this will come further in the course, but it's just a very visual example.
For instance, how do you distinguish between a dog and a cat?
For on the left side, on the approach that's, depicted on the left, you would,
program things in like, the cats ears have to be like this.
Look out for, whiskers.
Look out for this type of nose.
Look out for this type of shape. of, face.
Look out for these colors.
You kind of.
You describe all of these things, and you'd have conditions
like if if the ears are pointy, then cat, if the ears,
sloping down, then possibly dog and so on.
On the other hand, for a neural network, you'd just code the neural networks,
you code the architecture, and then you point the neural network
at a folder of all these cats and dogs
with images of cats and dogs, which are already categorized.
And you tell it, okay, I've got you.
I've got some images of cats and dogs.
Go and learn what a cat is.
Go and learn what a dog is.
And the neural network will on its own understand
everything it needs to understand
and then further down, once it's trained up, when you give it
a new image of a cat or a dog, it'll be able to understand what it was.
So they're there. They are.
Those are the two fundamentally different approaches.
And today we're going to slowly start
getting into how that second approach works.
All right. So let's get straight to it.
Here we have a very basic neural network with a one layer.
It's this is called a single layer feedforward neural network.
And it is also called a perceptron.
Now before we proceed one thing that we do need to adjust is that output value.
Right now you can see that it's just a why we need to put a y hat in there.
And the reason for that is usually y stands for the actual value.
And that's what we're going to be using.
So y is going to be the actual value which we see in reality
output value is the predicted value by the algorithm,
by the neural network y hat is the, output value.
Basically that's the denomination for the output value.
And the perceptron was first invented in 1957 by Frank Rosenblatt.
And his whole idea was to create something that can actually learn,
and, adjust itself.
And this is what we're going to be looking at now.
So, we've got a perceptron.
Let's see how a perceptron learns.
So let's say we have some input values, that have been supplied
to the perceptron, and, or basically to our neural network.
Then,
the activation function is applied, we have an output,
and now we're going to plot the output on a, chart.
So there it is, our output y hat.
Now, what we need
to do is in order to be able to learn, we need to compare the output value
to the actual value that we want the neural network to get right.
And that is, the value y.
And so if we plot it here, you'll see that there's a bit of a difference.
Now we're going to calculate a function called the cost
function is calculated as one half of the difference
of the squared difference between the actual value and output value.
Now there there are many ways you can come up with a cost function.
There are many different cost functions that you can use.
this is probably the most commonly used cost function.
And why it is specifically this function that we use.
We'll find out further down when we're talking about a gradient descent.
But for now we're just going to, agree that this is the cost function.
And basically what the cost function is telling us is,
what is the error that you have in your prediction?
And, our goal is to minimize the cost function because the,
the lower the cost function, the closer the y hat is to y.
Okay.
So as long as we agree on that, let's proceed.
So basically from here, what happens is, there's our cost function.
And from here what happens is now we're going to, once we've compared,
now we're going to feed this information back into, the neural network.
So there we go.
There's, the information going back into the neural network,
and it goes to the weights, and the weights get updated.
Basically, the only thing that we have control of
in this very simple neural network are the weights w1, w2 all the way to W1.
And, our goal is to minimize the cost function.
So all we can do is update the weights.
So we update the weights.
tweak them a little bit and how exactly we'll find out for the down.
But for now we, we agree that we update the weights and then we continue.
So but here I've put up this, screenshots of the data
just to make some one point very clear that right now, throughout this whole
experiment, everything we're doing right now, we're dealing with just the one row.
So we're dealing with we have a data set of one row where we have,
for instance, we're dealing with, how long you study like, the variable
that we're predicting is what, what, results you're going to get on an exam.
And the dependent independent variables that we have is how many hours
did you study for how many hours did you sleep,
and what did you get on the quiz in the mid semester.
So in in the middle of the semester it a quiz.
What percentage did you get there.
So based on those variables we're trying to predict what score
you'll get for the exam.
And in an exam the 93%.
That's the actual value. So that's why.
So, so we feed these three values
into our neural network again for the second time now,
and then we're going to be comparing the result to Y.
So let's see how this works.
We feed these values into the neural network.
Everything gets adjusted and weights get adjust.
So as you can see this is again we're going to feed the values again.
The point here is that we're feeding in these same values.
So we only have one row.
We're trying to we're training on one row.
This is because this is just a very simple basic example.
Then we'll see what happens when there's more rows.
So again we feed these rows in our cost functions get adjusted.
As you can see everything happens along those lines again.
So as you can see, every time our y hat is changing
because we've tweaked the weights, all I had is changing.
Our cost function is changing. Let's have a look again.
So we feed those in Y hat is changing.
Cost function is changing.
We get information back, feed back to the weights
so that the weights get adjusted. Again.
We feed in the same values.
Every time everything gets adjusted goes back to the weights and one more time
feed in. Okay
and another time.
So we've adjust the weight, adjusted the weights we feed in the information.
And there we go.
So, now this time the y hat is equal to y equals functional zero.
Usually we won't get cost function equal to zero.
But this is a very simple example.
So hopefully all that made sense every time we feed in exactly
that same row.
Because just in this case we're just dealing with that one row
into our neural network, where then, the weights get, the values get
well supplied by the weights, the activation function is applied.
We get y hat, Y hat is compared to y.
Then we see how the cost function has changed.
Feed back, feed that information
back into the neural network and then just adjust the weights again.
and then we repeat the same process again with the same exact row.
we're trying to minimize that cost function.
So up until now we've been dealing with just that one row.
Let's see what happens when you have multiple rows.
So here's the full data set.
We have eight rows of, how many hours you slept.
Or maybe these are, different students in a day taking the same exam.
How many other hours they studied, how many hours they slept before the exam?
What to get on the quiz and their final result on the test.
And as you can see here on the left, I've got eight of these perceptrons.
Actually, they are all the same perceptron.
So this is also important to understand.
I just multiplied it or like duplicated eight times just so that we can
conceptual understand.
But the important thing here, it's the same neural network.
We're going to be feeding these into one same neural network.
So let's go. Let's get started.
So one epoch,
as you all here had learned, mentioning one epoch is
when we go through our whole data set and we train our,
neural network on, on all of these, rows.
So let's go, let's get started.
So there's our first row, and there's y hat for the first row.
There's the second row, there's y hat for the second row.
So again it's being fed into the same neural network every time.
I've just copied them several times.
So we can visually see how this is happening.
Then again is it's happening again.
That's third row. Fourth row.
There's our y hat for the fourth row and so on basically.
Then we get the same values for the remaining four rows as well.
So every time we just feed in a row into our neural network, we get a value.
then we compare to the actual values.
So they are the actual values.
So for every single row we have an actual value.
And now based on all of these differences
between y hat and y, we can calculate the cost function
which is the sum of all of those
squared differences between y hat and y.
And all of that is halved.
And there's our cost function.
And basically now what we do after we have the full cost function,
we go back and we update the weights, we update w1, w2, w3.
And the important thing to remember here is that all of these,
perceptrons, all of these neural networks is actually one neural network.
So there's not eight of them, there's just one.
And when we update the weights we're going to update the weights
in that one neural network.
So basically the weights are going to be the same for all of the rows.
So it's not the case that every row has its own weights.
Now all the rows share the weights.
And so that's why we looked at the cost function,
which is the sum of the squared differences.
And then we updated the weights.
And now from here that was just one iteration.
Next we're going to
run this whole thing again.
We're going to, feed every single row into the neural network,
find out our cost function and do this whole process again.
So just as we saw previously, where we had just one row
and we were doing everything again and again, again, again, same thing here.
But now we're going to be doing it for rows or 800 rows or a thousand rows,
however many rows you have in your data set.
you do this process and then you calculate the cost function.
And the goal here is to minimize
the cost function, and to get as soon as you found
the minimum of the cost function, that is your final neural network.
That means your weights have been adjusted and you have,
found the optimal,
weights for
this, data set that you, you're training on and you're ready
to proceed to the testing phase or to the application phase.
And this whole process is called back propagation.
So some additional reading that you might want to do for the cost function.
And I know we just talked about one and there are many different ones.
A good article is located on Cross-Validated.
it's called a list of cost
functions used in neural networks alongside applications.
So the URL is there, but you can just Google
for that exact search term or a search phrase.
And you will that this one will be the first one that pops up.
It's actually got some good examples and application
or use cases for different cost function.
So if you're interested to learn more about cost functions
check out this article.
And on that note, I hope you enjoyed today's tutorial.
I look forward to seeing you next time.
Until then, enjoy deep learning.