In the last lesson we spoke about the inspiration and the general structure of a neural network and
how the learning happens.
We discussed the importance of the connection weights and we also spoke about how neural networks might
be good at things that traditional computers are not very good at.
In this lesson we'll talk more about layers in a neural network feature generation and learning artificial
neural networks are such a hot topic these days they are receiving a lot of hype when you come across
a news piece or an article on machine learning on say Hacker News or another outlet.
They're almost always talking about deep learning and artificial neural networks like the time that
Alpha go Pete the 9th then go champion Lee Siegel across several games of Go or Google's self-driving
cars or Google Translate which can sing by the way.
I a though though because they are using Nokia though.
Do you like your health though though.
How.
Put the link to that Google Translate music video into the course resources.
So if you're curious.
Check it out.
But now let's go back to the more important question at hand.
Why are neural networks so special.
What makes them so different.
Well let's think back to what we were doing in our regression tutorial when we were estimating the house
prices in Boston.
We were sifting through our data and we're choosing our features and we were specifying our model and
then we're using that model to estimate our house prices or make our predictions in this model that
we had we had eleven different features and by features I mean things like the number of rooms or the
amount of crime in the area or the amount of pollution.
The point is that we as the programmers we chose to include certain features in our model.
And similarly we chose what to do with those features.
Should we add them up should we multiply them together should we combine them.
These were all decisions that we took when specifying our model.
Now similarly when we were working with our native base spam classifier our features were the tokens
for the words in our emails.
We extracted these tokens we extract these features and then we use them to make our predictions.
Both of these approaches are what would be called a shallow learning algorithm.
We as the data scientists as the machine learning experts were choosing which features would help us
make sense of our data and make better predictions.
All that our algorithm had to do was learn the parameters for the model.
In the case of our regression we were training our model to learn these theta values.
In the case of naive bays we had to learn the probabilities to help classify our emails as spam or not
spam.
The thing is deciding which features to use in a model and how to use these features is actually a very
very challenging thing.
When we were working with our regression model we chose to exclude some features and transform others
and we did that to get a better model and to get a more reliable estimate.
And that's the crux of it right.
To get better results we needed to know something about the relationships in our data because certain
relationships are linear.
And that's pretty straightforward.
But other relationships are non-linear like the distance from employment centers and pollution in our
Boston house price data set.
This is where our relationship would best be represented by a curve.
In the case of classification we could also find nonlinear relationships right for example.
So you have some data points and they're distributed like this and the line that best separates them
would be a squiggly one would look like this.
In this case you've got a nonlinear decision boundary.
But what's the mathematical formula for this line that best fits the data.
Also will this formula play nicely with the model that we're using or do we need to choose a different
model.
Is there some kind of transformation that we can make to our features to get a better estimate.
Maybe a feature needs to be squared or a feature needs to be combined with one or other features to
get a better prediction.
These are all things that you as the model architect you as the programmer have to figure out when you're
using shallow learning algorithms Now in contrast with deep learning the neural network will do the
job of feature selection for the programmer.
The neural network will learn the features from the data by itself regardless of whether you've got
a linear relationship or a non-linear relationship in the data.
The neural network will learn to combine features in the most effective way and this is the reason that
we didn't have to go around programming neural networks explicitly.
This is why we can teach a car to drive itself without having to write the code to make the self-driving
car stop at a red light.
Let's talk about this whole process with a bit more of a concrete example.
Let's talk about image recognition with image recognition.
A neural network can learn to identify what is in the image.
Say that we want to identifying if an image contains cats or if it doesn't contain cats and that we
would do this if we would feed through labelled images into the network.
So a bunch of training data images containing cats labeled as cats images not containing cats labeled
not a cat.
And as these images are fed through the network and the network extracts the features that matter the
weights between the neurons in the network are updated and by the end of training you can use this network
to identify cats and images that the network has never seen before and at no point in this whole process
do you have to sit there and laboriously program the neural network to look for fur or tails or whiskers
or cat like faces.
You as the programmer don't have to supply the features.
Instead the neural network automatically generates the features from the training data set.
And this ability the ability of the neural network to learn the underlying identifying characteristics
of a cat from a set of images is one of the reasons why deep learning is both so powerful and also so
magical.
And I do say magical because it really feels that way.
Deep Learning removes the whole feature selection process.
It removes the learning one step away from the programmer.
And this is always a contrast to simpler algorithms where the machine learning algorithm is close to
the programmer and it feels a lot less like artificial intelligence in comparison to something like
deep learning.
But of course there is no magic.
So how does the neural network learn to generate its own features.
Let's stick with this cat example in our thought experiment.
The catch detection neural network is going to answer one of life's most important questions.
Do we have a kitty or do we not have a kitty.
That's the output that the network will generate for us.
Yay or Nay thumbs up or thumbs down.
Now neural networks are often represented in a chant like this.
This chant flows from left to right on the left.
We have the inputs to the neural network and on the right we have the output.
Now we've already talked a little bit about the individual neurons right.
We've talked about how they might fire or not fire.
Well this was kind of the model from the 1940s.
These were some of the earliest models the neuron what activate or not activate one or zero.
But we could also make our neuron a little bit more nuanced in the case where we just had a one or a
zero.
The function that would determine whether a neuron would activate would probably look something like
this would be a stepwise function for certain values would output a one for other values would output
a zero.
But what if instead our neuron actually gave us a probability between 0 and 1.
That way it could say things like well there's a 10 percent chance of this image containing a cat and
therefore it is not a cat or it could say things like well I'm 90 percent sure that this is a cat.
In this case you wouldn't have a stepwise function you'd have something a little bit more smooth something
maybe that looks like this.
This is a sigmoid function and you can see that it's continuous.
So if each individual neuron in the network would be using this function then it could fire with a weak
signal or with a slightly stronger signal depending on where it is on this function these functions
the ones that determine how if and how strong these neurons actually fire have a name.
These are called activation functions and it's the value of these activation functions that determine
the output.
An Iran.
Now if we take a look at this chart again what we can see is that these neurons are grouped into layers.
And in this case we've got three layers.
This is a very very simple network.
This first layer here is called the input layer.
Each node in the input layer represents a feature.
So in this example we've got six features.
The second layer here is called the hidden layer.
Very mysterious right.
Six nodes in the hidden layer.
Notice how each input feature is actually connected to each and every node in the hidden layer.
And that last layer is called the output layer.
And here again each neuron in that second layer connects to each neuron in the output layer.
In this case we've got one but we could have more right.
We could have two or three or ten or what have you.
The point I'm trying to make is that the neurons are grouped they're grouped into layers.
You can have different number of neurons in each layer and between the two layers all the neurons are
connected to each other.
So one thing that we can do is we can change the architecture of our network.
We can change the number of layers that it has the deeper the network the more layers it has.
That's where the deep and deep learning comes from.
So in this case we've got two hidden layers.
If we want an even deeper network then we could add a third layer yet again.
So in this case we have three hidden layers five total three hidden.
But here's the million dollar question what goes on in the hidden layers.
Well let's tackle that first hidden layer shall we.
In that first hidden layer you'll see that every input is connected to every single node.
And this is important because we were saying how the overall goal of the neural network will be to discover
the optimal combination of features.
So the fact that they're all connected means it gets to try out every single combination.
This is what this first hidden layer will do.
We'll combine each of the input features every which way and it will try to learn the best way to combine
these features.
Now in our thought experiment what we would do is we would show this neural network an image and we
will ask the neural network to tell us if this image contains a count or not count each pixel in the
image will be an input to the neural network.
Now I've drawn six inputs in the previous line but if we had a 30 by 30 pixel image then that would
actually be 900 different input nodes because we would convert the image into a matrix where each number
represents the value of a particular pixel when I say value would be something like the color.
Right.
Is it black is it white and red.
Green.
What have you now for the sake of argument.
This picture of a giraffe is what we're going to be sending over to our neural network.
We're going to convert it into numbers and then those numbers get passed on to the first hidden layer.
So what happens next.
That first hidden layer will combine all the pixels of this image and it will start to generate features
from those pixels.
Now what kind of features will it generate.
Well the neural network will probably start to detect simple patterns like lines or edges or textures
and those patterns those are going to be the features that the first hidden layer will generate.
The second hidden layer will then use those features that the first hidden layer outputs it.
And we'll try to work with them.
So that second hidden layer is no longer confronted with the individual pixels it's confronted with
the features that the first hidden layer generated.
So what will the second layer do with those things.
What it might do is try to combine these edges and lines and textures into something that's of a higher
level of complexity.
So it might start to detect things like shapes like rectangles circles or shadows or something that's
a little bit more complex than lines and edges and textures and those shapes in turn are going to feed
through to that third hidden layer.
The third hidden layer gets these shapes as an input and it will take these shapes and make its own
features say something like eyes or is or a tail or legs and these are the features that the output
layer then will use to identify if this image contains a cat or not a cat.
So this brings us to the output layer.
If a neural network we're a company then this output layer would be the CEO.
The output layer will look at what that last hidden layer sending over and make its decision.
So see that top neuron in that third hidden layer files.
And it says it detected an AI and then that second neuron in that last hidden layer of files and says
it detected some is the next one down reports that it detected a tail.
But the fourth one is silent no legs says the fourth neuron to the CEO how the output layer then has
to make a decision.
Was this a picture of a cat.
Well the CEO will take a good hard look at his managers and then he'll take a weighted average of their
outputs so the output layer comes back to us and says Yes I am 75 percent certain that we have a cat
in this image but we know the true answer right.
We fed them this image and we say well sorry Mark but that's not good enough a cat.
It is not.
You have made the wrong call and I'm afraid we still have a long way to go well what happens now.
Well our network just made a prediction but it got it wrong.
So our CEO is angry and he has to figure out how far off he was with his prediction.
So he looks at his loss and he adjusts his weights.
And maybe next time he'll be more suspicious of his first two managers and he'll listen more closely
to his fourth manager.
Maybe that will result in a better prediction.
So he runs back inside his company calls a meeting and while he starts yelling at his managers and they're
all shifting around uncomfortably in their three thousand dollar ergonomic chairs and they're looking
at the ground this is embarrassing right.
But the managers know exactly who's to blame for this fiasco.
If the associates reporting to them so the managers in that third hidden layer adjust their weights.
Call a meeting and they start yelling at their associates.
Now the associates are vexed.
So the associates adjust their weights and start giving the juniors and the interns a hung time.
But the juniors they only have the inputs to blame right.
Stupid pixels but pixels can't talk back.
So the juniors just adjust their weights and try to generate slightly different features.
The next time round for the next image and this whole process is called back propagation.
The era is passed down through the network from the output node so that each node in each layer adjusts
its weights.
So at this point the question is Well how are the weights adjusted.
Are they just it down or they adjust that up.
Well this depends on the loss function the slope or the gradient of this lost function will determine
the adjustment and we actually cover this in detail in our separate module that is dedicated to gradient
descent.
The point I'm trying to make here is that through trial and error and lots of yelling from the higher
ups in the company the network is able to start to generate its own features and detect patterns in
the data.
Now even though I spoke of lines and of shapes at one end and a higher level features like eyes and
tails at the other end of the network the truth is that we don't know exactly what features the neural
network actually generates but what we do know is that the neural network breaks down the input data
into chunks.
It creates a hierarchy but we don't know exactly what goes on under the hood.
And therein also lies a problem because this makes neural networks a bit like a black box.
We don't know exactly what's going on inside the neural networks.
So a neural network isn't exactly what you would call a tractable model.
And this is a big problem when you're not just interested in the accuracy of a prediction but you need
to understand why a neural network has given a particular output.
There is in fact a whole subfield of machine learning dedicated to analyzing neural networks to better
understand why they do what they do.
So I know this is a very very dense lesson on the theory.
So here's the executive summary.
Each neuron in a neural network will be activated based on a mathematical formula called the activation
function.
The activation function determines how strong this neuron will fire and then through trial and error
the neural network is able to generate its own features from the input data.
This allows the neural network to solve both linear problems and nonlinear problems because it tries
all these combinations the deeper the network the more complex and the more high level the features
are that are generated at each layer.
One piece of good news is that the pattern of learning for a neural network is very similar to our other
machine learning algorithms.
It makes a prediction.
It figures out how far off the prediction was by looking at the loss and then it adjusts its parameters
it adjusts its weights between the neurons.
In this case and the process by which the error gets sent back down through the network so that each
node can adjust its weights is called back propagation.
So in summary neural networks are very powerful and a reasonable question as well if they're so powerful
can we use neural networks for everything.
And the answer is yes you can.
You can solve almost every machine learning problem with the neural network but would you want to use
a neural network to solve every problem.
In this case the answer is no.
Why.
Well let's talk about that in the next lesson.
I'll see you there.
Take care.