In this lecture we will discuss about the hyper parameters of our neural network architecture and the
lectures in law you have seen that there are so many hybrid parameters in neural networks and these
hyper parameters give us the flexibility of creating similar types of architectures but the flexibility
of neural networks is also one of their main drawbacks.
We have to decide on so many hybrid parameters in our model.
Not only can we use any imaginable network architecture but even in a simple multilevel perception you
can change the number of layers.
The number of neurons per layer the type of activation function to use in each layer the rate initialization
logic and many more although still a lot of exciting research is going on in the field of hyper parameter
tuning for neural networks.
It will still help to have an idea of what values are reasonable for each hyper parameter so you can
build a quick prototype and restrict the search space.
Here are a few guidelines for choosing the number of the layers and neurons in a multilevel perception.
Let's first discuss about the number of hidden layers for most of the problems.
You can just begin with a single hidden layer and you will get reasonable results.
It has actually been shown that an MLP with just one hidden layer can model even the most complex functions
provided it has enough neurons for a long time.
These fact convince these voters that there is no need to investigate any deep neural network but it
was later found that deep networks have a much higher parameter efficiency than shallow ones.
They can model complex functions using exponentially fewer neurons than shallowness allowing them to
read much better performance with the same amount of trading data to understand why this happens.
Suppose you are asked to draw a forest using some drawing software but you are forbidden to use copy
paste.
You have to draw each tree individually branch by branch leaf belief.
If you could instead draw one leaf copy pasted to a broader branch then copy paste the branches to create
the tree and finally copy paste this tree to make a forest.
You would be finished in no time.
Real world data is often structured in such a hierarchical way and deep neural networks automatically
take advantage of this fact.
Lower hidden layers model lower level structures intermediate hidden layers.
Combine these lower level structures to model intermediate level structures and higher state layers
and the output layer.
Combine these intermediate structures to model high level structures.
Not only does this hierarchical structure help deep neural networks converge faster it also improves
the ability to generalize to new data.
For example if you have already trained a model to recognize faces in a picture and you know want to
train a new neural network model to recognize hair styles then you can kick start training by reusing
the lower levels of the first network instead of randomly initializing the weights and biases of the
first few layers of the new neural network.
You can initialize them to the value of the weights and biases of the lower layers of the first network.
This way the network will not have to learn from scratch and it will only have to learn the higher level
structures.
This is called transfer learning.
So in summary for most problems you can start with just one or two hidden layers and it will work just
fine for more complex problems.
You can gradually ramp up the number of hidden layers until you start all fitting the training data.
Next we discuss the number of neurons but you then layer obviously the number of neurons in the input
and output layers is determined by the type of input and output.
Your task requires for example the M NASD fashion dataset that we used that required 24 into 24.
That is 780 for input neurons and then output neurons.
As for the hidden layers earlier it was a common practice to size them to form a pyramid.
That is the first layer had the most number of neurons.
For example in the M NASD fashion dataset with three layers.
You can have 300 neurons in the first data layer two hundred neurons and the second one and hundred
and deterred the rationale being that many low level features can coalesce into far fewer higher level
features however this practice has been largely abandoned now as it seems that simply using these same
number of neurons in all hidden layers performs just as well.
In most cases or maybe even better
also it has the advantage of having only one hyper parameter to tune instead of one.
But earlier so instead of having three hundred two hundred and 100 neurons in the tree and layers you
can have 150 neurons in all three of them if you think that the problem at hand is really complex then
you can try increasing the number of neurons gradually until the network starts over putting in general
increasing the depth of the network has better results on the accuracy then increasing the number of
neurons.
Butler another approach could be to pick a model with large number of layers and large number of neurons
per head and there and then use at least hoping to prevent that model from all fitting
next type of barometer.
We are going to discuss learning great learning that is arguably the most important hyper parameter.
In general the optimal learning rate is about half of the maximum learning rate that is in learning
grade about which the training algorithm diverges
so a simple approach for tuning the learning rate is to start with large value that makes the algorithm
they would then divide this value by three and try again and repeat this until the training algorithm
stops diverging at that point.
You generally won't be too far from the optimal learning grade.
Then there is bad sites for bad essays.
The general rule general rules of thumb try to keep bad sides lower than 32 because a small bad size
ensures that each training ideation is very fast and on the lower end.
Try to keep a bad size more than 20 because this helps take advantage of the hardware and software optimizations
in particular for matrix multiplications.
So this will also help in speeding up training so a good range is between 20 to 32.
Lastly there is another hyper parameter called epoch that is the number of training ideations that we
will do instead of tuning it.
We would suggest that you use a large number for epochs and use the early stopping technique to prevent
or predict that's all about selecting high but parameters.