In this lecture, we will discuss about the hyper parameters of what neural network architecture and
the lectors didn't know.
You have seen that there are so many hybrid barometers in neural networks, and these hybrid barometers
give us the flexibility of creating several types of architectures.
But the flexibility of neural networks is also one of their main drawbacks.
We have to decide on so many hybrid parameters in our model.
Not only can we use any imaginable network architecture, but even in our simple, multi-level Perceptron,
you can change the number of layers, the number of neurons, parlier the type of activation function
to use in each layer, duvid, initialization logic and many more.
Although still a lot of exciting research is going on in the field of hyper barometer tuning for neural
networks.
It will still help to have an idea of what values are reasonable for each hyper barometer.
So you can build a quick prototype and restrict the search space.
He had a few guidelines for choosing the number of layers and neurons in a multilevel Perceptron.
Let's first discuss about the number of hidden layers.
For most of the problems, you can just begin with a single earlier and you will get reasonable results.
It has actually been shown that an MLP with just one hidden layer can model even the most complex functions,
provided it has enough neurons for a long time.
These facts convinced researchers that there is no need to investigate any deep neural network.
But it was later found that deep networks have a much higher parameter efficiency than shallow ones.
They can model complex functions using exponentially fewer neurons than shallowness, allowing them
to read much better performance with the same amount of training data to understand why this happens.
Suppose you are asked to draw a forest using some drawing software, but you are forbidden to use copy
paste.
You have brought each tree individually branch by branch leaf believe if you could instead draw one
leaf copy pasted to a broader branch, then copy paste the branches to create the tree and finally copy
paste this tree to make a forest.
You would be finished in no time.
Real raw data is often structured in such a hierarchical way.
And deep neural networks automatically take advantage of this fact.
Lower head and layers, more lower level structures, intermediate hidden layers.
Combine these lower level structures to model intermediate level structures and higher state layers
and the output layer.
Combine these intermediate structures to model high level structures.
Not only does this hierarchical structure help deep neural networks converge faster, it also improves
the ability to generalize to new data.
Say, for example, if you've already trained a model to recognize faces in a picture and you know,
want to train a new neural network model to recognize hairstyles, then you can kick start training
by reusing the lower levels of the first network instead of randomly initializing DVDs and biases of
the first few layers of the new neural network.
You can initialize them to the value of DVDs and biases of the lower layers of the first network.
This way, the network will not have to learn from scratch and it will only have to learn the higher
level structures.
This is called transfer learning.
So in summary, for most problems, you can start with just one or two hidden layers and it will work
just fine for more complex problems.
You can gradually ramp up the number of hidden layers until you start overfitting the training data.
Next, we discuss the number of neurons, but he didn't listen.
Obviously, the number of neurons in the input and output layers is determined by the type of input
and output.
Your task requires, for example, the M.
NASD fashion dataset that we used that required 24 input, 24.
That is 780 for input neurons and then output neurons.
As for the hidden layers, unclear.
It was a common practice to size them to formal.
But I mean, that is the first layer had the most number of neurons.
So, for example, in the M&A zesty fashion dataset with three layers, you can have 300 neurons in
the first data layer, 200 neurons and the second one and hundred, and deterred the rationale being
that many low level features can coalesce into far fewer higher level features.
However, this practice has been largely abandoned now, as it seems that simply using these same number
of neurons in all hidden lives performs just as well in most cases, or maybe even better.
Also, it has the advantage of having only one hyper barometer to tune instead of one but leered.
So instead of having three hundred, two hundred and one hundred neurons in the tree and layers, you
can have 150 neurons in all three of them.
If you think that the problem at hand is really complex, then you can try increasing the number of
neurons gradually until the network starts overfitting in general, increasing the depth of the network
has better reser on the accuracy, then increasing the number of neurons.
Bollier.
Another approach could be to become model with a large number of layers and large number of neurons
but hidden layer and then use at least hoping to prevent that model from overfitting.
Next, Apple Barometer.
We are going to discuss those learning great learning.
It is arguably the most important hyper barometer.
In general, the optimal learning rate is about half of the maximum learning rate.
That is the learning rate about which the training algorithm diverges.
So a simple approach for joining the learning rate is to start with large value.
That makes the algorithm diverge.
Then divide this value by three.
And try again and repeat this until the training algorithm stops diverging.
At that point, you generally won't be too far from the optimal learning rate.
Then there is bad sites for bad taste.
The general rule, general rule of thumb, trying to keep bad sales lower than 32 because a small bad
size ensures that each training iteration is very fast and on the lower end.
Try to keep a bad size more than 20 because this helps take advantage of the hardware and software optimizations,
in particular for matrix multiplications.
So this will also help in speeding up training.
So a good range is between twenty to thirty two.
Lastly, there is another hyper parameter called epoch.
That is the number of training iterations that we will do instead of tuning it.
We would suggest that you use a large number for epochs and usually at least topping technique to prevent
overfitting.
So that's all about selecting hyper parameters.