A comprehensive guide to understanding hyper-parameter optimization using Bayesian optimization with GPyOpt library in deep neural networks

A comprehensive guide to understanding hyper-parameter optimization using Bayesian optimization with GPyOpt library in deep neural networks

If you are new into the machine learning or deep learning field I suggest you read first some of my previous *articles* in which I give an approach to some of the concepts covered here. Nevertheless, if you want to have fun and get your mind blown for the next couple minutes sit and let’s start this journey in the machine learning field

The first time I started to work with neural networks I used to tune hyperparameters of my machine learning model based on trial and error or based on what other peers found in their tuning. However, what if you want to have more control over your neural network model and search for a value that can reduce your uncertainty to achieve a reliable accuracy with your data?

Indeed this question is attractive when designing and implementing your neural network and maybe you will think “I want to do this but it seems hard”. Well, partially it is. But, in this article, we are going to have a friendly approach on how to achieve the hyper-parameter optimization and not fail in the way. Once you understand this, believe me, either this technique or other technique you want to use for optimizing hyper-parameters, it will become your best friend to succeed in building better neural networks.

When you build a neural network to address a practical problem you came across a concept called hyper-parameters. Depending on how your tune these parameters your network will give you an approximation on your target accuracy or not. Think in parameters as variables that determine your network structure and how it will be trained. e.g how many layers does your neural network will have? does my network will have a learning rate variable when it’s being trained?

Despite the definition of hyper-parameter may sound simple it is important to notice that these are different from the model parameters. The model parameters are the properties of the training data that are learned during training by the classifier or other machine learning models. e.g. for a neural network the weights of the connections in the model are parameters because they differ for each experiment and depend on the type of input data.

machine-learning optimization neural-networks deep-learning software-engineering

In this article, we explore gradient descent - the grandfather of all optimization techniques and it’s variations. We implement them from scratch with Python.

On Optimization of Deep Neural Networks: A random-walk to classical developments that made deep learning the ultimate learning machines.

In this article, we will only focus on the Better Optimizing algorithm for Deep Neural Network (DNN). We will call this optimizing algorithm as a Learning algorithm for this article.

Various Optimization Algorithms For Training Neural Network: The right optimization algorithm can reduce training time exponentially.

The past few decades have witnessed a massive boom in the penetration as well as the power of computation, and amidst this information.