Hyperparameter Tuning

Parameters and Hyperparameters:

Parameters:

Probabilistic models are estimated by unknown quantities, called parameters. These are adjusted using an optimization technique so that in the training sample it is possible to find a pattern in the best possible way. In a simple way, parameters are estimated by the algorithm and the user has little / nothing control over them.

In a simple linear regression, the model parameters are betas (ẞ).

Image for post

Font: https://pt.slideshare.net/vitor_vasconcelos/regresso-linear-mltipla

BAM!!! : In statistics jargon, parameters are defined as population characteristics. So strictly speaking, when we want to talk about sample characteristics, we use the term estimators. This makes little difference in this context, but it is important to note.

Hyperparameters:

Hyperparameters are sets of information that are used to control way of learn an algorithm. Their definitions impact parameters of the models, seen as a way of learning, change from the new hyperparameters. This set of values affects performance, stability and interpretation of a model.

Each algorithm requires a specific hyperparameters grid that can be adjusted according to the business problem.

#hyperparameter-tuning #machine-learning #statistics #deep learning

What is GEEK

Buddha Community

Jerad Bailey

1597275960

Hyperparameters Tuning Using GridSearchCV And RandomizedSearchCV

While building a Machine learning model we always define two things that are model parameters and model hyperparameters of a predictive algorithm. Model parameters are the ones that are an internal part of the model and their value is computed automatically by the model referring to the data like support vectors in a support vector machine. But hyperparameters are the ones that can be manipulated by the programmer to improve the performance of the model like the learning rate of a deep learning model. They are the one that commands over the algorithm and are initialized in the form of a tuple.

In this article, we will explore hyperparameter tuning. We will see what are the different parts of a hyperparameter, how it is done using two different approaches – GridSearchCV and RandomizedSearchCV. For this experiment, we will use the Boston Housing Dataset that can be downloaded from Kaggle. We will first build the model using default parameters, then we will build the same model using a hyperparameter tuning approach and then will compare the performance of the model.

What We Will Learn From This Article?

What is Hyper Parameter Tuning?
What steps to follow to do Hyper Parameter Tuning?
Implementation of Regression Model
Implementation of Model using GridSearchCV
Implementation of Model using RandomizedSearchCV
Comparison of Different Models

1. What Is Hyperparameter Tuning?

Hyperparameter tuning is the process of tuning the parameters present as the tuples while we build machine learning models. These parameters are defined by us which can be manipulated according to programmer wish. Machine learning algorithms never learn these parameters. These are tuned so that we could get good performance by the model. Hyperparameter tuning aims to find such parameters where the performance of the model is highest or where the model performance is best and the error rate is least. We define the hyperparameter as shown below for the random forest classifier model. These parameters are tuned randomly and results are checked.

#developers corner #hyperparameter tuning #hyperparameters #machine learning #parameter tuning

Zakary Goyette

1600966800

Hyperparameter tuning with Keras and Ray Tune

In my previous article, I had explained how to build a small and nimble image classifier and what are the advantages of having variable input dimensions in a convolutional neural network. However, after going through the model building code and training routine, one can ask questions such as:

How to choose the number of layers in a neural network?
How to choose the optimal number of units/filters in each layer?
What would be the best data augmentation strategy for my dataset?
What batch size and learning rate would be appropriate?

Building or training a neural network involves figuring out the answers to the above questions. You may have an intuition for CNNs, for example, as we go deeper the number of filters in each layer should increase as the neural network learns to extract more and more complex features built on simpler features extracted in the earlier layers. However, there might be a more optimal model (for your dataset) with a lesser number of parameters that might outperform the model that you have designed based on your intuition.

In this article, I’ll explain what these parameters are and how do they affect the training of a machine learning model. I’ll explain how do machine learning engineers choose these parameters and how can we automate this process using a simple mathematical concept. I’ll be starting with the same model architecture from my previous article and will be modifying it to make most of the training and architectural parameters tunable.

#data-science #machine-learning #deep-learning #hyperparameter-tuning #bayesian-optimization

Rusty Shanahan

1598062920

Fine Tuning XGBoost model

Tuning the model is the way to supercharge the model to increase their performance. Let us look into an example where there is a comparison between the untuned XGBoost model and tuned XGBoost model based on their RMSE score. Later, you will know about the description of the hyperparameters in XGBoost.

Below is the code example for untuned parameters in XGBoost model:

#Importing necessary libraries
	import pandas as pd
	import numpy as np 
	import xgboost as xg

	#Load the data
	house = pd.read_csv("ames_housing_trimmed_pricessed.csv")
	X,y = house[house.columns.tolist()[:-1]],
	            house[house.columns.tolist()[-1]]

	#Converting it into DMatrix
	house_dmatrix = xgb.DMatrix(data = X, label = y)

	#Parameter configuration
	param_untuned = {"objective":"reg:linear"}

	cv_untuned_rmse = xg.cv(dtrain = house_dmatrix, params = param_untuned, nfold = 4, 
	                        metrics = "rmse", as_pandas = True, seed= 123)
	print("RMSE Untuned: %f" %((cv_untuned_rmse["test-rmse-mean"]).tail(1)))
view raw
tune_1.py hosted with ❤ by GitHub

Output: 34624.229980

Now let us look to the value of RMSE when the parameters are tuned to some extent:

#Importing necessary libraries
	import pandas as pd
	import numpy as np 
	import xgboost as xg

	#Load the data
	house = pd.read_csv("ames_housing_trimmed_pricessed.csv")
	X,y = house[house.columns.tolist()[:-1]],
	            house[house.columns.tolist()[-1]]

	#Converting it into DMatrix
	house_dmatrix = xgb.DMatrix(data = X, label = y)

	#Parameter Configuration
	param_tuned = {"objective":"reg:linear", 'colsample_bytree': 0.3,
	               'learning_rate': 0.1, 'max_depth': 5}

	cv_tuned_rmse = xg.cv(dtrain = house_dmatrix, params = param_tuned, nfold = 4,
	                      num_boost_round = 200, metrics = "rmse", as_pandas = True, seed= 123)
	print("RMSE Tuned: %f" %((cv_tuned_rmse["test-rmse-mean"]).tail(1)))
view raw
tune_2.py hosted with ❤ by GitHub

Output: 29812.683594

It can be seen that there is around 15% reduction in the RMSE score when the parameters got tuned.

#machine-learning #hyperparameter #artificial-intelligence #hyperparameter-tuning #xgboost #deep learning

Mathias Murazik

1600891200

A Novice’s Guide to Hyperparameter Optimization at Scale

Despite the tremendous success of machine learning (ML), modern algorithms still depend on a variety of free non-trainable hyperparameters. Ultimately, our ability to select quality hyperparameters governs the performance for a given model. In the past, and even some currently, hyperparameters were hand selected through trial and error. An entire field has been dedicated to improving this selection process; it is referred to as hyperparameter optimization (HPO). Inherently, HPO requires testing many different hyperparameter configurations and as a result can benefit tremendously from massively parallel resources like the Perlmutter system we are building at the National Energy Research Scientific Computing Center (NERSC). As we prepare for Perlmutter, we wanted to explore the multitude of HPO frameworks and strategies that exist on a model of interest. This article is a product of that exploration and is intended to provide an introduction to HPO methods and guidance on running HPO at scale, based on my recent experiences and results.

Disclaimer; this article contains plenty of general non-software specific information about HPO, but there is a bias for free open source software that is applicable to our systems at NERSC.

In this article, we will cover …

Scalable HPO with Ray Tune
Schedulers vs Search Algorithms
Not All Hyperparameters Can Be Treated the Same
Time-to-Solution Study
Optimal Scheduling with PBT
Cheat Sheet for Selecting an HPO Strategy
Technical Tips— Ray Tune, Dragonfly, Slurm, TB, W&B
Key Takeaways

#editors-pick #machine-learning #hyperparameter #hyperparameter-tuning #deep-learning

Eldora Powlowski

1597805700