Chesley Labadie

Chesley Labadie

1649852760

Goptuna: A Hyperparameter Optimization Framework, inspired By Optuna

Goptuna

Decentralized hyperparameter optimization framework, inspired by Optuna [1]. This library is particularly designed for machine learning, but everything will be able to optimize if you can define the objective function (e.g. Optimizing the number of goroutines of your server and the memory buffer size of the caching systems).

Supported algorithms:

Goptuna supports various state-of-the-art Bayesian optimization, evolution strategies and Multi-armed bandit algorithms. All algorithms are implemented in pure Go and continuously benchmarked on GitHub Actions.

  • Random search
  • TPE: Tree-structured Parzen Estimators [2]
  • CMA-ES: Covariance Matrix Adaptation Evolution Strategy [3]
  • IPOP-CMA-ES: CMA-ES with increasing population size [4]
  • BIPOP-CMA-ES: BI-population CMA-ES [5]
  • Median Stopping Rule [6]
  • ASHA: Asynchronous Successive Halving Algorithm (Optuna flavored version) [1,7,8]
  • Quasi-monte carlo sampling based on Sobol sequence [10, 11]

Projects using Goptuna:

Installation

You can integrate Goptuna in wide variety of Go projects because of its portability of pure Go.

$ go get -u github.com/c-bata/goptuna

Usage

Goptuna supports Define-by-Run style API like Optuna. You can dynamically construct the search spaces.

Basic usage

package main

import (
    "log"
    "math"

    "github.com/c-bata/goptuna"
    "github.com/c-bata/goptuna/tpe"
)

// ① Define an objective function which returns a value you want to minimize.
func objective(trial goptuna.Trial) (float64, error) {
    // ② Define the search space via Suggest APIs.
    x1, _ := trial.SuggestFloat("x1", -10, 10)
    x2, _ := trial.SuggestFloat("x2", -10, 10)
    return math.Pow(x1-2, 2) + math.Pow(x2+5, 2), nil
}

func main() {
    // ③ Create a study which manages each experiment.
    study, err := goptuna.CreateStudy(
        "goptuna-example",
        goptuna.StudyOptionSampler(tpe.NewSampler()))
    if err != nil { ... }

    // ④ Evaluate your objective function.
    err = study.Optimize(objective, 100)
    if err != nil { ... }

    // ⑤ Print the best evaluation parameters.
    v, _ := study.GetBestValue()
    p, _ := study.GetBestParams()
    log.Printf("Best value=%f (x1=%f, x2=%f)",
        v, p["x1"].(float64), p["x2"].(float64))
}

Link: Go Playground

Furthermore, I recommend you to use RDB storage backend for following purposes.

  • Continue from where we stopped in the previous optimizations.
  • Scale studies to tens of workers that connecting to the same RDB storage.
  • Check optimization results via a built-in dashboard.

Built-in Web Dashboard

You can check optimization results by built-in web dashboard.

  • SQLite3: $ goptuna dashboard --storage sqlite:///example.db (See here for details).
  • MySQL: $ goptuna dashboard --storage mysql://goptuna:password@127.0.0.1:3306/yourdb (See here for details)
Manage optimization resultsInteractive live-updating graphs
state-of-the-art-algorithmsvisualization

Advanced Usage

Parallel optimization with multiple goroutine workers

Optimize method of goptuna.Study object is designed as the goroutine safe. So you can easily optimize your objective function using multiple goroutine workers.

package main

import ...

func main() {
    study, _ := goptuna.CreateStudy(...)

    eg, ctx := errgroup.WithContext(context.Background())
    study.WithContext(ctx)
    for i := 0; i < 5; i++ {
        eg.Go(func() error {
            return study.Optimize(objective, 100)
        })
    }
    if err := eg.Wait(); err != nil { ... }
    ...
}

full source code

Distributed optimization using MySQL

There is no complicated setup to use RDB storage backend. First, setup MySQL server like following to share the optimization result.

$ docker pull mysql:8.0
$ docker run \
  -d \
  --rm \
  -p 3306:3306 \
  -e MYSQL_USER=goptuna \
  -e MYSQL_DATABASE=goptuna \
  -e MYSQL_PASSWORD=password \
  -e MYSQL_ALLOW_EMPTY_PASSWORD=yes \
  --name goptuna-mysql \
  mysql:8.0

Then, create a study object using Goptuna CLI.

$ goptuna create-study --storage mysql://goptuna:password@localhost:3306/yourdb --study yourstudy
yourstudy

$ mysql --host 127.0.0.1 --port 3306 --user goptuna -ppassword -e "SELECT * FROM studies;"
+----------+------------+-----------+
| study_id | study_name | direction |
+----------+------------+-----------+
|        1 | yourstudy  | MINIMIZE  |
+----------+------------+-----------+
1 row in set (0.00 sec)

Finally, run the Goptuna workers which contains following code. You can execute distributed optimization by just executing this script from multiple server instances.

package main

import ...

func main() {
    db, _ := gorm.Open(mysql.Open("goptuna:password@tcp(localhost:3306)/yourdb?parseTime=true"), &gorm.Config{
        Logger: logger.Default.LogMode(logger.Silent),
    })
    storage := rdb.NewStorage(db)
    defer db.Close()

    study, _ := goptuna.LoadStudy(
        "yourstudy",
        goptuna.StudyOptionStorage(storage),
        ...,
    )
    _ = study.Optimize(objective, 50)
    ...
}

Full source code is available here.

Receive notifications of each trials

You can receive notifications of each trials via channel. It can be used for logging and any notification systems.

package main

import ...

func main() {
    trialchan := make(chan goptuna.FrozenTrial, 8)
    study, _ := goptuna.CreateStudy(
        ...
        goptuna.StudyOptionIgnoreObjectiveErr(true),
        goptuna.StudyOptionSetTrialNotifyChannel(trialchan),
    )

    var wg sync.WaitGroup
    wg.Add(2)
    go func() {
        defer wg.Done()
        err = study.Optimize(objective, 100)
        close(trialchan)
    }()
    go func() {
        defer wg.Done()
        for t := range trialchan {
            log.Println("trial", t)
        }
    }()
    wg.Wait()
    if err != nil { ... }
    ...
}

full source code

Links

References:

Presentations:

Blog posts:

Status:

Author: C-bata
Source Code: https://github.com/c-bata/goptuna 
License: MIT License

#go #golang #machinelearning 

What is GEEK

Buddha Community

Goptuna: A Hyperparameter Optimization Framework, inspired By Optuna
Zakary Schmeler

Zakary Schmeler

1596745500

Efficient Hyperparameter Optimization for XGBoost model Using Optuna

Introduction :

Hyperparameter optimization is the science of tuning or choosing the best set of hyperparameters for a learning algorithm. A set of optimal hyperparameter has a big impact on the performance of any machine learning algorithm. It is one of the most time-consuming yet a crucial step in machine learning training pipeline.

A Machine learning model has two types of tunable parameter :

· Model parameters

· Model hyperparameters

Image for post

Model parameters vs Model hyperparameters (source)

Model parameters are learned during the training phase of a model or classifier. For example :

  • coefficients in logistic regression or linear regression
  • weights in an artificial neural network

**_Model Hyperparameters _**are set by the user before the model training phase. For example :

  • ‘c’ (regularization strength), ‘penalty’ and ‘solver’ in logistic regression
  • ‘learning rate’, ‘batch size’, ‘number of hidden layers’ etc. in an artificial neural network

The choice of Machine learning model depends on the dataset, the task in hand i.e. prediction or classification. Each model has its own unique set of hyperparameter and the task of finding the best combination of these parameters is known as hyperparameter optimization.

For solving hyperparameter optimization problem there are various methods are available. For example :

  • Grid Search
  • Random Search
  • Optuna
  • HyperOpt

In this post, we will focus on Optuna library which has one of the most accurate and successful hyperparameter optimization strategy.

#hyperparameter-tuning #optimization-algorithms #xgboost #optuna #machine-learning #algorithms

Everett Collier

Everett Collier

1624260780

Hyperparameter Optimization Run Time and Cost using AWS and Optuna

Following our companion blog on sequential hyperparameter optimization, here we discuss the engineering considerations taken with respect to run time and cost. We specifically dive into approaches to speed up parameter search using parallel or distributed computing. This is important since hyperparameter optimization (HPO) is often one of the costliest and slowest aspects of model development.

We optimized our hyperparameters using AWS virtual machines (EC2 instances) as the hardware and Optuna as the software framework. Optuna is a relatively new open-source framework for HPO developed by Preferred Networks, Inc.

Parallel and distributed computing

Both parallel and distributed computing can shorten run durations. Image by author.

The goal of parallel and distributed computing is to optimally use hardware resources to speed up computational tasks. While these two terms sound similar, and both indeed refer to running multiple processes simultaneously, there is an important distinction.

  • Parallel computing refers to running multiple tasks simultaneously on the different processors of a single machine.
  • Distributed computing refers to the ability to run tasks simultaneously on multiple autonomous machines.

#optuna #aws #hyperparameter optimization

AppClues Infotech

AppClues Infotech

1618480618

Best Android Mobile App Development Frameworks

Are you looking for the best Android app development frameworks? Get the best Android app development frameworks that help to build the top-notch Android mobile app.

For more info:
Website: https://www.appcluesinfotech.com/
Email: info@appcluesinfotech.com
Call: +1-978-309-9910

#best android mobile app development frameworks #top mobile app development frameworks #android app development frameworks #top frameworks for android app development #most popular android app development frameworks #app development frameworks

AppClues Infotech

AppClues Infotech

1619522346

Top Mobile App Development Frameworks in 2021

Do you need a high-quality and reliable framework to optimize the process? AppClues Infotech has created a list of top mobile app development frameworks to consider working with in the year 2021.

For more info:
Website: https://www.appcluesinfotech.com/
Email: info@appcluesinfotech.com
Call: +1-978-309-9910

#top mobile app development frameworks #top mobile app frameworks in 2021 #best mobile app development frameworks #best mobile app development frameworks #mobile development framework

TopDevelopers Co

TopDevelopers Co

1594963828

Top 15 Free JavaScript Frameworks for Web Applications

List of some useful JavaScript Frameworks and libraries for website, web apps, and mobile apps development, that developers should know about to make selection easier.
This article will help you understand the various types of JavaScript Framework available in the market. When it comes to choosing the best platform for you, it’s not only the number of features you need to consider but also its functionality. The ease with which it fits within your project is also an essential factor. The next step is to choose the framework that best fits your company requirements or you can select the best from the list of top web development companies to develop your product based on your requirements.

#javascript frameworks for web applications #web applications development companies #progressive javascript framework #javascript frameworks #javascript #frameworks