Following our companion blog on sequential hyperparameter optimization, here we discuss the engineering considerations taken with respect to run time and cost. We specifically dive into approaches to speed up parameter search using parallel or distributed computing. This is important since hyperparameter optimization (HPO) is often one of the costliest and slowest aspects of model development.

We optimized our hyperparameters using AWS virtual machines (EC2 instances) as the hardware and Optuna as the software framework. Optuna is a relatively new open-source framework for HPO developed by Preferred Networks, Inc.

Parallel and distributed computing

Both parallel and distributed computing can shorten run durations. Image by author.

The goal of parallel and distributed computing is to optimally use hardware resources to speed up computational tasks. While these two terms sound similar, and both indeed refer to running multiple processes simultaneously, there is an important distinction.

• Parallel computing refers to running multiple tasks simultaneously on the different processors of a single machine.
• Distributed computing refers to the ability to run tasks simultaneously on multiple autonomous machines.

#optuna #aws #hyperparameter optimization