Neural Architecture Search with NNI

Neural Architecture Search with NNI

In this post, we will show how to perform hyper-parameter search using an automated machine learning (AutoML) tool — NNI (for Neural Network Intelligence) open-sourced by Microsoft.

Neural architecture search (NAS) is a difficult challenge in deep learning. Many of us have experienced that for a given dataset, a network may initially struggle to learn. But with a simple change of a hyper-parameter, the learning can become very effective. Manually tweaking hyper-parameters including architecture is time-consuming and challenging even though to some it can also be a lot of fun. Recently, automatic hyper-parameter tuning has become more and more popular, as it provides an efficient mechanism to solve NAS at scale.

In this post, we will show how to perform hyper-parameter search using an automated machine learning (AutoML) tool — NNI (for Neural Network Intelligence) open-sourced by Microsoft. I just got started to play with NNI and I have liked it so far. Here I want to share how I use NNI to search for optimal hyper-parameters and architectures.

We will build a neural network to approximate math functions, as an example. Neural networks with one or more hidden layers are known as function approximators for continuous functions [1], assuming certain non-linearity conditions of activation functions are met. Shallow networks such as the ones with a single hidden layer are not as efficient as width-bound deep networks (for example [2]). Michael Nielsen has given a nice, accessible explanation of how neural nets can approximate functions.

Simple Nonlinear Function

We begin with a simple nonlinear target function y = x². We use a fully connected network with a few hidden layers to learn this function within the range [0, 1].

Below is code to implement this function as a PyTorch dataset. Input data and ground-truth labels are represented by tensors of shape (1,1), where the two components are channels and data dimension, respectively. They are both 1 since x and y hold only scalar values. If you have experience in image-based neural networks, you can think of the data as a single-channel, single-pixel image.

function-approximation automl artificial-intelligence nas neural-networks

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

All About Artificial Neural Networks – What, Why, and How?

Artificial Neural Network is a learning system that seeks to emulate the human brain. Here is an overview of ANN, critical to the discipline of Artificial Intelligence.

Activation Functions in Neural Network

A concise analysis on roles and types of activation functions in neural networks. Detailed pros and cons of 5 popularly used ones in Artificial Neural Network.

Introduction to Artificial Neural Networks for Beginners

Introduction to Artificial Neural Networks for Beginners. Understanding the concepts of Neural Networks.

Introduction to Neural Networks

There has been hype about artificial intelligence, machine learning, and neural networks for quite a while now. This will not be a math-heavy introduction because I just want to build the idea here.

Types Of Activation Functions used in Neural Network?

Discuss various types of activation functions and what are the types of problems one might encounter while using each of them.