Keeping up with PyTorch Lightning and Hydra

Keeping up with PyTorch Lightning and Hydra

How I shrank my training script by 50% using the new features from PyTorch lightning 0.9 and Hydra 1.0.0rc4

Introduction

Last week, PyTorch Lightning 0.9.0 and Hydra’s fourth release candidate for 1.0.0 were released with a choke-full of new features and mostly final APIs. I thought it’d be a good time for me to revisit my side project Leela Zero PyTorch to see how these new versions can be integrated into it. In this post, I’ll talk about some of the new features of the two libraries, and how they helped Leela Zero PyTorch. I’m not going to talk about the details about Leela Zero PyTorch all too much here, so if you want to read more about my side project for more context, you can read my previous blog post about it here.

PyTorch Lightning 0.9.0

This is a major milestone for the PyTorch Lightning team as they diligently work toward the 1.0.0 release. It introduces a number of new features and an API that is ever closer to the final one. Before we jump in, if you want to read more about this release, check out the official blog post. If you want to learn more about PyTorch Lightning in general, check out the Github page as well as the official documentation.

Result

Have you found yourself repetitively implementing *_epoch_end methods just so that you can aggregate results from your *_step methods? Have you found yourself getting tripped on how to properly log the metrics calculated in your *_step and *_epoch_end methods? You’re not alone, and PyTorch Lightning 0.9.0 has introduced a new abstraction called Result to solve these very problems.

There are two types of ResultTrainResult and EvalResult. As the names suggest, TrainResult is used for training and EvalResult is used for validation and testing. Their interfaces are simple: you specify the main metrics to act on during instantiation (for TrainResult, the metrics to minimize, for EvalResult, metrics to checkpoint or early stop on), then you specify additional metrics to log. Let’s take a look at how they’re used in my project:

machine-learning neural-networks deep-learning artificial-intelligence pytorch

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Artificial Intelligence, Machine Learning, Deep Learning 

Artificial Intelligence (AI) will and is currently taking over an important role in our lives — not necessarily through intelligent robots.

AI(Artificial Intelligence): The Business Benefits of Machine Learning

Enroll now at CETPA, the best Institute in India for Artificial Intelligence Online Training Course and Certification for students & working professionals & avail 50% instant discount.

Deep Learning 101 —  Neural Networks Explained

The past few decades have witnessed a massive boom in the penetration as well as the power of computation, and amidst this information.

Deep Learning Explained in Layman's Terms

Deep Learning Explained in Layman's Terms. In this post, you will get to learn deep learning through a simple explanation (layman terms) and examples.

Alibaba’s Mobile Neural Network: A deep learning framework for mobile and embedded devices

A peek at Alibaba’s Mobile Neural Network (MNN) and how it achieves balance between high performance, flexibility, and ease-of-use.