What's new in PyTorch 1.3?

What's new in PyTorch 1.3?

Facebook just released PyTorch v1.3 and it is packed with some of the most awaited features. You can find it in this article

Support for Android and iOS, Named Tensor, TPU Support, Quantization and more. Facebook just released PyTorch v1.3 and it is packed with some of the most awaited features. The three most attractive ones are:

  1. Named Tensor — Something that would make the life of machine learning practitioners much easier.
  2. Quantization — For performance critical systems like IoT devices and embedded systems.
  3. Mobile Support — For both Android and iOS devices.

I would be talking in brief about all of these and will link to some other important features.

Named Tensors

PyTorch v1.3 finally added the support for named tensors which allows users to access tensor dimensions using explicitly associated names rather than remembering the dimension number. For example, up until now in computer vision related tasks, we had to remember the general structure of a batch as follows — [N, C, H, W]. Where N is batch size, C is number of channels, H and W are height and width of images respectively. We had to keep track of such a structure while performing operations on this batch but now we can just use the dimension name rather keeping track of its index. Additionally, these named representations can provide augmented runtime error checks. I will discuss them further in this article.

import torch
batch = torch.zeros(64, 3, 100, 100, names=('N', 'C', 'H', 'W'))
print(batch.names) 

batch.namesprints the name of each dimension on tensor batch in order.

Alignment by name

Use [align_as()](https://pytorch.org/docs/master/named_tensor.html#torch.Tensor.align_as) or [align_to()](https://pytorch.org/docs/master/named_tensor.html#torch.Tensor.align_to) to align tensor dimensions by name to a specified ordering.

In computer vision models often the representation of a batch needs to change in between [N, C, H, W] (for forward and backward pass of model) and [N, H, W, C] (for plotting and saving images). Until now this had to be done counter intuitively as batch.permute([0, 2, 3, 1]) but now it can be done in a much easier way by using the [align_as()](https://pytorch.org/docs/master/named_tensor.html#torch.Tensor.align_as) or [align_to()](https://pytorch.org/docs/master/named_tensor.html#torch.Tensor.align_to)operator.

import torch

batch = torch.zeros(64, 3, 100, 100, names=('N', 'C', 'H', 'W'))
print(batch.shape) #torch.Size([64, 3, 100, 100])
batch = batch.align_to('N', 'H', 'W', 'C')
print(batch.shape) #torch.Size([64, 100, 100, 3])

For a large number of dimensions, normal permute operator needs an explicit list of all dimensions even for exchanging the positions of just two dimensions. However, in the case of named tensors, the permutation or reordering of the dimensions can be done in an easier and simpler way as follows:

#####################
# Before PyTorch v1.3
#####################
import torch

batch = torch.zeros(2, 3, 2, 2, 2, 2, 2, 2, 2, 2)
print(batch.shape)
batch = batch.permute([0, 2, 1, 3, 4, 5, 6, 7, 8, 9])
print(batch.shape)

#####################
# After PyTorch v1.3
#####################
import torch
alphas = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j']
batch = torch.zeros(2, 3, 2, 2, 2, 2, 2, 2, 2, 2, names=alphas)
print(batch.shape)
batch = batch.align_to('a', 'c', 'b', ...)
print(batch.shape)

Check names

Apart from making tensors more intuitive, named tensors also provide additional error checks. An operator, when applied on named tensors (in case of binary operator any one or both named tensors) will implicitly check that certain dimension names match at the run time. This provides extra security against errors. This is shown in the example below:

import torch

# The following will produce no error as the dimensions match
batch1= torch.zeros(64, 3, 100, 100, names=('N', 'C', 'H', 'W'))
batch2 = torch.zeros(64, 3, 100, 100, names=('N', 'C', 'H', 'W'))
batch3 = batch1 + batch2

# The following will produce an error as the dimensions don't match
batch1= torch.zeros(64, 3, 100, 100, names=('N', 'C', 'H', 'W'))
batch2 = torch.zeros(64, 3, 100, 100, names=('N', 'C', 'W', 'H'))
batch3 = batch1 + batch2

In the above example, if not for named tensors, batch1 and batch2 can be added without any error because height = width = 100. However, batch2 has the height and width dimension interchanged and adding it to batch1 might not be the intended operation. Thanks to named tensors, this logical error will be caught by name checking as (’N’, ‘C’, ‘H’, ‘W’) and (’N’, ‘C’, ‘W’, ‘H’) are not the same.

When are the names matched?

The rules are pretty much like the broadcasting rules for the dimensions in numpy or PyTorch. Quoting the official PyTorch docs:

Two names match if they are equal (string equality) or if at least one is _None__. Nones are essentially a special “wildcard” name._”

This is showcased in the example below:

import torch

batch1= torch.zeros(64, 3, 100, 100, names=('N', 'C', 'H', 'W'))
batch2 = torch.zeros(64, 3, 100, 100, names=('N', 'C', 'H', 'W'))
batch3 = torch.zeros(64, 3, 100, 100)
batch4 = torch.zeros(64, 3, 100, 100)
batch5 = torch.zeros(64, 3, 100, 100, names=('N', 'C', 'W', 'H'))

#Name tuple strings are equal so name matches
res1 = batch1 + batch2 
#Name of one of the inputs (batch3) is None so matches 
res2 = batch1 + batch3 
#Name of both the inputs are None so matches
res2 = batch3 + batch4 
#Name tuple strings are not equal so no name match and error
res3 = batch1 + batch5 

Name propagation

After performing operations on tensors you don’t need to enter the dimension names again, they will automatically be propagated. PyTorch makes use of two operators — match and unify for name propagation.

  • match is the same operator as defined above, it checks whether the two named tensors can be matched or not.
  • unify is an operator used to determine which of the two input tensor’s name shall be propagated the resulting tensor. Quoting the official PyTorch docs:

_unify(A, B)_ determines which of the names _A_ and _B_ to propagate to the outputs. It returns the more specific of the two names, if they match. If the names do not match, then it errors.

Name propagation is showcased in an example below:

  • Unary operator:
    import torch
    

Unary operator on named tensors

t = torch.randn(4, 2, names=('N', 'C')) t = t.abs() t.names #output: ('N', 'C')


*   Binary operator:

import torch

Binary operator on named tensors

t1 = torch.randn(4, names=('X')) t2 = torch.randn(4) t3 = t1 * t2 t3.names #output: ('X',)


## _Limitations_

At the time of writing this article, the named tensor functionality is in experimental mode and maybe subjected to many changes. However, one of the biggest current limitations of named tensors is that they don’t fully support the [Autograd](/pytorch-autograd-understanding-the-heart-of-pytorchs-magic-2686cd94ec95) engine. Although the gradient computations for named tensors is perfectly same but the autograd engine completely ignores the name and along with it, the additional security they provide.

## **Quantization**

PyTorch 1.3 now supports Quantization of tensors. This is something Tensorflow already supports and was much awaited in PyTorch. Quantization is a fairly simple yet elaborate concept. I will try to write about them in brief with a high level of abstraction here by answering the three questions — What, Why and How.

**What is Quantization?**

Quantization is the technique of performing operations in a low-precision format or converting high precision data formats to low precision data formats. This is done for example by treating a 32-bit floating point format as an 8-bit fixed point format. If interested, you might read about the [fixed-point](https://en.wikipedia.org/wiki/Fixed-point_arithmetic) and [floating-point arithmetic](https://en.wikipedia.org/wiki/Floating-point_arithmetic) and their related complexities to better understand the need of Quantization.

**Why** **Quantization?**

The whole purpose of pursuing research and creating neural network models is to deploy them and make them available for the public good. While the model training need only grows proportionally to the number of researchers and machine learning practitioners, the need for mode inference grows proportional to the number of consumers. To allow more and better access to the end users, the representation of the models deployed for inference needs to be much more compact than their representation while training. Another thing to keep in mind is that backpropagation needs high precision representations for the model weights and biases. However, during inference, the models are much more robust and do not require high precision representation. Thus, [a model of size 113.9MB in 32 bit floating point representation can be quantized as int8 to be of size 76.8MB.](https://pytorch.org/tutorials/advanced/dynamic_quantization_tutorial.html#test-dynamic-quantization)

**How to use Quantization in PyTorch v1.3?**

Quoting the official PyTorch documentation:

> PyTorch supports INT8 quantization compared to typical FP32 models allowing for a 4x reduction in the model size and a 4x reduction in memory bandwidth requirements. Hardware support for INT8 computations is typically 2 to 4 times faster compared to FP32 compute.

For Quantization, PyTorch introduced three new data types as follows:

*   `torch.quint8` — 8-bit unsigned integer.
*   `torch.qint8` — 8-bit signed integer.
*   `torch.qint32` — 32-bit signed integer.

PyTorch now offers three kinds of quantization methods for models:

1.  **Post Training Dynamic Quantization** — Quantize weights ahead of time but dynamically quantize the network activations at runtime. It is done as follows: `torch.quantization.quantize_dynamic(model, dtype=torch.qint8)`
2.  **Post Training Static Quantization** — Quantize both weights and activation bias and scale factors ahead of time. This is done by calling a calibration function. For more details refer the original [documentation.](https://pytorch.org/docs/master/quantization.html#torch.quantization.quantize_dynamic)
3.  **Quantization Aware Training** — In this model is trained on FP32 representation but is then quantized to a lower precision precision representation like INT8\. This is used in the rare cases where normal quantization can not provide accurate results. Thus, we get down to training the model with quantized values.

Another thing to notice is that PyTorch support Quantization from ground up. This means, we can also quantize tensors with the following equation (very intuitive) and code:

![ What's new in PyTorch 1.3](https://miro.medium.com/max/887/1*5lAWsBs3S4_KSKrN7XaiIg.png " What's new in PyTorch 1.3")
<figcaption class="au ei nu nv hv dn dl dm nw nx ap cu">This equation is used for the int representation of the quantized tensor that can be accessed using `t.int_repr()`</figcaption>

import torch

t = torch.tensor([1.111111111]) t_q = torch.quantize_per_tensor(t, 0.1, 10, torch.quint8) #output: tensor([21], dtype=torch.uint8) print(t_q.int_repr()) #output: tensor([1.1000], size=(1,), dtype=torch.quint8, quantization_scheme=torch.per_tensor_affine, scale=0.1, zero_point=10) print(t_q)


[Here](https://pytorch.org/docs/master/quantization.html#quantized-torch-tensor-operations) is the list of all supported operations on quantized tensors.

## Mobile Support

The natural purpose of allowing quantization is shipping PyTorch models on performance critical mobile phones (and other devices). They have implemented an end-to-end PyTorch API for both Android and iOS. This will have a great impact on reducing inference latency and user privacy. However, PyTorch mobile is currently in an early experimental stage where it has several limitations. For example, the current version only supports forward propagation (inference) and no backward operations are supported.

You can access the hello world tutorials for both [iOS](https://pytorch.org/mobile/ios/) and [Android](https://pytorch.org/mobile/android/) on the official PyTorch website.

## More updates

Apart from these three major updates, PyTorch v1.3 has implemented several other changes and bug fixes. You can view the list of all the changes on the official PyTorch [Github repository](https://github.com/pytorch/pytorch/releases). Some of these features are:

*   TPU support for PyTorch on Google Cloud. Here’s a [Github repo](https://github.com/pytorch/xla/tree/master/contrib/colab) showing how to use TPUs with PyTorch.
*   Extended support for TensorBoard: 3D Mesh and Hyperparameter
*   Major updates to the torchscript (mostly for mobile phones)
*   Performance improvements in `torch.nn`, `torch.nn.functional`, Autograd engine and more.

Additionally, if you want to port your code from a previous version of PyTorch to PyTorch v1.3, you need to take of care features which might cause errors or unintended behavior. Some of these features are (I am directly quoting these changes from the previously mentioned release notes):

1.  Data Type promotion: For example, `torch.tensor(5) + 1.5` outputs a tensor with value 6.5\. In earlier versions, the output would’ve been 6.
2.  Data Type promotion for in-place operators: For example, consider the following code: `a = torch.tensor(0); a.add_(0.5)` in earlier versions, this would’ve given a tensor with value 1 as the output. But as of now, PyTorch no more support in-place operations with a lower datatype hence the above piece of code outputs an error.
3.  torch.flatten: Output of `torch.flatten(torch.tensor(0))` is `tensor([0])` compared to `tensor(0)` earlier. Rather than returning a 0D tensor, it now returns a 1D tensor.
4.  `nn.functional.affine_grid`: when align_corners = True, changed the behavior of 2D affine transforms on 1D data and 3D affine transforms on 2D data (i.e., when one of the spatial dimensions has unit size). Previously, all grid points along a unit dimension were considered arbitrarily to be at -1, now they are considered to be at 0 (the center of the input image).
5.  torch.gels: removed deprecated operator, use `torch.lstsq` instead.
6.  `utils.data.DataLoader` made a number of Iterator attributes private (e.g. num_workers, pin_memory).
7.  [Additional changes in pyTorch for C++](https://github.com/pytorch/pytorch/releases)..

python tensorflow machine-learning deep-learning

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Keras with TensorFlow Course - Python Deep Learning and Neural Networks

Keras with TensorFlow Course - Python Deep Learning and Neural Networks for Beginners Tutorial: How to use Keras, a neural network API written in Python and integrated with TensorFlow. We will learn how to prepare and process data for artificial neural networks, build and train artificial neural networks from scratch, build and train convolutional neural networks (CNNs), implement fine-tuning and transfer learning, and more!

Deep Learning from Scratch and Using Tensorflow in Python

In this article, we will learn how deep learning works and get familiar with its terminology — such as backpropagation and batch size

PyTorch for Deep Learning | Data Science | Machine Learning | Python

PyTorch for Deep Learning | Data Science | Machine Learning | Python. PyTorch is a library in Python which provides tools to build deep learning models. What python does for programming PyTorch does for deep learning. Python is a very flexible language for programming and just like python, the PyTorch library provides flexible tools for deep learning.

Deep Learning Tutorial with Python | Machine Learning with Neural Networks

In this video, Deep Learning Tutorial with Python | Machine Learning with Neural Networks Explained, Frank Kane helps de-mystify the world of deep learning and artificial neural networks with Python!

Machine Learning, Data Science and Deep Learning with Python

Complete hands-on Machine Learning tutorial with Data Science, Tensorflow, Artificial Intelligence, and Neural Networks. Introducing Tensorflow, Using Tensorflow, Introducing Keras, Using Keras, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Learning Deep Learning, Machine Learning with Neural Networks, Deep Learning Tutorial with Python