The biggest hurdle to clear when developing machine learning solutions has always been the data. Large-scale, clean, fully-annotated data sets like ImageNet and COCO are not easy to come by, especially for niche tasks. This is especially true of deep learning and will become more true as deep learning is increasingly adopted. To overcome the labeled data bottleneck, researchers and developers are developing a variety of techniques like transfer learning, domain adaptation, synthetic data generation, and many semi- and self-supervised techniques.

The world of semi- and self-supervised techniques is a particularly fascinating field as it seems almost like magic. How can we use the world’s seemingly endless supply of unlabeled data to help us solve supervised learning problems? It turns out that these techniques are more accessible than you might think and you can start applying them right away.

Self-Supervised Learning

Before we dive in, let’s first define what these terms mean. Self-supervised learning in essence is the practice of extracting supervisory information from completely unlabeled data to create a supervised learning task. Basically we create an “artificial” supervised learning task which has the following properties:

• It encourages the network to learn semantically useful information about the data.
• Its labels can be derived from data augmentation.

One of the simplest, but still very effective techniques for learning from unlabeled data comes from a paper called “Unsupervised Representation Learning by Predicting Image Rotations” by Gidaris et al. This auxiliary task is very intuitive: given an input image, randomly rotate it by either 0, 90, 180, or 270 degrees. Then train your model to predict a label of “0”, “90”, “180”, or “270.”

#google-colab #pytorch #machine-learning #deep-learning