Virgil  Hagenes

Virgil Hagenes

1601539200

From Research to Production with Deep Semi-Supervised Learning

The success of most deep learning algorithms today is largely the result of decades of research, the growing availability of GPUs, and data. But not just any kind of data — the kind that is abundant, clean, and labeled.

Datasets like ImageNet, CIFAR10, SVHN, and others, have allowed researchers and practitioners to make remarkable progress on computer vision tasks and were immensely useful for our own experimentation. Yet the elephant in the room for many applications that seek to benefit from this progress, such as medicine, is precisely the fact that the data must be abundant, clean, and labeled.

**Semi-supervised learning (SSL), **a subfield that combines both supervised and unsupervised learning, has grown in popularity in the deep learning research community over the past few years. It’s very possible that, at least in the short-term, SSL approaches could be the bridge between label-heavy supervised learning and a future of data-efficient modeling.

In this post, we talk about when you should consider using SSL approaches in your production environments and the lessons we’ve learned using them to improve our object detection models at Uizard. Of course, we’ll do our best to share the big picture but keep some details of the wizardry to ourselves.

Our hope is that by displaying how and when SSL worked and didn’t work for us and by sharing tips learned on our journey from research to production, we can inspire you to take a chance on SSL for your work and unlock the potential of your unlabeled data.

In short, here are a few lessons we emphasize:

  • Simplicity is king. The most successful approaches in SSL that translated from research to production were those that were the simplest to reproduce. Specifically, we’ll elaborate on how “Self-Training with Noisy Student” (Xie et al., 2019) worked for us.
  • Pseudo-label refinement with heuristics can be extremely effective. Pseudo-labeling is a popular component of SSL approaches — we find that using simple heuristics to refine our pseudo-labels in the unlabeled data improves performance across different sizes of unlabeled datasets.
  • **Progress in semi-supervised image classification is difficult to translate to object detection. **Much of the progress in SSL that we followed measured performance on image classification with promises of similar improvements on object detection, but we found it difficult to adapt them appropriately in practice. As a result, more work and research is needed in the semi-supervised object detection space.

#production #semi-supervised-learning #machine-learning #research #deep-learning

What is GEEK

Buddha Community

From Research to Production with Deep Semi-Supervised Learning
Virgil  Hagenes

Virgil Hagenes

1601539200

From Research to Production with Deep Semi-Supervised Learning

The success of most deep learning algorithms today is largely the result of decades of research, the growing availability of GPUs, and data. But not just any kind of data — the kind that is abundant, clean, and labeled.

Datasets like ImageNet, CIFAR10, SVHN, and others, have allowed researchers and practitioners to make remarkable progress on computer vision tasks and were immensely useful for our own experimentation. Yet the elephant in the room for many applications that seek to benefit from this progress, such as medicine, is precisely the fact that the data must be abundant, clean, and labeled.

**Semi-supervised learning (SSL), **a subfield that combines both supervised and unsupervised learning, has grown in popularity in the deep learning research community over the past few years. It’s very possible that, at least in the short-term, SSL approaches could be the bridge between label-heavy supervised learning and a future of data-efficient modeling.

In this post, we talk about when you should consider using SSL approaches in your production environments and the lessons we’ve learned using them to improve our object detection models at Uizard. Of course, we’ll do our best to share the big picture but keep some details of the wizardry to ourselves.

Our hope is that by displaying how and when SSL worked and didn’t work for us and by sharing tips learned on our journey from research to production, we can inspire you to take a chance on SSL for your work and unlock the potential of your unlabeled data.

In short, here are a few lessons we emphasize:

  • Simplicity is king. The most successful approaches in SSL that translated from research to production were those that were the simplest to reproduce. Specifically, we’ll elaborate on how “Self-Training with Noisy Student” (Xie et al., 2019) worked for us.
  • Pseudo-label refinement with heuristics can be extremely effective. Pseudo-labeling is a popular component of SSL approaches — we find that using simple heuristics to refine our pseudo-labels in the unlabeled data improves performance across different sizes of unlabeled datasets.
  • **Progress in semi-supervised image classification is difficult to translate to object detection. **Much of the progress in SSL that we followed measured performance on image classification with promises of similar improvements on object detection, but we found it difficult to adapt them appropriately in practice. As a result, more work and research is needed in the semi-supervised object detection space.

#production #semi-supervised-learning #machine-learning #research #deep-learning

Marget D

Marget D

1618317562

Top Deep Learning Development Services | Hire Deep Learning Developer

View more: https://www.inexture.com/services/deep-learning-development/

We at Inexture, strategically work on every project we are associated with. We propose a robust set of AI, ML, and DL consulting services. Our virtuoso team of data scientists and developers meticulously work on every project and add a personalized touch to it. Because we keep our clientele aware of everything being done associated with their project so there’s a sense of transparency being maintained. Leverage our services for your next AI project for end-to-end optimum services.

#deep learning development #deep learning framework #deep learning expert #deep learning ai #deep learning services

Mikel  Okuneva

Mikel Okuneva

1603735200

Top 10 Deep Learning Sessions To Look Forward To At DVDC 2020

The Deep Learning DevCon 2020, DLDC 2020, has exciting talks and sessions around the latest developments in the field of deep learning, that will not only be interesting for professionals of this field but also for the enthusiasts who are willing to make a career in the field of deep learning. The two-day conference scheduled for 29th and 30th October will host paper presentations, tech talks, workshops that will uncover some interesting developments as well as the latest research and advancement of this area. Further to this, with deep learning gaining massive traction, this conference will highlight some fascinating use cases across the world.

Here are ten interesting talks and sessions of DLDC 2020 that one should definitely attend:

Also Read: Why Deep Learning DevCon Comes At The Right Time


Adversarial Robustness in Deep Learning

By Dipanjan Sarkar

**About: **Adversarial Robustness in Deep Learning is a session presented by Dipanjan Sarkar, a Data Science Lead at Applied Materials, as well as a Google Developer Expert in Machine Learning. In this session, he will focus on the adversarial robustness in the field of deep learning, where he talks about its importance, different types of adversarial attacks, and will showcase some ways to train the neural networks with adversarial realisation. Considering abstract deep learning has brought us tremendous achievements in the fields of computer vision and natural language processing, this talk will be really interesting for people working in this area. With this session, the attendees will have a comprehensive understanding of adversarial perturbations in the field of deep learning and ways to deal with them with common recipes.

Read an interview with Dipanjan Sarkar.

Imbalance Handling with Combination of Deep Variational Autoencoder and NEATER

By Divye Singh

**About: **Imbalance Handling with Combination of Deep Variational Autoencoder and NEATER is a paper presentation by Divye Singh, who has a masters in technology degree in Mathematical Modeling and Simulation and has the interest to research in the field of artificial intelligence, learning-based systems, machine learning, etc. In this paper presentation, he will talk about the common problem of class imbalance in medical diagnosis and anomaly detection, and how the problem can be solved with a deep learning framework. The talk focuses on the paper, where he has proposed a synergistic over-sampling method generating informative synthetic minority class data by filtering the noise from the over-sampled examples. Further, he will also showcase the experimental results on several real-life imbalanced datasets to prove the effectiveness of the proposed method for binary classification problems.

Default Rate Prediction Models for Self-Employment in Korea using Ridge, Random Forest & Deep Neural Network

By Dongsuk Hong

About: This is a paper presentation given by Dongsuk Hong, who is a PhD in Computer Science, and works in the big data centre of Korea Credit Information Services. This talk will introduce the attendees with machine learning and deep learning models for predicting self-employment default rates using credit information. He will talk about the study, where the DNN model is implemented for two purposes — a sub-model for the selection of credit information variables; and works for cascading to the final model that predicts default rates. Hong’s main research area is data analysis of credit information, where she is particularly interested in evaluating the performance of prediction models based on machine learning and deep learning. This talk will be interesting for the deep learning practitioners who are willing to make a career in this field.


#opinions #attend dldc 2020 #deep learning #deep learning sessions #deep learning talks #dldc 2020 #top deep learning sessions at dldc 2020 #top deep learning talks at dldc 2020

Tia  Gottlieb

Tia Gottlieb

1595356740

Converting deep learning research papers to useful code

If deep learning is a super power, then turning theories from a paper to usable code is a hyper power

Image for post

Why should I learn to implement machine learning research papers?

As I’ve said, being able to convert a paper to code is definitely a hyper power, especially in a field like machine learning which is moving faster and faster each day.

Most research papers come from people within giant tech companies or universities who may be PhD holders or the ones who are working on the cutting edge technologies.

What else can be more cool than being able to reproduce the research done by these top notch professionals. Another thing to note is that the ones who can reproduce research papers as code is in huge demand.

Once you get the knack of implementing research papers, you will be in a state on par with these researchers.

These researchers too has acquired these skills through the practice of reading and implementing research papers.

How do I read and implement papers?

You might say, “Hm, I have a general understanding of the deep learning algorithms like fully connected networks, convolutional neural networks, recurrent neural networks, but the problem is that I would like to develop SOTA(state of the art) voice cloning AI but I know nothing about voice cloning :( ”.

Okay, here is your answer(some parts of my method is taken from Andrew Ng’s advice on reading papers).

If you want to learn about a specific topic:

  1. Collect 5–6 papers related to the specific topic(you can simply search arxiv or similar websites to get papers related to a topic).
  2. Don’t read a single paper completely, instead skim through all of the papers and pick a paper that interests you or if you had a specific paper in mind, go pick it up, no one can stop you.
  3. Read the abstract carefully and understand the idea from a high level and see whether your interest still persists, if so continue to skim through the images and see whether you can make assumptions on what the paper might be about.
  4. Now read the introduction carefully line by line because most of what the paper contains will be explained here in the most simplest manner with minimal math.
  5. If you wish, you can skip the math equations in the first pass, don’t skip the math if the Greek letters are familiar.
  6. At any situation, if you get stuck or some words are confusing, never hesitate to google it. No one is born as master of everything ;)
  7. After completing the first pass, you will be in a state where you understand the high level view of what the paper is trying prove or improve.
  8. In the second pass, try to understand almost everything in the paper and if you encounter any pseudo-code, try to convert it into your python library of choice(PyTorch, TensorFlow…)
  9. You can get more papers to read and get a better understanding of the field by going to the references section of each paper(same as connecting the dots).

💡 Some tips for effectively understanding a paper:

  • If you are a beginner to reading research papers, it’s good to read some blog posts and videos related to that topic/research paper before reading the paper itself. This makes your job easier and you will not be discouraged by all that Greek letters.
  • Always take notes and highlight important points in the research paper for easier reference while implementing the code for the paper.
  • If you are new to implementing research papers and get stuck any where, it’s not a bad idea to go through open source implementations and see how others have done this.

#deep-learning #research #unsupervised-learning #machine-learning #deep learning

Snorkel: Build ML Models without Labeled Data

With bunches of hands-on tools, building models on labeled data has already been an easy task for data scientists. However, in the real world, many tasks are not well-formatted supervised learning problems: labeled data may be expensive or even impossible to obtain. An alternative approach is to leverage cheap and low-quality data to achieve supervision, which is the topic of this article: weak supervision

In the following sections, I will go through the concepts of weak supervision. I will also introduce a tool called Snorkel, which is developed by Stanford. Finally, I will show you how HK01 uses Snorkel to capture the trend topics on Facebook, and therefore enhance our recommender engine.


There are several paradigms of algorithm to remedy the situation if a large amount of high-quality, hand-labeled training data is not available. As you can see in the following diagram, if you don’t have enough data, you have to find another source of knowledge to achieve a comparable level of supervision to traditional supervision.

source: http://ai.stanford.edu/blog/weak-supervision/

Choosing one among these paradigms is pretty tricky. It depends on what you have on your hands. Transfer learning is great for tasks with a well-trained model in similar tasks, like fine-tuning ImageNet model with your own categories; while you may have some presumptions on the topological structure, such as the shape of clusters, you may prefer semi-supervised learning.

So, what kind of situation is the best suit for weak supervision?

You may have some ideas after reading the definition of weak supervision. Yes, if you have plenty of domain experts but lack of data, weak supervision is your pick.

The reason behind is revealed in the definition: weak supervision enables learning from low-quality and noisy labels. In other words, you can still find patterns, just like what supervised learning do, unless you should supplement multiple noisy labels for each training sample so that the model can generalize knowledge from them.


weak supervision enables supervision by multiple noisy labels

The rationale of weak supervision relies on the fact that noisy data is usually much easier and cheaper to obtain than high-quality data. Imagine you are working for an insurance company and your boss asks for a recommender engine of a whole-new product line which, of course, you don’t have data. With sales experts, we can set up some rules which are “mostly correct” like the new product is more attractive to the elderly. These rules are not perfectly correct; but, they are good enough to provide your models collective intelligence. And, most importantly, these rules are easier to obtain than perfectly hand-labeled data.

So, the next question is: **how can we inject these rules into our ML models? **The answer is Snorkel.


Snorkel is a system developed by Stanford which allows you to program the rules into ML models. The key idea of Snorkel is to build the generative model which represents the causal relationship between the true label and the noisy labels.

The left-hand side of the above diagram is the probabilistic model representing the generative process from the true label to the noisy labels. Although the true label is unobservable, we can still learn the accuracies and correlations by the agreements and disagreements from different noisy labels. Hence, we can estimate the P(L|y) of each noisy label, which is essentially an indicator of quality. By aggregating the noisy labels, we get the estimated true label and use it to train our model.

In Snorkel, noisy labels are programmed as labeling functions. A label function is basically a python function which hard-codes a rule to determine the label. For example, if you’re writing a program to determine which an email is spam, the program should be something like:

from snorkel.labeling import labeling_function

SPAM = 1
NORMAL = 0
ABSTAIN = -1
@labeling_function()
def contain_hyperlink(x):
    if 'http' in x:
        return SPAM 
    else:
        return NORMAL
@labeling_function()
def contain_foul_language(x):
    for each in x:
        if each in foul_language:
            return SPAM
        else:
            return NORMAL

In this toy example, you can see the basic elements of Snorkel.

  • define the labels. In this example, the labels are SPAM, NORMAL & ABSTAIN. ABSTAIN is the label used when you cannot determine the label.
  • define labeling functions. Add a decorator @labeling_function() to declare.

#machine-learning #deep-learning #transfer-learning #semi-supervised-learning #weak-supervision #deep learning