HTML to Markdown
Just last month (June 2020) Snapchat released a major update for its Lens creation software — Lens Studio 3.0. Out of all the new things that came along with the release, one feature that stood apart was SnapML.
SnapML allows lens creators to use their own custom machine learning models inside filters. Snapchat filters already use ML for a variety of things like face tracking, surface tracking, segmentation, facial gestures, and more.
But with SnapML, developers can now go beyond those limits, using the breadth of their imaginations to combine custom ML models with Snapchat’s feature-rich augmented reality tools—and an audience of millions!
Lens Studio 3.0 introduces SnapML for adding custom neural networks directly to Snapchat
Including a conversation with Hart Woolery, CEO of 2020CV and a SnapML creator
Getting started with SnapML in Lens Studio
Snapchat has made a lot of initial efforts to make the experience of implementing ML models as smooth and straightforward as possible. To do this, they’ve provided several ready-to-use templates that could be turned into AR experiences in only a few minutes. Let’s explore these templates and learn how to use them to create filters.
Lens Studio Templates
There are 5 main Lens Studio templates provided by Snap that cover some of the most common mobile machine learning tasks. We’ll cover the ML side of things a bit later, but first, here’s a look at the template projects inside Lens Studio:
Style Transfer
Style transfer is used to modify the overall visual appearance of the camera texture. Essentially, this task allows us to recompose the content of one image in the style of another.
This could be used to create unique artistic styles or color filters. Imagine a selfie in the style of your favorite visual artists. This is similar to what we see in many image and video editing apps like Prisma or Looq.
This project template contains an example ML model with a style transfer visual effect. We’ll discuss in more detail below how to implement your own custom style transfer model.
Classification
This template uses the image classification ML technique, which is used to classify images into different labels. This task is also sometimes referred to as image labeling or image recognition.
This could be used to tell, for example, whether user is wearing a hat or not in a given image or video frame. However, to create this kind of model, you’ll need to train it on a large dataset. There’s a wide variety of image classification datasets out there, but if there’s no dataset available for your use case, you’ll need to collect and label one.
This template contains an example ML model that can tell if the user is wearing glasses or not, showing a particle effect if true. We’ll discuss in more detail below how to implement your own custom image classification model.
Object Detection
The object detection template could be used to identify and locate one or more target objects in a given image or video frame. Object detection is somewhat similar to image classification, but it’s important to quickly distinguish the two:
- Image classification assigns a label to an image. A picture of a dog receives the label “dog”. A picture of two dogs, still receives the label “dog”
- Object detection, on the other hand, draws what’s called a bounding box around each dog and labels the box “dog”. The model predicts where each object is and what label should be applied. In that way, object detection provides more information about an image than recognition.
In Lens Studio, you could use this template to identify the location of the ring finger in an image or video frame, and use that prediction to place an AR ring for virtual product try-ons.
Again, here you’ll need a large dataset of objects you want to detect—and their bounding box coordinates—to create a custom model for this effect. Here are some available datasets to consider working with.
This template contains an example ML model that recognizes cars from the COCO dataset and draws bounding boxes around them. We’ll discuss in more detail below how to implement your own custom object detection model.
Custom Segmentation
Segmentation is an ML technique used to differentiate between the specific parts or elements of an image. Specifically, it works at a pixel-level—the goal is to assign each pixel in an image to the object to which it belongs (i.e. class label).
Segmentation could be used to add effects to a given segmented region. Snap Lense already had built-in segmentation to separate the user from the background, but with custom ML models you extend this task to anything you can think of—for example, segmenting the sky and changing it to into a beautiful, sunset-like pink-orange color.
This template contains an example ML model that recognizes a pizza and adds a sizzling effect to it. We’ll discuss in more detail below how to implement your own custom segmentation model.
33
This project template contains an example ML model with a style transfer visual effect. We’ll discuss in more detail below how to implement your own custom style transfer model.
34
35
Classification
36
37
This template uses the image classification ML technique, which is used to classify images into different labels. This task is also sometimes referred to as image labeling or image recognition.
38
39
40
41
Image Source
42
43
This could be used to tell, for example, whether user is wearing a hat or not in a given image or video frame. However, to create this kind of model, you’ll need to train it on a large dataset. There’s a wide variety of image classification datasets out there, but if there’s no dataset available for your use case, you’ll need to collect and label one.
44
45
This template contains an example ML model that can tell if the user is wearing glasses or not, showing a particle effect if true. We’ll discuss in more detail below how to implement your own custom image classification model.
46
47
Object Detection
48
49
The object detection template could be used to identify and locate one or more target objects in a given image or video frame. Object detection is somewhat similar to image classification, but it’s important to quickly distinguish the two:
50
51
- Image classification assigns a label to an image. A picture of a dog receives the label “dog”. A picture of two dogs, still receives the label “dog”
52 - Object detection, on the other hand, draws what’s called a bounding box around each dog and labels the box “dog”. The model predicts where each object is and what label should be applied. In that way, object detection provides more information about an image than recognition.
53
54
55
56
In Lens Studio, you could use this template to identify the location of the ring finger in an image or video frame, and use that prediction to place an AR ring for virtual product try-ons.
57
58
Again, here you’ll need a large dataset of objects you want to detect—and their bounding box coordinates—to create a custom model for this effect. Here are some available datasets to consider working with.
59
60
This template contains an example ML model that recognizes cars from the COCO dataset and draws bounding boxes around them. We’ll discuss in more detail below how to implement your own custom object detection model.
61
62
Custom Segmentation
63
64
Segmentation is an ML technique used to differentiate between the specific parts or elements of an image. Specifically, it works at a pixel-level—the goal is to assign each pixel in an image to the object to which it belongs (i.e. class label).
65
66
67
68
Segmentation could be used to add effects to a given segmented region. Snap Lense already had built-in segmentation to separate the user from the background, but with custom ML models you extend this task to anything you can think of—for example, segmenting the sky and changing it to into a beautiful, sunset-like pink-orange color.
69
70
This template contains an example ML model that recognizes a pizza and adds a sizzling effect to it. We’ll discuss in more detail below how to implement your own custom segmentation model.
#heartbeat #mobile-app-development #machine-learning #snapchat #lens-studio
