In my last post, we took a look at Snapchat’s new framework- SnapML that brings the power of Machine Learning to its AR Lens Creating platform- Lens Studio. I had covered the various available templates and what they had to offer. In this tutorial, we are going to use one of these templates and create a Snapchat filter using SnapML from start to finish
SnapML was introduced in the latest version of Snapchat’s Lens Creation tool Lens Studio (3.0). It allows Lens creators to combine their own custom machine learning models with the powerful augmented reality tools inside Lens Studio.
SnapML enables this via pre-configured templates that could be used as starting points for Lens development. Developers and engineers can use these templates to easily incorporate their own custom models inside Lens Studio.
Object detection is a popular machine learning task that has several applications in daily life. It involves identifying instance of certain objects in an image or video frame, and locating them by drawing bounding boxes around them. This is commonly used for tasks like face detection or pedestrian density and movement detection.
Using the Object Detection SnapML Template, we’re going to create a Lens that detects healthy and unhealthy food items and draws a bounding box around them, along with a thumbs up or down sign.
Our final effect will look somewhat like this:
So let’s get started by taking a look at the Object Detection Template inside Lens Studio.
Adding custom ML models to Lenses opens a whole new world of immersion and interactivity. But building those ML models is still tricky business. That’s why we’re building an integration between our no-code model-building platform and SnapML. Learn more and request early access.
The Object Detection Template
Default recommended templates in Lens Studio
When you open the template, you’ll see that it’s actually a complete project that you can publish directly or build on top of, according to your needs. This template comes with a pre-trained ML model that detects a variety of vehicles (cars, buses, and trucks) and draws bounding boxes around them on the screen. It also includes a counter that shows the number of objects detected in the current camera frame.
Now, since we’re making a Lens that detects healthy and unhealthy food items, this model that detects vehicles is not really useful to us. So we’ll need to train our own custom model.
To do this, we could use any machine learning framework to train an object detection model and export it as a .pb or .onnx file. This could take a lot of time if you aren’t already an ML expert. Luckily, if we want to avoid building a model from scratch, Snapchat has provided us with easy-to-use ML templates in the form of Jupyter Notebooks.
You can use these templates directly in Google Colab, tweak and fine-tune some of the values, and start training your model on a custom dataset very quickly.
For our effect, we’re going to use the Object Detection Template to train 2 different models. One that detects the healthy food items, and another for unhealthy items.
To do this, use the healthy and unhealthy food labels from the COCO dataset.
#heartbeat #lens-studio #machine-learning #snapchat #machine learning
