Scaling Data Science and Machine Learning Infrastructure Like Netflix

Scaling Data Science and Machine Learning Infrastructure Like Netflix

Scaling Data Science and Machine Learning Infrastructure Like Netflix | Would you move your data science project from a laptop to the cloud? We have Savin Goyal from Netflix. Savin is the technical lead for machine learning infrastructure at Netflix. He joins us to talk about Metaflow, an open-source tool to simplify building, managing, and scaling data science projects.

Would you move your data science project from a laptop to the cloud? Would you also like to have snapshots of your project saved along the way so that you can go back in time or share the state of your project with another team member? This week on the show, we have Savin Goyal from Netflix. Savin is the technical lead for machine learning infrastructure at Netflix. He joins us to talk about Metaflow, an open-source tool to simplify building, managing, and scaling data science projects.

Metaflow addresses the needs of the numerous data scientists who work at Netflix. Machine learning is key strength for the streaming service. They tried several existing tools to scale their own internal infrastructure and after this experimentation developed Metaflow.

We talk about the history of the project and how someone could get started with the open-source version. Savin also contrasts the cost of infrastructure as compared to data scientists and the cost of their time.

Topics:

  • 00:00:00 -- Introduction
  • 00:01:53 -- What is Metaflow?
  • 00:04:15 -- Savin's background in data science and infrastructure
  • 00:06:06 -- Democratization of infrastructure and iteration of tools
  • 00:10:34 -- What information is saved about the infrastructure requirements for a project?
  • 00:17:17 -- How are the requirements annotated?
  • 00:18:39 -- Sponsor: Digital Ocean's App Platform
  • 00:19:15 -- How do project snapshots work?
  • 00:29:33 -- Cost of infrastructure vs data scientists
  • 00:32:28 -- Working with data at Netflix scale
  • 00:37:55 -- Video Course Spotlight
  • 00:39:06 -- Getting an organization to use new tools and then making open-source
  • 00:49:51 -- Documentation of Metaflow and getting started on solving infrastructure problems
  • 00:53:57 -- What made you interested in working on infrastructure tools?
  • 00:55:13 -- What is something you are excited about in the world of Python?
  • 00:56:18 -- What do you want to learn next?
  • 00:58:14 -- Thanks and goodbye

๐Ÿ‘‰ Links from the show: https://realpython.com/podcasts/rpp/61/

data-science machine-learning cloud netflix

What is Geek Coin

What is GeekCash, Geek Token

Best Visual Studio Code Themes of 2021

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

How To Build A Data Science Career In 2021

In Conversation With Dr Suman Sanyal, NIIT University,he shares his insights on how universities can contribute to this highly promising sector and what aspirants can do to build a successful data science career.

Hire Machine Learning Engineer | Offshore Machine Learning Experts

We are a Machine Learning Services provider offering custom AI solutions, Machine Learning as a service & deep learning solutions. Hire Machine Learning experts & build AI Chatbots, Neural networks, etc. 16+ yrs & 2500+ clients.

The Difference between Data Science, Machine Learning and Big Data!

Many professionals and 'Data' enthusiasts often ask, โ€œWhat's the difference between Data Science, Machine Learning and Big Data?โ€. Let's clear the air. If you are still wondering about it then this article is for you.

5 stages of learning Data Science

5 stages of learning Data Science and how to ace each of them

Applications of Data Science and Machine Learning in NETFLIX

Businesses are using Data science in exciting and creative ways. This blog talks about how Netflix is leveraging Data Science and Machine learning. Do you know yet? Let's find out the article now