Arvel  Miller

Arvel Miller


Automate Your ML Model Retraining with Kubeflow

This is the first part of a 2 parts series where I explain how you can build a cost-efficient and automated ML retraining system with Kubeflow. Along the way, we’ll also pick some best practices around building pipelines.

While Kubeflow Pipelines isn’t yet the most popular batch jobs orchestrator, a growing number of companies is adopting it to handle their data and ML jobs orchestration and monitoring. Actually, Kubeflow is designed to benefit from Kubernetes strengths and that’s what makes it very attractive.

In this article, I’ll show you how you can build an automated and cost-efficient ML model retraining pipeline using Kubeflow Pipelines. As you might already know, retraining ML models is necessary to keep them accurate and cure the model drift curse. In case you are familiar with Airflow or planning to use it, make sure to have a look at this article where I demo how to build the exact retraining system using Airflow.

Now, let’s say you have created a nice ML model to predict the taxi fare of a car drive and serve a first version of the model. The retraining system you’ll be building is made of 2 pipelines:

  1. The first pipeline taxi-fare-predictor trains the ML model and serves it for prediction if the model outperforms the live version of the model.
  2. The second pipeline retraining-checker runs every now and then and checks if the model has become obsolete. If so, it triggers the first pipeline.

#tensorflow #machine-learning #kubeflow #kubernetes #developer

Automate Your ML Model Retraining with Kubeflow