How to use Docker Desktop for Mac or Windows to run Kubeflow

How to use Docker Desktop for Mac or Windows to run Kubeflow

In this post, I will show you how to use Docker Desktop for Mac or Windows to run Kubeflow. To make this easier, I used my Depend on Docker project.

Run Kubeflow natively on Docker Desktop for Mac or Windows

In this post, I will show you how to use Docker Desktop for Mac or Windows to run Kubeflow. To make this easier, I used my Depend on Docker project.


Even though we are experiencing a tectonic shift of development workflows in the cloud era towards hosted and remote environments, a substantial amount of work and experimentation still happens on developer’s local machines. The ability to scale down allows us to mimic a cloud deployment locally and enables us to play, learn quickly, and make changes in a safe, isolated environment. A good example of this rationale is provided by Kubeflow and MiniKF.


Since Kubeflow was first released by Google in 2018, adoption has increased significantly, particularly in the data science world for orchestration of machine learning pipelines. There are various ways to deploy Kubeflow both on desktops and servers as described in its Getting Started guide. However, the desktop deployments for Mac and Windows rely on running virtual machines using Vagrant and VirtualBox. If you do not wish to install Vagrant and VirtualBox on your Mac or PC but would still like to run Kubeflow, then you can simply depend on Docker! This article will show you how to deploy Kubeflow natively on Docker Desktop.



Kubeflow has a hard dependency on Kubernetes and the Docker runtime. The easiest way to satisfy both of these requirements on Mac or Windows is to install Docker Desktop (version 2.1.x.x or higher). In the settings of Docker Desktop, navigate to the Kubernetes tab and check “Enable Kubernetes”:

Enabling Kubernetes in Docker Desktop by selecting it under settings.

Enabling the Kubernetes feature in Docker Desktop creates a single node Kubernetes cluster on your local machine.

This article offers a detailed walkthrough of setting up Kubeflow on Docker Desktop for Mac. Deploying Kubeflow on Docker Desktop for Windows using Linux containers requires two additional prerequisites:

  1. Linux shell – to run the bash commands from the Kubeflow installation instructions
  2. Kfctl and kubectl CLI – to initialize, generate, and apply the Kubeflow deployment

The easiest way to satisfy both of these dependencies is to run a Linux container that has the kfctl and kubectl utilities. A Depend on Docker project was created for this purpose. To start a bash shell with the two CLI’s available, just execute:

docker run -it --rm -v <kube_config_folder_path>:/root/.kube iankoulski/kfctl bash

The remaining setup steps for both Mac and Windows are the same.

Resource Requirements

The instructions for deployment of Kubeflow on a pre-existing Kubernetes cluster specify the following resource requirements:

  • 4 vCPUs
  • 50 GB storage
  • 12 GB memory

The settings in Docker Desktop need to be adjusted to accommodate these requirements as shown below.

Setting the resource limits available to Docker Engine in Docker Desktop.

Configuring the disk image size in Docker Desktop.

Note that the settings are adjusted to more than the minimum required resources to accommodate system containers and other applications that may be running on the local machine.


We will follow instructions for the kfctl_k8s_istio configuration.

  1. Download your preferred version from the release archive: curl -L -o kfctl_v0.6.2_darwin.tar.gz
  2. Extract the archive: tar -xvf kfctl_v0.6.2_darwin.tar.gz
  3. Set environment variables: export PATH=$PATH:$(pwd) export KFAPP=localkf export CONFIG=
  4. Initialize deployment: kfctl init ${KFAPP} --config=${CONFIG} cd ${KFAPP} kfctl generate all -V

Note: The above instructions are for Kubeflow release 0.6.2 and are meant to use as an example. Other releases would have slightly different archive filename, environment variable names and values, and kfctl commands. Those would be available in the release-specific deployment instructions.

  1. Pre-pull container images (optional)

To facilitate the deployment of Kubeflow locally, we can pre-pull all required Docker images. When the container images are already present on the machine, the memory usage of Docker Desktop stays low. Pulling all images at the time of deployment may cause large spikes in memory utilization and can cause Docker Daemon to run out of resources. Pre-pulling images is especially helpful when running Kubeflow on a 16GB laptop.

To pre-pull all container images, execute the following one-line script in your $KFAPP/kustomize folder:

for i in $(grep -R image: . | cut -d ':' -f 3,4 | uniq | sed -e 's/ //' -e 's/^"//' -e 's/"$//'); do echo "Pulling $i"; docker pull $i; done;

The CLI output when you pre-pull Kubeflow container images.

Depending on your Internet connection, this could take several minutes to complete. Even if Docker Desktop runs out of resources, restarting it and running the script again will resume pulling the remaining images from where you left off.

If you are using the kfctl container on Windows, you may wish to modify the one-line script above so it saves the docker pull commands to a file and then execute them from your preferred Docker shell.

  1. Apply Kubeflow deployment to Kubernetes:
    cd ${KFAPP}
    kfctl apply all -V
    The deployment output and Kubeflow pods when running in Docker Desktop.

Note: An existing deployment can be removed by executing “kfctl delete all -V”

  1. Determine the Kubeflow entrypoint

To determine the endpoint, list all services in the istio-system namespace: kubectl get svc -n istio-system

The Istio Ingress Gateway service CLI output.

The Kubeflow end-point service is through the ingress-gateway service on the NodePort connected with the default HTTP port (80). The Node Port number is 31380. To access Kubeflow use:

Using Kubeflow

The Kubeflow central dashboard is now accessible:

The Kubeflow central dashboard.

We can run one of the sample pipelines that is included in Kubeflow. Select Pipelines, then Experiments, and choose Conditional expression (or just click the [Sample] Basic – Conditional expression link on the dashboard screen).

The Kubeflow conditional execution pipeline view.

Next, click the +Create run button, enter a name (e.g. conditional-execution-test), choose an experiment, and then click Start to initiate the run. Navigate to your pipeline by selecting it from the list of runs.

The Kubeflow Conditional execution pipeline run.

The completed pipeline run looks similar to Fig. 9 above. Due to the random nature of the coin flip in this pipeline, your actual output is likely to be different. Select a node in the graph to review various assets associated with that node, including its logs.


Docker Desktop enables you to easily run container applications on your local machine, including ones that require a Kubernetes cluster. Kubeflow is a deployment that typically targets larger clusters either in cloud or on-prem environments. In this article we’ve demonstrated how to deploy and use Kubeflow locally on your Docker Desktop.

You can find on Github.

Docker devops Kubernetes

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

DevOps Tutorial - Docker, Kubernetes, and Azure DevOps

In this article, we break down everything you need to know about DevOps, so that you can get started building your own CI/CD pipeline. DevOps Tutorial - Docker, Kubernetes, and Azure DevOps

Kubernetes vs Docker

Get Hands-on experience on Kubernetes and the best comparison of Kubernetes over the DevOps at your place at Kubernetes training

Docker vs. Kubernetes | Docker vs. Kubernetes Difference

Docker and Kubernetes are two orchestration tools that very popular. Many people have trouble picking one. In this video on Docker vs Kubernetes, we will be comparing these two tools end to end and see which one will suit your needs better.

DevOps and Docker Live Show (Ep 91)

Join me with guest Docker Captain Elton Stoneman to talk about the state of Docker Desktop and Docker Hub. Support this show on Patreon! It's the #1 way to support me interviewing DevOps and container experts, and doing this Live Q&A.

Docker manifest - A peek into image's manifest.json files

The docker manifest command does not work independently to perform any action. In order to work with the docker manifest or manifest list, we use sub-commands along with it. This manifest sub-command can enable us to interact with the image manifests. Furthermore, it also gives information about the OS and the architecture, that a particular image was built for. The image manifest provides a configuration and a set of layers for a container image. This is an experimenta