Justen  Hintz

Justen Hintz

1573108220

How to use Docker Desktop for Mac or Windows to run Kubeflow

Run Kubeflow natively on Docker Desktop for Mac or Windows

In this post, I will show you how to use Docker Desktop for Mac or Windows to run Kubeflow. To make this easier, I used my Depend on Docker project.

Rationale

Even though we are experiencing a tectonic shift of development workflows in the cloud era towards hosted and remote environments, a substantial amount of work and experimentation still happens on developer’s local machines. The ability to scale down allows us to mimic a cloud deployment locally and enables us to play, learn quickly, and make changes in a safe, isolated environment. A good example of this rationale is provided by Kubeflow and MiniKF.

Overview

Since Kubeflow was first released by Google in 2018, adoption has increased significantly, particularly in the data science world for orchestration of machine learning pipelines. There are various ways to deploy Kubeflow both on desktops and servers as described in its Getting Started guide. However, the desktop deployments for Mac and Windows rely on running virtual machines using Vagrant and VirtualBox. If you do not wish to install Vagrant and VirtualBox on your Mac or PC but would still like to run Kubeflow, then you can simply depend on Docker! This article will show you how to deploy Kubeflow natively on Docker Desktop.

Setup

Prerequisites

Kubeflow has a hard dependency on Kubernetes and the Docker runtime. The easiest way to satisfy both of these requirements on Mac or Windows is to install Docker Desktop (version 2.1.x.x or higher). In the settings of Docker Desktop, navigate to the Kubernetes tab and check “Enable Kubernetes”:

Enabling Kubernetes in Docker Desktop by selecting it under settings.

Enabling the Kubernetes feature in Docker Desktop creates a single node Kubernetes cluster on your local machine.

This article offers a detailed walkthrough of setting up Kubeflow on Docker Desktop for Mac. Deploying Kubeflow on Docker Desktop for Windows using Linux containers requires two additional prerequisites:

  1. Linux shell – to run the bash commands from the Kubeflow installation instructions
  2. Kfctl and kubectl CLI – to initialize, generate, and apply the Kubeflow deployment

The easiest way to satisfy both of these dependencies is to run a Linux container that has the kfctl and kubectl utilities. A Depend on Docker project was created for this purpose. To start a bash shell with the two CLI’s available, just execute:

docker run -it --rm -v <kube_config_folder_path>:/root/.kube iankoulski/kfctl bash

The remaining setup steps for both Mac and Windows are the same.

Resource Requirements

The instructions for deployment of Kubeflow on a pre-existing Kubernetes cluster specify the following resource requirements:

  • 4 vCPUs
  • 50 GB storage
  • 12 GB memory

The settings in Docker Desktop need to be adjusted to accommodate these requirements as shown below.

Setting the resource limits available to Docker Engine in Docker Desktop.

Configuring the disk image size in Docker Desktop.

Note that the settings are adjusted to more than the minimum required resources to accommodate system containers and other applications that may be running on the local machine.

Deployment

We will follow instructions for the kfctl_k8s_istio configuration.

  1. Download your preferred version from the release archive:
    curl -L -o kfctl_v0.6.2_darwin.tar.gz https://github.com/kubeflow/kubeflow/releases/download/v0.6.2/kfctl_darwin.tar.gz
  2. Extract the archive:
    tar -xvf kfctl_v0.6.2_darwin.tar.gz
  3. Set environment variables:
    export PATH=$PATH:$(pwd) export KFAPP=localkf export CONFIG=https://raw.githubusercontent.com/kubeflow/kubeflow/v0.6-branch/bootstrap/config/kfctl_k8s_istio.0.6.2.yaml
  4. Initialize deployment:
    kfctl init ${KFAPP} --config=${CONFIG} cd ${KFAPP} kfctl generate all -V

Note: The above instructions are for Kubeflow release 0.6.2 and are meant to use as an example. Other releases would have slightly different archive filename, environment variable names and values, and kfctl commands. Those would be available in the release-specific deployment instructions.

  1. Pre-pull container images (optional)

To facilitate the deployment of Kubeflow locally, we can pre-pull all required Docker images. When the container images are already present on the machine, the memory usage of Docker Desktop stays low. Pulling all images at the time of deployment may cause large spikes in memory utilization and can cause Docker Daemon to run out of resources. Pre-pulling images is especially helpful when running Kubeflow on a 16GB laptop.

To pre-pull all container images, execute the following one-line script in your $KFAPP/kustomize folder:

for i in $(grep -R image: . | cut -d ':' -f 3,4 | uniq | sed -e 's/ //' -e 's/^"//' -e 's/"$//'); do echo "Pulling $i"; docker pull $i; done;

The CLI output when you pre-pull Kubeflow container images.

Depending on your Internet connection, this could take several minutes to complete. Even if Docker Desktop runs out of resources, restarting it and running the script again will resume pulling the remaining images from where you left off.

If you are using the kfctl container on Windows, you may wish to modify the one-line script above so it saves the docker pull commands to a file and then execute them from your preferred Docker shell.

  1. Apply Kubeflow deployment to Kubernetes:
cd ${KFAPP}
kfctl apply all -V

The deployment output and Kubeflow pods when running in Docker Desktop.

Note: An existing deployment can be removed by executing “kfctl delete all -V”

  1. Determine the Kubeflow entrypoint

To determine the endpoint, list all services in the istio-system namespace:
kubectl get svc -n istio-system

The Istio Ingress Gateway service CLI output.

The Kubeflow end-point service is through the ingress-gateway service on the NodePort connected with the default HTTP port (80). The Node Port number is 31380. To access Kubeflow use: http://127.0.0.1:31380

Using Kubeflow

The Kubeflow central dashboard is now accessible:

The Kubeflow central dashboard.

We can run one of the sample pipelines that is included in Kubeflow. Select Pipelines, then Experiments, and choose Conditional expression (or just click the [Sample] Basic – Conditional expression link on the dashboard screen).

The Kubeflow conditional execution pipeline view.

Next, click the +Create run button, enter a name (e.g. conditional-execution-test), choose an experiment, and then click Start to initiate the run. Navigate to your pipeline by selecting it from the list of runs.

The Kubeflow Conditional execution pipeline run.

The completed pipeline run looks similar to Fig. 9 above. Due to the random nature of the coin flip in this pipeline, your actual output is likely to be different. Select a node in the graph to review various assets associated with that node, including its logs.

Conclusion

Docker Desktop enables you to easily run container applications on your local machine, including ones that require a Kubernetes cluster. Kubeflow is a deployment that typically targets larger clusters either in cloud or on-prem environments. In this article we’ve demonstrated how to deploy and use Kubeflow locally on your Docker Desktop.

You can find on Github.

#Docker #devops #Kubernetes

What is GEEK

Buddha Community

How to use Docker Desktop for Mac or Windows to run Kubeflow
Iliana  Welch

Iliana Welch

1595249460

Docker Explained: Docker Architecture | Docker Registries

Following the second video about Docker basics, in this video, I explain Docker architecture and explain the different building blocks of the docker engine; docker client, API, Docker Daemon. I also explain what a docker registry is and I finish the video with a demo explaining and illustrating how to use Docker hub

In this video lesson you will learn:

  • What is Docker Host
  • What is Docker Engine
  • Learn about Docker Architecture
  • Learn about Docker client and Docker Daemon
  • Docker Hub and Registries
  • Simple demo to understand using images from registries

#docker #docker hub #docker host #docker engine #docker architecture #api

Turner  Crona

Turner Crona

1595743500

Performance Tips: Docker Desktop on Mac

Native NFS

A quick test reveals that we can see that we can extract ~100% performance improvement.

Host mount

/home # dd if=/dev/zero of=./output bs=8k count=10k; rm ./output
10240+0 records in
10240+0 records out
83886080 bytes (80.0MB) copied, 3.084522 seconds, 25.9MB/s

Host (NFS volume) mount

/home # dd if=/dev/zero of=./output bs=8k count=10k; rm ./output
10240+0 records in
10240+0 records out
83886080 bytes (80.0MB) copied, 1.426896 seconds, 56.1MB/s

If you prefer these tests can be performed locally on your machine using docker-mac-perf repo.

Thanks to this script created by Sean Handley we can quickly setup Native NFS on Mac OS. To enable NFS on your Mac run the below commands:

#docker-for-mac #docker #docker-volume

Justen  Hintz

Justen Hintz

1573108220

How to use Docker Desktop for Mac or Windows to run Kubeflow

Run Kubeflow natively on Docker Desktop for Mac or Windows

In this post, I will show you how to use Docker Desktop for Mac or Windows to run Kubeflow. To make this easier, I used my Depend on Docker project.

Rationale

Even though we are experiencing a tectonic shift of development workflows in the cloud era towards hosted and remote environments, a substantial amount of work and experimentation still happens on developer’s local machines. The ability to scale down allows us to mimic a cloud deployment locally and enables us to play, learn quickly, and make changes in a safe, isolated environment. A good example of this rationale is provided by Kubeflow and MiniKF.

Overview

Since Kubeflow was first released by Google in 2018, adoption has increased significantly, particularly in the data science world for orchestration of machine learning pipelines. There are various ways to deploy Kubeflow both on desktops and servers as described in its Getting Started guide. However, the desktop deployments for Mac and Windows rely on running virtual machines using Vagrant and VirtualBox. If you do not wish to install Vagrant and VirtualBox on your Mac or PC but would still like to run Kubeflow, then you can simply depend on Docker! This article will show you how to deploy Kubeflow natively on Docker Desktop.

Setup

Prerequisites

Kubeflow has a hard dependency on Kubernetes and the Docker runtime. The easiest way to satisfy both of these requirements on Mac or Windows is to install Docker Desktop (version 2.1.x.x or higher). In the settings of Docker Desktop, navigate to the Kubernetes tab and check “Enable Kubernetes”:

Enabling Kubernetes in Docker Desktop by selecting it under settings.

Enabling the Kubernetes feature in Docker Desktop creates a single node Kubernetes cluster on your local machine.

This article offers a detailed walkthrough of setting up Kubeflow on Docker Desktop for Mac. Deploying Kubeflow on Docker Desktop for Windows using Linux containers requires two additional prerequisites:

  1. Linux shell – to run the bash commands from the Kubeflow installation instructions
  2. Kfctl and kubectl CLI – to initialize, generate, and apply the Kubeflow deployment

The easiest way to satisfy both of these dependencies is to run a Linux container that has the kfctl and kubectl utilities. A Depend on Docker project was created for this purpose. To start a bash shell with the two CLI’s available, just execute:

docker run -it --rm -v <kube_config_folder_path>:/root/.kube iankoulski/kfctl bash

The remaining setup steps for both Mac and Windows are the same.

Resource Requirements

The instructions for deployment of Kubeflow on a pre-existing Kubernetes cluster specify the following resource requirements:

  • 4 vCPUs
  • 50 GB storage
  • 12 GB memory

The settings in Docker Desktop need to be adjusted to accommodate these requirements as shown below.

Setting the resource limits available to Docker Engine in Docker Desktop.

Configuring the disk image size in Docker Desktop.

Note that the settings are adjusted to more than the minimum required resources to accommodate system containers and other applications that may be running on the local machine.

Deployment

We will follow instructions for the kfctl_k8s_istio configuration.

  1. Download your preferred version from the release archive:
    curl -L -o kfctl_v0.6.2_darwin.tar.gz https://github.com/kubeflow/kubeflow/releases/download/v0.6.2/kfctl_darwin.tar.gz
  2. Extract the archive:
    tar -xvf kfctl_v0.6.2_darwin.tar.gz
  3. Set environment variables:
    export PATH=$PATH:$(pwd) export KFAPP=localkf export CONFIG=https://raw.githubusercontent.com/kubeflow/kubeflow/v0.6-branch/bootstrap/config/kfctl_k8s_istio.0.6.2.yaml
  4. Initialize deployment:
    kfctl init ${KFAPP} --config=${CONFIG} cd ${KFAPP} kfctl generate all -V

Note: The above instructions are for Kubeflow release 0.6.2 and are meant to use as an example. Other releases would have slightly different archive filename, environment variable names and values, and kfctl commands. Those would be available in the release-specific deployment instructions.

  1. Pre-pull container images (optional)

To facilitate the deployment of Kubeflow locally, we can pre-pull all required Docker images. When the container images are already present on the machine, the memory usage of Docker Desktop stays low. Pulling all images at the time of deployment may cause large spikes in memory utilization and can cause Docker Daemon to run out of resources. Pre-pulling images is especially helpful when running Kubeflow on a 16GB laptop.

To pre-pull all container images, execute the following one-line script in your $KFAPP/kustomize folder:

for i in $(grep -R image: . | cut -d ':' -f 3,4 | uniq | sed -e 's/ //' -e 's/^"//' -e 's/"$//'); do echo "Pulling $i"; docker pull $i; done;

The CLI output when you pre-pull Kubeflow container images.

Depending on your Internet connection, this could take several minutes to complete. Even if Docker Desktop runs out of resources, restarting it and running the script again will resume pulling the remaining images from where you left off.

If you are using the kfctl container on Windows, you may wish to modify the one-line script above so it saves the docker pull commands to a file and then execute them from your preferred Docker shell.

  1. Apply Kubeflow deployment to Kubernetes:
cd ${KFAPP}
kfctl apply all -V

The deployment output and Kubeflow pods when running in Docker Desktop.

Note: An existing deployment can be removed by executing “kfctl delete all -V”

  1. Determine the Kubeflow entrypoint

To determine the endpoint, list all services in the istio-system namespace:
kubectl get svc -n istio-system

The Istio Ingress Gateway service CLI output.

The Kubeflow end-point service is through the ingress-gateway service on the NodePort connected with the default HTTP port (80). The Node Port number is 31380. To access Kubeflow use: http://127.0.0.1:31380

Using Kubeflow

The Kubeflow central dashboard is now accessible:

The Kubeflow central dashboard.

We can run one of the sample pipelines that is included in Kubeflow. Select Pipelines, then Experiments, and choose Conditional expression (or just click the [Sample] Basic – Conditional expression link on the dashboard screen).

The Kubeflow conditional execution pipeline view.

Next, click the +Create run button, enter a name (e.g. conditional-execution-test), choose an experiment, and then click Start to initiate the run. Navigate to your pipeline by selecting it from the list of runs.

The Kubeflow Conditional execution pipeline run.

The completed pipeline run looks similar to Fig. 9 above. Due to the random nature of the coin flip in this pipeline, your actual output is likely to be different. Select a node in the graph to review various assets associated with that node, including its logs.

Conclusion

Docker Desktop enables you to easily run container applications on your local machine, including ones that require a Kubernetes cluster. Kubeflow is a deployment that typically targets larger clusters either in cloud or on-prem environments. In this article we’ve demonstrated how to deploy and use Kubeflow locally on your Docker Desktop.

You can find on Github.

#Docker #devops #Kubernetes

Instructions for installing Docker on Ubuntu, Mac and Windows

How to Install Docker on Ubuntu, Mac & Windows

Everything is moving into Cloud these days and with Docker(container) things have become faster and smoother to be achieved. Docker's goal is to ease the creation, to deploy, and to deliver an application using the so-called Containers.

#docker #windows #ubuntu #mac #kubernetes 

Iliana  Welch

Iliana Welch

1597368540

Docker Tutorial for Beginners 8 - Build and Run C++ Applications in a Docker Container

Docker is an open platform that allows use package, develop, run, and ship software applications in different environments using containers.
In this course We will learn How to Write Dockerfiles, Working with the Docker Toolbox, How to Work with the Docker Machine, How to Use Docker Compose to fire up multiple containers, How to Work with Docker Kinematic, Push images to Docker Hub, Pull images from a Docker Registery, Push stacks of servers to Docker Hub.
How to install Docker on Mac.

#docker tutorial #c++ #docker container #docker #docker hub #devopstools