Kubernetes on MIPS

Kubernetes on MIPS

MIPS (Microprocessor without Interlocked Pipelined Stages) is a reduced instruction set computer (RISC) instruction set architecture (ISA). Kubernetes has officially supported a variety of CPU architectures such as x86, arm/arm64, ppc64le, s390x. However, it’s a pity that Kubernetes doesn’t support MIPS. With the widespread use of cloud native technology, users under MIPS architecture also have an urgent demand for Kubernetes on MIPS.


MIPS (Microprocessor without Interlocked Pipelined Stages) is a reduced instruction set computer (RISC) instruction set architecture (ISA), appeared in 1981 and developed by MIPS Technologies. Now MIPS architecture is widely used in many electronic products.

Kubernetes has officially supported a variety of CPU architectures such as x86, arm/arm64, ppc64le, s390x. However, it’s a pity that Kubernetes doesn’t support MIPS. With the widespread use of cloud native technology, users under MIPS architecture also have an urgent demand for Kubernetes on MIPS.


For many years, to enrich the ecology of the open-source community, we have been working on adjusting MIPS architecture for Kubernetes use cases. With the continuous iterative optimization and the performance improvement of the MIPS CPU, we have made some breakthrough progresses on the mips64el platform.

Over the years, we have been actively participating in the Kubernetes community and have rich experience in the using and optimization of Kubernetes technology. Recently, we tried to adapt the MIPS architecture platform for Kubernetes and achieved a new a stage on that journey. The team has completed migration and adaptation of Kubernetes and related components, built not only a stable and highly available MIPS cluster but also completed the conformance test for Kubernetes v1.16.2.

Figure 1 Kubernetes on MIPS

K8S-MIPS component build

Almost all native cloud components related to Kubernetes do not provide a MIPS version installation package or image. The prerequisite of deploying Kubernetes on the MIPS platform is to compile and build all required components on the mips64el platform. These components include:

  • golang
  • docker-ce
  • hyperkube
  • pause
  • etcd
  • calico
  • coredns
  • metrics-server

Thanks to the excellent design of Golang and its good support for the MIPS platform, the compilation processes of the above cloud native components are greatly simplified. First of all, we compiled Golang on the latest stable version for the mips64el platform, and then we compiled most of the above components with source code.

During the compilation processes, we inevitably encountered many platform compatibility problems, such as a Golang system call compatibility problem (syscall), typecasting of syscall. Stat_t from uint32 to uint64, patching for EpollEvent, and so on.

To build K8S-MIPS components, we used cross-compilation technology. Our process involved integrating a QEMU tool to translate MIPS CPU instructions and modifying the build script of Kubernetes and E2E image script of Kubernetes, Hyperkube, and E2E test images on MIPS architecture.

After successfully building the above components, we use tools such as kubespray and kubeadm to complete kubernetes cluster construction.

Note: CKE is a Kubernetes-based cloud container engine launched by Inspur

Figure 2 K8S-MIPS Cluster Components

Figure 3 CPU Architecture

Figure 4 Cluster Node Information

Run K8S Conformance Test

The most straightforward way to verify the stability and availability of the K8S-MIPS cluster is to run a Kubernetes conformance test.

Conformance is a standalone container to launch Kubernetes end-to-end tests for conformance testing.

Once the test has started, it launches several pods for various end-to-end tests. The source code of those images used by these pods is mostly from kubernetes/test/images, and the built images are at gcr.io/kubernetes-e2e-test-images. Since there are no MIPS images in the repository, we must first build all needed images to run the test.

Build needed images for test

The first step is to find all needed images for the test. We can run sonobuoy images-p e2e command to list all images, or we can find those images in /test/utils/image/manifest.go. Although Kubernetes officially has a complete Makefile and shell-script that provides commands for building test images, there are still a number of architecture-related issues that have not been resolved, such as the incompatibilities of base images and dependencies. So we cannot directly build mips64el architecture images by executing these commands.

Most test images are in golang, then compiled into binaries and built as Docker image based on the corresponding Dockerfile. These images are easy to build. But note that most images are using alpine as their base image, which does not officially support mips64el architecture for now. For this moment, we are unable to make mips64el version of alpine, so we have to replace the alpine to existing MIPS images, such as Debian-stretch, fedora, ubuntu. Replacing the base image also requires replacing the command to install the dependencies, even the version of these dependencies.

Some images are not in kubernetes/test/images, such as gcr.io/google-samples/gb-frontend:v6. There is no clear documentation explaining where these images are locaated, though we found the source code in repository github.com/GoogleCloudPlatform/kubernetes-engine-samples. We soon ran into new problems: to build these google sample images, we have to build the base image it uses, even the base image of the base images, such as php:5-apache, redis, and perl.

After a long process of building an image, we finished with about four dozen images, including the images used by the test pod, and the base images. The last step before we run the tests is to place all those images into every node in the cluster and make sure the Pod image pull policy is imagePullPolicy: ifNotPresent.

Here are some of the images we built:

  • docker.io/library/busybox:1.29
  • docker.io/library/nginx:1.14-alpine
  • docker.io/library/nginx:1.15-alpine
  • docker.io/library/perl:5.26
  • docker.io/library/httpd:2.4.38-alpine
  • docker.io/library/redis:5.0.5-alpine
  • gcr.io/google-containers/conformance:v1.16.2
  • gcr.io/google-containers/hyperkube:v1.16.2
  • gcr.io/google-samples/gb-frontend:v6
  • gcr.io/kubernetes-e2e-test-images/agnhost:2.6
  • gcr.io/kubernetes-e2e-test-images/apparmor-loader:1.0
  • gcr.io/kubernetes-e2e-test-images/dnsutils:1.1
  • gcr.io/kubernetes-e2e-test-images/echoserver:2.2
  • gcr.io/kubernetes-e2e-test-images/ipc-utils:1.0
  • gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0
  • gcr.io/kubernetes-e2e-test-images/kitten:1.0
  • gcr.io/kubernetes-e2e-test-images/metadata-concealment:1.2
  • gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0
  • gcr.io/kubernetes-e2e-test-images/mounttest:1.0
  • gcr.io/kubernetes-e2e-test-images/nautilus:1.0
  • gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0
  • gcr.io/kubernetes-e2e-test-images/nonroot:1.0
  • gcr.io/kubernetes-e2e-test-images/resource-consumer-controller:1.0
  • gcr.io/kubernetes-e2e-test-images/resource-consumer:1.5
  • gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.10
  • gcr.io/kubernetes-e2e-test-images/test-webserver:1.0
  • gcr.io/kubernetes-e2e-test-images/volume/gluster:1.0
  • gcr.io/kubernetes-e2e-test-images/volume/iscsi:2.0
  • gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0
  • gcr.io/kubernetes-e2e-test-images/volume/rbd:1.0.1
  • k8s.gcr.io/etcd:3.3.15
  • k8s.gcr.io/pause:3.1

Finally, we ran the tests and got the test result, include e2e.log, which showed that all test cases passed. Additionally, we submitted our test result to k8s-conformance as a pull request.

Figure 5 Pull request for conformance test results

What’s next

We built the kubernetes-MIPS component manually and finished the conformance test, which verified the feasibility of Kubernetes On the MIPS platform and greatly enhanced our confidence in promoting the support of the MIPS architecture by Kubernetes.

In the future, we plan to actively contribute our experience and achievements to the community, submit PR, and patch for MIPS. We hope that more developers and companies in the community join us and promote Kubernetes on MIPS.

Contribution plan:

  • contribute the source of e2e test images for MIPS
  • contribute the source of hyperkube for MIPS
  • contribute the source of deploy tools like kubeadm for MIPS

Originally published at https://kubernetes.io

Angular 9 Tutorial: Learn to Build a CRUD Angular App Quickly

What's new in Bootstrap 5 and when Bootstrap 5 release date?

What’s new in HTML6

How to Build Progressive Web Apps (PWA) using Angular 9

What is new features in Javascript ES2020 ECMAScript 2020

How Kubernetes Helps to Enable DevOps

How Kubernetes Helps to Enable DevOps

The automation and infrastructural capabilities of Kubernetes makes it an ideal technological partner for DevOps. In this article, you'll see 10 ways Kubernetes enables DevOps

The automation and infrastructural capabilities of Kubernetes makes it an ideal technological partner for DevOps. In this article, you'll see 10 ways Kubernetes enables DevOps

DevOps was an idea before its time.

It actually took a while for technology to catch up and fully implement the principles and vision of DevOps, but that is how innovation works.

A new set of tools in our toolboxes (though it's now taking up more and more mindshare) is end-to-end automation, and Kubernetes helps you manage that. Kubernetes is an open source framework for "automating deployment, scaling, and management of containerized applications." Originally introduced by Google, Red Hat, and others pushed it forward the last 3-4 years.

At the 2017 All Day DevOps conference, Siamak Sadeghianfar laid out 10 ways Kubernetes enables DevOps. We thought it was worth revisiting what he had to say as **Kubernetes **only becomes more prominent.

**So, how exactly does Kubernetes enable DevOps? Let's dive into what Siamak had to say: **

1. Deployment automation.

You should automate every step of your delivery pipeline. Kubernetes automates the deployment of containers. Each component becomes a container image.

**2. Infrastructure as code. **

With **Kubernetes **your entire infrastructure is code. This means any part in your application (databases, ports, access controls, etc.) can be described in a way **Kubernetes **can use. For example, you store your infrastructure code in a version-control repository. **Kubernetes **takes the code, and based on instruction, deploys, and maintains your infrastructure. This happens automatically and consistently.

3. Configuration as code.

**Kubernetes **allows you to "configure as code." Traditionally, an admin had to run configuration scripts manually to make sure they get the right one. **Kubernetes **keeps the file in the source repo. This allows you to describe where the file needs to go in the container and how the application consumes it. Additionally, it can be version controlled.

4. Immutable infrastructure.

Since the inception of servers, there's been one problem: every time you put out a fire you change the state of the virtual machine. Eventually, you don't know exactly what the server looks like and you can't recreate it, even though you know it works. (These virtual machines are referred to as snowflakes.)

In Kubernetes, new containers are immutable; they are created when there is a problem with the original state. So, you can be confident that you know exactly how the new environment is configured.

5. On-Demand Infrastructure.

Developers can create hybrid services and infrastructure on-demand from the self-service catalog. This gives control to developers to get the resources they need, yet allows operations to control the configuration of the services. Kubernetes follows open service and API standards so you can expose cloud services.

6. Environment consistency.

Build once, and deploy into production-like environments everywhere. Kubernetes allows you to build a golden image and use the exact same image for every single place you deploy your container. This gives you a consistent, production-like environment so it is the exact same whether you are on your local Windows or Mac development machine, a test server, etc. This helps you "shift left" so that you can see issues when you are in development instead of in production.

7. Continuous Delivery pipeline.

This is a series of automated steps to test code before it goes in production. Jez Humble says the role of Continuous Delivery is, "an automated process to prove to you a change is a bad change and it should not go into production." Continuous Integration (CI) means every change is tested. Continuous Delivery (CD) automates all the way to production.

How can you tell if you have fully implemented Continuous Delivery? Siamak says that the test is if someone walks in and says, "Can you go to production right now?" can you do it without breaking a sweat?

8. Zero downtime deployments.

When you have several deployments a day, you can't pull down production to deploy. You must have safe, rolling updates without disrupting the production traffic. Kubernetes helps with blue/green deployments so that you can set up a new environment and switch to the new one without downtime.

9. A/B Testing.

How can you evaluate proposed changes, such as changing copy or the color of buttons? A/B testing delivers different versions to different customers so you can get real-world test results. Kubernetes manages routing traffic to different versions. The same version with slight modifications are possible.

10. Cross-functional collaboration.

This is shared access to environments with granular control. Kubernetes goes beyond the silos so everyone has the same set of environments, but you can grant access to different roles and allow different roles to do different things. For instance, developers may be able to push to production, while Infosec may have view access, and QA may have access to live containers, but not building or deploying. The entire team can access production, but operations is the only one who can make containers.

The growing popularity of Kubernetes is undeniable. Luckily, with these 10 capabilities, Siamak has laid out a great path for us all to follow as they continue to not only make their way into the DevOps pipeline, but in many ways enable it.

What is Kubernetes | Kubernetes Tutorial For Beginners

What is Kubernetes | Kubernetes Tutorial For Beginners

This video on "What is Kubernetes | Kubernetes Tutorial For Beginners" will give you an introduction to one of the most popular Devops tool in the market - Kubernetes, and its importance in today's IT processes. This tutorial is ideal for beginners who want to get started with Kubernetes & DevOps

What Is Kubernetes | Kubernetes Introduction | Kubernetes Tutorial For Beginners

The following topics are covered in this training session:

  1. Need for Kubernetes
  2. What is Kubernetes and What it's not
  3. How does Kubernetes work?
  4. Use-Case: Kubernetes @ Pokemon Go
  5. Hands-on: Deployment with Kubernetes

Building and Managing Kubernetes with Kubernetes

Building and Managing Kubernetes with Kubernetes

Building and Managing Kubernetes with Kubernetes: Kubernetes as a declarative and portable system can be used to do many things in different ways.

Kubernetes as a declarative and portable system can be used to do many things in different ways.

At eBay we built a fleet management system based on k8s. Everything(server, subnet, OS, package and state) is declarative and can be modeled as CRDs in k8s, or referred to as a commit id in git from the objects. By running various controllers on top of these CRD objects, we use k8s to manage k8s, and the entire eBay data center. - Our system provisions hosts the same way k8s creates and manages pods. - We build k8s clusters with Salt. each host has a set of states defined in its salt CRD object. controllers pull states from git based on commit ids to apply. - We build both schedulers and deployment transactions to manage the k8s clusters for both config deployments and upgrades. This declarative, highly scalable, auto healing, and cloud native system is what we think can unify eBay’s fleet.

Thanks for reading

If you liked this post, share it with all of your programming buddies!

Follow us on Facebook | Twitter

Further reading about Kubernetes

An illustrated guide to Kubernetes Networking

Google Kubernetes Engine By Example

An Introduction to the Kubernetes DNS Service

Deploying a Laravel app in Kubernetes on Google Cloud

How to build a Microservice Architecture with Spring Boot and Kubernetes?