Upgrading to Kubernetes 1.16

What’s new and deprecated in Kubernetes 1.16? Use kube-no-trouble or pluto to find and address issues before the upgrade.

Image for post

Kubernetes 1.18 was released earlier this year, which means the rollout for Kubernetes 1.16 across managed Kubernetes platforms (i.e. GKE, EKS, AKS) is quickly approaching (the majority of providers only support the most recent three minor releases). The upgrade to 1.16 is significant because several APIs are finally deprecated and removed from Kubernetes. This means that any attempts to create or upgrade any Kubernetes resources containing deprecated APIs will fail. In this post, we’ll highlight some new features on Kubernetes 1.16, point out all the deprecated APIs, and list a few tools to help with the upgrade process.

What’s New on Kubernetes 1.16

Three major themes in version 1.16 include:

Promoting Custom Resource Definitions (CRDs) to GA (CRDs have been in beta since 1.7).
Overhauling Kubernetes metrics: aligning metrics to the official Kubernetes standard (e.g. removing duplicates like pod_name and container_name from cAdvisor metrics, changing API latency histogram buckets, client-go metrics changes)
Container Storage Interface (CSI) graduates to Beta, giving more support for expanding PersistentVolumeClaims (PVC)

You can read the official release notes, CNCF’s presentation, or Sysdig’s summary to find more information on new features and enhancements.

What’s Removed on Kubernetes 1.16

More importantly, 1.16 removed the following APIs in favor of stable API versions:

NetworkPolicy: extensions/v1beta1 → networking.k8s.io/v1
**PodSecurityPolicy: **extensions/v1beta1 → policy/v1beta1
**DaemonSet: **extensions/v1beta1, apps/v1beta1, and apps/v1beta2 → apps/v1
**Deployment: **extensions/v1beta1, apps/v1beta1, and apps/v1beta2 → apps/v1
StatefulSet: apps/v1beta1 and apps/v1beta2→ apps/v1
ReplicaSet: extensions/v1beta1, apps/v1beta1, and apps/v1beta2 → apps/v1

In most cases, the fix is to simply update the API version to the stable API. However, app/v1 behaves slightly differently in certain cases:

DaemonSet: spec.templateGeneration is removed, spec.selector is now required and immutable, spec.updateStrategy.type defaults to RollingUpdate
**Deployment: **spec.rollbackTo is removed, spec.selector is now required and immutable, spec.progressDeadlineSeconds defaults to 600s, spec.revisionHistoryLimit defaults to 10, maxSurge and maxUnavailable default to 25%
**StatefulSet: **spec.selector is now required and immutable, spec.updateStrategy.type defaults to RollingUpdate

Make note of these behaviors (especially for StatefulSets that changed the updateStrategy) to ensure minimal disruption to your workloads.

How to Find Deprecated APIs

If you are using GKE and have Stackdriver Logging enabled on your cluster, you can identify all the workloads using deprecated APIs using the following filter:

resource.type="k8s_cluster"
resource.labels.cluster_name="$CLUSTER_NAME"
protoPayload.authenticationInfo.principalEmail:("system:serviceaccount" OR "@")
protoPayload.request.apiVersion=("extensions/v1beta1" OR "apps/v1beta1" OR "apps/v1beta2")
protoPayload.request.kind!="Ingress"
NOT ("kube-system")

Otherwise, you can use a tool called Kube No Trouble** (**kubent**)** by DoiT International or FairwindsOps’sPluto if you are using other tools like polaris or goldilocks. Both tools work with kubectl and Helm and lists deprecated APIs:

kubent example

$./kubent
6:25PM INF >>> Kube No Trouble `kubent` <<<
6:25PM INF Initializing collectors and retrieving data
6:25PM INF Retrieved 103 resources from collector name=Cluster
6:25PM INF Retrieved 132 resources from collector name="Helm v2"
6:25PM INF Retrieved 0 resources from collector name="Helm v3"
6:25PM INF Loaded ruleset name=deprecated-1-16.rego
6:25PM INF Loaded ruleset name=deprecated-1-20.rego
__________________________________________________________________________________________
>>> 1.16 Deprecated APIs <<<
------------------------------------------------------------------------------------------
KIND         NAMESPACE     NAME                    API_VERSION
Deployment   default       nginx-deployment-old    apps/v1beta1
Deployment   kube-system   event-exporter-v0.2.5   apps/v1beta1
Deployment   kube-system   k8s-snapshots           extensions/v1beta1
Deployment   kube-system   kube-dns                extensions/v1beta1
__________________________________________________________________________________________
>>> 1.20 Deprecated APIs <<<
------------------------------------------------------------------------------------------
KIND      NAMESPACE   NAME           API_VERSION
Ingress   default     test-ingress   extensions/v1beta1

pluto example

$ pluto detect-helm --helm-version 3 
KIND          VERSION        DEPRECATED   RESOURCE NAME 
StatefulSet   apps/v1beta1   true         prod-rabbitmq-ha Deployment    apps/v1        false        goldilocks-controller Deployment    apps/v1        false        goldilocks-dashboard

Once you have identified the deprecated APIs, you can use kubectl convert :

$ kubectl convert -f <file> --output-version <group>/<version>

(example)
$ kubectl convert -f deployment.yaml --output-version apps/v1

or update the Helm chart with the new apiVersions and run helm upgrade. If for some reason, you upgraded the cluster before migrating all the Helm charts or if Helm fails due to deprecated APIs, you can use [**helm-mapkubeapis**](https://github.com/hickeyma/helm-mapkubeapis)tool to patch the Helm release manifests stored in configmap or secret.

Finally, if you are running the upgrade on EKS or other providers where other Kubernetes components such as kube-proxy and CNI are not upgraded automatically, make sure to patch the versions manually as well.

#coding #software-development #kubernetes #software-engineering #technology