What’s the big fuss over the latest Kubernetes apiserver vulnerability?
Early on Monday December 3rd, a boulder splashed into the placidly silent Kubernetes security channels. A potentially high severity authentication bypass was disclosed with scant explanation the same day that K8s version 1.13 went golden master. For Kubernetes administrators with PTSD from 2014’s HeartBleed, the CVE blast and its 37-line fix triggered palpitations in anticipation of sleepless patchfests to come.
In this post, we’ll explain the “verify backend upgrade connection” commit and the bug’s actual impact. We have also whipped up a proof of concept of the vulnerability, which we could not find elsewhere, in case you want to see if your clusters are affected.
Explanation of CVE-2018-1002105 root cause
Kubernetes apiserver has the ability to proxy http requests to other kubernetes services, allowing for the K8s API itself to become extensible through its Aggregation Layer. This is the same facility that allows RBAC or namespace-constrained users to magically kubectl exec, kubectl attach, and kubectl port-forward directly from their laptops to pods running within live clusters.
Kubernetes cluster-local authentication model is largely based around full mutual TLS authentication (mTLS), and the various “microservice” components that make up a live K8s cluster use signed certificates to trust each other. When a “master” apiserver process establishes a connection to an “aggregate” layer, the master uses its certificate to authenticate the connection with this peer. The lower-level peer, aka “aggregate layer”, verifies the certificate and trusts that the apiserver has validated required credentials on the other side of the proxied connection.
Where the palpitations start is that the kubernetes API isn’t just basic HTTPS. To support remote administrative tasks, K8s also allows upgrading apiserver connections to full, live, end-to-end HTTP/1.1 websockets.
The CVE-2018-1002105 vulnerability comes from the way this websocket upgrade was handled: if the request contained the Connection: Upgrade http header, the master apiserver would forward the request and bridge the live socket to the aggregate. The problem occurs in the event that the websocket connection fails to complete. Prior to the fix, the apiserver could be tricked into assuming the pass-thru connection successfully landed even when it had triggered an error code. From that “half-open” and authenticated websocket state, the connected client could send follow-up HTTP requests to the aggregated endpoint, essentially masquerading itself as the master apiserver.
Because the apiserver has bridged the connections between client and server, this allowed the client to continue to use the connection to make requests, bypassing all security and audit controls on the master. Finding the exploit in logs would be extremely difficult.
#kubernetes #cve
