Graceful Node Shutdown Goes Beta

Graceful Node Shutdown Goes Beta

Graceful Node Shutdown Goes Beta - Graceful node shutdown, beta in 1.21, enables kubelet to gracefully evict pods during a node shutdown. The best practice prior to bringing your node down is to safely drain and cordon your node. (when graceful node shutdown was introduced as an alpha feature)

Authors: David Porter (Google), Mrunal Patel (Red Hat), and Tim Bannister (The Scale Factory)

Graceful node shutdown, beta in 1.21, enables kubelet to gracefully evict pods during a node shutdown.

Kubernetes is a distributed system and as such we need to be prepared for inevitable failures — nodes will fail, containers might crash or be restarted, and - ideally - your workloads will be able to withstand these catastrophic events.

One of the common classes of issues are workload failures on node shutdown or restart. The best practice prior to bringing your node down is to  safely drain and cordon your node. This will ensure that all pods running on this node can safely be evicted. An eviction will ensure your pods can follow the expected  pod termination lifecycle meaning receiving a SIGTERM in your container and/or running preStopHooks.

Prior to Kubernetes 1.20 (when graceful node shutdown was introduced as an alpha feature), safe node draining was not easy: it required users to manually take action and drain the node beforehand. If someone or something shut down your node without draining it first, most likely your pods would not be safely evicted from your node and shutdown abruptly. Other services talking to those pods might see errors due to the pods exiting abruptly. Some examples of this situation may be caused by a reboot due to security patches or preemption of short lived cloud compute instances.

Kubernetes 1.21 brings graceful node shutdown to beta. Graceful node shutdown gives you more control over some of those unexpected shutdown situations. With graceful node shutdown, the kubelet is aware of underlying system shutdown events and can propagate these events to pods, ensuring containers can shut down as gracefully as possible. This gives the containers a chance to checkpoint their state or release back any resources they are holding.

Note, that for the best availability, even with graceful node shutdown, you should still design your deployments to be resilient to node failures.

How does it work?

On Linux, your system can shut down in many different situations. For example:

  • A user or script running shutdown -h now or systemctl poweroff or systemctl reboot.
  • Physically pressing a power button on the machine.
  • Stopping a VM instance on a cloud provider, e.g. gcloud compute instances stop on GCP.
  • A Preemptible VM or Spot Instance that your cloud provider can terminate unexpectedly, but with a brief warning.

Many of these situations can be unexpected and there is no guarantee that a cluster administrator drained the node prior to these events. With the graceful node shutdown feature, kubelet uses a systemd mechanism called  "Inhibitor Locks" to allow draining in most cases. Using Inhibitor Locks, kubelet instructs systemd to postpone system shutdown for a specified duration, giving a chance for the node to drain and evict pods on the system.

Kubelet makes use of this mechanism to ensure your pods will be terminated cleanly. When the kubelet starts, it acquires a systemd delay-type inhibitor lock. When the system is about to shut down, the kubelet can delay that shutdown for a configurable, short duration utilizing the delay-type inhibitor lock it acquired earlier. This gives your pods extra time to terminate. As a result, even during unexpected shutdowns, your application will receive a SIGTERM,  preStop hooks will execute, and kubelet will properly update Ready node condition and respective pod statuses to the api-server.

node node.js

What is Geek Coin

What is GeekCash, Geek Token

Best Visual Studio Code Themes of 2021

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Hire Top Node JS Developers | Best Node.js Development Company India

Hire dedicated Node JS developers & programmers in India for custom full-stack NodeJS web development projects on hourly/full-time basis. Strict NDA, 16+ years exp & 2500+ clients|450+ Experts

Hire Dedicated Node.js Developers - Hire Node.js Developers

Get business-centric Node.Js development services from expert Node.JS developers. We have expertise in developing & maintaining Node JS apps as per the business requirements.

Node JS Development Company| Node JS Web Developers-SISGAIN

SISGAIN is the top rated node js development company providing professional services on node js web and mobile development.

Why use Node.js for Web Development? Benefits and Examples of Apps

Get Node.js developers & programmers, and dedicated Node coders in India with average 5 years of experience. 15+ Yrs Exp, 500+ Staff, 13800+ Projects, 6800+ Clients.

Node JS Development Company | Hire Node.js Developers

Looking to hire Node js developers? One of the top Node js development companies in India & USA offers cost-effective Node js web development services.