How We Minimized the Overhead of Kubernetes in our Job System

How We Minimized the Overhead of Kubernetes in our Job System

Learn How We Minimized the Overhead of Kubernetes in our Job System. Solving performance problems when moving an application to Kubernetes.

If you’re running a lot of machines, you probably know that Kubernetes can offer significant management and scalability benefits. But these benefits aren’t free: Kubernetes and container runtime overhead can be significant. Compounding the issue, a poorly-configured or naively-deployed Kubernetes can lead to underutilization of all of your machines. At Datadog, we moved our existing job system to Kubernetes. It took substantially more CPU time than before, yet completed jobs at a 40-50% slower rate.

This post describes how we solved this performance regression. The solution involved some performance experiment design, light performance tuning, and timing analysis to get back to parity. We also answered a key deployment question: what does the per-pod overhead look like?

Experiment setup

Our initial Kubernetes experiment was mainly about deployment, not about performance. Our job system used a worker-coordinator pattern with coordinators called “parents”, which dispatched work to each node and monitored their progress.

tbd kubernetes

What is Geek Coin

What is GeekCash, Geek Token

Best Visual Studio Code Themes of 2021

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

50+ Useful Kubernetes Tools for 2020 - Part 2

Our original Kubernetes tool list was so popular that we've curated another great list of tools to help you improve your functionality with the platform.

Kubernetes in the Cloud: Strategies for Effective Multi Cloud Implementations

This article explains how you can leverage Kubernetes to reduce multi cloud complexities and improve stability, scalability, and velocity.

Kubernetes vs Docker

Get Hands-on experience on Kubernetes and the best comparison of Kubernetes over the DevOps at your place at Kubernetes training

Typical flow for deploying applications to Kubernetes

Get Hands-on experience on Kubernetes and the best comparison of Kubernetes over the DevOps at your place at Kubernetes training

Microsoft Announces General Availability Of Bridge To Kubernetes

Microsoft announced the general availability of Bridge to Kubernetes, formerly known as Local Process with Kubernetes.