Coinbase recently wrote about why Kubernetes is not part of their technology stack. Coinbase uses containers, but they run them in VMs. For deployments, they use Odin, its open-source solution for deploying their services in VMs as auto-scaling groups. Adopting Kubernetes adds unnecessary complexity to their current deployment pipeline. Moreover, they’d prefer to explore other options like Fargate or ECS before directly choosing Kubernetes. Coinbase expressed that Kubernetes is not the right tool for them at the moment.
From a technological perspective, Kubernetes doesn’t solve any of Coinbase’s customer problems. Conversely, for them, Kubernetes creates a new set of challenges. For instance, they’ll need to dedicate a team to build the necessary infrastructure to run their services. Also, they’ll have to translate their current security practices for Kubernetes. Moreover, Coinbase said that existing managed services from cloud providers, like EKS or GKE, are not mature enough yet. For instance, if they had to upgrade a cluster, they said it would require “a much more operationally-heavy focus” than they currently have.
When someone asked Kelsey Hightower what his thoughts were on Coinbase not using Kubernetes, he replied:
Coinbase built and maintains their own platform that’s working for them. Coinbase provided an analysis worth studying. The major takeaway for me: asking people to manage their own Kubernetes cluster is like asking people to manage their own hypervisors when they just want VMs.
Coinbase’s existing technology stack consists primarily of containers running in EC2 instances. For application service discovery, they use Route53 in conjunction with application load balancers and Envoy. They scale their services through auto-scaling groups (ASGs). They also use lambda functions to schedule deployments through step functions. Coinbase uses Odin, the orchestration platform they built to deploy their services as ASGs, and Codeflow, their internal tool to manage deployments through a UI. Odin has all the logic for doing deployments progressively using health checks and can even perform rollbacks when needed.
Coinbase defines the desired state specifications of their services like instance type or security groups through a JSON manifest, somewhat similar to a Kubernetes YAML manifest. In this regard, Drew Rothstein, director of engineering at Coinbase, said:
We enable the same key features of Kubernetes: a single deploy + rollback button in Codeflow, scaling based on some defined heuristic (we support custom AWS metrics or standard CPU metrics), and re-scheduling/moving of your containers if your VM dies or becomes unhealthy in your ASG.
#aws #kubernetes #devops #coinbase
