Migrating Cassandra from one Kubernetes cluster

Migrating Cassandra from one Kubernetes cluster

Our experience withing changing the K8s operator for Cassandra. For about the last six months, we have been successfully using the Rook operator to operate the Cassandra cluster in Kubernetes.

For about the last six months, we have been successfully using the Rook operator to operate the Cassandra cluster in Kubernetes. However, recently we have faced a seemingly trivial task: to change some parameters in the Cassandra config. And that is when we have discovered that the Rook operator is not flexible enough. To make changes, we would have had to clone the repository, modify source code, and rebuild the operator (good knowledge of Go is also a must since the config itself is built into the operator). Obviously, all these activities are very time-consuming.

We have already reviewed the existing operators, and for this time we have decided to give CassKop by Orange a try. Why? It has all the necessary features, such as custom configurations and monitoring, right out-of-the-box.

The challenge

In the real-life story, which we will discuss below, we have decided to combine shifting to another operator with the need to migrate the entire customer’s infrastructure to a new cluster. After the migration of vital workloads was complete, Cassandra remained the only essential application in the old cluster. Of course, we could not afford to lose its data.

So, here are the requirements for the migration and some limitations:

  • The maximum downtime is limited to 2–3 minutes. The idea is to perform this migration simultaneously with rolling out the application to a new cluster;
  • We need to transfer all data without any losses or extra manipulations.

How do you perform such a maneuver? Similarly to the methods described in our article about RabbitMQ migration, we have decided to create a new Cassandra installation in the new Kubernetes cluster, merge two Cassandra installations (in the old and the new cluster), and migrate the data. After the migration is complete, we can remove the old installation.

However, our task was complicated by the fact that the networks in Kubernetes overlap, so it was difficult to establish a connection between clusters. We had to configure routes for each pod on every node, which is a very time-consuming and, as a matter of fact, unreliable approach. The thing is that IP communication is possible between master nodes only. However, Cassandra is running on dedicated nodes. So, we have to first set up a route to the master, and then configure a route to another cluster on the master. Moreover, any pod restart leads to an IP address change, which is another problem. Why? Well, keep on reading to find an answer.

In the rest of the article, we will be using the following definitions for Cassandra clusters:

  • Cassandra-new —our new installation. We will start it up in the new Kubernetes cluster;
  • Cassandra-current — the “former” but still active installation (it is being used by applications at the moment);
  • Cassandra-temporary — the temporary installation that will be deployed next to Cassandra-current during the migration process only.

So what is the catch?

Since the Cassandra-current cluster uses localstorage, you cannot migrate its data to a new cluster directly (as is the case, for example, with vSphere volumes). We will create a temporary installation of Cassandra to solve this problem and use it as kind of a buffer to migrate data.

The overall sequence of actions includes the following steps:

  1. Create the Cassandra-new cluster in the target Kubernetes cluster.
  2. Scale down the Cassandra-new cluster to zero.
  3. Mount new volumes created by PVC to the old cluster.
  4. Create Cassandra-temporary alongside Cassandra-current using the operator so that the temporary cluster uses _Cassandra-new_’s disks.
  5. Scale down the Cassandra-temporary_’s operator to zero (otherwise it will restore the original state of the cluster) and modify the _Cassandra-temporary config in order to merge Cassandra-temporary with Cassandra-current. In the end, we should get a single Cassandra cluster and two datacenters (you can learn more about that and other peculiarities of Cassandra in our [previous_](https://medium.com/flant-com/running-cassandra-in-kubernetes-challenges-and-solutions-9082045a7d93) article)_.
  6. Migrate data between Cassandra-temporary and Cassandra-current datacenters.
  7. Scale Cassandra-current and Cassandra-temporary clusters down to zero and run Cassandra-new in the target K8s cluster (while switching volumes in the process). At the same time, we move applications to the new cluster.

By following the above sequence of actions, you can keep downtime to a minimum.

devops cassandra database-administration kubernetes

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

50+ Useful Kubernetes Tools for 2020 - Part 2

Our original Kubernetes tool list was so popular that we've curated another great list of tools to help you improve your functionality with the platform.

How to Extend your DevOps Strategy For Success in the Cloud?

DevOps and Cloud computing are joined at the hip, now that fact is well appreciated by the organizations that engaged in SaaS cloud and developed applications in the Cloud. During the COVID crisis period, most of the organizations have started using cloud computing services and implementing a cloud-first strategy to establish their remote operations. Similarly, the extended DevOps strategy will make the development process more agile with automated test cases.

DevOps for Databases: Collaboration for Convenience or Practicality?

Devops for Databases look like a perfect match. Database Devops can function like a well-oiled machine that ensures successful implementation of your devops strategy.

What Is DevOps and Is Enterprise DevOps Any Good?

What is DevOps? How are organizations transitioning to DevOps? Is it possible for organizations to shift to enterprise DevOps? Read more to find out!

▷ Kubernetes Training | Certified Kubernetes Administrator Course

Learn Certified Kubernetes Administrator (CKA) Course from the World's Best Online Training Platform - Mindmajix ✔️12 Hrs ✔️Real-Time Use Cases ✔️Job Placement