We all make mistakes and we make them quite often. Deleting a pod, a persistence volume or even a whole namespace is an easy thing to do in Kubernetes and if you delete the right one you can say goodbye to your Elasticsearch cluster.

Today we will be going through how to backup our Elasticsearch cluster (Running on Kubernetes) data into Google Cloud Storage and protect it from unexpected data losses.

What will we be seeing?

• Create Google Cloud Storage Bucket
• Create Service Account
• Create a Kubernetes Secret to Hold Your Key
• Configure Elasticsearch To Connect To GCS
• Take a snapshot through Elasticsearch
• Take a snapshot through Kibana

Create Google Cloud Storage Bucket

Let’s get started by creating a Google Cloud Storage Bucket, you can do it by clicking here.

For this example, we will name our bucket elasticsearch-backup .

Create Service Account

Now that we have our GCS bucket, we need to set up a service account that will be used by Elasticsearch to Backup.

Give it a name and grant it permissions as storage.objectAdmin

Download the JSON API key and save it i your computer as gcs_backup_key.json

Create a Kubernetes Secret To Hold Your Key

Now that you have your JSON key file in hands, let’s create a Kubernetes Secret to store that file and allow Elasticsearch to access it.

On the directory where you saved your JSON file, run:

kubectl create secret generic gcs-backup-key --from-file=gcs_backup_key.json=gcs_backup_key.json

Configure Elasticsearch To Connect To GCS

In order to do it, we will need to:

1. Install the Google Cloud Storage plug in
2. Add the JSON key to the configurations of Elasticsearch

Both actions must be done before the cluster actually starts and in order to accomplish it, we will use Init Containers.

Init containers allow us to run commands before the actual entrypoint is run and the Elasticsearch cluster has started. If you followed my Deploy the Elastic Stack with the Elastic Cloud On Kubernetes (ECK) story, you already have an init container configuring the vm.max_map_count that you can see as an example:

initContainers:
  - name: sysctl
  securityContext:
    privileged: true
  command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']

First, let’s use the secret we created before as a volume in our manifest. You can do it by adding the following to the volume sections:

volumes:
- name: gcs-backup-key
  secret:
    secretName: gcs-backup-key

We will need to add two more initContainers to this spec, the first one to install the plugin:

- name: install-plugins
  command:
  - sh
  - -c
  - |
  bin/elasticsearch-plugin install --batch repository-gcs

We will also need another init container to add the JSON key where we will mount our file specified in the previous step and add it to the Elasticsearch Keystore:

- name: add-gcs-key
  command:
  - sh
  - -c
  - |
  echo y | bin/elasticsearch-keystore add-file
  gcs.client.default.credentials_file ./key/gcs_backup_key.json
  volumeMounts:
  - name: gcs-backup-key
    mountPath: "/usr/share/elasticsearch/key"
    readOnly: true

#backup #elasticsearch #snapshot #kubernetes #google-cloud-storage