Our product is equipped with built-in, preconfigured backups to support business continuity. In the event of disruptions from internal processes, accidental user changes, or other incidents that may threaten data integrity, it’s essential to know that our system ensures all critical areas are protected. These backups enable fast and effective restoration of data, minimizing downtime and safeguarding your information. This document provides some background to underlying processes behind how we store and backup data.

Backup configuration

Every deployed product comes with a backup preconfigured for all three vital parts required to work properly. We perform daily backups with 7 day retention, but product can be reconfigured to meet any reasonable requirements. In case this document does not cover the scenario customer is interested in please contact Support.

During product deployment

Backup configuration is done during installation/upgrade of gv-essentials chart. When deploying chart with Rancher this is done in Step 2:

Step 2 - ElasticSearch configuration	Step 2 - Backup time configuration

Step 2 - ElasticSearch configuration	Step 2 - Backup time configuration
Open Enabling ElasticSearch backup	Open Setting up time when backup starts

If not utilising Rancher we utilize --set backup argument during gv-essentials installation/upgrade, this is true for both Helm and Airgap deployments.

Todo: Backup Jobs

Responsible for performing backups are the following three jobs, which can be revieved in Rancher visiting Cluster > Workloads > Jobs or inside terminal by:

• consul-backup-cronjob.yaml

• postgres-backup-cronjob.yaml

• elastic-backup-hook.yaml

Todo: this needs to be updated

MinIO and MinIO Console

We are using MinIO for accessing backups, either via web interface (MinIO Console) or utilizing terminal. It provides a user-friendly interface for managing product backups, making it easy to view, retrieve, and organize data efficiently. It supports S3-compatible storage, enabling seamless access and integration with other backup solutions for reliable data redundancy and high availability. Additionally, MinIO’s console allows for simplified configuration of storage policies, versioning, and data protection settings.

Retrieving MinIO credentials

Using terminal to retrieve MinIO credentials

If you are connected to the cluster you can run the following command to retrieve MinIO username:

And for the password:

Using Rancher to retrieve MinIO credentials

To retrieve username and password for MinIO using Rancher, locate your cluster and then click on

Storage > Secrets > gv-essentials-minio

Using backup pod

Backed-up data can only be accessed from terminal from within the backup pod, providing an additional level of security and preventing unauthorized access from outside the pod environment. This isolation ensures that data is only accessible within a controlled scope, reducing exposure to potential breaches. Additionally, it limits interaction with the data, further protecting it from malicious threats.

Running these two commands will result in two strings being produced on the screen, username and password:

We can now connect to the pod and initialize MinIO:

Accessing Consul backup using terminal

Let’s see what it takes to retrieve latest Consul backup from MinIO.

What we’ve done here is:

• retrieve MinIO username and password

• started backup pod

• set alias for mc (minio client, a cli tool) to access our backups

• listed all the files inside consul-backup/

• copied the most recent file from a MinIO bucket to inside the pod

• archive it using tar into consul-backup-2024-10-25.snap.tar.gz

Because we are inside backup pod, we need to open another window and copy that file outside of the backup pod:

And we have our Consul backup ready to be transferred inside consul-backup-2024-10-25.snap.gz

Accessing PostgreSQL backup using terminal

Let’s see what do we need for a full backup of PostgreSQL. This task is a bit more difficult, because we are dealing with multipl PostgreSQL databases. Here’s a complete list:

And here’s a sammple capture process:

The […] above means there were more files listed but for brevity we are only showing the ones with the latest date - the rest is of no interest for us.

We used mc mirror here to copy to a local folder named postgres. Because we of limitations of mc, we needesd to use a little trick: we copied over all postgresql backups into one local folder and archived using tar only the files from the day we are interested in.

We can transfer the archive outside of the pod with:

Our postgres.tar.gz archive now only includes the .dump files created on the 2024-10-25. You can now transfer your backup to the target machine.

Accessing Elasticsearch backup using terminal

Using mc for creating a snapshot

First we need to retrieve Elasticsearch password, see https://getvisibility.atlassian.net/wiki/spaces/GS/pages/871989261/Backup+and+restore#Getting-Elasticsearch-credentials

You will also need MinIO credentials from https://getvisibility.atlassian.net/wiki/spaces/GS/pages/871989261/Backup+and+restore#Retrieving-MinIO-credentials

Run snapshot manually

You can manually run the SLM policy to immediately create a snapshot. This is useful for testing a new policy or taking a snapshot before an upgrade. Manually running a policy doesn’t affect its snapshot schedule.

Check status of last ran snapshot

To check status of the latest Elasticsearch snapshot you can run:

Backing up elasticsearch-backup bucket

Let’s mirror the MinIO bucket elasticsearch-backup locally and archive it for later:

From another terminal:

Our file elastic.tar.gz now contains the full content of our backed up Elasticsearch.

Accessing your backups using MinIO console

Navigate to https://10.20.30.40/gv-essentials-minio/ and enter username/password retrieved in previous chapter:

After logging in you will see the main screen of MinIO console:

Downloading your backup with MinIO Console

Regardless of the method used to access your backups, you need to know how to pick the backup you are interested in. If the product is working as expected - use the latest one. Otherwise, use the one dated last time the product worked as expected.

For a full backup you need all three services backup files: Consul, Elasticsearch and PostgreSQL. Navigate to respective folders, sort by date, find your files and transfer them onto target machine.

Assuming we are interested in latest backup, here is what we need:

If we get into each of these folders (or buckets - this is how they are called here) we see which backups are available, as an example:

Using MinIO Console you can download backups directly from the browser’s window

Restoring from a backup

Full system restore (migration)

Restoring PostgreSQL content

Getting PostgreSQL credentials

To connect to Postgres you will need to get the credentials generated during the installation. The password is located in the secret gv.gv-postgresql.credentials.postgresql.acid.zalan.do in the default namespace.

To get the password run:

The default username is gv. Before we start we need to connect to the backup pod:

In chapter related to backing up postgresql we ended up with archive named postgres.tar.gz that include all the .dump files created on a particular day, 2024-10-25, so let’s now restore that backup into current configuration:

We now need to transfer both the script and files into the backup pod using a second terminal:

In first terminal we need to unpack tar archive, then make our script executable:

Now we have two options:

Replace all content of PostgreSQL with data from your full backup

That is it - you have completed restoring PostgreSQL from a backup.

If you see an error related to password, go back to the beginning of this chapter.

Replace partial PostgreSQL content from previously created backup

In this example we will replace content of bunzi database.

Repeat this for all the databases you want to restore.

Additional information about PostgreSQL backups

If you are looking for additional information about PostgreSQL backups please refer to:

• https://www.postgresql.org/docs/current/backup.html

Restoring Consul content

Getting Consul credentials

You do not need credentials when restoring Consul backups (or in Consul terminology: snapshots).

Replace Consul with data from your full backup

Previously we ended up with a file named consul-backup-2024-10-25.snap.tar.gz containing Consul backed up information. We will now use it to replace current content in Consul. This procedure is a little different that we used for postgres.

First let’s start backup pod:

Using second terminal let’s transfer the file into the backup pod:

Back to the backup pod we can now execute:

If you see response HTTP/1.1 200 OK your action has been successful.

Additional information about Consul backups

For more information about Consul snapshots please refer to:

- https://developer.hashicorp.com/consul/api-docs/snapshot

Restoring Elasticsearch content

Getting Elasticsearch credentials

Elasticsearch username you need is always gv. You can retrieve Elasticsearch password from either Rancher or using the terminal. In Rancher, find your cluster and navigate to:

Storage > Secrets > gv-es-elastic-user

In terminal, use:

Replace all content of Elasticsearch with data from your full backup

First let’s start the backup pod:

From a second terminal copy the required backup into the restore pod:

Inside the pod we need to unpack that archive:

From inside the pod show all available snapshots with:

You can also view a lot more information than this with command below:

But we are really interested in the following:

• snapshot name (“name”)

• if it was successful (“state”)

• and when it started, so we know which one to select (“time_started”)

Let’s put all snaphot related info into a file we can search:

Now let’s grep for the lines we are looking for:

And there it is - we have now a list from which we can choose, let’s assume we are going to pick the newest snapshot, which in this case will be:

Backup pod contains a helper script to restore a snapshot, let’s run it, but first we need to populate the following:

Additional information about Elasticsearch snapshots

For more information about Elasticsearch snapshots please refer to:

• https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html

Troubleshooting

Starting backup pod

Error from server (AlreadyExists): pods "debug" already exists

This error occurs because there's already a pod, named in this example "debug", running in your Kubernetes cluster. The --rm flag is meant to remove the pod after it exits, but if a previous session was interrupted or didn't clean up properly, the pod might still exist.
Here are three ways to resolve this:

• connect to the existing pod with exec -it debug -- /bin/bash

• delete existing pod then run the command again with kubectl delete pod debug

• use different pod name than debug