BRIX On-Premises > BRIX On-Premises Enterprise / Back up and recover databases with WAL-G and Rclone

Back up and recover databases with WAL-G and Rclone

You can configure BRIX data backup and recovery using the following programs:

WAL-G is a database backup and recovery utility that supports several DBMSs, including PostgreSQL and MongoDB.

It utilizes Continuous Archiving and Point-in-Time Recovery (PITR) to provide reliable data archiving and recovery.

WAL-G supports storing backups in various cloud storage services such as Amazon S3, Google Cloud Storage, Azure Blob Storage, and others.

Rclone is an open-source, multi-threaded, command-line content management program. It allows you to move data to the cloud and other storage services.

This article describes how to back up and restore PostgreSQL 15 and MongoDB 6 in Ubuntu Linux 22.04.

Configure WAL-G for PostgreSQL

Follow the steps below to back up your data using the WAL-G:

First install WAL-G from the official GitHub repository:

wget https://github.com/wal-g/wal-g/releases/download/v3.0.5/wal-g-pg-ubuntu-22.04-amd64.tar.gz
tar -xzf wal-g-pg-ubuntu-22.04-amd64.tar.gz
sudo mv wal-g-pg-ubuntu-22.04-amd64 /usr/local/bin/wal-g-pg

Configure environment variables to access the cloud storage. To do this, open the configuration file /etc/postgresql/15/main/.walg.json and add the following parameters:

{
"WALG_S3_PREFIX": "s3://elma365-backup/postgres",
"AWS_ENDPOINT": "https://storage.yandexcloud.net"
"AWS_ACCESS_KEY_ID": "<your-access-key-id>",
"AWS_SECRET_ACCESS_KEY": "<your-secret-access-key>",
"AWS_REGION": "us-east-1",
"AWS_S3_FORCE_PATH_STYLE": "true",
"WALG_COMPRESSION_METHOD": "brotli",
"WALG_DELTA_MAX_STEPS": "6",
"PGHOST": "/var/run/postgresql",
"PGDATA": "/var/lib/postgresql/15/main",
"PGSSLMODE": "disable"
}

Here:

WALG_S3_PREFIX specifies the path to the S3 repository where backups and .WAL files (Write-Ahead Logs) will be stored.
AWS_ENDPOINT specifies the endpoint for the S3-compatible storage.
AWS_ACCESS_KEY_ID is the access Key ID for authentication to S3-compliant storage.
AWS_SECRET_ACCESS_KEY is the Secret Access Key for authentication in S3-compliant storage.
AWS_REGION is the region where the S3-compatible storage is located.
AWS_S3_S3_FORCE_PATH_STYLE indicates that the path style should be used to access S3 buckets.
WALG_COMPRESSION_METHOD is the compression method used for backups. In this case, the Brotli algorithm is used, which provides a good balance between compression ratio and speed.
WALG_DELTA_MAX_STEPS is the maximum number of steps (delta steps) that WAL-G can use for progressively larger (delta) backups. It allows you to save space by storing only changes between backups.
PGHOST specifies the path to the PostgreSQL socket through which WAL-G will connect to the database. It is used to execute commands that require a connection to PostgreSQL.
PGDATA is the path to the PostgreSQL data directory. WAL-G uses this parameter to determine where PostgreSQL data are stored for correct backup and recovery.
PGSSLMODE indicates that SSL/TLS encryption for connection to PostgreSQL is disabled. This can be useful for local connections where encryption is not required.

Change the file owner:

chown postgres: /etc/postgresql/15/main/.walg.json

To use WAL-G, configure PostgreSQL to work with continuous archiving. To do this, open the configuration file /etc/postgresql/15/main/postgresql.conf and add the following parameters:

archive_mode = on
archive_timeout = 600s
archive_command = '/usr/local/bin/wal-g-pg wal-push "%p" --config /etc/postgresql/15/main/.walg.json >> /var/log/postgresql/archive_command.log 2>&1'
wal_level = replica
restore_command = '/usr/local/bin/wal-g-pg wal-fetch "%f" "%p" --config /etc/postgresql/15/main/.walg.json >> /var/log/postgresql/restore_command.log 2>&1'

To create a full database backup, run the command:

sudo -u postgres /usr/local/bin/wal-g-pg backup-push --config /etc/postgresql/15/main/.walg.json /var/lib/postgresql/15/main/

To restore a database from a backup using WAL-G:

начало внимание

Restore from backup to an empty database. First, make sure you have a backup copy of the database you want to delete. Only then proceed with deletion. For more information, see the PostgreSQL article.

конец внимание

Stop PostgreSQL if it is running:

sudo systemctl stop postgresql

Move or delete the contents of the /var/lib/postgresql/15/main directory:

sudo mv /var/lib/postgresql/15/main /var/lib/postgresql/15/main_old

Create an empty directory for recovery:

sudo mkdir /var/lib/postgresql/15/main
sudo chown postgres:postgres /var/lib/postgresql/15/main

To view the list of database backups, run the command:

sudo -u postgres /usr/local/bin/wal-g-pg backup-list --config /etc/postgresql/15/main/.walg.json

Run the restore command:

sudo -u postgres /usr/local/bin/wal-g-pg backup-fetch /var/lib/postgresql/15/main LATEST --config /etc/postgresql/15/main/.walg.json

Here LATEST points to the last backup, or you can enter the name of the desired backup obtained from the list in the previous step, for example: base_000000000100000000000000000000000000000009.

Start PostgreSQL:

sudo systemctl start postgresql

Configure WAL-G for MongoDB

To back up your data using WAL-G, follow these steps:

Pre-install WAL-G from the official GitHub repository:

wget https://github.com/wal-g/wal-g/releases/download/v3.0.5/wal-g-mongo-ubuntu-22.04-amd64.tar.gz
tar -xzf wal-g-mongo-ubuntu-22.04-amd64.tar.gz
sudo mv wal-g-mongo-ubuntu-22.04-amd64 /usr/local/bin/wal-g-mongo

Configure environment variables to access the cloud storage. Open the configuration file /home/user/.walg.json and add the following parameters:

{
“WALG_S3_PREFIX": ‘s3://elma365-backup/mongo’,
“AWS_ENDPOINT": ‘https://storage.yandexcloud.net’,
“AWS_ACCESS_KEY_ID": ‘<your-access-key-id>’,
“AWS_SECRET_ACCESS_KEY“:”<your-secret-access-key>”,
“AWS_REGION": ‘us-east-1’,
“AWS_S3_FORCE_PATH_STYLE": ‘true’,
“WALG_COMPRESSION_METHOD": ‘brotli’,
“MONGODB_URI": ‘mongodb://superuser:SecretPassword@mongodb-server.your_domain:27017/?authSource=admin&connect=direct&socketTimeoutMS=60000&connectTimeoutMS=10000’
“WALG_STREAM_CREATE_COMMAND“: ‘mongodump --archive --oplog --uri=’mongodb://superuser:SecretPassword@mongodb-server.your_domain:27017/?authSource=admin&connectTimeoutMS=10000'”
“WALG_STREAM_RESTORE_COMMAND": ‘mongorestore --archive --oplogReplay --uri=’mongodb://superuser:SecretPassword@mongodb-server.your_domain:27017/?authSource=admin&connectTimeoutMS=10000'”
}

Here:

WALG_S3_PREFIX specifies the path to S3 where the backups and .WAL files (Write-Ahead Logs) will be stored.
AWS_ENDPOINT specifies the endpoint for S3-compatible storage.
AWS_ACCESS_KEY_ID access Key ID for authentication to S3-compliant storage.
AWS_SECRET_ACCESS_KEY is the secret Access Key for authentication in S3-compliant storage.
AWS_REGION is the region where the S3-compatible storage is located.
AWS_S3_S3_FORCE_PATH_STYLE indicates that the path style should be used to access S3 buckets.
WALG_COMPRESSION_METHOD is the compression method used for backups. In this case, the Brotli algorithm is used, which provides a good balance between compression ratio and speed.
MONGODB_URI is the URI to connect to MongoDB.
WALG_STREAM_CREATE_COMMAND is the command used by WAL-G to back up MongoDB.
ALG_STREAM_RESTORE_COMMAND is the command used by the WAL-G to restore a MongoDB backup.

To create a full database backup, run the command:

/usr/local/bin/wal-g-mongo backup-push --config /home/user/.walg.json

To view the list of database backups, run the command:

/usr/local/bin/wal-g-mongo backup-lis --config /home/user/.walg.json

To restore a database from a backup, run the command:

/usr/local/bin/wal-g-mongo backup-fetch LATEST --config /home/user/.walg.json

начало внимание

Restore from backup to an empty database. First, make sure you have a backup of the database you want to delete. Only then proceed with deletion. For more information, see the MongoDB article.

конец внимание

Here LATEST indicates the last backup, or you can specify the name of the desired backup obtained from the list in the previous step, for example: stream_20250214T093724Z.

Configure Rclone

To install Rclone, use the command:

sudo -v ; curl https://rclone.org/install.sh | sudo bash

To configure Rclone to work with S3:

To configure the new S3 storage, run the command:

rclone config

In the opened window, select the n option to create a new storage.
Enter a name for your storage. For example source-s3 for the source repository and backup-s3 for the target repository.
Select the type of storage (s3).
Select the provider (Other).
Enter the access parameters for your s3 storage:

access_key_id is your access key identifier.
secret_access_key is your secret key.
region is the region of your S3 storage.
endpoint specifies the access point of your S3 storage.

Repeat the steps to configure the second S3 storage.

To copy data from one S3 storage to another, use the rclone copy command:

rclone copy source-s3:bucket-name/path backup-s3:backup-bucket-name/path -v --ignore-errors --fast-list --checksum

Here:

source-s3 is the name of the remote storage configured for source S3.
bucket-name/path is the bucket name and path to the data in the source storage.
backup-s3 is the name of remote storage configured for target S3.
backup-bucket-name/path is the bucket name and path for the backup in the target storage.
-v enables detailed output (verbose mode).
--ignore-errors ignores errors occurring during command execution.
--fast-list speeds up the process of creating a list of files and directories, especially when working with cloud storage such as S3, Google Drive and others.
--checksum enables checksum (hash) checking of files when copying or synchronizing.

database-backup-and-recovery.html dump_db.html

Was this helpful?