search string side menu

BRIX On-Premises > BRIX On-Premises Enterprise / Back up and recover databases with WAL-G and Rclone

Back up and recover databases with WAL-G and Rclone

You can configure BRIX data backup and recovery using the following programs:

  • WAL-G is a database backup and recovery utility that supports several DBMSs, including PostgreSQL and MongoDB.
     
    It utilizes Continuous Archiving and Point-in-Time Recovery (PITR) to provide reliable data archiving and recovery.
     
    WAL-G supports storing backups in various cloud storage services such as Amazon S3, Google Cloud Storage, Azure Blob Storage, and others.
  • Rclone is an open-source, multi-threaded, command-line content management program. It allows you to move data to the cloud and other storage services.

This article describes how to back up and restore PostgreSQL 15 and MongoDB 6 in Ubuntu Linux 22.04.

Configure WAL-G for PostgreSQL

With WAL-G you can:

Backup for PostgreSQL

Follow the steps below to back up your data using WAL-G:

  1. First install WAL-G from the official GitHub repository:

wget https://github.com/wal-g/wal-g/releases/download/v3.0.5/wal-g-pg-ubuntu-22.04-amd64.tar.gz
tar -xzf wal-g-pg-ubuntu-22.04-amd64.tar.gz
sudo mv wal-g-pg-ubuntu-22.04-amd64 /usr/local/bin/wal-g-pg

  1. Configure environment variables to access the cloud storage. To do this, open the configuration file /etc/postgresql/15/main/.walg.json and add the following parameters:

{
"WALG_S3_PREFIX": "s3://elma365-backup/postgres",
"AWS_ENDPOINT": "https://storage.yandexcloud.net"
"AWS_ACCESS_KEY_ID": "<your-access-key-id>",
"AWS_SECRET_ACCESS_KEY": "<your-secret-access-key>",
"AWS_REGION": "us-east-1",
"AWS_S3_FORCE_PATH_STYLE": "true",
"WALG_COMPRESSION_METHOD": "brotli",
"WALG_DELTA_MAX_STEPS": "6",
"PGHOST": "/var/run/postgresql",
"PGDATA": "/var/lib/postgresql/15/main",
"PGSSLMODE": "disable"
}

Where:

  • WALG_S3_PREFIX specifies the path to the S3 repository where backups and .WAL files (Write-Ahead Logs) will be stored.
  • AWS_ENDPOINT specifies the endpoint for the S3-compatible storage.
  • AWS_ACCESS_KEY_ID is the access Key ID for authentication to S3-compliant storage.
  • AWS_SECRET_ACCESS_KEY is the Secret Access Key for authentication in S3-compliant storage.
  • AWS_REGION is the region where the S3-compatible storage is located.
  • AWS_S3_S3_FORCE_PATH_STYLE indicates that the path style should be used to access S3 buckets.
  • WALG_COMPRESSION_METHOD is the compression method used for backups. In this case, the Brotli algorithm is used, which provides a good balance between compression ratio and speed.
  • WALG_DELTA_MAX_STEPS is the maximum number of steps (delta steps) that WAL-G can use for progressively larger (delta) backups. It allows you to save space by storing only changes between backups.
  • PGHOST specifies the path to the PostgreSQL socket through which WAL-G will connect to the database. It is used to execute commands that require a connection to PostgreSQL.
  • PGDATA is the path to the PostgreSQL data directory. WAL-G uses this parameter to determine where PostgreSQL data are stored for correct backup and recovery.
  • PGSSLMODE indicates that SSL/TLS encryption for connection to PostgreSQL is disabled. This can be useful for local connections where encryption is not required.
  1. Change the file owner:

chown postgres: /etc/postgresql/15/main/.walg.json

  1. To use WAL-G, configure PostgreSQL to work with continuous archiving. To do this, open the configuration file /etc/postgresql/15/main/postgresql.conf and add the following parameters:

archive_mode = on
archive_timeout = 600s
archive_command = '/usr/local/bin/wal-g-pg wal-push "%p" --config /etc/postgresql/15/main/.walg.json >> /var/log/postgresql/archive_command.log 2>&1'
wal_level = replica
restore_command = '/usr/local/bin/wal-g-pg wal-fetch "%f" "%p" --config /etc/postgresql/15/main/.walg.json >> /var/log/postgresql/restore_command.log 2>&1'

  1. To create a full database backup, run the command:

sudo -u postgres /usr/local/bin/wal-g-pg backup-push --config /etc/postgresql/15/main/.walg.json /var/lib/postgresql/15/main/

Data restore for PostgreSQL

Important: restore from backup to an empty database. First, make sure you have a backup copy of the database you want to delete. Only then proceed with deletion. Read more about preparing for restoration in the PostgreSQL article.

To restore a database from a backup using WAL-G, do the following:

  1. Stop PostgreSQL if it is running:

sudo systemctl stop postgresql

  1. Move or delete the contents of the /var/lib/postgresql/15/main directory:

sudo mv /var/lib/postgresql/15/main /var/lib/postgresql/15/main_old

  1. Create an empty directory for recovery:

sudo mkdir /var/lib/postgresql/15/main
sudo chown postgres:postgres /var/lib/postgresql/15/main

  1. To view the list of database backups, run the command:

sudo -u postgres /usr/local/bin/wal-g-pg backup-list --config /etc/postgresql/15/main/.walg.json

  1. Run the restore command:

sudo -u postgres /usr/local/bin/wal-g-pg backup-fetch /var/lib/postgresql/15/main LATEST --config /etc/postgresql/15/main/.walg.json

Here LATEST points to the last backup. You can also enter the name of the desired backup obtained from the list in the previous step, for example: base_000000000100000000000000000000000000000009.

  1. Start PostgreSQL:

sudo systemctl start postgresql

Configure WAL-G for MongoDB

With WAL-G you can:

Backup for MongoDB

To back up your data using WAL-G, follow these steps:

  1. Pre-install WAL-G from the official GitHub repository:

wget https://github.com/wal-g/wal-g/releases/download/v3.0.5/wal-g-mongo-ubuntu-22.04-amd64.tar.gz
tar -xzf wal-g-mongo-ubuntu-22.04-amd64.tar.gz
sudo mv wal-g-mongo-ubuntu-22.04-amd64 /usr/local/bin/wal-g-mongo

  1. Configure environment variables to access the cloud storage. Open the configuration file /home/user/.walg.json and add the following parameters:

{
“WALG_S3_PREFIX": ‘s3://elma365-backup/mongo’,
“AWS_ENDPOINT": ‘https://storage.yandexcloud.net’,
“AWS_ACCESS_KEY_ID": ‘<your-access-key-id>’,
“AWS_SECRET_ACCESS_KEY“:”<your-secret-access-key>”,
“AWS_REGION": ‘us-east-1’,
“AWS_S3_FORCE_PATH_STYLE": ‘true’,
“WALG_COMPRESSION_METHOD": ‘brotli’,
“MONGODB_URI": ‘mongodb://superuser:SecretPassword@mongodb-server.your_domain:27017/?authSource=admin&connect=direct&socketTimeoutMS=60000&connectTimeoutMS=10000’
“WALG_STREAM_CREATE_COMMAND“: ‘mongodump --archive --oplog --uri=’mongodb://superuser:SecretPassword@mongodb-server.your_domain:27017/?authSource=admin&connectTimeoutMS=10000'”
“WALG_STREAM_RESTORE_COMMAND": ‘mongorestore --archive --oplogReplay --uri=’mongodb://superuser:SecretPassword@mongodb-server.your_domain:27017/?authSource=admin&connectTimeoutMS=10000'”
}

Where:

  • WALG_S3_PREFIX specifies the path to S3 where the backups and .WAL files (Write-Ahead Logs) will be stored.
  • AWS_ENDPOINT specifies the endpoint for S3-compatible storage.
  • AWS_ACCESS_KEY_ID access Key ID for authentication to S3-compliant storage.
  • AWS_SECRET_ACCESS_KEY is the secret Access Key for authentication in S3-compliant storage.
  • AWS_REGION is the region where the S3-compatible storage is located.
  • AWS_S3_S3_FORCE_PATH_STYLE indicates that the path style should be used to access S3 buckets.
  • WALG_COMPRESSION_METHOD is the compression method used for backups. In this case, the Brotli algorithm is used, which provides a good balance between compression ratio and speed.
  • MONGODB_URI is the URI to connect to MongoDB.
  • WALG_STREAM_CREATE_COMMAND is the command used by WAL-G to back up MongoDB.
  • ALG_STREAM_RESTORE_COMMAND is the command used by the WAL-G to restore a MongoDB backup.
  1. To create a full database backup, run the command:

/usr/local/bin/wal-g-mongo backup-push --config /home/user/.walg.json

  1. To view the list of database backups, run the command:

/usr/local/bin/wal-g-mongo backup-list --config /home/user/.walg.json

Data restore for MongoDB

Important: restore from backup to an empty database. First, make sure you have a backup of the database you want to delete. Only then proceed with deletion. Read more about preparing for restoration in the MongoDB article.

To restore a database from a backup, run the command:

/usr/local/bin/wal-g-mongo backup-fetch LATEST --config /home/user/.walg.json

Here LATEST indicates the last backup. You can also specify the name of the desired backup obtained from the list in the previous step, for example: stream_20250214T093724Z.

Configure Rclone

  1. Install Rclone using the command:

sudo -v ; curl https://rclone.org/install.sh | sudo bash

  1. Configure Rclone to work with S3:
  1. To configure the new S3 storage, run the command:

rclone config

  1. In the opened window, select the n option to create a new storage.
  2. Enter a name for your storage. For example source-s3 for the source repository and backup-s3 for the target repository.
  3. Select the s3 storage type.
  4. Select the Other provider.
  5. Enter the access parameters for your s3 storage:
  • access_key_id is your access key identifier.
  • secret_access_key is your secret key.
  • region is the region of your S3 storage.
  • endpoint specifies the access point of your S3 storage.
  1. Repeat the steps to configure the second S3 storage.
  1. Copy data from one S3 storage to another using the rclone copy command:

rclone copy source-s3:bucket-name/path backup-s3:backup-bucket-name/path -v --ignore-errors --fast-list --checksum

Where:

  • source-s3 is the name of the remote storage configured for source S3.
  • bucket-name/path is the bucket name and path to the data in the source storage.
  • backup-s3 is the name of remote storage configured for target S3.
  • backup-bucket-name/path is the bucket name and path for the backup in the target storage.
  • -v enables detailed output (verbose mode).
  • --ignore-errors ignores errors occurring during command execution.
  • --fast-list speeds up the process of creating a list of files and directories, especially when working with cloud storage such as S3, Google Drive and others.
  • --checksum enables checksum (hash) checking of files when copying or synchronizing.  
database-backup-and-recovery.html dump_db.html