# Back up and recover databases with WAL-G and Rclone

> [HTML Version](backup-and-recovery-with-wal-g-and-rclone.html)

You can configure BRIX data backup and recovery using the following programs:

- WAL-G is a database backup and recovery utility that supports several DBMSs, including [PostgreSQL](#walg-pgsql) and [MongoDB](#walg-mongo).  
  
It utilizes Continuous Archiving and Point-in-Time Recovery (PITR) to provide reliable data archiving and recovery.  
  
WAL-G supports storing backups in various cloud storage services such as Amazon S3, Google Cloud Storage, Azure Blob Storage, and others.

- [Rclone](#rclone) is an open-source, multi-threaded, command-line content management program. It allows you to move data to the cloud and other storage services.

This article describes how to back up and restore PostgreSQL 15 and MongoDB 6 in Ubuntu Linux 22.04.

## Configure WAL-G for PostgreSQL

With WAL-G you can:

- Back up your data.

- [Restore data from a backup](#psql-restore).

### Backup for PostgreSQL

Follow the steps below to back up your data using WAL-G:

1. First install WAL-G from the [official GitHub repository](https://github.com/wal-g/wal-g):

````
wget https://github.com/wal-g/wal-g/releases/download/v3.0.5/wal-g-pg-ubuntu-22.04-amd64.tar.gz  
tar -xzf wal-g-pg-ubuntu-22.04-amd64.tar.gz   
sudo mv wal-g-pg-ubuntu-22.04-amd64 /usr/local/bin/wal-g-pg

2. ````
Configure environment variables to access the cloud storage. To do this, open the configuration file \[OBJECT\] and add the following parameters:

````
\{  
"WALG\_S3\_PREFIX": "s3://brix365-backup/postgres",  
"AWS\_ENDPOINT": "https://storage.yandexcloud.net"  
"AWS\_ACCESS\_KEY\_ID": "<your-access-key-id>",  
"AWS\_SECRET\_ACCESS\_KEY": "<your-secret-access-key>",  
"AWS\_REGION": "us-east-1",  
"AWS\_S3\_FORCE\_PATH\_STYLE": "true",  
"WALG\_COMPRESSION\_METHOD": "brotli",  
"WALG\_DELTA\_MAX\_STEPS": "6",  
"PGHOST": "/var/run/postgresql",  
"PGDATA": "/var/lib/postgresql/15/main",  
"PGSSLMODE": "disable"  
\}

````
Where:

- \[OBJECT\] specifies the path to the S3 repository where backups and **.WAL** files (Write-Ahead Logs) will be stored.

- \[OBJECT\] specifies the endpoint for the S3-compatible storage.

- \[OBJECT\] is the access Key ID for authentication to S3-compliant storage.

- \[OBJECT\] is the Secret Access Key for authentication in S3-compliant storage.

- \[OBJECT\] is the region where the S3-compatible storage is located.

- \[OBJECT\] indicates that the path style should be used to access S3 buckets.

- \[OBJECT\] is the compression method used for backups. In this case, the \[OBJECT\] algorithm is used, which provides a good balance between compression ratio and speed.

- \[OBJECT\] is the maximum number of steps (delta steps) that WAL-G can use for progressively larger (delta) backups. It allows you to save space by storing only changes between backups.

- \[OBJECT\] specifies the path to the PostgreSQL socket through which WAL-G will connect to the database. It is used to execute commands that require a connection to PostgreSQL.

- \[OBJECT\] is the path to the PostgreSQL data directory. WAL-G uses this parameter to determine where PostgreSQL data are stored for correct backup and recovery.

- \[OBJECT\] indicates that SSL/TLS encryption for connection to PostgreSQL is disabled. This can be useful for local connections where encryption is not required.

3. Change the file owner:

````
chown postgres: /etc/postgresql/15/main/.walg.json

4. ````
To use WAL-G, configure PostgreSQL to work with continuous archiving. To do this, open the configuration file \[OBJECT\] and add the following parameters:

````
archive\_mode = on  
archive\_timeout = 600s  
archive\_command = '/usr/local/bin/wal-g-pg wal-push "%p" --config /etc/postgresql/15/main/.walg.json >> /var/log/postgresql/archive\_command.log 2>\&1'  
wal\_level = replica  
restore\_command = '/usr/local/bin/wal-g-pg wal-fetch "%f" "%p" --config /etc/postgresql/15/main/.walg.json >> /var/log/postgresql/restore\_command.log 2>\&1'

5. ````
To create a full database backup, run the command:

````
sudo -u postgres /usr/local/bin/wal-g-pg backup-push --config /etc/postgresql/15/main/.walg.json /var/lib/postgresql/15/main/

### ````
Data restore for PostgreSQL

**Important**: restore from backup to an empty database. First, make sure you have a backup copy of the database you want to delete. Only then proceed with deletion. Read more about preparing for restoration in the [PostgreSQL](postgresql.md#prepare-to-restore) article.

To restore a database from a backup using WAL-G, do the following:

1. Stop PostgreSQL if it is running:

````
sudo systemctl stop postgresql

2. ````
Move or delete the contents of the \[OBJECT\] directory:

````
sudo mv /var/lib/postgresql/15/main /var/lib/postgresql/15/main\_old

3. ````
Create an empty directory for recovery:

````
sudo mkdir /var/lib/postgresql/15/main  
sudo chown postgres:postgres /var/lib/postgresql/15/main

4. ````
To view the list of database backups, run the command:

````
sudo -u postgres /usr/local/bin/wal-g-pg backup-list --config /etc/postgresql/15/main/.walg.json

5. ````
Run the restore command:

````
sudo -u postgres /usr/local/bin/wal-g-pg backup-fetch /var/lib/postgresql/15/main LATEST --config /etc/postgresql/15/main/.walg.json

````
Here \[OBJECT\] points to the last backup. You can also enter the name of the desired backup obtained from the list in the previous step, for example: **base\_000000000100000000000000000000000000000009**.

6. Start PostgreSQL:

````
sudo systemctl start postgresql

## ````
Configure WAL-G for MongoDB

With WAL-G you can:

- Back up your data.

- [Restore data from a backup](#mongo-restore).

### Backup for MongoDB

To back up your data using WAL-G, follow these steps:

1. Pre-install WAL-G from the [official GitHub repository](https://github.com/wal-g/wal-g):

````
wget https://github.com/wal-g/wal-g/releases/download/v3.0.5/wal-g-mongo-ubuntu-22.04-amd64.tar.gz  
tar -xzf wal-g-mongo-ubuntu-22.04-amd64.tar.gz  
sudo mv wal-g-mongo-ubuntu-22.04-amd64 /usr/local/bin/wal-g-mongo

2. ````
Configure environment variables to access the cloud storage. Open the configuration file \[OBJECT\] and add the following parameters:

````
\{  
“WALG\_S3\_PREFIX": ‘s3://brix365-backup/mongo’,  
“AWS\_ENDPOINT": ‘https://storage.yandexcloud.net’,  
“AWS\_ACCESS\_KEY\_ID": ‘<your-access-key-id>’,  
“AWS\_SECRET\_ACCESS\_KEY“:”<your-secret-access-key>”,  
“AWS\_REGION": ‘us-east-1’,  
“AWS\_S3\_FORCE\_PATH\_STYLE": ‘true’,  
“WALG\_COMPRESSION\_METHOD": ‘brotli’,  
“MONGODB\_URI": ‘mongodb://superuser:SecretPassword@mongodb-server.your\_domain:27017/?authSource=admin\&connect=direct\&socketTimeoutMS=60000\&connectTimeoutMS=10000’  
“WALG\_STREAM\_CREATE\_COMMAND“: ‘mongodump --archive --oplog --uri=’mongodb://superuser:SecretPassword@mongodb-server.your\_domain:27017/?authSource=admin\&connectTimeoutMS=10000'”  
“WALG\_STREAM\_RESTORE\_COMMAND": ‘mongorestore --archive --oplogReplay --uri=’mongodb://superuser:SecretPassword@mongodb-server.your\_domain:27017/?authSource=admin\&connectTimeoutMS=10000'”  
\}

````
Where:

- \[OBJECT\] specifies the path to S3 where the backups and **.WAL** files (Write-Ahead Logs) will be stored.

- \[OBJECT\] specifies the endpoint for S3-compatible storage.

- \[OBJECT\] access Key ID for authentication to S3-compliant storage.

- \[OBJECT\] is the secret Access Key for authentication in S3-compliant storage.

- \[OBJECT\] is the region where the S3-compatible storage is located.

- \[OBJECT\] indicates that the path style should be used to access S3 buckets.

- \[OBJECT\] is the compression method used for backups. In this case, the \[OBJECT\] algorithm is used, which provides a good balance between compression ratio and speed.

- \[OBJECT\] is the URI to connect to MongoDB.

- \[OBJECT\] is the command used by WAL-G to back up MongoDB.

- \[OBJECT\] is the command used by the WAL-G to restore a MongoDB backup.

3. To create a full database backup, run the command:

````
/usr/local/bin/wal-g-mongo backup-push --config /home/user/.walg.json

4. ````
To view the list of database backups, run the command:

````
/usr/local/bin/wal-g-mongo backup-list --config /home/user/.walg.json

### ````
Data restore for MongoDB

**Important**: restore from backup to an empty database. First, make sure you have a backup of the database you want to delete. Only then proceed with deletion. Read more about preparing for restoration in the [MongoDB](mongodb.md#prepare-to-restore) article.

To restore a database from a backup, run the command:

````
/usr/local/bin/wal-g-mongo backup-fetch LATEST --config /home/user/.walg.json

````
Here \[OBJECT\] indicates the last backup. You can also specify the name of the desired backup obtained from the list in the previous step, for example: **stream\_20250214T093724Z**.

## Configure Rclone

1. Install Rclone using the command:

````
sudo -v ; curl https://rclone.org/install.sh | sudo bash

2. ````
Configure Rclone to work with S3:

1. To configure the new S3 storage, run the command:

````
rclone config

2. ````
In the opened window, select the **n** option to create a new storage.

3. Enter a name for your storage. For example **source-s3** for the source repository and **backup-s3** for the target repository.

4. Select the **s3 **storage type.

5. Select the **Other **provider.

6. Enter the access parameters for your s3 storage:

- \[OBJECT\] is your access key identifier.

- \[OBJECT\] is your secret key.

- \[OBJECT\] is the region of your S3 storage.

- \[OBJECT\] specifies the access point of your S3 storage.

7. Repeat the steps to configure the second S3 storage.

3. Copy data from one S3 storage to another using the \[OBJECT\] command:

````
rclone copy source-s3:bucket-name/path backup-s3:backup-bucket-name/path -v --ignore-errors --fast-list --checksum

````
Where:

- \[OBJECT\] is the name of the remote storage configured for source S3.

- \[OBJECT\] is the bucket name and path to the data in the source storage.

- \[OBJECT\] is the name of remote storage configured for target S3.

- \[OBJECT\] is the bucket name and path for the backup in the target storage.

- \[OBJECT\] enables detailed output (verbose mode).

- \[OBJECT\] ignores errors occurring during command execution.

- \[OBJECT\] speeds up the process of creating a list of files and directories, especially when working with cloud storage such as S3, Google Drive and others.

- \[OBJECT\] enables checksum (hash) checking of files when copying or synchronizing.