CREATIVE CHAOS   ▋ blog

restic backup to B2 / Backblaze

PUBLISHED ON 26/03/2022 — EDITED ON 11/12/2023 — SYSOPS

Intro

Replacing duply/duplicity with restic.

Implementation

B2 stuff

One needs to:

  • create a bucket
  • generate and obtain app keys
  • set lifecycle

All of this is out of scope for this article, instructions can be obtained from official B2 documentation if needed.

restic

Create a new directory for all restic configuration files:

mkdir /etc/restic

Create restic-env file that contains the common settings.

vim /etc/restic/restic-env 
export B2_ACCOUNT_ID="xxxxxxxxxxx"
export B2_ACCOUNT_KEY="xxxxxxxxxxxx"
export RESTIC_REPOSITORY="b2:sablun-eu-restic"
export RESTIC_PASSWORD_FILE=/etc/restic/restic-password

You can now run this to load the settings every time you want to do something with restic:

source /etc/restic/restic-env

Or rather put it in your shell rc file (.zshrc, .bashrc,…) so it is allways loaded.

Create a password file and fill it with a password of your selection, do not loose this password, as backup is dead without it:

vim /etc/restic/restic-password

Create exclude list:

vim /etc/restic/restic-exclude
proc/
dev/
mnt/
run/
sys/
tmp/
var/log/
var/cache/apt/archives/

Secure the settings folder, you don’t want anyone else reading it:

chown -R root:root /etc/restic
chmod -R 700 /etc/restic

Initialize the backup:

restic init

backup

Simple backup:

restic backup /

Backup with the use of tags and exclude file:

restic --exclude-file=/etc/restic/restic-exclude --tag auto backup /

mysql

Dump all databases and backup them:

mysqldump --all-databases | restic backup --stdin --stdin-filename all_databases.sql --tag mysql

prune and forget

restic forget --prune --keep-last 60 --keep-tag keep --keep-daily 60 --keep-monthly 24 --keep-yearly 10

validation

$ restic check

using temporary cache in /tmp/restic-check-cache-309345860
repository d0444097 opened (repository version 1) successfully, password is correct
created new cache in /tmp/restic-check-cache-309345860
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[1:22] 100.00%  79 / 79 snapshots
no errors were found

You can use tag keep when manually creating important point in time backup/snapshot, so the prune and forget does not remove it.

backup info

restic snapshots
repository d0444097 opened successfully, password is correct
ID        Time                 Host        Tags        Paths
-------------------------------------------------------------------------
8331ca07  2022-03-25 21:03:20  sablun.org              /
f2ff363f  2022-03-25 21:20:16  sablun.org              /
929759b7  2022-03-25 21:21:01  sablun.org  auto        /
5d568b22  2022-03-25 21:22:59  sablun.org  auto        /
bb9b4e58  2022-03-25 21:26:44  sablun.org  auto        /
1f62a54d  2022-03-25 21:27:43  sablun.org  auto        /
beb141d4  2022-03-25 21:36:21  sablun.org  auto        /
0d90d87e  2022-03-25 21:38:58  sablun.org  mysql       /all_databases.sql
8833e852  2022-03-25 21:41:38  sablun.org  auto        /
72cb95af  2022-03-25 21:42:10  sablun.org  mysql       /all_databases.sql
990927ee  2022-03-26 00:00:01  sablun.org  auto        /
8e7942cc  2022-03-26 00:00:54  sablun.org  mysql       /all_databases.sql
00aff052  2022-03-26 08:24:25  sablun.org  auto        /
b1a6bafc  2022-03-26 08:30:37  sablun.org  mysql       /all_databases.sql
-------------------------------------------------------------------------
14 snapshots

shush

To automate backup, prune and validation, we will use shush.

vim /etc/shush/restic-backup.conf
# "set -o pipefail" is here because we want bash to return an error if any of
# the commands in the pipeline fails.

command=nice -n19 ionice -c3 bash -c -- 'set -o pipefail; (source /etc/restic/restic-env && restic --exclude-file=/etc/restic/restic-exclude --tag auto backup / ; mysqldump --all-databases | restic backup --stdin --stdin-filename all_databases.sql --tag mysql ; restic forget --keep-last 60 --keep-tag keep --keep-daily 60 --keep-monthly 24 --keep-yearly 10 ) 2>&1 | ts "[%b %d %H:%M:%S]"'
lock=notify=root,abort
lockfile=/var/lock/shush-restic-backup.lock
lockmsg=WARNING: backup on host %h has been invoked again, but the previous invocation is still running! (via restic, ran by shush with config file %N)
format=text

[success]
if=$exit == 0
subject=OK: backup on host %h completed. (via restic, ran by shush with config file %N)
to=root
format=text

[failure]
if=$exit != 0
to=root
subject=ERROR: backup on host %h failed! (via restic, ran by shush with config file %N)
format=text

You can test the configuration manually:

shush -c /etc/shush -m restic-backup.conf

Implementing retention and validation once per week.

vim /etc/shush/restic-prune.conf
# "set -o pipefail" is here because we want bash to return an error if any of
# the commands in the pipeline fails.

command=nice -n19 ionice -c3 bash -c -- 'set -o pipefail; (source /etc/restic/restic-env && restic prune && restic check) 2>&1 | ts "[%b %d %H:%M:%S]"'
lock=notify=root,abort
lockfile=/var/lock/shush-restic-prune.lock
lockmsg=WARNING: prune on host %h has been invoked again, but the previous invocation is still running! (via restic, ran by shush with config file %N)
format=text

[success]
if=$exit == 0
subject=OK: prune on host %h completed. (via restic, ran by shush with config file %N)
to=root
format=text

[failure]
if=$exit != 0
to=root
subject=ERROR: prune on host %h failed! (via restic, ran by shush with config file %N)
format=text

crontab

vim /etc/cron.d/restic-backup
# run the shush configuration file for the restic backup.
# Settings are in the /etc/shush/restic-backup.conf and /etc/restic/*.

# unset MAILTO, as shush sends its own mail.
MAILTO=""

# set PATH
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# m h   dom mon dow   user    command
00  00  *   *   *     root    shush -c /etc/shush -m restic-backup.conf
00  03  1   *   *     root    shush -c /etc/shush -m restic-prune.conf

restore

TODO

https://restic.readthedocs.io/en/stable/050_restore.html

Edit 11.12.2023: First time I had to do a recovery, after a mistake with rm.

source /etc/restic/restic-env && restic restore latest --target /tmp/restore --include "/home/b4d/important-data"