Encrypted Linux VPS Backups with restic and a systemd Timer

calendar Oct 31, 2026 / · 14 min read · restic systemd backups encryption vps  ·
Share on: linkedin copy

The restic backup program runs when you execute it and then exits. It is not a daemon, and it has no built-in way of scheduling backups. On a Linux VPS, systemd handles the scheduling; systemd timers in place of cron covers the timer basics, and rsync and cron backups is the unencrypted alternative. Each run produces a snapshot, which is restic's name for the contents of a directory at a specific point in time.

The setup has four parts: an encrypted repository, a script that backs up and prunes, a daily timer that makes up runs missed while the VPS was down, and an integrity check on its own timer. Commands and options below are for restic 0.19.1 and systemd 262.

What restic Encrypts and What You Must Keep

All the data restic stores in the repository is encrypted with AES-256 in counter mode and authenticated with Poly1305-AES (design reference). Your password goes through scrypt, a key derivation function, before restic uses it.

There is no way for restic to recover the password for you. If you lose it, the data in the repository is gone for good. This is the main risk of the setup. The password file sits on the VPS, so if the VPS disk is what you need to restore, that copy of the password is gone too. Keep a second copy somewhere off the server.

The key command lets one repository have several access keys or passwords. One password can live in a file on the server for the timer, and a second one can live in a password manager, so losing the server does not lock you out of the repository.

Initialise the Repository

An SFTP repository on a second machine:

1$ restic -r sftp:user@host:/srv/restic-repo init

For automated backups, restic reads the repository location from the RESTIC_REPOSITORY environment variable. It reads the password from a file named with --password-file or RESTIC_PASSWORD_FILE. A scheduled job has no terminal to type a password into, so use the file.

The scheduler used below is erikw/restic-automatic-backup-scheduler, released under the BSD-3-Clause licence. It needs restic v0.9.6 or later and bash 4.0.0 or later. It stores these settings in a profile, /etc/restic/default.env.sh, which sets RESTIC_PASSWORD_FILE="/etc/restic/pw.txt". pw.txt holds the repository password on a single line. Once you have filled in the profile, source it and initialise the repository:

1# source /etc/restic/default.env.sh
2# restic init

sudo make install-systemd installs the scripts, configs and units. If you copy the files to /etc and /bin by hand, first remove the {{ INSTALL_PREFIX }} placeholder from the source files. Every code block below still contains it.

The project targets Backblaze B2 by default. To use another backend, such as the SFTP repository above, change a few settings variables in the profile and some restic arguments inside restic_backup.sh.

The Backup Script

This is the main section of restic_backup.sh, Copyright 2018 Erik Westrup, used under the BSD-3-Clause licence, with the environment-variable checks included. Left out: the helper functions that log backup stats and write the notification file, and the optional summary-stats block that uses them at the end of the file.

  1#!/usr/bin/env bash
  2# Copyright 2018 Erik Westrup. BSD-3-Clause licence:
  3# https://github.com/erikw/restic-automatic-backup-scheduler/blob/main/LICENSE
  4set -o errexit
  5set -o pipefail
  6[[ "${TRACE-0}" =~ ^1|t|y|true|yes$ ]] && set -o xtrace
  7
  8# Clean up lock if we are killed.
  9# If killed by systemd, like $(systemctl stop restic), then it kills the whole cgroup and all it's subprocesses.
 10# However if we kill this script ourselves, we need this trap that kills all subprocesses manually.
 11exit_hook() {
 12	echo "In exit_hook(), being killed" >&2
 13	jobs -p | xargs kill
 14	restic unlock
 15}
 16trap exit_hook INT TERM
 17
 18# Assert that all needed environment variables are set.
 19assert_envvars() {
 20	local varnames=("$@")
 21	for varname in "${varnames[@]}"; do
 22		if [ -z ${!varname+x} ]; then
 23			printf "%s must be set for this script to work.\n\nDid you forget to source a {{ INSTALL_PREFIX }}/etc/restic/*.env.sh profile in the current shell before executing this script?\n" "$varname" >&2
 24			exit 1
 25		fi
 26	done
 27}
 28
 29warn_on_missing_envvars() {
 30	local unset_envs=()
 31	local varnames=("$@")
 32	for varname in "${varnames[@]}"; do
 33		if [ -z "${!varname-}" ]; then
 34			unset_envs=("${unset_envs[@]}" "$varname")
 35		fi
 36	done
 37
 38	if [ ${#unset_envs[@]} -gt 0 ]; then
 39		printf "The following env variables are recommended, but have not been set. This script may not work as expected: %s\n" "${unset_envs[*]}" >&2
 40	fi
 41}
 42
 43assert_envvars \
 44	RESTIC_BACKUP_PATHS RESTIC_BACKUP_TAG \
 45	RESTIC_BACKUP_EXCLUDE_FILE RESTIC_BACKUP_EXTRA_ARGS RESTIC_REPOSITORY RESTIC_VERBOSITY_LEVEL \
 46	RESTIC_RETENTION_HOURS RESTIC_RETENTION_DAYS RESTIC_RETENTION_MONTHS RESTIC_RETENTION_WEEKS RESTIC_RETENTION_YEARS
 47
 48warn_on_missing_envvars \
 49	B2_ACCOUNT_ID B2_ACCOUNT_KEY B2_CONNECTIONS \
 50	RESTIC_PASSWORD_FILE
 51
 52# Convert to arrays, as arrays should be used to build command lines. See https://github.com/koalaman/shellcheck/wiki/SC2086
 53IFS=':' read -ra backup_paths <<< "$RESTIC_BACKUP_PATHS"
 54
 55# Convert to array, an preserve spaces. See #111
 56backup_extra_args=( )
 57if [ -n "$RESTIC_BACKUP_EXTRA_ARGS" ]; then
 58	while IFS= read -r -d ''; do
 59		backup_extra_args+=( "$REPLY" )
 60	done < <(xargs printf '%s\0' <<<"$RESTIC_BACKUP_EXTRA_ARGS")
 61fi
 62
 63B2_ARG=
 64[ -z "${B2_CONNECTIONS+x}" ] || B2_ARG=(--option b2.connections="$B2_CONNECTIONS")
 65
 66# If you need to run some commands before performing the backup; create this file, put them there and make the file executable.
 67PRE_SCRIPT="{{ INSTALL_PREFIX }}/etc/restic/pre_backup.sh"
 68test -x "$PRE_SCRIPT" && "$PRE_SCRIPT"
 69
 70# Set up exclude files: global + path-specific ones
 71# NOTE that restic will fail the backup if not all listed --exclude-files exist. Thus we should only list them if they are really all available.
 72## Global backup configuration.
 73exclusion_args=(--exclude-file "$RESTIC_BACKUP_EXCLUDE_FILE")
 74## Self-contained backup exclusion files per backup path. E.g. having an USB disk at /mnt/media in RESTIC_BACKUP_PATHS,
 75# then a file /mnt/media/.backup_exclude.txt will automatically be detected and used:
 76for backup_path in "${backup_paths[@]}"; do
 77	if [ -f "$backup_path/.backup_exclude.txt" ]; then
 78		exclusion_args=("${exclusion_args[@]}" --exclude-file "$backup_path/.backup_exclude.txt")
 79	fi
 80done
 81
 82# --one-file-system is not supported on Windows (=msys).
 83FS_ARG=
 84test "$OSTYPE" = msys || FS_ARG=--one-file-system
 85
 86# NOTE start all commands in background and wait for them to finish.
 87# Reason: bash ignores any signals while child process is executing and thus the trap exit hook is not triggered.
 88# However if put in subprocesses, wait(1) waits until the process finishes OR signal is received.
 89# Reference: https://unix.stackexchange.com/questions/146756/forward-sigterm-to-child-in-bash
 90
 91# Remove locks from other stale processes to keep the automated backup running.
 92restic unlock &
 93wait $!
 94
 95# Do the backup!
 96# See restic-backup(1) or http://restic.readthedocs.io/en/latest/040_backup.html
 97# --one-file-system makes sure we only backup exactly those mounted file systems specified in $RESTIC_BACKUP_PATHS, and thus not directories like /dev, /sys etc.
 98# --tag lets us reference these backups later when doing restic-forget.
 99restic backup \
100	--verbose="$RESTIC_VERBOSITY_LEVEL" \
101	$FS_ARG \
102	--tag "$RESTIC_BACKUP_TAG" \
103	"${B2_ARG[@]}" \
104	"${exclusion_args[@]}" \
105	"${backup_extra_args[@]}" \
106	"${backup_paths[@]}" &
107wait $!
108
109# Dereference and delete/prune old backups.
110# See restic-forget(1) or http://restic.readthedocs.io/en/latest/060_forget.html
111# --group-by only the tag and path, and not by hostname.
112restic forget \
113	--verbose="$RESTIC_VERBOSITY_LEVEL" \
114	--tag "$RESTIC_BACKUP_TAG" \
115	"${B2_ARG[@]}" \
116	--prune \
117	--group-by "paths,tags" \
118	--keep-hourly "$RESTIC_RETENTION_HOURS" \
119	--keep-daily "$RESTIC_RETENTION_DAYS" \
120	--keep-weekly "$RESTIC_RETENTION_WEEKS" \
121	--keep-monthly "$RESTIC_RETENTION_MONTHS" \
122	--keep-yearly "$RESTIC_RETENTION_YEARS" &
123wait $!
124
125# Check repository for errors.
126# NOTE this takes much time (and data transfer from remote repo?), do this in a separate systemd.timer which is run less often.
127#restic check &
128#wait $!
129
130echo "Backup & cleaning is done."

The script runs under errexit and pipefail, so the first failing command stops it. Setting TRACE=1 turns on xtrace. If a required variable is unset, assert_envvars prints its name and exits with status 1 before restic runs. warn_on_missing_envvars only prints a warning, for the B2 credentials and RESTIC_PASSWORD_FILE.

If a signal reaches the script, the INT/TERM trap kills the child jobs and runs restic unlock. Every restic command runs in the background followed by wait $!. While a foreground child is running, bash ignores signals, so without this pattern the trap would never fire.

The script runs restic unlock first to clear locks left behind by stale processes. On its own, restic also tests existing locks on its own. A lock with a timestamp older than 30 minutes is stale. For a lock created on the same machine, restic also checks whether the owning process is still alive.

--one-file-system stops restic from crossing filesystem boundaries and subvolumes. Virtual filesystems such as /proc count as separate filesystems, so they are excluded too. On a VPS where a data volume is mounted separately, put that mount point in RESTIC_BACKUP_PATHS (colon-separated). Otherwise a backup of / skips it.

--tag marks each snapshot so that forget can keep or remove snapshots by tag. --keep-daily n works per day: for the last n days that have one or more snapshots, it keeps only the most recent snapshot of each day. --prune runs prune automatically if forget removed any snapshots. The repository is locked while prune runs, and backups cannot complete. If two profiles write to one repository, one profile's prune can block the other's backup.

By default, restic groups snapshots by hostname and backup paths. --group-by "paths,tags" drops the hostname, so renaming the VPS does not start a second set of snapshots that retention handles separately.

Two failure points to know about:

  • A backup fails if any listed --exclude-file is missing. The per-path files are only added when they exist, but RESTIC_BACKUP_EXCLUDE_FILE must always exist.
  • B2_ARG= assigns an empty string. If B2_CONNECTIONS is unset, "${B2_ARG[@]}" passes restic one empty argument instead of none. On a non-B2 backend, test a run with B2_CONNECTIONS unset before you rely on the timer.

Exclude Files and Dry Runs

The exclude file ignores empty lines and lines starting with #. The file does not expand a tilde (~), so write $HOME or the full path. --exclude-caches excludes the contents of any folder that contains a CACHEDIR.TAG file but keeps the tag file itself. To add it, put it in RESTIC_BACKUP_EXTRA_ARGS, which the script splits into an array.

Excludes do not apply to sources passed on the command line by name. That covers only those exact paths, so files inside a directory you back up should still be filtered. So an exclude pattern cannot remove an entry you listed in RESTIC_BACKUP_PATHS.

To test the rules, run a dry run. --dry-run (-n) reports what restic would do without writing to the repository. Add -vv to get a list of the changes restic would make, and grep "added" narrows that list:

1# source /etc/restic/default.env.sh
2# restic backup --dry-run -vv --one-file-system --exclude-file "$RESTIC_BACKUP_EXCLUDE_FILE" /home | grep "added"

Databases: Use --stdin-from-command

If you pipe a dump into restic backup --stdin and mysqldump fails to connect, restic still creates a snapshot, and that snapshot is empty. With --stdin-from-command, a non-zero exit code from the dump command makes restic cancel the backup:

1$ restic -r /srv/restic-repo backup --stdin-from-command -- mysqldump --host example mydb

The -- stops restic from parsing the dump command's arguments as its own. If you have to use --stdin, set set -o pipefail in the calling shell. Version 0.19.1 of restic has --stdin-from-command, but the scheduler script does not use it. Run restic backup --help to confirm that your installed restic has the option.

The Service Unit

restic-backup@.service:

 1[Unit]
 2Description=Backup with restic to Backblaze B2
 3# Email on failure require special setup. See README.md
 4#OnFailure=status-email-user@%n.service
 5# Prevent backup on unmetered connection. Needs special setup. See README.md.
 6#Requires=nm-unmetered-connection.service
 7#After=nm-unmetered-connection.service
 8
 9[Service]
10Type=simple
11Nice=10
12# $HOME or $XDG_CACHE_HOME must be set for restic to find /root/.cache/restic/
13Environment="HOME=/root"
14# pipefail: so that redirecting stderr from the script to systemd-cat does not hide the failed command from OnFailure above.
15# Random sleep (in seconds): in the case of multiple backup profiles. Many restic instances started at the same time could case high load or network bandwidth usage.
16# `systemd-cat` allows showing the restic output to the systemd journal
17ExecStart=/bin/bash -c 'set -o pipefail; ps cax | grep -q restic && sleep $(shuf -i 0-300 -n 1); source {{ INSTALL_PREFIX }}/etc/restic/%I.env.sh && {{ INSTALL_PREFIX }}/bin/restic_backup.sh 2>&1 | systemd-cat'

The text after @ selects the profile. restic-backup@default.service sources /etc/restic/default.env.sh. For a second profile, copy the env file and enable the timer under the new name.

The unit sets Environment="HOME=/root" because without HOME or XDG_CACHE_HOME, restic cannot find its cache under /root/.cache/restic/. The script's output goes through systemd-cat into the journal. Without pipefail, the pipeline's exit status would be systemd-cat's, and a failed backup would look like a success to OnFailure=. The OnFailure= and unmetered-connection lines are commented out. Each needs extra setup before it works.

A scheduled backup should check for restic instances that are already running before it starts. Here, systemd covers part of that. A timer activates the service with the same name, and if that service is still active when the timer fires again, systemd leaves it running and does not restart it. So a slow default backup does not stack up behind its own timer.

Separate profiles are a different case. The ps cax | grep -q restic && sleep line waits a random 0 to 300 seconds when another restic process is running. That spreads out start times but does not take a lock, so two profiles can still overlap.

The Timer: OnCalendar and Persistent=true

restic-backup@.timer:

1[Unit]
2Description=Backup with restic on schedule
3
4[Timer]
5OnCalendar=daily
6Persistent=true
7
8[Install]
9WantedBy=timers.target

OnCalendar= sets a wall-clock timer. daily expands to *-*-* 00:00:00. To change the schedule, edit OnCalendar in /usr/lib/systemd/system/restic-backup@.timer, then check the expression and see its next run time:

1$ systemd-analyze calendar daily

With Persistent=true, systemd stores the time the service last ran on disk. If the VPS was powered off when a run was due, the timer runs the service once it comes back up. This only works for timers that use OnCalendar=. The setting defaults to false and was added in systemd 212. A catch-up run still waits for RandomizedDelaySec=, which defaults to 0. If you add a random delay to spread out the load on a shared backend, the catch-up run is delayed by the same amount.

Other timer behaviour that affects backups:

  • A timer with OnCalendar= gets After= dependencies on time-set.target and time-sync.target, so a VPS whose clock is wrong at boot does not fire at a bogus time.
  • AccuracySec= defaults to 1min, so a run can start up to a minute after 00:00:00.
  • If a calendar timer elapsed several times while the system was sleeping the whole time, the service runs only once.
  • systemctl clean --what=state on the timer unit removes the timestamp file that Persistent= keeps.

Enable the timer. --now also starts the first backup:

1# systemctl enable --now restic-backup@default.timer
2# systemctl list-timers | grep restic

Verify: restic check on Its Own Timer

Run restic check after a backup to verify that all data is stored properly. By default, check does not verify that the pack files on disk are unmodified, because that means reading a copy of every pack file. --read-data does read every pack. On a remote repository, that means downloading all of them, which takes longer than the default check and costs extra bandwidth.

--read-data-subset checks part of the data at a time. n/t splits the packs into t parts and checks part n, so 1/5 through 5/5 cover the whole repository over five runs. x% checks a random percentage, and a size such as 50M checks a random subset of that size:

1$ restic -r /srv/restic-repo check --read-data-subset=1/5
2$ restic -r /srv/restic-repo check --read-data-subset=10%

The backup script leaves restic check commented out, with a note to run it from a separate timer that fires less often. The project ships one:

1# systemctl enable --now restic-check@default.timer

Reading the Results

Follow a run in the journal:

1# journalctl -f --lines=50 -u restic-backup@default

restic backup returns one of these exit codes:

  • 0: the backup succeeded and the snapshot contains all source files.
  • 1: fatal error, no snapshot was created.
  • 3: some source files could not be read. A snapshot was created with the remaining files.

Because the script uses errexit, a 3 from wait $! stops the script before forget runs. The unit fails even though a partial snapshot exists. Read the journal for the files restic could not open before you treat that run as a lost day.

If the repository runs out of disk space partway through a run, no new snapshot is created, because restic writes the snapshot only at the successful end of the backup. Earlier snapshots are still there and still work.

By default, restic creates a new snapshot on every run even when nothing changed since the parent snapshot. --skip-if-unchanged skips that empty snapshot. If you set it through RESTIC_BACKUP_EXTRA_ARGS, remember that --keep-daily counts days that have snapshots, so quiet days no longer use up retention slots.

References

Posts in this series