A backup you can restore from runs on a schedule without you, keeps one copy off-site where a compromised server cannot delete it, captures databases in a consistent state, and has been restored and timed recently. With restic that means an encrypted repository on S3-compatible storage protected by object lock or an append-only server, database dumps before each run, a systemd timer with failure alerts, and a regular restore drill. This guide builds that setup for a Linux server, and the same pieces work on a developer machine.
Decide RPO and RTO before choosing tools
The recovery point objective (RPO) is how much data you can afford to lose, measured in time. The recovery time objective (RTO) is how long the service may stay down during a restore. The owner of a service can answer both in plain words: "we can lose one day of orders at most" and "the shop must be back within four hours".
The worst-case loss of a scheduled backup is roughly the interval between runs plus the time a run needs to reach off-site storage. A nightly job that starts at 02:00 and finishes uploading at 02:40 gives a worst-case RPO of about 24 hours and 40 minutes. If that is too much for a database, back it up more often or add WAL archiving.
RTO is dominated by transfer and rebuild time. Transfer seconds equal the data size in megabits divided by the link speed in Mbit/s. A 200 GB restore is 1,600,000 megabits, so at 100 Mbit/s it takes 16,000 seconds, about 4.4 hours, before the database restore. If that already exceeds the RTO, a remote bucket alone cannot meet it, and you need a local copy or a standby.
The 3-2-1 rule and its 3-2-1-1-0 extension
The 3-2-1 rule means three copies of the data, on two different kinds of storage, with one copy off-site. On a VPS, the production disk is copy one, a local dump or repository on separate storage is copy two, and a bucket at another provider or in another account is copy three. A second partition on the same drive dies with the first, so put the local copy on another device (the NVMe SSD buying guide helps with choosing one) or another host.
Backup vendors popularized the 3-2-1-1-0 extension. The extra 1 is a copy that is offline or immutable, so an attacker with root on the server and its cloud credentials still cannot delete it. The 0 means zero errors in integrity checks and restore tests. Most setups skip that last digit, yet it shows whether the other four are real.
What to back up and what to leave out
Back up what you cannot recreate from a public source in reasonable time: /etc, application code and uploads under /srv or /var/www, home directories with SSH and GPG keys, crontabs, TLS material, database dumps, and a list of installed packages from apt-mark showmanual. On a developer machine, the valuable data is unpushed work, stashes, dotfiles, keys, and documents.
Leave out large or reproducible data: caches, node_modules, virtual environments, build output, container images, model weights, and swap. Exclude live database directories such as /var/lib/postgresql, because a copy taken from a running server is not a usable backup. In restic exclude files, a pattern that starts with / matches an absolute path, and a bare name such as node_modules matches at any depth:
/var/lib/postgresql
/var/lib/docker
/var/cache
/var/tmp
/swapfile
/home/*/.cache
node_modules
.venv
__pycache__
*.tmp
--exclude-caches also skips directories that contain a standard CACHEDIR.TAG file, and --one-file-system keeps restic from crossing into other mounted file systems.
restic basics: repository, encryption, and password
restic stores deduplicated snapshots in a repository on a local disk, SFTP, its own REST server, or object storage, and uploads only new chunks. Encryption cannot be disabled: according to the restic design documentation, all data is encrypted with AES-256 in counter mode and authenticated with Poly1305-AES, and the key is protected by a password stretched with scrypt.
Keep the settings in a root-only environment file shared by interactive commands and scheduled jobs. The repository documentation lists URL formats such as s3:https://server:port/bucket for S3-compatible services.
# /etc/restic/restic.env, mode 600
RESTIC_REPOSITORY=s3:https://s3.example.com/web1-backups
RESTIC_PASSWORD_FILE=/etc/restic/password
RESTIC_CACHE_DIR=/var/cache/restic
AWS_ACCESS_KEY_ID=replace-me
AWS_SECRET_ACCESS_KEY=replace-me
HEARTBEAT_URL=https://monitor.example.net/ping/web1-backup
ALERT_WEBHOOK_URL=https://chat.example.net/hooks/backups
install -d -m 700 /etc/restic
(umask 077; openssl rand -base64 32 > /etc/restic/password)
set -a; . /etc/restic/restic.env; set +a
restic init
restic key add # second password for an offline recovery copy
restic backup --one-file-system --exclude-caches \
--exclude-file=/etc/restic/excludes.txt /etc /root /home /srv /var/www
restic snapshots
The documentation is blunt: losing the password means the data is irrecoverably lost. Keep the password, repository URL, and bucket credentials in a password manager and a sealed offline copy, never only on the server you protect. With a secret manager, RESTIC_PASSWORD_COMMAND can print the password instead of a file.
Retention with forget and prune, integrity with check
restic forget applies a retention policy to each group of snapshots (host and paths by default), and prune deletes data that no snapshot references. The --prune flag runs both:
restic forget --prune --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --keep-yearly 3
restic check --read-data-subset=10%
This keeps the newest snapshot for each of the last 14 days that have one, then one per week for 8 weeks, per month for 12 months, and per year for 3 years. Preview policy changes by adding --dry-run. The forget documentation notes that the repository is locked during prune, so schedule it away from backups and give the backup --retry-lock.
restic check verifies structure and metadata. --read-data downloads every pack, which costs time and egress, while --read-data-subset reads a part: 10% picks a random subset each run, and 3/12 reads a fixed twelfth, as the repository maintenance guide describes. A passing check proves consistency, not that you can restore a service.
An off-site repository that ransomware cannot erase
An attacker with root can read /etc/restic/restic.env. If those credentials can delete objects, the off-site copy can be deleted too. Scope bucket credentials to one bucket, keep administrative credentials off the server, and apply the least-privilege habits from the Linux VPS hardening guide. Then pick one of two protections.
Append-only REST server
The restic rest-server has an --append-only mode that accepts new backups but prevents deleting or modifying existing ones. The restic documentation recommends running forget and prune for such repositories from a separate, well-secured machine and using --keep-within instead of rules such as --keep-weekly, because a compromised client could add fake snapshots with crafted timestamps that take the slots of real ones.
Object lock on S3-compatible storage
S3 Object Lock keeps object versions write-once-read-many. It requires versioning, and in compliance mode nobody, including the account root user, can delete a locked version before its retention date. Since version 0.13, restic sends the upload checksums that locked buckets require, so it works with a default bucket retention (add --endpoint-url for other providers):
aws s3api create-bucket --bucket web1-backups --object-lock-enabled-for-bucket
aws s3api put-object-lock-configuration --bucket web1-backups \
--object-lock-configuration '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"COMPLIANCE","Days":30}}}'
When restic deletes a file in such a bucket, S3 adds a delete marker and the locked version remains. After an attack, copy the bucket as it was, for example with rclone copy --s3-version-at "2026-09-20" offsite:web1-backups /srv/recovered-repo, and restore from the copy. Pruned data keeps using storage until retention ends, so expire noncurrent versions with a lifecycle rule no earlier than that. Test the whole cycle against your provider, since S3-compatible implementations differ.
Consistent PostgreSQL backups
The PostgreSQL documentation on file system level backups is explicit: the server must be shut down for a usable copy of its data directory, and blocking connections is not enough, because tools such as tar do not take an atomic snapshot and the server buffers data internally. An atomic volume snapshot with all data files and WAL does work, since PostgreSQL replays WAL as after a crash.
For most servers, logical dumps are the simplest correct choice. pg_dump produces a consistent export while the database is in use and does not block other users. The directory format allows parallel dumps and restores. Roles are not included, so dump globals separately. This script runs before each backup and keeps the latest dumps on local disk:
#!/usr/bin/env bash
# /usr/local/sbin/pg-dump-for-backup
set -euo pipefail
out=/var/backups/postgresql
install -d -m 700 -o postgres -g postgres "$out"
runuser -u postgres -- pg_dumpall --globals-only --file="$out/globals.sql"
runuser -u postgres -- psql -Atc \
"SELECT datname FROM pg_database WHERE datallowconn AND NOT datistemplate" |
while read -r db; do
rm -rf "$out/$db.new"
runuser -u postgres -- pg_dump --format=directory --jobs=2 --file="$out/$db.new" "$db"
rm -rf "$out/$db"
mv "$out/$db.new" "$out/$db"
done
Use pg_basebackup when a logical restore of a large cluster breaks the RTO or when you need point-in-time recovery. It copies the whole cluster into an empty directory over the replication protocol, needs a role with REPLICATION and a pg_hba.conf entry, and streams the WAL needed to start the copy. pg_verifybackup checks it against the manifest:
sudo -u postgres pg_basebackup --pgdata=/var/backups/pg-base --wal-method=stream --checkpoint=fast
sudo -u postgres pg_verifybackup /var/backups/pg-base
pg_restore rebuilds indexes from their definitions. With large vector indexes, such as the HNSW indexes in the pgvector hybrid search guide, that rebuild can dominate restore time.
Scheduling with a systemd service and timer
A oneshot service runs the dump, then the backup. If the dump fails, the backup does not start and the unit fails. Save the units in /etc/systemd/system/:
# restic-backup.service
[Unit]
Description=restic backup to the off-site repository
Wants=network-online.target
After=network-online.target postgresql.service
OnFailure=backup-alert@%n.service
[Service]
Type=oneshot
EnvironmentFile=/etc/restic/restic.env
CacheDirectory=restic
Nice=10
IOSchedulingClass=idle
ExecStartPre=/usr/local/sbin/pg-dump-for-backup
ExecStart=/usr/local/bin/restic backup --retry-lock 30m \
--one-file-system --exclude-caches \
--exclude-file=/etc/restic/excludes.txt --tag scheduled \
/etc /root /home /srv /var/www /var/backups/postgresql
ExecStartPost=-/usr/bin/curl -fsS --max-time 10 --retry 3 ${HEARTBEAT_URL}
# restic-backup.timer
[Unit]
Description=Nightly restic backup
[Timer]
OnCalendar=*-*-* 02:15:00
RandomizedDelaySec=30m
Persistent=true
[Install]
WantedBy=timers.target
Enable it with systemctl enable --now restic-backup.timer and confirm with systemctl list-timers. Per systemd.timer, Persistent=true catches up on a run missed while the machine was off, and RandomizedDelaySec= spreads hosts that share a bucket. Oneshot services have no start timeout by default. On a Linux laptop, the same pair works as user units for the home directory.
A weekly restic-maintenance.service with the same header runs retention and checks:
ExecStart=/usr/local/bin/restic forget --prune --retry-lock 2h \
--keep-daily 14 --keep-weekly 8 --keep-monthly 12 --keep-yearly 3
ExecStart=/usr/local/bin/restic check --read-data-subset=10%%
Write %% for a literal percent sign, because % starts a specifier in unit files.
Monitoring and alerting on failure
Use two independent signals, because a silent failure is otherwise found during the incident. The first is OnFailure=. restic's exit codes are specific: 1 is a failed command, 3 means some source files were unreadable, 10 a missing repository, 11 a lock failure, and 12 a wrong password. systemd treats any non-zero code as failure, which is right for 3: the snapshot exists but has gaps.
# [email protected]
[Unit]
Description=Alert for failed %i
[Service]
Type=oneshot
EnvironmentFile=/etc/restic/restic.env
ExecStart=/usr/bin/curl -fsS --max-time 10 --retry 3 \
--data-urlencode "text=%H: %i failed, see journalctl -u %i" \
${ALERT_WEBHOOK_URL}
One template serves every job, and systemd passes it variables such as MONITOR_EXIT_STATUS. Test the path with systemctl start [email protected].
The second signal is a dead man's switch: ExecStartPost= pings a heartbeat URL only after success, and the monitor alerts when no ping arrives within, say, 26 hours. That catches a disabled timer or a powered-off host. For an outside view, run restic snapshots --latest 1 --json --no-lock daily from another machine and alert when the newest snapshot is older than the RPO.
Run a timed restore drill
A drill turns assumptions into numbers. Run one after setup, after major changes, and at least quarterly, on a clean VM rather than the production host, using only what an engineer would have in a real incident: the password manager entry and the runbook.
- Start a stopwatch when the simulated incident begins.
- Retrieve credentials, install restic, and list snapshots. The newest snapshot's age is your achieved RPO.
- Restore files and dumps, then the database, timing each step.
- Start a staging copy of the application and complete one real workflow.
- Record durations and problems, and update the runbook.
set -a; . ./restic.env; set +a
restic snapshots --latest 3
time restic restore latest --target /restore --verify
sudo -u postgres psql -f /restore/var/backups/postgresql/globals.sql
sudo -u postgres createdb app
time sudo -u postgres pg_restore --jobs=4 --dbname=app /restore/var/backups/postgresql/app
sudo -u postgres psql -d app -c "SELECT max(created_at) FROM orders;"
--verify rereads restored files and compares them with the snapshot. The last query shows how much data the restore actually lost. Errors about roles that already exist are expected. Also practice a single-file restore with --include /etc/nginx/nginx.conf, the most common real request.
Compare the total with the RTO. Typical surprises are missing credentials, a password that lived only on the dead server, slow index rebuilds, and forgotten files such as TLS keys or an .env outside the backed-up paths.
restic, BorgBackup, or file system snapshots
Many setups combine these: a ZFS, Btrfs, or LVM snapshot for fast local rollback plus file-level backups off-site. BorgBackup 1.4 is the current stable series.
| Property | restic | BorgBackup 1.4 | File system snapshots |
|---|---|---|---|
| Storage targets | Local, SFTP, REST server, S3-compatible and other cloud storage | Local, or SSH, best with Borg on the server | Same pool; off-site needs replication such as zfs send |
| Encryption | Always on | Chosen at repository creation, optional | Depends on the storage |
| Compromised client | rest-server --append-only, bucket object lock |
borg serve --append-only |
Safe only when replicated to a separately administered system |
| Database consistency | Needs dumps | Needs dumps | Crash-consistent if data and WAL share one atomic snapshot |
| Restore | Single file to full tree, FUSE mount | Single file to full tree, FUSE mount | Volume rollback or copy from a mounted snapshot |
| Clients | Linux, macOS, Windows, BSD | Linux, macOS, BSD; Windows only via experimental WSL or Cygwin | Depends on the file system |
A snapshot on the same pool protects against a bad deploy, not against a failed pool or a compromised host, so it is not a 3-2-1 copy by itself.
Checklist
- RPO and RTO are written down and agreed with the service owner.
- Three copies exist, one off-site and one immutable or append-only.
- Password, repository URL, and credentials are in a password manager and offline.
- Live database directories are excluded, and dumps are included.
- PostgreSQL is covered by
pg_dumpplus globals, or a verifiedpg_basebackup. - The timer uses
Persistent=true, and theOnFailure=handler has fired in a test. - A heartbeat alerts on a missing success, and an outside check on stale snapshots.
forget --pruneandcheck --read-data-subsetrun weekly.- Server credentials cannot remove locked versions.
- A timed restore drill ran this quarter, and the runbook reflects its findings.