Every volume, every night: how our estates back up

The problem

A Docker estate keeps everything that matters in named volumes: database files, uploads, configuration. The containers can be thrown away and pulled again. The volumes cannot.

Backups tend to fail quietly. A job stops running and nobody notices. A copy exists, but nobody can open it. A path points at the wrong folder, and every night a perfectly successful backup of nothing is taken. We have met the last one ourselves, and it shaped how we set backups up.

What runs on a host

A host’s backups run in one container: Backrest, with restic inside it. Backrest keeps the schedule and a web console. restic does the work.

The nightly plan lists the host’s named volumes one by one. Volumes that hold persistent data are in. Caches, logs, monitoring data and anonymous volumes are left out on purpose: they can be rebuilt, and copying them only makes the repository bigger.

At 02:00 each night the plan takes a snapshot of those volumes. The snapshot is encrypted with a passphrase, compressed, and deduplicated against everything already stored, so only what changed takes new space. The result is kept on a ladder: 14 daily snapshots, 8 weekly and 12 monthly. That is about a year of history, thinning as it gets older.

A good night leaves three successful operations behind: the snapshot, the step that applies the ladder, and an index of what was kept. That is what a healthy night looks like.

Checked every week

Every Sunday, two more jobs run. A prune drops the snapshots that have fallen off the ladder. A check then reads back a portion of the stored data to confirm it is intact. A damaged repository is found on an ordinary Sunday, not on the day a restore is needed.

When the backup tool is what broke

A backup that only the tool that made it can open has a single point of failure. So each host also carries a standalone copy of restic, outside the container. If Backrest itself is down, the repository still opens, with its passphrase.

That passphrase is the one thing that cannot be recovered. Lose it and the backups are unreadable, by design. It is generated once, at setup, and kept in a password manager, not only on the host.

Restores land beside, not on top

Each volume is its own path inside a snapshot, so one volume can come back without touching the others. A restore is written to a new volume next to the live one. The live data changes only when someone decides to copy the restored data in. The restore itself overwrites nothing.

The trap that shaped the runbook

Docker installed as a Snap package keeps its volumes in a different place from a standard install. Point the backup at the usual path on such a host and it binds, without complaint, to an empty folder. Every night then succeeds, and nothing is saved. So the first question for any host is which kind of Docker it runs, and the plan is written for that answer.

What it means for you

Every volume that holds your data has a history about a year deep. Part of it is read back every week. It opens without the backup service, and any one volume can come back without disturbing the rest.

It also costs less disk than people expect. Compressed and deduplicated, the repository has run at roughly a quarter of the raw data in practice. It grows with how much data changes, not with how many snapshots are kept.

What we don’t claim

The repository sits on the same host as the volumes it protects. A second, off-site destination is an open item in our runbook, so no off-site copy is claimed here. Regular restore drills are planned but not yet routine, so tested restores are not claimed either. The snapshots copy volumes at the file level: a database caught mid-write can be restored, but it is not guaranteed to be transactionally consistent, and taking database dumps before each snapshot is still open. Nothing pages a person when a night fails; that alerting is not wired yet.