↓ Skip to main content
steeman.be
  1. Posts/

Offsite Backups on a Raspberry Pi, with a Key That Lives Only at Home

· David Steeman · Linux, DIY, Raspberry Pi

The first two parts of the 3-2-1 backup rule are easy: three copies, on two different media. I had that covered with the Borg backup strategy on my home server — nightly Borg archives on a local USB disk, mirrored to a NAS. The hard part is the “1”: one copy offsite, somewhere physically separate from the house, so a fire or a theft doesn’t take the data and every backup of it in one go.

The obvious answer is “put a disk at a relative’s house.” The interesting question is how to make that offsite copy encrypted, automatic, and safe — and specifically, what do you do about the encryption key? If the key lives on the offsite machine next to the disk, then a thief who takes the disk and the machine has both, and your encryption is theatre. I wanted the offsite disk encrypted with a key that is provably never present at the offsite location.

The solution is a Raspberry Pi sitting at a remote site, a LUKS-encrypted USB disk hanging off it, and a decryption key that lives only on my home server. At boot the Pi reaches home over Tailscale, fetches the key, and pipes it straight into cryptsetup — it is never written to any filesystem on the Pi. Take the disk and the Pi together and you still can’t read a byte, because the Pi can no longer reach home to ask for the key.

The threat model first
#

Before any of the how, the why. The point of offsite encryption isn’t to defend against a skilled attacker who has infinite time — it’s to make the common threats harmless:

  • Disk stolen alone — it’s encrypted, unreadable. Easy.
  • Disk and Pi stolen together — this is the one most “encrypted offsite” setups get wrong. If the key is on the Pi’s SD card, the thief has everything. My key is not on the Pi; it’s on the home server. With the Pi unplugged from its network it can never fetch the key, so the disk stays locked.
  • Pi stolen while running — the key only ever exists in RAM for the moment of the cryptsetup call, and it’s gone the instant power is lost.
  • Somebody extracts the Pi’s SSH key from the SD card — that key is locked to a single forced command that only returns the LUKS key, and only when the home server is reachable over Tailscale. No shell, no port forwarding, nothing else.

There’s a kill switch, too: revoke the Pi’s Tailscale authorisation and the Pi can never reach home again, so the disk can never be unlocked — useful if the remote site is ever compromised and you want to neutralise the box remotely.

The architecture
#

Home server (john-ai)
  00:00  Borg create  →  local USB disk
  02:00  rsync over Tailscale  →  offsite Pi (LUKS-encrypted USB disk)
                                ↘  also mirrors Borg repos to the NAS

The Pi is deliberately a dumb storage target. It runs no Borg, does no processing — it just receives data via rsync and writes it to the encrypted disk. All the cleverness lives on the home server; the Pi is a sealed box that holds bytes. Everything between the two travels over Tailscale’s WireGuard tunnel, and the Pi’s firewall admits SSH only on the tailscale0 interface — nothing is exposed to the public internet.

The boot unlock sequence
#

This is the heart of it. When the Pi powers on:

  1. Tailscale connects to the tailnet (with retries — it’ll wait up to five minutes, and a systemd timer retries the whole unlock every 30 minutes if home is down).
  2. A systemd service SSHes to the home server using a key whose only allowed action is cat-ing the keyfile. The forced command in the home server’s authorized_keys enforces that — no shell, no -t, no forwarding of any kind.
  3. The 32 bytes that come back are piped directly into cryptsetup luksOpen --key-file -. They exist in the pipeline and in RAM for that instant, and nowhere else — not on the SD card, not on the disk.
  4. The LUKS container opens, the ext4 filesystem inside mounts at /mnt/backup, and a sentinel file marks it ready.

If any step fails, the Pi emails me (via a separate mail path) and tries again in half an hour. Crucially, if home is unreachable the disk simply stays locked — which is exactly the failure mode you want for an offsite box.

Two keys, each good for exactly one thing
#

There are two SSH relationships, and each is locked down to a single purpose:

  • Home → Pi: an rsync-only key, restricted with rrsync to the /mnt/backup/ directory. It can push backup data and nothing else — no shell, no commands. This is the key the nightly sync uses.
  • Pi → Home: the key-fetch key above, good for one cat and nothing more.

The subtlety that bit me: because the rsync key is rrsync-restricted, you cannot use it for admin. sudo ssh offsite-backup runs the forced rsync command and exits — it will never give you a shell. So I keep a third, unrestricted key (my normal john@home key) on the Pi specifically as the escape hatch for hands-on admin, and it has to stay without a forced command. It’s the only way back in if rsync breaks or the disk fails to mount. Removing it to “tighten security” would lock me out of my own recovery path.

A dead-man’s switch that works when home is the thing that died
#

Monitoring an offsite backup has a bootstrapping problem: if the home server dies, who sends the alert? The home server can’t — it’s dead. So the alert has to come from somewhere else.

After every successful offsite sync, the home server rsyncs a heartbeat file to the Pi. The Pi runs a watchdog hourly: if that heartbeat is more than 30 hours stale — or if its own disk is unmounted — it emails me, from the Pi, over a separate mail account. So if the home server dies, the sync stops, the heartbeat goes stale, and the Pi raises the alarm through a path that doesn’t depend on the home server being alive. The same watchdog warns pre-emptively when the disk hits 95% full, before a sync can fail with “no space left.”

The heartbeat reflects offsite freshness specifically — a fresh heartbeat means the offsite copy is genuinely current, not merely that the local backup ran.

The bandwidth misdiagnosis
#

When the Pi first went offsite, the nightly sync crawled at about 200 KB/s, and each run took six to ten hours. tailscale ping briefly showed the connection going via DERP(ams) — relayed through a Tailscale relay server in Amsterdam instead of direct — so I confidently blamed the relay and went looking for why direct peering wasn’t establishing.

I was wrong. tailscale ping can transiently report DERP even when the actual data path is already a direct peer-to-peer connection. The real governor was never the relay — it was the ISP upload bandwidth. The remote site was on a 2 Mbps upstream line, and 2 Mbps up is about 200 KB/s, direct connection or not. A direct P2P tunnel over a 2 Mbps uplink is still only 2 Mbps.

Upgrading that line from 20/2 to 100/40 Mbps took the sync from ~200 KB/s (6–10 h) to about 4–5 MB/s (30–40 minutes) overnight. The lesson: when a Tailscale link is slow, look at the uplink bandwidth first. The relay is almost never the bottleneck; the slowest uplink on either end is.

The key that unlocks the key
#

There’s one more failure to plan for: what if the home server’s disk dies and takes the only copy of the LUKS key with it? Then the offsite backup — the thing meant to rescue me from exactly that — becomes unrecoverable.

So the key is escrowed in three places: in my password manager, and in an offline recovery bundle (the keyfile, a base64 copy, a LUKS header backup, and a runbook for manual unlock that doesn’t depend on the home server) stored both on the NAS and on a BitLocker-encrypted USB stick kept off-site. That bundle is deliberately excluded from the sync that pushes data to the Pi — the keyfile must never land on the very disk it unlocks.

Gotchas worth passing on
#

A 4.5 TB WD Elements drive needs a powered USB hub. The Pi 4’s USB ports can’t supply enough current for it, even with usb_max_current_enable=1. Symptoms are undervoltage warnings in dmesg, I/O errors, and — under heavy load — outright reboots. A powered USB 3.0 hub fixed it completely.

A directory of 65,536 empty subdirectories will OOM the Pi. The Proxmox Backup Server datastore contains a .chunks/ tree of 65,536 hex-named folders. When rsync enumerates them, the kernel page cache fills to 3.5 GB just loading the directory blocks, and the Pi reboots — even on a dry run. The fix is --exclude=.chunks/ in the rsync flags. (And a note on shell quoting: write --exclude=.chunks/ inside a variable, not --exclude='.chunks/' — the single quotes are passed literally and never match.)

LUKS2’s default key-derivation memory can OOM a small Pi. Argon2id defaults to 1 GB, which is fatal on a 1 GB Pi 2. I reduced it to 128 MB with cryptsetup luksConvertKey (with a temporary swap file on the SD card; it took about 50 minutes). The Pi 4 with 4 GB handles it fine, but always check the PBKDF memory when moving an encrypted disk onto low-RAM hardware.

Borg repos must be owned by the rsync user. The repos are seeded as root on the home server, but rsync runs as john on the Pi. Without chown-ing the repos to john and adding --no-owner --no-group to the rsync flags, every root-owned source file causes an exit-23.

A volatile journal means no post-reboot forensics. I run journald in RAM (for SD-card longevity), so previous-boot logs are gone after a reboot. When diagnosing an unexpected reboot, check vcgencmd get_throttled (0x0 means no undervoltage ever seen this boot) and dmesg | grep undervoltage immediately on the next boot, before the evidence evaporates.

Lessons learned
#

Design the failure modes before the happy path. The two decisions that make this safe — the key that never lives offsite, and the dead-man’s switch that alarms from a separate machine — both came from asking “what if the thing I’m protecting against is the thing that fails.” A backup system is only as good as its worst-day behaviour.

Keep an unrestricted escape hatch, and know which key it is. Locking the rsync key down to rrsync is correct, but it means that key will never give you a shell. Document the one key that does, and protect it — it’s your way back in when the automated path breaks.

Blame bandwidth before relays. A slow Tailscale link is almost always a slow uplink, not a relay fallback. tailscale ping lies sometimes; iperf and the ISP speed tier don’t.

Encrypt for the physical-threat model, and escrow the key against your own disk failure. The offsite copy’s job is to survive the home server being gone — including the case where the home server held the only key.

Resources
#