🍄Server Backup Of Openbsd

It is necessary to keep a backup of a server in case of emergency. We can use simple tools to keep our data backed up, just rsync(1) will do for now. In order to run the task periodically, I will use the stock option on Arch Linux and that is systemd's Timers.

I usually want to backup:

  • Any data I keep on the server – currently wikipedia at /var/mycorrhiza and bookmarks manager at /var/shiori.

  • Configuration that I don't want to restore manually – currently /etc/httpd.conf, /etc/relayd.conf and /etc/acme-client.conf.

  • And anything that is just convenient to quickly restore – currently my websites at /var/www/htdocs.

Authentication

Since I can't restore the ssh private key from gpg-agent (more on this in How to never use GPG), let's generate a new one without a password exactly for the purpose of backing things up. 

ssh-keygen
# enter filename such as id_ed_backup
# press enter for empty password
# enter again to repeat an empty password
ssh-copy-id -i ~/.ssh/id_ed_backup.pub alice@greenfork.me

After sending it to the server, I'd like to leave a comment on the server in ~/.ssh/authorized_keys that the last key is for backing up. This key is without a password, so it could be wise to have an ability to quickly delete it from the server.

Backup script

First we should make sure that all the files are going to be readable by the backing up user. I use my personal account to back up because it is in a wheel group. It is relatively hard to set up a "backup user" because it requires quite a lot of privileges in order to backup all the relevant files, so I just use the user I use to log in.

Currently the only offenders to backing up are the /etc/relayd.conf file and /var/shiori, we add reading to their group permissions, on the server: 

doas -s
chmod g+r /etc/relayd.conf
find /var/shiori/ -print0 | xargs -0 chmod o+r

Now back to our computer. Create an appropriate directory: 

sudo mkdir /var/backups
sudo chmod 750 /var/backups
sudo chown grfork:grfork /var/backups
cd /var/backups

and create a file backup.sh with this content: 

#!/bin/sh

set -ex

mkdir -p current_backup/

rsync -v --links --perms --recursive --times --delete --rsync-path=openrsync \
	-e "ssh -i /home/grfork/.ssh/id_ed_backup" \
	alice@greenfork.me:/var/www/htdocs \
	:/var/mycorrhiza \
	:/var/shiori \
	:/etc/httpd.conf \
	:/etc/relayd.conf \
	:/etc/acme-client.conf \
	current_backup/

  • --rsync-path option is used because OpenBSD by default ships their own version of rsync with a limited set of features.

  • Unfortunately we can't keep the directories such as var because openrsync doesn't support the --relative option.

Make this script executable: 

chmod 750 backup.sh

and check that it works, run it: ./backup.sh.

systemd timer

Next let's create a systemd timer and service. Create a file /etc/systemd/system/backup.timer: 

[Unit]
Description=Back up the OpenBSD server

[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=1m

[Install]
WantedBy=timers.target

and a file /etc/systemd/system/backup.service: 

[Unit]
Description=Back up the OpenBSD server
After=network.target

[Service]
Type=oneshot
ExecStart=/var/backups/backup.sh
WorkingDirectory=/var/backups
User=grfork
Group=grfork

Check that it works with systemctl start backup. If that is successful, we can enable the execution of the server on a daily schedule with systemctl start backup.timer and systemctl enable backup.timer. Check that the timer is present in the list systemctl list-timers.

Bonus: compress and encrypt

If we want to save the archive somewhere, like Google Drive, it would be wise to keep it compressed for reducing transfer bandwidth and encrypt it so no one can peek at what's inside.

We will use tar with xz compression and age for encryption. 

tar --create --xz current_backup/ backup.sh > "backup-$(date --iso-8601=seconds).tar.xz"

this line should compress everything to a single archive of the format like backup-2024-06-05T16:22:54+03:00.tar.xz. Note that we also include the backup.sh file itself so that we can reconstruct what's inside the archive.

For the encryption we will generate a pair of public and private keys: 

age-keygen -o backup_key.txt

Copy the generated public key (should be visible in the terminal after creating a pair) and you can encrypt the file using the following command (here I copied age1pe7m57cycvjm9mw77sxhvvs2f4yy9wz4ph96nr2222w79rftjeaskcwr89): 

age -r age1pe7m57cycvjm9mw77sxhvvs2f4yy9wz4ph96nr2222w79rftjeaskcwr89 backup-2024-06-05T16:22:54+03:00.tar.xz > backup-2024-06-05T16:22:54+03:00.tar.xz.age

All-in-all we can expand our script to the following: 

#!/bin/sh

set -ex

mkdir -p current_backup/

rsync -v --links --perms --recursive --times --delete --rsync-path=openrsync \
	-e "ssh -i /home/grfork/.ssh/id_ed_backup" \
	alice@greenfork.me:/var/www/htdocs \
	:/var/mycorrhiza \
	:/var/shiori \
	:/etc/httpd.conf \
	:/etc/relayd.conf \
	:/etc/acme-client.conf \
	current_backup/

tar --create --xz current_backup/ backup.sh | \
	age -r age1pe7m57cycvjm9mw77sxhvvs2f4yy9wz4ph96nr2222w79rftjeaskcwr89 \
	> "openbsd-$(date --iso-8601=seconds).tar.xz.age"

Bonus: rotate files

There's a notion of "log rotation" – when the log file gets too large, it is compressed and renamed so that new logs are stored in a new file. And there's a maximum number of files that is stored on disk, too old files are deleted.

With the compressed and encrypted backups above there will be a daily tarball of all the files. Currently a compressed encrypted tarball takes 15 MiB of space, but it can grow larger. I think 7 most recent files is fine for me.

I will use the Raku programming language with this simple script: 

#!/bin/env raku

my $count = 7;

# Keep only $count latest files.
.unlink for dir.grep(/^ "openbsd-" /).sort.reverse[$count..*];