Backstory
A few weeks ago (not today, and by the time I published this post it's a few
months ago), I fucked up:
During irregular disk cleanup, I accidentally removed the config folder!
Fortunately, I "backed up" the config with git, didn't I? 😌
Well yes, but actually no! 🥲 I did commit, but I didn't push.
The latest pushed config had been 9 months before this incident.
On top of that, many programs generate unreadable configs, which I prefer not
to put into version control, or put sensitive information or large data into
.config, which shouldn't be pushed into a remote repository on a server I
don't own. For example, nheko puts authentication info there, and after this
incident I lost the session with several messages---well, matrix's irregularly
regular decryption failure could be expanded into a post on its own, so let's
not digress.
And then there are more things to back up than just configs, say, my photos and
music. I mirror them between my devices with syncthing, providing
redundancy, but syncthing is not backup,
and neither is RAID
I should back them up, or I might one day be doomed to repeat the mistake.
Preparation
Despite the bitterness from the partial data loss, I kinda didn't have the
spoon to do what I must, so I've been delaying it hitherto.
Hardware
One of the reason I was cleaning up the disk in the first place was that it's
start to be filled up, so I bought a large hard drive to store more data and
backup. Doesn't seem too safe, I know: if the hard drive gets lost or broken,
I'll lose both the main data and the backup, but with a limited budget and
space, that's good enough for now. I am planning for some off-site backup.
I got a Seagate One Touch 2 TB1 HDD. It seems to come with builtin backup
software, which doesn't support Linux of course, and I likely wouldn't use it
even if it did. A USB hub is also needed, as my laptop has a limited number of
USB ports.
Formatting
Coming with the hard drive are several files, such as an .exe program to
initialize the backup, I suppose. I don't need these, and the disk can be
formatted right away after I do full-disk encryption on that.
cryptsetup luksFormat /dev/sdb
cryptsetup open --type luks /dev/sdb extern
No partitioning is needed: while I do seek vegan alternative, I
intended to use btrfs as the file system, which can create
subvolumes, which can act as separate mounted partitions. I would just
reformat the existing partition to btrfs.
mkfs.btrfs -L margarine /dev/mapper/extern
I intended to have three subvolumes, one for big data (not Big Data™) such as
movies or music, one for more personal data and another for backup:
mkdir /data
mount /dev/mapper/extern /data
btrfs subvolume create /data/hoard
btrfs subvolume create /data/perso
btrfs subvolume create /data/backup
ACCIDENT!
OK, I'll admit I don't know what the heck went on, but the main user, xarvos,
and root were apparently deleted, but not actually. I cannot log in to either
account, but they were still listed in the /etc/passwd file. Huh? I
cancelled a rebuild, which I have done a dozen times before and nothing went
wrong so it should be fine this time, right? Right? (Looking at
the laptop Padmely)
Per later investigation, it looks like for some unknown reason, the NixOS
configurations were removed mistakenly, and the failure was triggered by a
rebuild.
Rescue
Well, it'd be a lie to say I didn't panic. But this is not something not
having happened before (translation for people confused by double negatives:
similar thing happened before).
The easiest solution is to reinstall the thing (though it will leave me
wondering why this happenned). At the time of my NixOS installation, I didn't
make use of the service configurations in nix, only declaring those packages
and invoke them manually. Additionally, like cnx, I also didn't think of
full-disk encryption during installation. So, this is an opportunity to
rectify those wrongs
Backing up
Thanks to the available external disk as well as a ready bootable disk (you
should always have those, for occasions like these), I can simply back up my
whole home directory. I don't do stuff elsewhere, so no needs to back those
up.
mkdir /mnt/{int,ext}
mount /dev/sda1 /mnt/int
cryptsetup open --type luks /dev/sdb extern
mount /dev/mapper/extern /mnt/ext
rsync -a /mnt/int/home/xarvos /mnt/ext/backup/manual-$(date)-home-xarvos
This also mean I can do the rescue while documenting this process at the same
time 😃.
The bootable USB drive is a NixOS installer, but it's not the latest version,
so I burn a newer installer to a second one (so handy), after backing up
the second one's content to the disk using similar process.
Reinstallation
OK, it's time to remount. I can install right now with nixos-install --root=/mnt/int, but I want to include the external disk in the same root, so
that their hardware configuration can be generated together. Though, before
remounting, I should encrypt the main partition.
Previously, I only have two partition---the main one and the swap, but now I
would need three, because the /boot can't be encrypted
parted /dev/sda
(parted) mklabel msdos
(parted) mkpart primary 128MB -8GB
(parted) mkpart primary linux-swap -8GB 100%
(parted) mkpart primary 1MB 128MB
(parted) set 3 boot on
mkswap /dev/sda2
swapon /dev/sda2
mkfs.ext4 -L boot /dev/sda3
cryptsetup luksFormat /dev/sda1
cryptsetup open --type luks /dev/sda1 intern
mkfs.btrfs -L pain /dev/mapper/intern
(pain as in French for bread, because of course we spread margarine on bread :wink:)
mount /dev/mapper/intern /mnt
mkdir /mnt/data
mount /dev/mapper/extern /mnt/data
Before generating the config, I'd like to make some more subvolumes. I didn't
intend to run with my root on tmpfs like cnx, but I believe it'd be better to
have different partitions to be backupped (with snapshots) separately.
btrfs subvolume create /mnt/etc
btrfs subvolume create /mnt/home
btrfs subvolume create /mnt/root
btrfs subvolume create /mnt/var
After that, I copied the configuration to /etc and generated the config.
mkdir /mnt/etc/nixos
cp -r /mnt/data/backup/.../{services,configuration.nix} /mnt/etc/nixos
nixos-generate-config --root /mnt
(Because nixos-generate-config does not overwrite configuration.nix file if
it already exists, I can copy the configs there first.)
While in an urgence (there was one day and a half before the holiday period
ended and I needed the machine back for work), I still spent the time to rewrite
some services, namely syncthing, transmission, and mpd. For example, I set mpd
home to /data/hoard
services.mpd = {
enable = true;
user = "xarvos";
dataDir = "/data/hoard/music";
startWhenNeeded = true;
};
Finally, I ran nixos-install to finish the setup, and sync'ed the backup back home.
Setting up
(About two months later)
After rebuilding the system, I kinda ran out of energy for doing the backup
(again), so I postponed the setup to the week after, but then the next week
something happened, which delayed it further and further, and since the
momentum died, it took quite a while for me to pick up this again.
My initial plan was to use snapper, as I had good experience with it back when
I was using openSUSE.
Snapper (HTTP without TLS link, no idea why) is a tool by openSUSE
project that helps managing file system snapshots, which was intended for
btrfs, but also supports several other file systems. See also its
source, the tutorial on opensuse.org, and
ArchLinux wiki article for more information. My main use
of snapper was pre-post snapshots, which was integrated into zypper, but I have
no use of that on NixOS, since I already manage system changes in git, and
NixOS also create different boot options on rebuild as well.
Timeline snapshot is also great, but it seems a bit
overkill for me. Additionally, snapper doesn't seem to provide a way to send
snapshots to the external drive, so I'd have to set that up, and it'd probably
be harder to get the names correctly.
So, I'd just copy cnx here, which is less thinking
btrfs subvolume create /data/backup/home
today=$(date --iso-8601)
btrfs subvolume snapshot -r /home /home/$today
sync
btrfs send /home/$today | btrfs receive /data/backup/home
sync
Now, I have to setup a cron job to automate backup, as well as deletion. I
modified the script a little so running it twice does not accidentally delete
last snapshot.
previous=$(find /home/ -maxdepth 1 -regex '.*20[0-9][0-9]-[0-1][0-9]-[0-3][0-9]')
today=$(date --iso-8601)
if [[ "$previous" == "/home/$today" ]]; then
echo "Today is already backed up"
else
echo "Running backup"
btrfs subvolume snapshot -r /home /home/$today
btrfs send -p $previous /home/$today | btrfs receive /data/backup/home
btrfs subvolume delete $previous
sync
fi
Then, I set up the fcron service. Following the example in
fcrontab manual, I can use %hours keyword so that this runs once
in the day from 9h to 16h:
services.fcron = {
enable = true;
systab = "%hours 0 9-16 * * * /path/to/backup.sh"
};
I think this is pretty much it. I will wait till tomorrow to see if the cron
will run properly.
Footnotes
That means 1.81 TiB and I feel cheated sometimes.↩