https://s3hh.wordpress.com/feed

It’s amazing how quickly public opinion changes. Or, how quickly
people cave. Remember just 10 years ago, how people felt about
google glass?

• https://www.businessinsider.com/i-was-assaulted-for-wearing-google-glass-2014-4
• https://www.youtube.com/watch?v=m8coAuWZL20

Or how they felt when they found out target was analyzing their
purchases and perhaps knew them better than themselves or their
family?

• https://techland.time.com/2012/02/17/how-target-knew-a-high-school-girl-was-pregnant-before-her-parents/
• https://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-father-did/

People used to worry about being tracked by companies and government.
Today, they feel insecure unless they are certain they are being
tracked. For their own well-being of course. If an online store does
*not* send an email 10 hours after you’ve “left items in your cart,
don’t miss out!”, and another the next day, you feel disappointed. I
believe they’re now seen as sub-par.

OCI (open container initiative) images are the standard format based on
the original docker format. Each container image is represented as an
array of ‘layers’, each of which is a .tar.gz. To unpack the container
image, untar the first, then untar the second on top of the first, etc.

Several years ago, while we were working on a product which ships its
root filesystem (and of course containers) as OCI layers, Tycho Andersen
(https://tycho.pizza/) came up with the idea of ‘atomfs’ as a way to
avoid some of the deficiencies of tar
(https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar). In
‘atomfs’, the .tar.gz layers are replaced by squashfs (now optionally
erofs) filesystems with dm-verity root hashes specified. Mounting an
image now consists of mounting each squashfs, then merging them with
overlay. Since we have the dmverity root hash, we can ensure that the
filesystem has not been corrupted without having to checksum the files
before mounting, and there is no tar unpacking step.

This past week, Ram Chinchani presented atomfs at the OCI weekly
discussion, which you can see here
https://www.youtube.com/watch?v=CUyH319O9hM starting at about 28
minutes. He showed a full use cycle, starting with a Dockerfile,
building atomfs images using stacker, mounting them using atomfs, and
then executing a container with lxc. Ram mentioned his goal is to have
a containerd snapshotter for atomfs soon. I’m excited to hear that, as
it will make it far easier to integrate into e.g. kubernetes.

Exciting stuff!

Project Machine is the home for our "OCI-native operating system". We started this work in 2017.  As many people, including cyphar and Tycho have pointed out, the .tar.gz nature of OCI layer files has many problems.  So starting early in 2019 Tycho started adding support to stacker for building OCI images with squash layers.

When we started - long before OCI artifacts support etc was discussed - there was some concern of whether squash based layers would have support issues.  Docker registries would not store them, for instance.  So Tycho coined 'atomfs' (*1) as the name for OCI images using squash layers.  Point being - it's not a filesystem, just a way of making use of filesystems.  We (wrote, and) used zot to store them, enabled stacker to build them, and used our internal OS build and management tool, atomix, to extract or mount them.  But while stacker and zot are external tools, we didn't have an external tool to make mounting these images easy.

To rectify this, we present atomfs: a program which can be used to easily examine atomfs images, that is, OCI images with squashfs layers.

If you are root host, it will use the kernel to mount each squash layer, with fsverity if the image specifies verity root hashes.  If you are root in a container, it will try to use squashfuse.  If that is not available, then it will use unsquashfs to unpack the full filesystem. The layers are then combined using overlay into one mount.  This is the step which requires root (*2), at least in your namespace, to accomplish.  So if you are not root, you'll need to unshare a user namespace.

Here are two asciicinema sessions showing some usage.

Happy spelunking!

---

*1 - atomix is the name of the internal project which begat project-machine.
*2 - we could use fuse-overlay for this, but requiring a new namespace has a certain tidyness, so we are still considering.

Let’s say you had one of those huge keyrings, with 50 keys on it. (Let’s say 41 keys – number chosen either randomly or bc its the current number of capabilities) On the plus side, when you want to let someone use one, you can magically, instantly make a copy to give to that person. On the down side, that person can do the same (give a copy to *anyone*), once you hand it to them. Further on the down side, well, you have 41 keys.

Now if you want to send the valet to take your car out, you have to give him a key to the elevator, key to the garage entry door, key to the car door, key to the car ignition, and a key to the garage-exit. You’ve done this yourself plenty so you know what keys to give.

But other parts of the building were designed by other people, who have placed gates in various places you’ve never visited, and know nothing about. So now you need the maid to go clean the fifth bedroom. How do you know what keys to give her?

You can start her on her way with the key to the elevator and the key to the bedroom. But she needed a key to pass a gate in the first hallway. So she comes back… all she knows to say is EPERM! She can’t even tell you where the gate is. So you walk with her to the first gate (strace), see the gate, figure out which key it needs, give it to her. Send her on her way.

Of course she hits another gate, comes back, and says EPERM!

Now, you’re not a quitter. And this is important. So you’ll keep trying until you figure out all the keys she needs. Unfortunately, there are 30 other people – chauffeurs, mechanics, janitors, butlers, maids, some delivery people, a gardener… They all need access to various places.

It gets worse. Some gates are only locked at certain times. So you think you’ve gotten them sorted out, but at 2am it turns out half of them are newly blocked. Oh, and next week the person in charge of the second floor decides a few more gates are really needed. Hopefully their newsletter will tell you where, and which keys they’ll use – but they’re busy, so probably not.

So, sure, you’re not usually a quitter, but it sure will be temping to give the full 41 keys to anyone who comes by.

Containers are most commonly distributed in two ways:

1. ‘Image based’: lxc and lxd distribute their container images as full images, a simple representation of root filesystem and some configuration info.
2. OCI: based on the original docker format, this has become an open standard for publishing not only container images, but any artifacts.

Our products are created, distributed, and installed as OCI. All services run as containers. Each container rootfs is re-created from its OCI image at every start. A physical machine’s rootfs is also shipped as an OCI image, and is recreated on every boot. A system representation therefore consists of a manifest specifying the OCI references for services to run. To make this secure,

1. Images must be verifiable. An fs extraction step, such as un-tarring, prevents us from verifying the result on next boot without re-extracting. Therefore we distribute OCI layers as squashfs instead of tarballs, and mount them using overlayfs.
2. Squashfs layers ship with their dmverity root hash in the image manifest.
3. The system manifest which lists the content-addressed OCI images is signed with a product key.
4. The certificate for a product’s manifest signing public key is stored with the system manifest. All product manifest signing certificates are signed by one manifest signing CA.
5. The manifest signing CA certificate is stored in initrd.
6. The initrd, ‘smooshed’ together with the kernel and kernel command-line into one kernel.efi, are signed with a kernel signing key.
7. The TPM keys for root filesystems and machine-identifying unique key are only unlocked for the pcr7 resulting from (our shim and) a kernel signed with the right kernel signing key certificate.

In this way, we can ship a single ‘kernel.efi’ for all TPM-enabled hardware and VM products. To protect different groups’ products from each other, products are provisioned with a product ID, which must match product ID in the product manifest signing certificate. Each machine is also provisioned with a unique keypair, supporting secure cluster bringup and remote attestation.

This allows us to use OCI as the source for (verifiably) securely installed and booted products. We can install the OS on a host in the traditional way, or we can pxe-boot specifying on the kernel command-line an OCI URL to a layer containing the manifest to boot into.

We hope to present the full solution (with source) at FOSDEM 2023.

# References
1. For more details on the OCI specification, see https://github.com/opencontainers/image-spec/blob/main/spec.md.
2. The very code for generating and mounting squashfs based OCI images is at https://github.com/project-stacker/stacker and https://github.com/project-stacker/stacker/tree/master/atomfs.
3. The in-development replacement for atomfs is puzzlefs, at https://github.com/anuvu/puzzlefs and https://github.com/anuvu/puzzlefs/blob/master/doc/index.md.
4. The TPM-based unattended encrypted filesystem solution was presented in full at LSS 2021: ‘Securing TPM secrets in the datacenter’: https://www.youtube.com/watch?v=wfJDmfPP1OA.

Sorry, I should have posted this weeks ago to save others some time.

If you are running openconnect-sso to connect to a Cisco anyconnect VPN, then when you upgrade to Ubuntu Jammy, openssl 3.0 may stop openconnect from working. The easiest way to work around this is to use a custom configuration file as follows:


cat > $HOME/ssl.cnf
openssl_conf = openssl_init

[openssl_init]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Options = UnsafeLegacyRenegotiation
EOF


Then use this configuration file (only) when running openconnect:


OPENSSL_CONF=~/ssl.cnf openconnect-sso --server=your-server.whatever.com


If you are looking for a new phone that either respects your privacy, leaves you in control, or just has a different form factor from the now ubiquitous 6″ slab, there are quite a few projects in various states of readiness

Freedom:

• vollaphone
• oneplus
• pinephone
• librem 5
• fairphone

Different form factors:

• unihertz atom and titan
• gemini and cosmo

Earlier this year I bought a pinephone, braveheart edition. I’ve tried several OSes on it. Just yesterday, I tried:

• sailfish: looked great, but it would not recognize sim, and crashed when launching browser.
• ubports (ubuntu touch): looked good, texting worked, but crashed when launching app store and would not ring on incoming calls.
• mobian: nice set of default apps, but again would not ring on incoming calls.

So I’m back to running what I’ve had on it for a month or two – sxmo, the suckless mobile operating system. It’s an interesting, different take on> interacting with the phone, and I quite like it. More importantly, for now it’s the most reliable as a communication devvice. With it, I can

• make and receive calls and texts.
• send texts using vi :).
• easily send/receive mail using mbsync, mutt, and msmtp.
• easily customize using scripts – editing existing ones, and adding new ones to the menu system.
• use a cozy, known setup (dwm, st, tmux, sshd)
• change call and text ringtone based on the caller – few other phones I’ve had have done that, not one did it well.
• have a good browsing experience.
• use both wifi and 4G data. I’ve not hotspotted, but can see no reason why that will be a problem.

The most limiting thing about this phone is the battery. It drains very quickly, charges slowly, and if I leave the battery in while turned off, it continues to discharge until, after a day, it doesn’t want to turn back on. An external battery charger helps enormously with this. There is also an apparent hardware misfeature which will prevent the modem from waking the cpu during deep sleep – this will presumably be fixed in later hardware versions, remember mine is the braveheart .

User namespaces were designed from the start to meet a requirement that unprivileged users be able to make use of them. Eric accomplished this by introducing subuid and subgid delegations through shadow. These are defined by the /etc/subuid and /etc/subgid files, which only root can write to. The setuid-root programs newuidmap and newgidmap, which ship with shadow, respect the subids delegated in those two files.

Until recently, programs which wanted to query available mappings, like lxc-usernsexec, have each parsed these two files. Now, shadow ships a new library, libsubid, to facilitate more programatic querying of subids. The API looks like this:

struct subordinate_range **get_subuid_ranges(const char *owner);
struct subordinate_range **get_subgid_ranges(const char *owner);
void subid_free_ranges(struct subordinate_range **ranges);

int get_subuid_owners(uid_t uid, uid_t **owner);
int get_subgid_owners(gid_t gid, uid_t **owner);

/* range should be pre-allocated with owner and count filled in, start is
 * ignored, can be 0 */
bool grant_subuid_range(struct subordinate_range *range, bool reuse);
bool grant_subgid_range(struct subordinate_range *range, bool reuse);

bool free_subuid_range(struct subordinate_range *range);
bool free_subgid_range(struct subordinate_range *range);

The next step, which I’ve not yet begun, will be to hook these general queries into NSS. You can follow the work in this github issue </p.

This is one of those “for my own reference on my next (pine) phone” posts, but anyone using ubuntu phone (ubports.com) may find it useful.

I use mutt (in a libertine ‘container’) on the ubuntu phone for sending email. The terminal keyboard is not bad, but one annoying thing I’ve found is that the auxilliary keyboard rows were not optimal for use in vi. The top row, for those who haven’t seen it, is a single row of extra buttons. There are several top rows to choose from, i.e. Fn keys, control keys, scroll keys, and there’s even one where buttons type out full commands like ‘clear’, ‘rm’ and ‘chmod’.

The main buttons I use are the up arrow, tab, and escape. But escape is in the ‘fn’ row, while tab and up arrow are in the scroll list. So I kept having to switch between different rows. To switch rows, you hold down a button until a popup appears allowing you to choose. This is suboptimal.

To fix this, I went into /opt/click.ubuntu.com/com.ubuntu.terminal/0.9.4/qml/KeyboardRows/Layouts and edited ScrollKeys.json. I removed the _key suffix for all the labels, which just take up space so that fewer buttons show up in one line. I copied the escape key entry from FunctionKeys.json as the first entry in ScrollKeys.json. Then I moved all other entries which preceded the tab key to the end of the file (adjusting the trailing ‘,’ as needed). Finally, I copied ScrollKeys.json to AScrollkeys.json, to make this the first keyboard row whenever I fire up the terminal. (The file ~/.config/ubuntu-terminal-app/ubuntu-terminal-app.conf supposedly orders these, but it is re-written every time the terminal starts!)

Perhaps I should add a row for ‘|’, ‘!’, and a few others which I’m always going to the second number screen for. But for now, this should speed things up.

For the past few weeks I’ve been using a nexus 4 running ubuntu touch as, mostly, my daily driver. I’ve enjoyed it quite a bit. In part that’s just the awesome size of the nexus 4. In part, it’s the ubuntu touch interface itself. If you haven’t tried it, you really should. (Sailfish ambiances are so much prettier, but ubuntu touch is much nicer to use – the quick switch to switch between two apps, for instance. Would that I could have both.). And in part it’s just the fact that it really feels like – is – a regular ubuntu system.

There have been a few problems. The biggest has been to do with email. I need a phone to do only a few things well – texts, calls, and imap email. The only imap mailer available by default, dekko2, looks very nice and is promising, but was simply not yet reliable for me. It would simply stop getting updates for hours, with no warning, for instance. So I’ve taken to using an ubuntu-push notification system for email notifications, and mutt and offlineimap in a libertine container for reading and sending. The notification system is based on https://forums.ubports.com/topic/3126/facebook-messenger-push-notifications/2 and the python mailbox library. It runs on my mail server, checks for new mail, and, if there is any, sends a push notification to my phone. The code I’m using is here on launchpad and here on github. It can certainly stand to be made a bit smarter (the seenmsgs list should be pruned, for instance, and maildir and mh folder support should be trivial to add for those cool cats who use those). Using this service instead of having the phone try to check for emails not only ends up being very reliable, but also saves a lot of battery life.

All in all this could definately work as my permanent new phone! Now if I could just get my hands on a pinephone or librem 5. The nexus 4 hardware is great, but it would be awesome being able to run an uptodate, upstream kernel. More than that – now that my experiment has succeeded, I probably need to stop, because running the ancient kernel simply is not as safe as I’d like. But I digress.

A huge thanks to Mark and the original touch team for creating it, and to the ubports team for keeping it going.

Nice job, everyone!