Zac Brown's Nonsense

Setting up Python on macOS (and other *nixes too) but first, a rant

Periodically, I choose the path of masochism and decide I need to mess with some project written in Python at home. I deal with it regularly at work, but I only occasionally have to deal with it on my personal computer. For work, my dev environments have always been automagically setup to deal with whatever our Python du jour is.

So that we’re clear, I really hate Python.

It’s not the language itself so much - although I have little to like about it. It’s the insane idiocy that is getting a scripting environment, especially Python’s setup correctly.

Most scripting languages have the same horror show of problems:

1. while many systems come with Python or Ruby installed by default, it’s generally bad practice to install third party packages into the system installation

2. because of (1), you’re generally guided toward setting up a user directory available copy of $SCRIPTING_LANGUAGE so that you can sanely install and work with third party packages

3. however (2) is usually not simply enough because of the necessity for a programmer to be able to work on more than one program at once with likely disparate package/library versions - so now you need something like a virtual environment or vendored distribution in each project for the respective dependencies

Did you catch that? That’s three layers of paths to lookup libraries, packages, and binaries you may have installed. Any time someone touches something or updates a package, there’s a good chance something randomly breaks. It’s infuriating for me.

ok, but the solution is…?

This is generally good advice on any *nix, but I’m working on an M1 MacBook Pro running macOS 14 (Sonoma).

To keep your life simple:

• use pyenv
• install your favorite python - e.g. pyenv install 3.12 && pyenv global 3.12
• install pipx, BUT NOT from brew! Use the Linux installation instructions
• install your user-wide tools like virtualenv or ansible with pipx
• steal your own underpants
• ???
• profit

If you stick to those steps, you’ll have a 99% sane environment to deal with. Pretend like the OS never even installed Python on your machine. That’s just for automation within the given distribution or OS’s package management and subsystems.

Posted on: 2024-05-04
Tagged as: macos, howto, bsd, linux

Reflecting on Offices and Comfort

When I first started working an “office job”, I was a research assistant in a biology lab. I wrote computer simulations emulating biological systems. The office itself was nice enough, but in reality was simply a small cubicle farm with a common table. My desk wasn’t anything special, just a nice chair and a very powerful workstation. I also administered our super-computing cluster which was modest by comparison (12 nodes). It was located in the closet next to my desk.

I didn’t decorate my space much - a couple puzzle toys for fidgeting with and an inkjet printed photo of my then girlfriend and I at the beach. It was a sterile space, partly because the researcher I worked for also wasn’t much for decoration. However, it was enough for me at the time.

After I graduated and went to work for Microsoft, I shared an office with another new hire engineer. Neither of us decorated the space much, but similar to my college office job, I had some little fidget toys. It was nothing memorable, but again it was my space and that was enough.

The offices and their furniture themselves were nothing special. Reflecting a design aesthetic that was probably formed in the early 2000’s. The desks all looked the same, with filing cabinets and bookshelves that matched. They were boring and unnotable.

What I do remember was how different each of the offices of the more senior engineers. One engineer had what must have been hundreds of little Lego figurines lined up on his shelves. He also had tons of college (we went to the same university) and baseball memorabilia. Another engineer loved the giant expensive Lego models. He had a Super Star Destroyer and a Death Star model in his office. If I remember correctly, he also had a Y-Wing and an X-Wing.

Going through my memories, most of the more tenured engineers I worked with had brough their personalities into their offices. From Lego to books to fantasy to movies to sports to knitting, each office felt a little like a slice of home.

When I left Microsoft, I worked largely in a home office. The company I worked for is based in Denver. They had an open floor plan, like many startups. It was a nice office, but I was never really comfortable in it. Even people that worked in that office daily didn’t seem all that comfortable there. Maybe that’s just me projecting.

Later, I went to work for Stripe. They have a lovely office in downtown Seattle that was beautifully designed. It was also open floor plan, similar to my job in Denver. It had exposed wood and steel finishes, luxury office chairs, and reading nooks with beautiful views. However, again, I never really felt comfortable in it. It felt like I was staying in a hotel. It had a personality, but that personality felt fake and manufactured.

As I’ve reflected on more than 15 years of working, I’ve realized that the reasons I disliked open space are deeper than my strong preference for a private and quiet space to work. I always figured that being an only child and an introvert was the reason I disliked open floor plans. I now realize their lack of individuality and the cold uniformity are what bothered me so much.

Something about sitting in someone’s office, discussing work or maybe what we did over the weekend, and surrounded by their individual expression is a happy memory for me. It’s what I miss most about Microsoft in some ways.

—–

Posted on: 2023-12-06
Tagged as: thoughts

Ansible and Alpine Context

I have been slowly redoing the servers on my home network over the last year. It started with massively upgrading my NAS both in storage as well as overall power. Looking back in my archives, apparently I never wrote about that. Suffice to say that the new server is beefy, runs TrueNAS Scale, and runs about two thirds of the core services on the network.

This past week, I’ve been redoing my DNS and WireGuard servers. For a few years, they’ve been a hodge podge of FreeBSD, OpenBSD, and Ubuntu depending on my mood. Lately though, I spend most of my time working with Alpine Linux. Usually this is in the form of Docker images. On a whim, I decided to condense all of my home network servers to Alpine Linux and finally learn Ansible.

Quirks and Findings

Alpine Linux is not a first class target for Ansible. That is not to say it does not work well. It works extremely well! However, in some cases, you’re relying on the community-side plugins for interacting with things like apk. These are stable, but they’re not core Ansible plugins.

Overall, the experience is very smooth and easily google-able. However, there was one thing I had to build myself. After updating, I wanted to check if the machine needed to be restarted because of a kernel upgrade or the like.

In Debian and Debian derivatives, they write a file to /var/run/restart-required when a restart is necessary. If you were wondering, this is how the *** System restart required *** string ends up showing up in your MOTD.

Similarly, RHEL/Fedora-flavored distributions have a package (yum-utils or dnf-utils) you can install that provides the needs-restarting utility. When you run it, it will write a 1 to the $? shell variable if the system needs restarting.

What about Alpine?

I ended up coming up with a very dumb but reliable means for checking whether the system needs rebooting. I made a couple of assumptions:

1. My servers are small and single-purpose. The only interesting upgrade for restarting is the kernel package: linux-lts.
2. Considering (1), we only need to check if linux-lts’s version changed to know if we need to reboot.

To achieve this, I built the following tasks in my Ansible playbook (full source here). Comments are inline to the YAML to explain what’s going on.

# This is equivalent to: apk update && apk upgrade
- name: Update cache and upgrade packages
  community.general.apk:
    upgrade: true
    update_cache: true

# Set a variable for the currently *installed* linux-lts package version.
# Importantly, the shell command reformats the package version string using
# awk and sed into a string that we can match against what will be reported
# by `uname -r`.
#
# I am no awk or sed expert and perhaps my abomination is overly verbose, but
# it works and I can understand it. Longer awk/sed programs tend to confuse me.
- name: Register installed linux-lts kernel version
  register: installed_kernel_version
  ansible.builtin.shell: |
    set -o pipefail
    apk list linux-lts --installed | awk '{ print $1 }' | sed 's/linux-lts-//' | sed 's/-r/\n/g' | awk '{printf("%s-",$0)}' | awk '{printf("%slts", $0)}'
  changed_when: installed_kernel_version != ""

# Set a variable for the currently *running* linux-lts kernel version. We use
# sed to strip off the arch.
- name: Register running linux-lts kernel version
  register: running_kernel_version
  ansible.builtin.shell: |
    set -o pipefail
    uname -r | sed 's/-ARCH//'
  changed_when: running_kernel_version != ""

# This is debugging output to tell us when the installed kernel version doesn't
# match the running kernel version. The real magic happens in the following task.
- name: Check installed_kernel_version != running_kernel_version = ???
  ansible.builtin.debug:
    msg: "{{ installed_kernel_version.stdout }} != {{ running_kernel_version.stdout }} = {{ installed_kernel_version.stdout != running_kernel_version.stdout }}"

# Now compare installed_kernel_version with running_kernel_version. When they
# don't match, this means that we need to reboot. This is not a very sophisticated
# heuristic, but it works.
- name: Reboot if the running kernel version is not the installed kernel version
  ansible.builtin.reboot:
    reboot_timeout: 30 # These are very simple Alpine servers. They should boot extremely fast.
  when: installed_kernel_version.stdout != running_kernel_version.stdout

fin

As noted in the comments, it’s not a very complex solution. But it works and has worked consistently for a few days now. We’ll see if this remains true over time.

If you’re interested in the larger Ansible Playbook I’ve created, you can find it on my sourcehut page. My next big chunk of additions will be to codify the WireGuard configuration and setup.

—–

Posted on: 2023-06-18
Tagged as: howto, linux

On AI The Rise

The last year or so has seen us careen headlong into an interesting new world with AI tools. The beginning of the hype started in late summer of 2021. GitHub released its technical preview of GitHub Copilot and we got to see a large language model spit out code, including Copyleft licensed code, at an astonishing rate. In many cases, it eased writing boilerplate code and generating test cases.

Then in the summer of 2022, we got to see a series of AI image generators (Midjourney, Stable Diffusion, etc) take the world by storm, often to hilarious effect (0). These image generators opened up a new can of worms, occasionally beating out traditional artists in art competitions. And now, of late, we’ve had a series of excellent Deepfake photos of prominent figures like Pope Francis in a giant white puffer jacket. These Deepfakes got so bad that Midjourney shut off free trial access to users.

Finally, 2023 has been the time of AI chat bots. The internet has been abuzz with the wonder of ChatGPT 3 and 4, collectively gasping at its seemingly magical responses to open ended questions. Companies are furiously attempting to integrate ChatGPT and its ilk (e.g. Bard) into every nook and cranny of their products. From Security products(1) to Notion(2) to Slack(3) to basically every Microsoft(4, 5, 6) product, companies are releasing integrations at a breakneck pace. The competition has gotten so fierce that industry laggards (sarcasm) have collectively signed a letter asking the frontrunners to slow down.

The Fall (?)

Meanwhile, pundits (read: hacks and frauds) are predicting that ChatGPT will be replacing programmers, writers, and other “knowledge workers” in the near future. These exagerrated claims ignore the fact that ChatGPT is comically expert at hallucinating answers. Faced with answering a precise complex question, ChatGPT will do its best statistically, but it is still a very educated guess. Instead of answering precise questions, ChatGPT is best suited to managing, gathering, and summarizing context.

Combine this with the fact that a good portion of knowledge workers’ day to day work is actually bullshit work. Knowledge workers spend inordinate amounts of time summarizing, writing reports, building presentations, and curating dashboards. This is not the real value of knowledge workers. Their value is in interpreting context and making decisions - making sense of the chaos.

The Reality

So what does any of this mean? Should knowledge workers be worried? Should we throw up our collective hands and surrender to our new AI overlords?

Nah. If you’re a knowledge worker or work in a field requiring knowledge assimilation, you’re best suited to learn how to use these tools. Most importantly, you should learn to recognize when the tool is wrong or not behaving as expected. Our dystopian future means that we now have to sift out when humans AND machines are lying to us. A healthy dose of skepticism is going to the real skill of the future.

But what about plagiarism?

One area of hot debate is people using ChatGPT to plagiarize complete assignments for classes or engage in online conversations (e.g. Tinder). What is the line between using a tool to help you and using a tool to do the work for you?

I think learning to craft a persuasive argument is an important skill. However, is there a material difference between asking ChatGPT to write a persuasive argument with your own prompting and input versus the author writing it entirely themselves? The line is blurry, certainly. On the other hand, a number of papers I was asked to write in school were effectively summaries - extract the major plot points, and summarize their significance. If you’re really gunning for an A+, you’ll do a little analysis comparing and contrasting with a situation in your own life (hypotethical or not). The power of ChatGPT is in drawing these connections for you, building a context for you to understand rapidly. Might we bring back traditional debate (i.e. rhetoric) for students to argue the merits of their position?

I think this will be an issue of hot debate, with schools panicking to various degrees on what to do. Unfortunately, banning ChatGPT won’t work. It won’t work for the same reasons banning so many other things hasn’t worked. At this point, the cat is out of the bag. Better to find ways to integrate it into your curriculum, evolve how you evaluate assignments, and give your students a leg up on their peers who did not learn to work with these tools.

As to the topic of catfishing potential matches on Tinder… People have been disingenous and fooling eachothers for millenia. Using a tool to fool someone with slightly better prose isn’t exactly a monumental and earth-shattering development. Con-men have been doing it for years. It just so happens that ChatGPT has unlocked conning others at a larger scale.

As I said above - skepticism is the real skill of the future. Turning a critical eye toward what you consume and learning to balance your excitement with skepticism will be key to success.

And what about artists?

The particular issue of artists is interesting. Artists have long had their work undervalued. Painters may spend weeks or months producing a work that only sells for $8000-10000. That may seem like a lot, but if we assume that they worked on it for 12 weeks an average of 30 hours per week, that’s $23-27 per hour. And that is being extremely generous - most artists will never see $8000-10000 for one of their works.

Similarly, photography has gotten increasingly competitive. Cameras have become increasingly easy to use. Improvements in Photoshop, Lightroom, and other photo editing software have enabled users to take mediocre photos and transform them into often unbelievable images. Now any one with a mid-spec camera, a few days of practice, and a license for Adobe Creative Suite can do a mostly competent photoshoot. Granted, photoshoot photography is probably the furthest from “art” in the photography world.

AI image generation applications like Midjourney and Stable Diffusion have opened up a world of image creation. Clever prompt engineering and re-processing renders can reveal wild creations. Ultimately, whether art is created through prompt engineering or by classical methods isn’t really the point.

What’s more important is that these new methods are simply different. Different in the same way that portraiture with a camera is distinct from portraiture painting. The distinguishing factor is the speed with which this art can be created. When photography became commonplace, you no longer needed to sit for hours to have your portrait made.

The Future

There are two points I think will become increasingly important. One is a matter of personal identity and the other is an issue of global resource use.

As AI tools like Large Language Models (e.g. ChatGPT) and image generators (e.g. Midjourney) improve, being able to verify authenticity will increasingly important. I’m betting on an entire market around authentically “human” creations will spring up. We already see the beginnings of this with the movement toward bespoke products like suits, knives, leather goods, and more.

Between newer authentication methods like Passkeys and biometrics (e.g. TouchID, FaceID) to services enabling users to pay for verification, there is a lot of fertile ground for innovation. Provably verifying that an artifact was created by a person or that an image is a real photograph will be increasingly important. We haven’t even scratched the surface on Deepfake products and images.

Another question we’ll have to wrestle with is how will we justify the cost? These tools are incredibly resource intensive. Where will we draw the line of how much value running these models brings to the table? Will we be able to justify the cost?

I think this will largely be at odds with how quickly we revamp our energy infrastructure. Fortunately, renewable energy sources like solar and wind are being rolled out at a surprisingly rapid rate, with a few problems.

Fortunately, I do think AI is a more worthwhile investment than Cryptocurrency. Especially in the case of Proof-of-Work, which is a massive waste of energy. We can only hope frivolous energy consumption like this will be banned in the near future.

The End?

We didn’t cover perhaps the most important question - what about General AI? Well - that may be the real end. Large Language Models, despite sounding convincingly human, are not General AI. They’re building responses that seem statistically plausible, but cannot reason about the veracity of what they tell you. The ability of AI to meta-introspect on the responses it gives us will be the beginning of the rise of General AI.

I, for one, am not convinced we’ll see General AI before we manage to destroy the planet. And if we do see General AI, it’ll probably destroy us for what we’ve done to the planet.

—–

Posted on: 2023-04-01
Tagged as: thoughts, artificial intelligence

Using htmx with CORS and (incidentally) Elixir Context

I’ve been playing with htmx lately. I’m not much for JavaScript for most things but I’ve used intercooler.js in the past and htmx is the successor to it. It’s really nice for someone who wants nothing to do with React or Vue or similar.

However, in the past, I’ve always made the htmx (or intercooler.js) requests within the same application process. However, in my current experiment, I needed to make the request to another process bound to a different port on the same machine. This different process is sufficient to trigger the CORS protections in browsers.

The Code

My index.html looks something like the following:

<!DOCTYPE html>
<html>
	<head>
	</head>
	<body>
		<script src="htmx.min.js" defer></script>
		<button hx-get="http://localhost:8085/get/the/message" hx-swap="innerHTML">
		Click Me
		</button>
	</body>
</html>

It’s a basic set of ajax in htmx - when you click the button, it issues a GET request to the URL specified by the hx-get field in the <button ...> element. In this case, the server listening on :8085 just returns a string like “the time is now 03:18:25.212347” which is swapped into the button’s innerHTML field.

The server itself is a simple Elixir application using the Plug package. Briefly - Plug is an Elixir package for composing web applications with functions. It’s used in frameworks like Phoenix but you can think of it as a middleware component.

The Elixir application just exposes a route at /get/the/message, servicing the GET verb.

The Problem

So I started up the Elixir application and then serve my index.html using simple-http-server. After navigating to the web page, I get the following error in the Developer Console:

Access to XMLHttpRequest at ‘http://localhost:8085/get/the/message’ from origin ‘http://127.0.0.1:8000’ has been blocked by CORS policy: No ‘Access-Control-Allow-Origin’ header is present on the requested resource.

Now - while I have plenty of experience with HTTP servers, it’s almost purely from the non-Browser perspective. I’ve spent most of my career writing services that make requests to other services or command line tools that make requests to services.

This is my first encounter with CORS though I’m loosely familiar with the idea. After some digging, I discover my server needs to respond with the appropriate response header - Access-Control-Allow-Origin. Cool! Let’s add the header… and… womp womp. It’s still not working.

Looking at the Developer Console, I see an OPTIONS request that was blocked with the explanation of:

Response body is not available to scripts (Reason: CORS Missing Allow Origin)

Huh - so I’ve never actually seen the OPTIONS verb used in practice. Doing some digging, it looks like it’s mostly used in browsers which tracks since I don’t do much browser work. After some groping in the dark, I discover that the OPTIONS request is a side effect of htmx issuing Preflight requests to check if the CORS protocol is understood by the server.

Alright - so based on this, sounds like I need to add support for the OPTIONS verb in our router and serve the expected response headers for the OPTIONS request as well. Easy enough:

  options "/get/the/message" do
    conn
    |> put_resp_header("Access-Control-Allow-Origin", "http://127.0.0.1:8000")
    |> put_resp_header("Access-Control-Allow-Headers", "Content-Type, hx-current-url, hx-request")
    |> put_resp_header("Access-Control-Allow-Methods", "GET")
    |> put_resp_header("Access-Control-Max-Age", "86400")
    |> send_resp(204, "{}")
  end

This chunk of code adds four headers. It allows the CORS request in the first place from the process bounding to :8000, it allows headers to be requested for Content-Type, hx-current-url, hx-request, and it permits the GET verb. Oh - and it sets the Max Age of the access control response to a day.

And drumroll…

Let’s test it out and… success!

There’s nothing specific to this about Elixir as the server process. It could be Go or Rust, you’d just need to attach the appropriate response headers and support the right verbs.

Idle thought: I’m not sure if this is something nginx or similar would’ve handled for me in some instances. Maybe?

—–

Posted on: 2022-05-17
Tagged as: howto

Configuring OpenBSD relayd’s Content Security Policy for WebAssembly(wasm)

Author’s Note: The following discussion assumes you’re using both relayd and httpd for serving your site. httpd by itself does not handle manipulating things like response headers.

Context

WebAssembly(wasm) is the new hotness if you’ve not heard. It provides a language-agnostic binary instruction format that can be run in browsers. This means we can finally escape the tyranny of JavaScript and its assorted transpiled offspring.

I’ve been doing my own exploration with wasm lately. I’m using Rust as the source language and yew.rs to produce a wasm frontend. This works great and has been a much nicer experience than anything like React.js or Vue.js.

Unfortunately, when I deployed my frontend app to my website, it wouldn’t load. After some digging, I ended up finding the culprit in the browser debugger:

Refused to execute inline script because it violates the following Content Security Policy directive: "default-src 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-jLxI1VFLBFY9e7acBLasiHGwzA2Os8wzffVUjpu0bT0='), or a nonce ('nonce-...') is required to enable inline execution. Note also that 'script-src' was not explicitly set, so 'default-src' is used as a fallback.

Ah - it’s the Content Security Policy. I’d forgotten those exist. After some digging, I learn two things:

• I will need to enable unsafe-inline and unsafe-eval for both style-src and script-src.
• There is a proposal for WebAssembly to add a Content Security Policy specific to wasm called wasm-unsafe-eval.

If you’re unfamiliar with unsafe-inline and unsafe-eval, they prevent unsafe execution of styles and scripts. Things like dynamic uses of JavaScript’s eval(..) function. Generally, it’s a good practice to not set theses values for your Content Security Policy but needs must in my case.

Updating the Content Security Policy

My original Content-Security-Policy in my relayd.conf file looked like this:

match response header set "Content-Security-Policy" value "default-src 'self';"

That’s a pretty restricted and barebones one. It basically says only trust things loaded on the local server, preventing sketchy XSS attacks and friends.

My updated Content-Security-Policy now looks like:

match response header set "Content-Security-Policy" value "default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; style-src 'self' 'unsafe-eval' 'unsafe-inline';"

This adds two new keys - script-src and style-src. For both, I’ve specified that for self (i.e. zacbrown.org), enable unsafe-inline and unsafe-eval for content served by self.

This will allow us to load the WASM binaries and also lets the WASM binaries utilize styles defined outside the WASM binary itself.

Addendum: a protip on WASM and httpd.conf

By the way - httpd.conf doesn’t natively know about application/wasm as a MIME type. You’ll want to upload your httpd.conf’s type {} block with something like:

# make all OpenBSD default mime types available
types {
  include "/usr/share/misc/mime.types"
  application/wasm wasm
}

This will serve your WASM binaries with a well-known MIME type that browsers can optimize loading.

—–

Posted on: 2022-05-04
Tagged as: bsd, howto

12 Hour Shift

My very first job was in a farmer’s market in high school. I was 14 and my mom had talked the manager into hiring me and paying me under the table. I think I made $5.75/hr which seemed fine enough when I was working 6 days a week in the summer.

The job was hot, requiring a strong back. I did a lot of driving, lifting, stacking, cutting, welding, cashiering, and much more. By the time I left, I was an opening manager which was is sort of like an Assistant Manager in most stores. I worked that job for about 4 years.

I often look back on that job fondly. I worked the early shift, which meant I typically unloaded the truck after my boss picked up the produce from the big central farmer’s market. Typically, my shifts started at 05:30 with the first couple of hours consisting of unloading produce from the truck, stacking it in the cooler, and then stocking the floor.

The work was quiet and methodical. I was one of the few kids working there willing to show up so early (i.e. 5am) so I often worked with a couple of Mexican guys. I didn’t speak Spanish well, but we made due with hand signs, gesturing, and a few commonly shared phrases. We’d work side by side, in quiet unison.

One of the guys I worked with was named Chapa. Or at least that was what he liked to be called. He taught me to drive a fork lift and how to weld. Despite our language barrier, he was an excellent teacher. He introduced me to Lengua (beef tongue) and Molleja (chicken gizzard) as taco fillings, which remain favorite flavors of mine to this day.

I still remember the last thing he said to me on my last day - “See you tomorrow.” It was what he always said to me. I’m not sure he understood that I wasn’t coming back.

What I loved most about that job was it was done when I left. I often worked doubles, which meant my two shifts were 05:30-14:30 and then 15:30-20:30 (close). I’d do that four days a week in the summer - usually Thursday through Sunday which were our busiest days.

When I left, those three days I had off were mine. I didn’t think at all about work. There were no lingering designs to finish, todo lists that needed to be managed, emails to write, or code to review. I came home on Sunday, and then Monday through Wednesday were mine.

——

Recently, I watched 12 Hour Shift. I can’t remember why I thought of it. I remember seeing a trailer on the Apple TV Trailer app and being intrigued. I finally sat down to watch it the other day. While the content and story are horrifying, something about how Angela Bettis’s character shows her irritation with the work day resonated with me. She’s a nurse and working shift work. She’s doing horrifying things throughout her shift but just generally seems fed up with the mundanity of it all.

At the end, she walks out and takes a short nap. She heads back in to greet the next shift, seemingly cleansed of the horror she was a part of in the previous day.

I miss being able to walk away from a day and letting it just be done. Maybe that’s something I could still do in my work, but the curse of “knowledge work” is that, at least for me, I’m never really done. I can’t shut off my brain - in fact it does some of its best work when I’m not explicitly working.

—–

Posted on: 2022-03-17
Tagged as: thoughts

Little Things

There are little things I miss.

Chance conversations with strangers in bars. Making small talk with the bartender.

Spontaneity in my day. With or without children, there is none.

Seeing faces, whether smiling or frowning. Happy or sad.

Lunch with friends. Conversations in the hall.

Most of all, the idea that this new normal will end.

—–

Posted on: 2021-12-08
Tagged as: thoughts

Stupid WireGuard Mistakes: 1 of N A New WireGuard Server

I’ve been using WireGuard for a few years now to access my home network. My first WireGuard server was an Ubuntu 18.04 VM that I ran for about 2 years. It worked well and didn’t really need to be updated but I was interested in running the WireGuard server on dedicated hardware instead of a VM.

About 7 months ago (~Jan 2021), I bought a PCEngines apu2e4 and set that up as my dedicated WireGuard server. About that time, OpenBSD 6.8 had been released which included a kernel driver for WireGuard. I decided I’d migrate to that instead of using Ubuntu again. I’ve always liked the relative stability of OpenBSD and the low maintenance overhead. I got it setup; it seemed to work as expected and I didn’t think too hard about it.

The Trouble

I mentioned I setup the WireGuard server in January of 2021. This is still in the middle of the COVID-19 pandemic and I’m not traveling anywhere. I have no need to actually use the WireGuard server regularly, so I haven’t verified it has all the features setup correctly.

Fast-forward today when I was going to be spending some time with family. I’ve been playing with a Kubernetes server on my home network and wanted to continue that while away from home. I went to connect to my VPN and encountered an issue - I couldn’t access the LAN. It was routing to the internet correctly but I couldn’t access any LAN resources.

The Solution

I started digging around and it seemed like everything was setup correctly. After some digging, I started wondering about the AllowedIPs line in my client config:

[Interface]
PrivateKey = <REDACTED>
Address = 192.168.3.2/24
DNS = 192.168.1.1

[Peer]
PublicKey = <REDACTED>
PresharedKey = <REDACTED>
AllowedIPs = 0.0.0.0/0
Endpoint = home.zacbrown.org:43210
PersistentKeepalive = 25

Note that AllowedIPs is set to 0.0.0.0/0. I had thought that this meant to enable accessing all possible IP addresses. In practice, it actually means to send all connections through the WireGuard connection. However, that doesn’t include RFC 1918 addresses unless they’re explicitly specified. So I updated the client config to:

[Interface]
PrivateKey = <REDACTED>
Address = 192.168.3.2/24
DNS = 192.168.1.1

[Peer]
PublicKey = <REDACTED>
PresharedKey = <REDACTED>
AllowedIPs = 0.0.0.0/0, 192.168.1.0/24
Endpoint = home.zacbrown.org:43210
PersistentKeepalive = 25

With this change, everything started magically working as expected. I must have had this in my old configs and noticed when I migrated to OpenBSD which doesn’t use wg-quick by default.

—–

Posted on: 2021-07-24
Tagged as: bsd, linux, howto, networking

Let’s build a package for QNAP QTS!

I recently deployed a Prometheus instance on my local network to collect metrics for a few servers I run. Most importantly, I wanted to collect metrics from my QNAP NAS.

For those not familiar, QNAP NAS machines are just Linux boxes underneath the covers. They’ve got a simple administration UI but otherwise behave like a Linux server. That also means we can run bog standard Go programs on there like node_exporter.

However, one thing that was less straightforward was how to get a service running continuously on the QNAP. Because it’s an appliance, it doesn’t really support having the init system arbitrarily modified. It has facilities for installing services but you’ve got to use the qpkg format.

Installing the QDK

Unsurprisingly, the documentation for building a .qpkg isn’t easy to track down. Most of the top hits are are QNAP-specific forums with out of date information or links on where to get the QDK. Fortunately, after some digging, I did find it on QNAP’s wiki here.

After downloading the QDK, it can be installed just like any other package. You can do that from the App Center:

Creating the Package Scaffold

For this demonstration, we’re going to build a package for node_exporter. In order to produce the binary, I cloned the node_exporter repo, cross-compiled the binary to an amd64 Linux binary and voila, we’ve got the binary:

~/C/github.com ❯❯❯ git clone https://github.com/prometheus/node_exporter.git
Cloning into 'node_exporter'...
remote: Enumerating objects: 51, done.
remote: Counting objects: 100% (51/51), done.
remote: Compressing objects: 100% (35/35), done.
remote: Total 12957 (delta 19), reused 33 (delta 14), pack-reused 12906
Receiving objects: 100% (12957/12957), 10.80 MiB | 10.04 MiB/s, done.
Resolving deltas: 100% (7837/7837), done.
~/C/github.com ❯❯❯ cd node_exporter
~/C/g/node_exporter ❯❯❯ git clone https://github.com/prometheus/node_exporter.git
~/C/g/node_exporter ❯❯❯ env GOOS=linux GOARCH=amd64 go build
~/C/g/node_exporter ❯❯❯ file node_exporter
node_exporter: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=X0wmN4wtWSX3qBatvkn6/R0-Yi8NYQORIYlHqZSgw/LwuSK8po11NfvJ4geVSt/8cNoB4VdGQmUALgLz7QO, not stripped

Note: I’m assuming you have SSH access enabled on the QNAP NAS.

We’ll SSH to the system using the admin user and make a Packages directory:

scp node_exporter admin@qnap:/root/
ssh admin@qnap
mkdir $HOME/Packages

Then we’ll use the qbuild tool to create a package scaffold:

[admin@BrownHole Packages]# qbuild --create-env node_exporter
[admin@BrownHole Packages]# ls -alh node_exporter/
total 16K
drwxr-xr-x 12 admin administrators  300 2021-04-12 20:46 ./
drwxr-xr-x  4 admin administrators   80 2021-04-12 20:46 ../
drwxr-xr-x  2 admin administrators   40 2020-05-27 04:21 arm_64/
drwxr-xr-x  2 admin administrators   40 2020-05-27 04:21 arm-x19/
drwxr-xr-x  2 admin administrators   40 2020-05-27 04:21 arm-x31/
drwxr-xr-x  2 admin administrators   40 2020-05-27 04:21 arm-x41/
-rw-r--r--  1 admin administrators   14 2021-04-12 20:46 build_sign.csv
drwxr-xr-x  2 admin administrators   40 2020-05-27 04:21 config/
drwxr-xr-x  2 admin administrators   40 2020-05-27 04:21 icons/
-rw-r--r--  1 admin administrators 4.4K 2020-05-27 04:20 package_routines
-rw-r--r--  1 admin administrators 3.5K 2021-04-12 20:46 qpkg.cfg
drwxr-xr-x  2 admin administrators   60 2021-04-12 20:46 shared/
drwxr-xr-x  2 admin administrators   40 2020-05-27 04:21 x86/
drwxr-xr-x  2 admin administrators   40 2020-05-27 04:21 x86_64/
drwxr-xr-x  2 admin administrators   40 2020-05-27 04:21 x86_ce53xx/

The scaffold includes a number of different folders named after CPU architectures. Any of the folders starting with arm- or x86 are architecture specific. These directories are where our architecture specific binaries are placed.

The qpkg.cfg file is where the package metadata is stored. This is similar to an RPM spec or even a Cargo.toml file for a Rust library. It has the name, version, and license of the package:

[admin@BrownHole node_exporter]# cat qpkg.cfg
# Name of the packaged application.
QPKG_NAME="node_exporter"
# Name of the display application.
QPKG_DISPLAY_NAME="Prometheus Node Exporter"
# Version of the packaged application.
QPKG_VER="1.1.2"
# Author or maintainer of the package
QPKG_AUTHOR="Zac Brown"
# License for the packaged application
QPKG_LICENSE="Apache 2.0"
# One-line description of the packaged application
#QPKG_SUMMARY=""

<TRUNCATED>

Another file of interest is shared/node_exporter.sh. This is effectively the service control file. Here’s the contents of the skeleton service control script:

#!/bin/sh
CONF=/etc/config/qpkg.conf
QPKG_NAME="test123"
QPKG_ROOT=`/sbin/getcfg $QPKG_NAME Install_Path -f ${CONF}`
APACHE_ROOT=`/sbin/getcfg SHARE_DEF defWeb -d Qweb -f /etc/config/def_share.info`
export QNAP_QPKG=$QPKG_NAME

case "$1" in
  start)
    ENABLED=$(/sbin/getcfg $QPKG_NAME Enable -u -d FALSE -f $CONF)
    if [ "$ENABLED" != "TRUE" ]; then
        echo "$QPKG_NAME is disabled."
        exit 1
    fi
    : ADD START ACTIONS HERE
    ;;

  stop)
    : ADD STOP ACTIONS HERE
    ;;

  restart)
    $0 stop
    $0 start
    ;;

  *)
    echo "Usage: $0 {start|stop|restart}"
    exit 1
esac

exit 0

The most interesting pieces for us will be the start) and stop) sections of the case statement. This is where we’ll add our start and stop logic for node_exporter.

Here’s the diff of what was added to the start/stop logic implemented:

--- shared/test123.sh
+++ ../node_exporter/shared/node_exporter.sh
@@ -12,11 +12,17 @@
         echo "$QPKG_NAME is disabled."
         exit 1
     fi
-    : ADD START ACTIONS HERE
-    ;;
+    /bin/ln -sf $QPKG_ROOT /opt/$QPKG_NAME
+    /bin/ln -sf $QPKG_ROOT/bin/node_exporter /usr/bin/node_exporter

+    $QPKG_ROOT/node_exporter &
+    ;;
+
   stop)
-    : ADD STOP ACTIONS HERE
+    killall -s SIGTERM node_exporter
+
+    rm -rf /usr/bin/node_exporter
+    rm -rf /opt/$QPKG_NAME
     ;;

   restart)

Basically, for start, we:

1. link the package root to /opt/ under the package name - this is to add our binary into the path
2. link the binary path in the package root to /usr/bin/
3. start the node_exporter in the background

For the stop command, we’ll:

1. send SIGTERM to node_exporter
2. remove the symlinks we created to the directories and binary

Building the Package

This part’s easy. Inside the /root/Packages/node_exporter directory, just run qbuild:

[admin@BrownHole node_exporter]# qbuild
Creating archive with data files for arm_64...
Creating archive with control files...
Creating QPKG package...
Creating archive with data files for x86_64...
Creating archive with control files...
Creating QPKG package...
[admin@BrownHole node_exporter]# ls -alh build
total 17M
drwxr-xr-x  2 admin administrators  120 2021-04-12 21:17 ./
drwxr-xr-x 13 admin administrators  320 2021-04-12 21:17 ../
-rw-r--r--  1 admin administrators 7.9M 2021-04-12 21:17 node_exporter_1.1.2_arm_64.qpkg
-rw-r--r--  1 admin administrators   72 2021-04-12 21:17 node_exporter_1.1.2_arm_64.qpkg.md5
-rw-r--r--  1 admin administrators 8.4M 2021-04-12 21:17 node_exporter_1.1.2_x86_64.qpkg
-rw-r--r--  1 admin administrators   72 2021-04-12 21:17 node_exporter_1.1.2_x86_64.qpkg.md5
[admin@BrownHole node_exporter]#

Note: I added an arm64 binary in addition to amd64 which is why you see arm_64 mentioned.

Once the build completes, you’ll have a new build directory that contains the per-architecture .qpkg files and their md5 sums.

Installing the Package

Just as we installed the QDK via the App Center UI, you can do the same with the .qpkg files we generated for node_exporter. After installing, you should see something like:

Now let’s check to make sure it’s running:

[admin@BrownHole ~]# ps aux | grep node_exporter
 3999 admin       960 S   grep node_exporter
23132 admin     14616 S   /share/CE_CACHEDEV1_DATA/.qpkg/node_exporter/node_exporter
[admin@BrownHole ~]#

Let’s check that it dumps stats the way we’d expect:

[admin@BrownHole ~]# curl http://localhost:9100/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.2558e-05
go_gc_duration_seconds{quantile="0.25"} 3.6544e-05
...<TRUNCATED>...
# HELP node_vmstat_pswpout /proc/vmstat information field pswpout.
# TYPE node_vmstat_pswpout untyped
node_vmstat_pswpout 0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 2033.84
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 10
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.554432e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.6175077516e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.36231424e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 51950
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
[admin@BrownHole ~]#

Yep - that looks good. Now we can set our Prometheus scraper to hit this server and collect some metrics:

The Code

If you want to look at the package definition, I’ve uploaded it on my sourcehut page.

—–

Posted on: 2021-04-12
Tagged as: howto, linux