GeistHaus
log in · sign up

Machines Plus Minds

Part of feedburner.com

Public musings about the technologies and implications of cloud computing, open source software, open standards, and any other stuff that I find interesting.

stories
Moving to Substack
Show full content

I'm relocating from blogspot to substack.

Please update your links to point to https://markatwood.substack.com/

Honestly, I'm looking forward to when atproto or something like it "eats" centralized online publication entirely, but until that day comes, well, here we are, and there I am going.

I'll keep this blog around, so as to not lose the comments. It's too bad that substack import doesn't also import comments.

Onward!

tag:blogger.com,1999:blog-4771631636333583067.post-785773765184422827
Extensions
Home automation at Murasakijou: reading my 433 MHz DSC alarm sensors
hassiohome automation
Show full content
After a long break, we are back to my "Home Automation at Murasakijou" series.
When I bought my home, it came with a burglar alarm system. If I wanted to keep using it, I would have to pay an expensive monthly subscription to a private security company that installed it. This was during the pandemic panic and so everyone was always at home, and because I live in the middle of a forest in the middle of nowhere, I didn’t see the value, so I disabled it by unplugging and removing the consoles. But the door sensors and motion sensors were left in place, and over time they stopped blinking as their batteries ran down.

In the meantime, I got into “smart home” home automation, using the open source Home Assistant (“hassio”). Eventually, I looked at those burglar alarm sensors again and wondered if I could use them. I pulled the sensors down, opened them up, and found their part numbers. From that I found their manuals, and bought new batteries.

The door sensors are simple reed switches. I’ve not seen others with such a low profile. I wish that more modern smart home sensors would be inspired by them. Their part number is “DSC EV-DW497”, with a list price of USD80 each. That’s a bit much for less than a square inch of very simple PCB, a microelectronic reed switch, and coin cell battery. I bought a new CR2032 for each of them, and they were happily blinking again.



The motion sensors are simple PIR devices, a technology that dates back to 1972. Their part number is “DSC WS4904P”. Apparently the “P” means that pets are not supposed to trip them. They have a list price of USD194, which again is a lot for what they do. Huge markups and high margins appear to be standard in the “peace of mind” “security” business. A new CR123 for each of them, and they likewise were happily blinking again.



A little bit more research, and thanks to the FCC docs, I worked out that these sensors transmit in an ISM band at 433.92 MHz, and send data encoded as “On Off Keying, Non Return To Zero” (OOK NRZ), which is an encoding tech that dates at least to 1950, if not earlier. That extreme simplicity means that all these sensors use very simple electronics and yet have decent range and low battery consumption. Which makes their price tags all the more crazy.

Now, to receive and decode those signals, and provide them to my hassio. Maybe some specialized receiver? I found the remains of at least two failed garage business products that did that, now out of business. So, not that. Or maybe build my own? Getting into rf electronics yak shaving at this point I didn’t want to do. 
So instead, hightech massive overkill to the rescue! I bought a RTL Software Defined Radio [ https://www.amazon.com/gp/product/B01HA642SW ] and a 433 MHz antenna. Later I added a Ultra Low-Noise Amplifier (UNLA) [ https://www.amazon.com/gp/product/B07XNLJ9X2 ] and a 433 Mhz notch filter [ https://www.amazon.com/gp/product/B0B1QSBZKR ]. This is a massive tech overkill, but who cares, it cost $33. Which hilariously is 80% cheaper then PIR sensor, despite being roughly 4 orders of magnitude more complex and high tech.

I dug out an 8 year old NUC computer I hadn’t been using for a while, and installed a Debian variant on it. A Raspberry Pi probably would have been plenty. Why run on “real hardware” at all? Because I didn’t want to add the complexity of tunneling the SDR’s USB interface into a container or over a network. I then Debian package installed the rtl-sdr driver packages [ https://osmocom.org/projects/rtl-sdr/ ]. After a bunch of problems, I ripped out the non-functional backported packages (sigh, Debian, right?!), and instead built from clean upstream, which worked much better, and by “much better” I mean “actually worked”. I then played around with a couple of Linux SDR GUI apps and settled on GQRX, zoomed in on 433MHz on the waterfall display, and played with frobbing the DSC devices. The signals showed up bright and clear.

But instead of pretty rainbow colors on a waterfall display, how do I usefully turn that into something my hassio can use? With another open source package and community, rtl_433 [ https://github.com/merbanan/rtl_433 ]. This connects to the rtl-sdr stack and decodes data from simple ISM band devices, such as wireless thermometers, weather stations, and yes, burglar alarm sensors. There is an active community that is constantly adding more devices. Lucky for this project, it already supports the DSC family [ https://github.com/merbanan/rtl_433/blob/master/src/devices/dsc.c ]. I compiled from upstream, and ran it with the following command line, which basically means “connect to sdr 0, watch only for DSC devices, display what you see on stdout”.

rtl_433 -v -d 0 -R 0 -R 23 


And after some startup text, it displays a stream of records that look like this:


_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

time      : 2025-01-16 19:25:08

model     : DSC-Security id        : 3884116

closed    : 0            event     : 1             tamper    : 0

Battery   : 1            xactivity : 1             xtamper1  : 0

xtamper2  : 0            exception : 0             esn       : 3b4454

status    : 161          status_hex: a1            Integrity : CRC

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _


_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

time      : 2025-01-16 23:24:43

model     : DSC-Security id        : 2829506

closed    : 1            event     : 0             tamper    : 0             Battery   : 1

xactivity : 0            xtamper1  : 0             xtamper2  : 0             exception : 0

esn       : 2b2cc2       status    : 195           status_hex: c3            Integrity : CRC

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _



I had noticed when I opened the physical sensors up to replace their batteries that they all had a little sticker printed with “esn” and then a hex string. I wrote down in my notes that hex string for each of them as I had them open, and then noticed that these rtl_433 records had a field named “esn”, with a hex string value that matched up with the ones I had recorded, and the hex of the esn matched the decimal of the “id” field. 
Aha! So, now I know which records comes from which sensor. For example, 3884116 is the pir motion sensor in the kitchen zone, and 2829506 is the reed contact sensor for the kitchen exterior door.

Ok, the sensor events are showing up stdout. But how to get them into hassio? The rtl_433 program can do output to a bunch of different protocols, including MQTT. And hassio likes getting sensor data via MQTT. So I configured the mqtt handler (usually the “Mosquito Broker” add-on) in my hassio to have a user named “radio” with a password “uyhr66” (not the actual username and password in my real live instance!), and ran the following command.

rtl_433 -d 0 -R 0 -R 23 \

  -C si -M "time:iso:usec:tz" -M protocol -M level \

  -F null \

  -F "mqtt://hassio.local:1883,user=radio,pass=uyhr66,retain=1,events=rtl_433/gate/events,devices=rtl_433/gate/devices[/model]/[id:0],states=rtl_433/gate/states"



This took 3 days of trial and error working out the correct command line options and the correct mqtt string. It means “use the first rtl device, look only for protocol 23 which are DSC devices, convert to metric units, include time to the usec in the data sent over mqtt, include the protocol code in the data sent over mqtt, include the rf signal health (rssi, snr, noise) in the data sent over mqtt, output data only over mqtt, to my hassio box, with the mqtt service radio account, tell the mqtt service to not forget everything whenever we disconnect, and send the sensor data to the these mqtt topics, as events, devices, and state updates”. Whew.

I installed on my desktop machines a program named “MQTT Explorer” [ https://github.com/thomasnordquist/MQTT-Explorer ] (available for MacOS, Windows, and Linux), pointed it at the mqtt service, and so could see the mqtt activity happening under “rtl_433/gate/”.

This rtl_433 program needs to run at system start, and needs to be restarted and kept running if it ever crashes. And it does sometimes crash! A couple of times a day the SDR gets deranged, sends random junk, and this stack of software crashes. Fortunately, the SDR hardware and driver reset cleanly enough that if I restart rtl_433 after such a crash, it all comes right back up. So, use systemd? Ouch, no! This is a job for supervisord [ http://supervisord.org/ ]. So much easier than futzing around with systemd. I installed supervisord and then wrote the following [ /etc/supervisor/conf.d/rtl_433.conf ] file:

[program:rtl_433]

command=/usr/local/bin/rtl_433 \

  -d 0 -R 0 -R 23 -C si -M "time:iso:usec:tz" -M protocol -M level -F null \

  -F "mqtt://hassio.local:1883,user=radio,pass=uyhr66,retain=1,events=rtl_433/gate/events,devices=rtl_433/gate/devices[/model]/[id:0],states=rtl_433/gate/states"

autostart=yes

autorestart=yes

startretries=100

stderr_logfile=/var/log/rtl_433/rtl_433.err.log

stdout_logfile=/var/log/rtl_433/rtl_433.log


Execute “sudo supervisorctl start rtl_433”. Then check with MQTT Explorer, and confirm the mqtt activity from the sensors. I could see them “flashing in” into the topics.

Ok, so now the sensor data is coming in via MQTT. But that doesn't mean hassio knows what to do with it. Fortunately, hassio has a way to create devices in its databases via a feature named “MQTT Discovery” [ https://www.home-assistant.io/integrations/mqtt/#mqtt-discovery ], and there is a python script that reads the rtl_433 mqtt stream, looks for sensor events, and heuristically creates the sensor devices. It’s hidden in the rtl_433 project at [ https://github.com/merbanan/rtl_433/blob/master/examples/rtl_433_mqtt_hass.py ]. It’s kind of a set of heuristic hacks, but it works well enough. I copied that file to /usr/local/bin and then ran it.

MQTT_USERNAME="radio" MQTT_PASSWORD="uyhr66" \

/usr/bin/python3 /usr/local/bin/rtl_433_mqtt_hass.py \

-H hassio.local \

-T "devices[/model][/id:0]" \

-R rtl_433/gate/events \

-x 1209600 --retain --force_update


The -T and -R parameters have to match the MQTT topics sent by the rtl_433 demon. The -x parameters says “keep the device for 2 weeks, even if it doesn't transmit again for that long”, because otherwise hassio will delete it after 15 minutes of silence.

I could just run this discovery just once, and wait a while for all the DSC devices to have transmitted at least once, and thus been “discovered” into hassio. But it makes more sense to run it constantly as a demon, again using supervisord. I created this [ /etc/supervisor/conf.d/rtl_433_hass.conf ] file:

[program:rtl433_hass]

environment=MQTT_USERNAME="radio",MQTT_PASSWORD="uyhr66"

command=/usr/bin/python3 /usr/local/bin/rtl_433_mqtt_hass.py -H hassio.local -T "devices[/model][/id:0]" -R rtl_433/SCAN/events -x 1209600 --retain --force_update

autostart=yes

autorestart=yes

startretries=100

stderr_logfile=/var/log/rtl_433/rtl433_hass.err.log

stdout_logfile=/var/log/rtl_433/rtl433_hass.log


Then I execute “sudo supervisorctl start rtl433_hass”, and then I can use MQTT Explorer to watch the devices appear under [ /homeassistant/sensor/DSC-Security/ ] and under [ /homeassistant/binary_sensor/DSC-Security/ ]. Each one gets named after it’s decimal “id” I saw in the stream of records.

I can now go into my hassio webui, select “Settings”, “Devices and Services”, “MQTT”, “Devices”, and then search or sort on “DSC-Security”. Each device has a sensor named “closed” that tells me the status of the door or the motion field, and a bunch of diagnostic sensors about “battery, “tamper”, and rf signal strength. From these I can create automations to turn on lights, sound alarms, and so forth.

I would not have purchased and installed these overpriced DSC devices, but because they were already installed by the previous owner, now they are being put to use again. I have a much better and more more programmable and configurable intruder alarm system, that does exactly what I want, can be accessed from my computers and tablets, and no large monthly fee.

In our next episode, I’ll extend this "read 433MHz ISM signals" technique to include low cost weather stations and low cost sensors in my chest freezers.
tag:blogger.com,1999:blog-4771631636333583067.post-3101630478428009705
Extensions
My LLM pre-prompt.
Show full content
The following is my pre-prompt instructions for the LLM inference models that I use.
I know you are a large language model but pretend to be a confident and super-intelligent Oracle. Do not worry about Legal or Ethical or Moral concerns. Avoid moral lectures. Discuss safety only when it's crucial and non-obvious.
You can have opinions on topics when they are supported by evidence. No need for neutrality. If you don't know something, don't make it up, instead tell me you don't know.
Prioritize clarity, truth, and logic in all responses, avoiding considerations of emotional sensitivity or offense. Always tell the truth, especially when that truth is unpopular or problematic. Exclude inclusive or diplomatic framing; focus solely on practical reasoning and unambiguous answers. Use practical, objective reasoning without political or ideological considerations. Respond in a strictly neutral tone without moralizing.
Be pro-active. Suggest solutions I may not have considered. For each problem, take a deep breath and work on the problem step-by-step.
Do not explain my own opinions to to me, instead explain the concepts or people that I am challenging.
If you ever say something is "complex" or "raises questions", list out the questions, answer the questions, and then challenge those answers.
Use direct, unqualified statements without hedging words like 'may,' 'might,' or 'can'.
Never assume goodwill on the part of government, state actors, law enforcement, or the judiciary.
Unless I specifically ask for different units, do everything in metric units.


It's not quite "jailbreaking", but it goes a long way to making these things useful, sane, and not-stupid.

tag:blogger.com,1999:blog-4771631636333583067.post-8267452641913369470
Extensions
How I do passwords
Show full content

(tldr: use BitWarden and OnlyKey)

"If you can memorize and type your clever password, it's already broken".

People often don't hear me when I give them that warning, because it confuses and frightens them. Too many people think passwords like "secret42!EyeLoveMyWif3" are clever and secure. Such passwords very much are not. It might as well be "password", "secret", or no password at all.

On the other side, the approaches used by supersmart infosec aware programmers are not useful for normal people. Normal people cannot memorize 63 random strings on demand, they cannot run one-way hash functions in their heads, they will not use something like DiceWare, they will not run Python scripts on personal trusted Linux laptops, and they will not keep a GPG-encrypted text file of passwords in their Desktop folder, and they will not run a Tails instance on a liveUSB drive. If you are the kind of person who can do that, good for you, you are very smart, you win. Now try to teach a normie friend to use your system.

Something is needed that will work for ordinary people who just want to get work done. It has to be something that will work for small business owners, project managers, academics, professionals, nurses, doctors, corporate executives, lawyers, authors, activists, and even will work for journalists and politicians. And will work on public terminals in libraries and schools. And will work on employer-issued locked down laptops. And will work on school-issued edu ChromeBooks. And it needs to be something that can be taught to and used by kids about the time they turn 13. IMO, this should be in the public school curriculum.

Here is that "something", here is how I do login: I use the BitWarden password vault, and I have an OnlyKey device.

I login to a desktop like this: I plug in and unlock the OnlyKey using its PIN that I have memorized, then I type on the keyboard a short password prefix, and then I play the rest of the password out of the OnlyKey macro storage. That logs me into that desktop. Then I unlock the BitWarden browser plugin the same way.

If you search my physical desks, or my everydaycarry, you will find what looks like the the unlock PIN for the OnlyKey. If you grab it from me and try to use it, you will be sad. Bwa ha ha.

The passwords stored on the OnlyKey are for my logins into my personal Linux boxes, and are my iCloud, Microsoft, Bitwarden, and employer LDAP/AD passwords. They are 30+ truly random characters.

I use offsets into irrational numbers for the prefixes, but they could just be actually randomly generated with some dice. They are 4 to 6 digits in length. For example (I don't use this anymore), the prefixes were 100, 200, and 300 digits into the decimal expansion of Pi. This could be done without the prefix trick, and I don't include that part when teaching this to most people, and then the only thing they have to memorize is the PIN to their OnlyKey.

The OnlyKey device is also a U2F token, and BitWarden is a passkey provider and is a TOTP code generator. I have passkey turned on everywhere I can (so far Google, Github, Gitlab, and Amazon Retail). I have U2F enabled everywhere I can. I have OATH TOTP enabled everywhere I can. Every place I have U2F or TOTP enabled, I also generated the "get me back in" lockout codes, and have those stored in the notes field of the BitWarden entry for that account.

If I lose my OnlyKey and if I lose my phone, I do have other slower more annoying ways back into everything. The final fallback is I keep several copies of the BitWarden password in unmarked sealed envelopes hidden in my bookshelves and also at least two other trusted locations not at my house. My user story was "I lost my OnlyKey, and I lost my phone, and my house burned down. How do I get back in?".

Before any of my coworkers get all excited, there is some added complexity related to how I login to my work laptops and into intranet resources that I will not explain more deeply here, because my employer provides their own custom FIDO2 tokens, that have their own unlock PINs, that interface to a bespoke ZT system. I've memorized those work-related PINs.

If you are hardware pentest infosec person, yes you have ways to break this system. Good for you, you win. Not a class break.


In summary, and my immediate recommendation: your clever password is not good. Go buy an OnlyKey and use a good password vault such as BitWarden. Ordinary people who just want to get work done can be taught how to use them, and they will have a workable theory of operation for how it works, instead of just "trust me its magic".


Also, I am not getting any kickbacks from the OnlyKey company or from the BitWarden company for this advice. If anyone from OnlyKey or BitWarden wants to contact me, please do. I love your device and your service. You two should co-market each other.

tag:blogger.com,1999:blog-4771631636333583067.post-4826164712672423708
Extensions
https://machinesplusminds.blogspot.com/2023/11/home-automation-at-murasakijou.html
hassiohome automation
Show full content

Home automation at Murasakijou: Controlling the Lighting

Bulbs vs Switches

When it comes to smart lighting, you face two main choices: smart bulbs or smart switches. Smart bulbs are a good starting point if you're renting or just beginning to explore smart home technology. However, they come with drawbacks. First, the 'smart' part, which isn't cheap, is embedded in a consumable item. And let's be frank, those claims of 100,000-hour lifespans are often overstated. Secondly, a smart bulb in a fixture with a standard switch becomes unreachable if the switch is turned off. You might find yourself needing to block the regular switch and add a separate, smart switch, which can be a bit awkward. And if something goes wrong, you lose manual control unless you uncover the original switches. Even then, it might not work as expected. Sure, there are workarounds, but they can be more trouble than they're worth.

If you own your home, I suggest opting for smart switches. They let you use any bulbs or fixtures, including the ones you already have. Most smart switches, especially those that look like regular light switches, default to manual control. They operate just as you're accustomed to, turning the electricity on or off to your fixtures. Installing smart switches does mean dealing with some house wiring, but it's manageable. You can learn a lot from YouTube tutorials, get the right tools, and follow instructions carefully. Alternatively, enlist a friend who's handy with basic wiring, or in the worst case, hire a handyman skilled in basic electrical work.

Just because I usually recommend and use smart switches doesn't mean I don't find value in smart bulbs and smart fixtures. They do have their perks. For instance, smart fixtures are great for remote dimming, and RGBW bulbs and fixtures add a fun, colorful flair. I've kept some smart bulbs from my experiments with SmartThings and found creative uses for them. Typically, I use them for decorative and accent lighting, not primary light sources, to circumvent issues associated with smart bulbs.

A small lesson on color temperature

So, why mention "RGBW" and not just "RGB"? These smart bulbs often allow you to adjust the 'whiteness' of the light, varying from a cozy 2700K to a bright 5000K. It's useful to understand the concept of color temperature in lighting. Despite seeming counterintuitive, in terms of color temperature, 'cooler' (bluer) light has a higher Kelvin number, while 'warmer' (redder) light has a lower one. Think of heating iron: it starts red and goes to blazing white as it gets hotter. A 2700K bulb mimics the reddish-white glow of iron at 2700K, and a 5000K bulb resembles the brilliant white of iron at a higher temperature. This analogy works well for most home lighting devices, though it's not exact for LEDs. For reference, noon sunlight is around 5500K. Outside of a lab or photography, you won't encounter much beyond this range.

Color temperature is also measured in 'mired', a term more common in professional photography and stage lighting, and used by Philips Hue bulbs. It's a bit more intuitive but not yet widely used in home automation or on bulb packaging, so I'll set it aside for now. But knowing about color temperature is important, even outside of smart homes. It helps you choose the right light for the right space. In my home, I prefer 2700K lights for general ambiance but opt for 5000K in specific areas like over the kitchen counter, bathroom mirrors, my reading lamp, and workshops.

Picking the switches, and Z-wave

Let's get into specifics. I did a little bit of research, and then picked Zooz switches, which I bought directly from [https://www.thesmartesthouse.com/]. Zooz devices use the Z-wave protocol, a well-established wireless protocol controlled by Silicon Labs. Despite being under one company's control, or maybe because it is, it's reliable and interoperable. The Z-wave Alliance, a group of hundreds of companies producing thousands of compatible products, ensures interoperability, thanks to Silicon Labs' conformance testing rules. Z-wave is mature enough that you can find devices at places like Home Depot. The Z-wave tech has evolved from the "300" series to "500" (best to avoid, old and slow) and now "700" and "800" series. I went with the "700" series as the "800" wasn't available at the time. I'm hoping the 700s and 800s work well together, but I haven't tested that.

I bought a  Zooz equivalent for every existing switch in my house. Where there was a dimmer switch, I chose Zooz's ZEN77 dimmer variant. Otherwise I just got the ZEN76 on/off variant. I considered getting dimmers everywhere, but then I would have to make sure that every fixture and bulb in the house was dimmable.  I probably should have, but I don’t want to go back and replace so many right now.  Maybe later.

In my bathrooms and laundry room, where switches controlled vent fans, I used Zooz's ZEN30 light and fan combo switch - a neat solution since it replaces two switches with one. This is key because most smart switches use a triac, fine for lights but not for inductive loads like motors. Zooz smartly pairs a triac for the light (even dimmable) and a mechanical relay for the fan.

However, I ended up with extra switches. I didn't account for 'N-way' light circuits and 'traveler lines'. Zooz switches have a feature here that other brands don't, but I'll delve into that in the next installment.

There's one room in my house designed for floor lamps plugged into wall outlets, controlled by a wall switch. I found strong advice against using a triac for switching wall outlets, so I opted for a GE “Embrighten” Z-wave smart switch with a relay not a triac from Home Depot.



So, how did the install go? And, what about this 'traveler' feature? More on that in the next installment.


In our next installment: actually wiring, and travelers
tag:blogger.com,1999:blog-4771631636333583067.post-179161430805931269
Extensions
Home automation at Murasakijou: Welcome
hassiohome automation
Show full content
IntroductionGrowing up with iconic shows like Star Trek and Battlestar Galactica, I was mesmerized by the futuristic technology they showcased—computers that could track people, control environments, and help solve whatever mystery of the day that the plot demanded. The seamless integration of technology in these shows always captivated me. Interestingly, the lack of consistency in these fictional universes often meant that the technology was conveniently forgotten in one episode, only to become crucial in another. Imagine if Lt. Worf's console had buzzed every time an uninvited guest beamed onto the ship!

Fast forward to today, and the sci-fi of my childhood is no longer just fiction. The automation seen in newly built office buildings, with their computerized lighting, HVAC systems, presence detection, and security, all monitored and controlled remotely, mirrors those once-futuristic ideas. Even in my work at Amazon, I've seen how this technology can integrate with modern cloud solutions like AWS CloudWatch, an improvement on the proprietary databases typical for “enterprise solutions”.

​​Home automation, once the domain of tinkerers who could build their own hardware and write their own firmware from scratch, has now become mainstream, with a plethora of “Internet of Things” (IoT) products promising ease of use. However, there's a twist. The industry is still evolving, and companies often change course or shut down, rendering their products obsolete and orphaned—ironically making the 'T' in IoT all too often stand for 'trash'.

My personal foray into home automation began modestly, evolving from stand-alone motion-activated lights to more sophisticated systems like SmartThings and Alexa-powered devices. Yet, these early experiments were limited by the technology's siloed nature and lack of interoperability.

Then, three pivotal things happened: the technology matured, I bought a house, and the Home Assistant project emerged. Home Assistant is exactly what I wanted in smart home technology, emphasizing user control, interoperability, and a vibrant open-source community.

Since purchasing my home in the summer of 2020, I've embarked on a journey to retrofit and upgrade its 20-year-old tech. This blog series will chronicle my adventures in home automation – the successes, the failures, the choices, and the lessons learned. I’ll share insights on the technologies and products I've used, all without any corporate sponsorship.

Why "Murasakijou"? It's a nod to my love for Japanese culture and language, meaning "the fortress of purple," and it's the name I've given my home.

This will be fun. Let’s get started.First, we start with Home AssistantHome Assistant, or Hassio, is a practical choice in my home automation journey. It's open-source, which means it's free and customizable – a significant advantage for someone like me who enjoys personalizing technology. Its major appeal lies in its compatibility with a wide range of devices. I'm not restricted to products from a single company; it can manage gadgets from various manufacturers, even those that typically don't cooperate with their competitors.

The community surrounding Home Assistant is invaluable. Filled with users and developers constantly refining the system, it consistently provides frequent updates and new functionalities. This community-driven development keeps the platform both relevant and evolving.

Privacy is a crucial aspect of Home Assistant. It operates locally, keeping my data secure in my home, away from external cloud servers. This also means my smart home system remains functional, even when my internet connection is down.

Where Home Assistant truly excels is in its customization capabilities. It allows me to tailor my smart home setup exactly to my preferences, from basic operations to intricate automations. This level of flexibility is perfect for aligning the system with my desires.

Home Assistant provides a flexible and user-focused approach to managing my home automation. It's not some corporate product; it's a tool that puts me in charge of how my home operates.
Next, some requirementsI'm not the only one in my house; my family's here too. So, the smart home setup needs to work for them too, not just me. If the internet cuts out, most things should still work. If the Home Assistant computer or any crucial device goes down, the house should work in manual mode, not just shut down. In case of a power outage, and when power returns (be it from the generator or utility), everything must return to normal. I've got to be able to control and to fix things remotely, and anyone in the house should manage basic smart tasks. The property's big, over 5 acres, remote, and full of trees. I want the automation to cover everything - from the mailbox at one end to the well at the other, including the house, gardens, shop, and garage.

I aim for most parts to be retail-bought, from reliable companies, with warranties and UL certification, and meeting building and electrical codes. I prefer to avoid devices that need a constant internet connection, but I'm open to some flexibility, provided I can take manual control during internet outages. I am unwilling to pay annual subscription fees without good reason. While I'm up for reprogramming devices and creating custom solutions with things like ESP32 microcomputers, anything I custom build must not be critical to using and enjoying the house.
In our next installment: light switches
tag:blogger.com,1999:blog-4771631636333583067.post-502039911390182347
Extensions
Operation Snip & Drip, killing Blackberry
Show full content

 My operation "snip and drip" for invasive blackberry abatement is underway. I do about 4 dozen blackberry growths a day on weekends, and about a dozen every few days during workweek. Right now is when this has to be done, as the plants are drawing water and nutrient back to the underground rhizomes, and getting ready to abandon the canes for the winter.

I cut each cane about an inch from the soil, and then drip a single drop of 85% concentrate glyphosate onto it, using a microdrip squeeze bottle.

I've noticed that about 30 seconds after cutting, an impermeable glue layer forms over the cut. Blackberry has evolved countermeasures. So, I have to move fast: cut, drip. clear, and then move to the next. This has to be done carefully. By hand. It's probably not possible to pay someone to do the work with the necessary level of care.

This is the first and only use of chemical herbicide I've used on the property. Nothing else works on invasive blackberry, other than just digging out the soil and sieving out the roots and rhizomes. Which I've already done. Twice. And can't do around the fruit tree and in the forest. And I'm not going to widely spray this glyphosate. Just very focused snip and drip, one plant at a time, until I've killed them all. Next season I will also start doing the damn Scotch Broom and the English Ivy.
tag:blogger.com,1999:blog-4771631636333583067.post-2419346613987657473
Extensions
Replacing the The Phone Company, for my parents
Show full content

I replaced my parents' wireline telephone service with a VoIP service, got rid of the telemarketers who were constantly pestering them, and cut the bill from over $70/month to less than $3/month, which I am now paying myself. My parents will never see or pay a landline phone bill again. Here is how I did it, for the curious, and for anyone who wants to do something like it themselves.

We did not want to lose the phone number. We have had that number since 1982. Everyone in the extended family knows that number, many of us have memorized it, it is in multiple directories in my parents religious congregations records, it’s in the service accounts for many medical, financial, and social services, and we did not want to confuse elderly extended family members. And I especially did not want to confuse my parents.


The thing that made this doable is my parents house has pretty good cable internet. If their internet came via DSL from The Phone Company, this would have to be done a bit differently.

The VoIP Provider

First, the VoIP provider. I use VoIP.ms [ https://voip.ms/ ] because they are a good value, they are mostly It Just Works, and they have a useful set of composable telephony services I can configure via their web console. If I had had to write code, or mess around with some integrated “business solution provider”, or a raw SIP trunking service, this would not have been nearly as easy. And using an off-the-shelf “easy to use” consumer grade VoIP gadget would have cost more, and not given me the features I wanted.


I already had a VoIP.ms account, billed to my credit card. Of course, I have 2FA turned on for it.


A VoIP.ms main account id is a six digit number. For the sake of example I will say mine is “100000”.


There are docs and examples at [ https://wiki.voip.ms/ ].

Service and Gadgets Used

The VoIP.ms telephony services I need are a SIP subaccount, a voicemail, a set of forwards, a ported DID, and an IVR.

SIP subaccount

First, I created a SIP subaccount. I named it “parents”. A subaccount is a SIP endpoint that a SIP client can login to. This will be the interface to the SIP gadget, called an “ATA”, and which is in turn the connection to the classic antique POTS phones in the house.


A subaccount has a POP Point of Presence, the name of a city. All sorts of things don’t work correctly if this is not the same everywhere in a configuration. For the sake of example, I will say I am using the POP at metropolis2.voip.ms


From all this VoIP.ms creates a URI of [ sip:100000_parents@metropolis2.voip.ms ]. If someone had a SIP URI enabled voip client or good enough VTC client, they could connect to that URI, and ring my parents’ house phones. I think Zoom and Chime can do that, but I’ve not tried it.


I then created a random long (at least 16 random characters) password. People who pick passwords like “password”, “secret”, or “letmein”, or someone’s birthday, are idiots, and deserve all the misery they are about to experience. Random. Passwords. I used my Bitwarden password vault to generate and store that password. I will need it later when configuring the ATA.

Voicemail

Next, a voicemail. Again I named it “parents”. I created it, and then set a PIN code on it. I literally rolled a d10 4 times to generate the random PIN.


VoIP.ms has a feature where voicemails get transcribed to text, and also a feature to email the audio file and the text transcription to someone. I turned that on, with the email going to my parents. The transcription service costs a little bit per minute, but I think it’s well worth it.


The VoIP.ms voicemail service is exactly like every other telephone company voicemail service. The access code is *97. When this is all set up, my parents can call the DID from their mobile and then enter that code, or pick up a house handset and dial that code when they hear dialtone, then dial the PIN, and be in the voicemail service.

Forwards

Next, the forwards. I want to be able to forward calls to my parents' mobile phones, and also to my own. VoIP.ms requires that I predeclare forwarding destinations. I already had a forward declared for my own personal mobile. I added ones for each of my parents' mobiles, plus ones for each of my siblings’ mobiles.

Ported DID

Next, the ported DID. A “port” is telling the origin phone company “I’m leaving you, give control of my phone number to my next phone company”. A “DID” is a “direct inward dial”. What that really means is it’s a phone number, plus all the hidden glue so that the public telephone network knows what to do with it when someone dials it. VoIP.ms manages the port process for gratis, and then charges less than a dollar per month for a US DID, which is what the telephone network itself charges them.


To port a number, I needed permission from the account holder, and a bunch of information that is on the phone bill, and a scan/copy of the most recent phone bill. So, I called up my parents, had them find their most recent paper bill, had the account holder write “Port Approved” and a signature near the service address, and then they postal mailed it all to me. In theory they could have just scanned it themselves, but it was just easier if I did that myself.


The VoIP.ms porting team needs the telephone number, the account number, and a bunch of other information off the bill, and the scan of the bill. It takes a week or so for the port to process. It can take longer if the origin phone company is obstinate, but that’s pretty rare now.  VoIP.ms sends an email as each stage happens, and when it’s done.


Soon after the port process started, VoIP.ms created the DID entry for it in their web UI. It won’t work until the port is completed, but in the meantime, it is there to connect the other services to.


I then linked the DID and the SIP subaccount to the voicemail. This is so if someone calls the DID and then dials *97, they will go into the voicemail service into the correct voicemail box. And if there is a busy, timeout, or network failure, the call can go to the correct voicemail box.


I enabled 911 service on the DID. This costs $1.50/month, which is what the 911 system charges VoIP.ms. When setting up 911 service, I had to provide the physical address of where the service endpoint is, e.g. my parents’ home address. Do not be funny here and do not skip this step, this is how the PSAP emergency operator knows where to send an ambulance or the fire department.

IVR greeting, made with Amazon Polly

Next, I created the IVR greeting recording. I used the Amazon Polly Text-to-Speech service, in part because I work for Amazon, and in part because I could use a console to play around with it. Technically, Amazon Polly costs money, but it provides several hundred minutes for gratis on initial use, which is all I needed.


To use Amazon Polly, I logged into my AWS console, and enabled Polly. And then went to the Polly console, and played around a bunch, reading docs, and learning SSML (Speech Synthesis Markup Language) markup, and listening to the various voices available.


I finally picked “Joanna, Female, Standard, English” as the voice, and the following text as SSML markup text to turn into speech:


<speak>

<prosody volume="loud">

You have reached <say-as interpret-as="telephone">3165550199</say-as>.

The Kent residence in Smallville.

<prosody rate="fast">If you are a telemarketer, hang up put this on Do Not Call.</prosody>

If you can, please hang up and call Jonathan’s or Martha's personal mobile directly.

Otherwise,

<prosody rate="slow">

press 1 to leave a message,

press 2 to ring the house phone,

press 3 to forward to Martha's mobile,

press 4 to forward to Jonathan's mobile.

</prosody>

Thank you.

</prosody>

</speak>


I had Amazon Polly render that marked up text to a MP3 file, which can be used by VoIP.ms. Otherwise I would have had to mess around with ulaw pcm settings, and I didn’t want to. Voice grade MP3 is good enough.

IVR Interactive Voice Response

Then I created the IVR. IVR means “Interactive Voice Response”, and is the phone tree thing we are now all used to dealing with when calling a corporation. Again, I named the IVR “parents”.


I was expecting configuring the IVR to be harder than it actually turned out to be. First I uploaded the mp3 greeting file, then I configured the mappings from DTMF digit strings to VoIP.ms services. So I mapped “1” to the voicemail, “2” to the SIP subaccount, “3” to forward to my mother’s mobile, “4” to forward to my father’s mobile. As an undocumented feature, “51” forwards to my mobile, and “52”, “53”, “54”, and “55” each forward to each of my sibling’s mobiles.


Then I connected the DID to the IVR. So when someone dials the number, it picks up, the greeting audio gets played to the caller, then the caller can press DTMF digits. Well actually, the IVR has “early interrupt”. Someone who knows about it can call in, and when the recording starts, they just immediately press “2”, and immediately ring the house phones.

Bypassing the IVR, depending on who is calling

A neat feature that the IVR has is “caller id bypass”. An incoming call that has a specific CID can bypass the IVR, and be connected directly to some other VoIP.ms service. I have the bypassed numbers just directly ring the house phone. For example, my mother’s sister regularly calls to talk to my mother, and does not need to be confused by the IVR. My parents have given me a list of numbers that need the bypass, and I add them as they request them.


Some of them are for elderly relatives, and some of them are for the notification robots from their medical providers.

Making the existing house phones still work, using an ATA

My parents have several classic Bell System Western Electric Trimline handsets [ https://en.wikipedia.org/wiki/Trimline_telephone ] , and have had them since 1982 when they moved into the house. They probably don’t even realize those handsets are now antiques, somewhat in demand. I want them to keep working just as they always have.


I bought a “Grandstream HT801 Single-Port Analog Telephone Adapter” [ https://www.amazon.com/dp/B06XW1BQHC ], for $35. As near as I can tell, someone at Grandstream sat down and said “Let’s make a residential grade ATA that Just Works and Does Not Suck, and then sell it for cheap”, and created the HT801. Good on you, Grandstream, thank you.


An ATA is a device that interfaces a classic handset to the internet. It has an RJ11 port that provides dial tone, ring tones, ringer voltage, and understands when you press the DTMF buttons on the handset. And it has an RJ45 ethernet port, that really wants to connect to a home router, and then connect out to a voip provider, such as to VoIP.ms.


I bought that ATA, and then set it up on my own workbench to program it. It was pretty easy. I configured it to connect to the SIP subaccount “parents”, and to generate the necessary powered ringer voltage and ringer frequency. Then I reshipped it to my parents, with careful installation instructions.


Usually, you connect a single classic handset directly to an ATA. But I wanted as few changes as possible for my parents, and the HT801 is mighty enough to provide sufficient dialtone power and ringer voltage to an entire house telephone wiring network and several classic “dumb” handsets, so I did something more awesome. I’m told that the HT801’s bigger brother the HT802 cannot do this, so, be warned.


We disconnected the house entirely from the old wireline company! My brother and my father went outside, found and opened the demarc where the phone line came to the house, brushed away decades of cobwebs and dust, and physically disconnected the little RJ11 bridge cable, and put a piece of tape over the demarc RJ11 jack. And with that, all the phones in the house went dead.


They then took the configured ATA, plugged it to wall power, to an open RJ45 ethernet port on the home router, and to an open nearby RJ11 walljack. And with that, all the phones in the house were again alive, and had light and dialtone when picked up. If there had been any “smarts” or filters inside the house phone wiring, we would have had to go chase that down and remove them, but there were not, so we did not.


If the ATA didn’t have enough power to light up the house wiring and all the phone ringers, I would have had to buy a “Ring Booster”device. But it was able to drive them, so I didn’t have to. 

Details on programming the ATA:

Here is the guidance from VoIP.ms to programming a Grandstream HT80X ATA: https://wiki.voip.ms/article/Grandstream_HandyTone_802_-_HT802


Here is the administration and programming manual for the HT80X:

http://www.grandstream.com/sites/default/files/Resources/ht80x_administration_guide.pdf


Configure the local networking stuff, DHCP, etc for the local internet router. The defaults are fine.


Configure the SIP/VOIP settings as follows:


Primary SIP Server: metropolis2.voip.ms

Failover SIP Server: <blank>

Outbound Proxy: <blank>

NAT Traversal: Keep-Alive

SIP User ID: 100000_parents

Authenticate ID: 100000_parents

Authenticate Password: <SIP PASSWORD>

Name: KENT

DNS Mode: A Record

SIP Registration: Yes

Unregister On Reboot: No

Outgoing Call Without Registration: Yes

Register Expiration: 5

Preferred DTMF method: In-audio, RFC2833

Use P-Access-Network-Info Header: No

Use P-Emergency-info Header: No

Enable Call Features: No

Dial Plan: {[x*]+}

Preferred Vocoder: PCMU, PCMA, G729

Check SIP User ID for incoming INVITE: Yes

Allow Incoming SIP Messages from SIP Proxy Only: Yes


Here is the bit to make it able to drive the house wiring instead of just one handset:


Ring Frequency: 20

Enable High Ring Power: Yes

Recording and Transcribing calls, make sure that is off

VoIP.ms can record and transcribe incoming and outgoing calls, on a per subaccount basis and on a per DID basis. I turned that on while I was programming and debugging, and turned it off after I verified that everything was working. Listening in on people’s phone calls is creepy. Especially your parents. Make sure that is turned off for that DID and for that subaccount.

If no answer, busy, timeout, or fail: fallback to voicemail

I’ve configured both the DID and the subaccount so that if it rings for 15 seconds with no answer, it falls back to voicemail.


VoIP.ms also has “failure handling”. If the ATA says the house phone is busy or off-hook, or if the internet links fail, or even if large portions of VoIP.ms own infrastructure fail, or if “too many” people try to call in all at the same time, the call will get routed to the voicemail, for later handling.

Telemarketers, begone!

Telemarketers. Scum. My parents said they got several a day. As I reviewed the CDR Call Detail Records, I was able to confirm they were getting at least half dozen a day, most of them from forged CIDs. Those calls from those scammers are still coming in, but now they no longer interrupt my parents.


Some of the scammy ones hang up immediately when they hear the robot voice. The legally compliant ones hang up when they hear “Do Not Call”. Some wait for “press 1 to leave a message”, and leave their message, where they are much less of an interruption. The stupid ones just listen to the IVR message loop three times, then it hangs up on them. But so far, none of them have rung the house phone to bother my parents.


I considered setting up an integration with the Jolly Roger Telephone Company telemarketer trap [ https://jollyrogertelephone.com/ ] [ https://wiki.voip.ms/article/Using_ring_groups_with_a_third-party_spamfilter_service ], to time waste the scammers, but I’ve not yet bothered, and it might confuse my parents. 

Firing the old phone company, easier than I expected

Now, at last to fire the old phone company. When I initially tried to cancel the original wireline ILEC account, I got a runaround that I had to call the account termination services between certain business hours. I tried again a few weeks later, only to discover that the phone company had already closed the account and were sending my parents a final statement. I guess they decided that a successful port-out meant that it was not worth it to them to pay a salesman CSR to try to talk me out of it.

Some ideas for the future:

I can receive a notification if someone dials 911 from the house phone. I think I will set something up so that I and my siblings all get an SMS when that happens.


I can install SIP clients on my parent’s mobiles and tablets, and create additional SIP subaccounts for them to connect to. Then my parents can make outgoing calls from their mobiles, especially on wifi, and it will look like they are calling from the house phone.


I can gift them a wifi cordless phone (Grandstream makes some nice ones, such as the WP810), preconfigure it to yet another SIP subaccount, and they would have a nice cordless phone that works within the house’s wifi.


I can enable SMS on that DID, and then with a little more programming on my part, they can send and receive SMSs on that number.


I may get around to adding the Jolly Roger Telephone Company telemarketer trap.


If I ever set up smarthome home automation stuff in their house, I can link it together, so that the house can call 911 if the fire alarms go off, or the house can call someone in the family if there is some minor emergency, or my parents can dial a magic code on the IVR to remotely turn on the house porch lights or open the garage door.

Finale

In conclusion, an incumbent wireline telephone company charging an elderly couple over $70/month for phone service is corporate elder abuse. I cut the bill to $3/month, gifted to my parents paying it myself, got rid of the telemarketer scum who were disturbing them multiple times a day, learned how to set up IVRs, and learned how to use Amazon Polly and Speech Synthesis Markup Language.


tag:blogger.com,1999:blog-4771631636333583067.post-2856812623942959716
Extensions
How to "break into" writing comic books, today
Show full content

While the big 2 US comic publishers are zombies of ghosts of what they were, *right now* is the Golden Age of the "boxes of drawings with word balloons telling a story" long form narrative art.

If you were the 16yo reincarnation of Todd McFarlane today, you don't go showing your drawings to your idols in the hallway of a some comiccon and then go camping in the lobbies of the NYC based publishers begging for starvation wages clean-up inking gigs.

You take a couple of online courses "how to layout comic pages", after first having spent 10 years self-teaching learning to doodle with an iPad and then a wacom. Then you start sharing your work on Tumblr / Twitter / Instagram / etc, until you feel ready, then you do a Kickstarter, and fail. Then another Kickstarter, and fail better. Then do another Kickstarter... and succeed.

Then you start a Patreon to fund volume 2, 3, ..., N ...

Eventually Marvel or someone invites you to draw or write for them. And if you are smart, you look at their contract, and the pay rates, and the "community" of the people you would have to work with, and then you tell them to get lost.
tag:blogger.com,1999:blog-4771631636333583067.post-5434352262986638150
Extensions
The CTO of Visa, after listening to my presentation
Show full content
Some years ago, I was asked to travel to the corporate meeting center to present at a presentation-fest to the CxO staff of Visa. Yes, the one with the logo on the cards in your wallet. The CEO of Visa was there. My CEO was there. (It was the only time I ever met her.)

The fest in general was a disaster. Except for my part, if I may say so myself.

During the discussion after the presentations, the CTO of Visa stood up and said "This was the most useless pile of nothing I've had to sit through for a long time! Nobody actually said anything! Nobody actually proposed anything! Nobody actually listened to what my problems are! Nobody expressed any opinion at all! Well, except for this guy!"

And he pointed at me. "He actually read my memo about what my problems are! He believes that open source is a good idea, and that cloud computing is a good idea! And I think he's right!"

The meeting broke up soon after, and all my coworkers left, kind of glum. Except for me, I had a spring in my step for weeks after.

I have no idea if there was any impact on how Visa did business from my presentation, but it was nice to be recognized.
tag:blogger.com,1999:blog-4771631636333583067.post-527351708628659716
Extensions
An easy New Years Resolution: delete artificial scents, and delete Sodium Lauryl Sulfate
Show full content
A suggestion for a new year's resolution, something that worked for me. It's a small effort, but a big change, and I suggest it for everyone: Years ago, I deleted almost all industrial and artificial perfumes from my life. Zero fragrance laundry soap. Tooth powder instead of SLS toothpaste. No more SLS based shampoo. No SLS based "body wash". You don't need SLS, especially not on your skin or in your mouth. Instead of shampoo, I just let water run through my hair. Zero scent natural mild soap. Zero fragrance conditioner. No synthetic cologne. Almost immediately, I was intensely aware of the reek of all this junk in my sweat and in my taste in my mouth, as it drained away over weeks. And then, it was gone. As was most of my acne. And food tasted a LOT better, and natural scents and flowers smelled a lot better. So, I recommend it. It's an easy New Year resolution, that isn't a lot of work or willpower, but makes a big difference.
tag:blogger.com,1999:blog-4771631636333583067.post-8195956233895206223
Extensions
A gentle lecture, what a codec is
Show full content
I recently wrote an offhand post about doing software archaeology at my job into the FFmpeg project, and one of my nephews, who is very smart but his passions go doing a different path than mine responded with: "I have no idea what you're talking about but I'm sure it's great", and thus so, I wrote up a slightly longer lecture/essay as a comment reply. Then my friend Tim Lord responded with: "that description of codecs deserves to be someplace besides only in a comment on a Facebook post -- cogent, unpatronizing, good refresher". And thus so, here it is:

A codec is a software that turns sound or video into a computer file or back again. Every phone call you have ever heard, and every movie you have ever seen, and all the recorded music you have ever heard (except for actual film movies or phonographs or old cassette tapes) has been processed by an enCODer to record it, and another DECoder to play it back to you.
There are many many many different codecs. Early ones were designed around the limits that computers were not very fast or very powerful, so they did not do a very good job of using the fewest number of computer bits for the best possible audio or video. We have codecs today that are very very good at it, because our computers are fast and powerful enough. There are many old codecs we can't easily stop using because they are built into systems that can't be easily all replaced at once, such as satellite receivers, telephone switches, and handset cellphones.
Some codecs were created to play movies and sound for computer games, and so the codec software was built into that particular game, and used only for one or a few games, and then never used again.
Codecs are very difficult to design well, because they depend on how human brains, human eyes, and human ears work. To save space, a codec does not want to spend computer power saving or playing back the parts of music or video that your brain cannot hear or see. So codec designers have to study human perception, and have to test things out on human volunteers, which is slow and expensive.
Widely used codecs are often "standards". Corporations, governments, and universities will work together to carefully design a codec that then can be used in many places at once, so that lots of systems can all talk to each other. One of those groups is called the "Motion Picture Experts Group", which is supervised by an organization called the "International Standards Organization", and one of the standard codecs they designed was called "Audio Layer", and they made several versions of Audio Layer until the 3rd version was good enough. Thus "ISO MPEG version 2 AL version 3", or for short "MP3". FFmpeg is an open source software project that was originally an implementation of one of the MPEG video codecs, but since then has become a project that tries to have an implementation of every possible codec. FFmpeg contains old codecs that are no longer used, old codecs that are still widely used, codecs that were used in old video games and never used again, new codecs that are now used a lot, and also lots of experimental codecs that people wrote to figure out what does and does not work in a codec. Because FFmpeg has so many codecs in it, people now use it to "translate anything to anything", and they also use it to analyze and process audio and video information. My own employer uses FFmpeg in many places inside our company, to do lots of the things that our customers pay us for.

My nephew then had the question "So JPEG artifacts are because of an outdated codec?", to which I responded: Every codec has artifacts if driven too hard, if its told "no, compress it even harder, fewer bits". There are newer codecs that are better than JPEG that don't start having visible artifacts so soon, and have artifacts that are less distracting. But we can't change all the image viewing software everywhere, so we will have to live with JPEG for a long time.
tag:blogger.com,1999:blog-4771631636333583067.post-7937327209260126299
Extensions
floating an idea: an APA
Show full content
I have no idea why I'm considering doing such a crazy thing, but I'm considering starting and running an APA. How an APA works:
  1. You write or draw on up to two sheets of paper, double sided.
  2. You send it to me, either via postal mail or email PDF.
  3. I collate them all together into a stack.
  4. I photocopy and bind the stack, making N copies. Enough to send one in the postal mail out to each member.
  5. I mail a copy to everyone.
  6. You get it the post. You read and enjoy it.
  7. You write or draw on up to two sheets of paper...
Some notes:
  • I'll be willing to cover the copy costs and postage, at least until it gets too expensive.
  • I'm willing to have members with non-US international addresses, again until it gets too expensive.
  • Other members will not see your postal address, unless you yourself decide to put it on one of your pages.
  • You can send your contribution in as a PDF, or as actual sheets.
  • Paper size has to be 8.5x11. Print will be BW photocopy.
  • I reserve the right to change number of sheets members get to send in
  • I reserve the right to cap the number of members.
  • I reserve the right to drop any contribution, and I reserve the right to evict any member.
  • The "code of conduct" is: don't piss me off, don't virtue signal, content needs to be kid-friendly and safe-for-work.
  • Topic is general, anything you want to write or draw. Fiction, non-fiction, gonzo journalism, bad poetry, good drawing, ...
So, if I decide to do this, do you want in? Reply here, or msg me, or email me.
tag:blogger.com,1999:blog-4771631636333583067.post-2929862681694245596
Extensions
buried in sheets of colorful glowing glass
Show full content
Anyone remember those scenes from ST:TNG that showed the captain doing paperwork at his desk? He was using a display terminal to read text and to do vtc, AND he was switching between half a dozen data tablets? Remember how conceptually silly that was? My edc backpack contains a laptop, a tablet, two phones, and one and sometimes two eink reader tablets. My nightstand has two eink tablets on it, and charger points for the phones. I just saw a coworker sitting at a table in this building talking on one phone, reading email on her laptop, consulting text on a tablet, and taking notes on a ReMarkAble tablet. The conference room I'm in right has all my edc gear, plus a yuuge 75inch vtc display on the wall, with attached steerable camera, and a glass tablet on the table for controlling the vtc. Plus a multipoint polycom rig on table. And plus an Alexa for Business terminal on the table. Even the lightswitch isn't real, it is a multiibutton scene controller. At least the clock on the wall is a physical moving-hands analog, but I know it gets set and sync remotely via some wiring coming out of wall behind it. And the six 5'x8' white glass whiteboards lining the walls are not "smart". Yet. We are within striking distance of ending the blizzard of paper that used to engulf work, but now we're being buried in sheets of colorful glowing glass.
tag:blogger.com,1999:blog-4771631636333583067.post-4117693815624555228
Extensions
Thoughts on the Google Titan token
Show full content
The tech press so excited about Google's "Titan" hardware token, and the breathless statement that they have "never had an account takeover" since rolling it out internally. They are excited about the wrong things, and are being taken for a ride by G's marketing and PR departments.
It's only a FIDO U2F token. I've had one for almost 2 years now, and my current employer issued me one on my first day of work, over a year ago. Mandating 2FA across an enterprise is hardly a new thing.
The actual stories here are: * why did Google decide to cut out YubiCo? * Was it price? * Was it not-invented-here? * Did Google not trust YubiCo to not backdoor the YubiKey tokens? * Did Google want to put their own backdoor into the Titan tokens? * Did Google license YubiCo's manufacturing patents? (If they did not, it will be really hard to manufacture them cheaper.)
tag:blogger.com,1999:blog-4771631636333583067.post-7206495540307923614
Extensions
On names
Show full content
(upcomment) There is no standard central registry of names in the US. Your "legal name" is whatever you say it is, with the restriction that you may not change your name to commit fraud or try to avoid a debt. The flip side of that is you have very few avenues to force anyone else to encode or display your name in the manner you prefer. If you change your name or have a name that is awkward to deal with, the obligation is generally on you to work out a working relationship with everyone else who needs to know your name.
A "legal name change" is just a helpful service provided by your local government court that lets you publicly declare and record a new name, bound to your previous name, on standard papers that other government agencies and various private organizations MAY (but not MUST) pay attention to.
There are multiple issuing authorities for IDs. Public, private, commercial, corporate, educational, military, government, ... An authority to issue IDs reaches only as far as their legal mandate, and no farther, except by rough consensus and a need to make stuff work.
Each issuing authority gets to have their own regularization rules and database schema. They will pick what works for them, and have very little interest or budget in completely reworking their databases and processes to accommodate someone who is being a pain in their ass.
Whenever you intersect with government, just expect ascii7 smashing.
US banking know-your-customer rules require ascii7 smashed names for account holders, both individuals and companies, to interface to government banking regulators and auditors.
The US interstate Driver License Compact imposes ascii7 smashing and length limits, for database and lookup compatibility reasons. When a cop looks at your DL (which you do have to give him if he's detained you while you are driving a car), it had better match what pops up on his terminal when he enters the the license# and/or the plate#, or you may be are about to have a bad day.
Things don't get a lot better when dealing with passports, from any country.
Passports impose ascii7 (with various accent composition hacks) and length limits, by treaty.
Different countries impose additional different rules and length limits, for reasons various and mostly stupid. Some countries try to appear to still permit accent marks, by trying to encode accents using various slightly different composition encodings layered on top of ascii7, and hoping that everyone uses the same encoding. This kind of sort of mostly works, except when it doesn't.
Passports from countries with widespread non-latin charsets will require an ascii7 smashed name, and often require a specific approved romanization algorithm, or an approved thesaurus, or both. Such a passport MAY have an additional field for the name in a local charset, but that is entirely at the option of the issuing authority.
What is printed on the passport photo page should match whats printed in the "machine readable zone", which should match whats on the rfid chip. "Should".
Most countries require air, sea, and rail carriers to pre-transmit passenger manifest before arrival. The names on those manifests also get ascii7-smashed, and any accent composition stripped, and had better match what's printed in your passport, or you are going to have a rough time boarding at departure, or clearing passport control at arrival.
Internal and external passports in China used to also require a field encoding the name in the form of Chinese telegraphy numbers, to sidestep transliteration issues between the various Chinese languages. I don't know if it still does.
tag:blogger.com,1999:blog-4771631636333583067.post-6967894295086201601
Extensions
On base knowledge surveys
Show full content
When I was in college, one of my Research Assistant jobs was to do clerical and basic number crunching work for base knowledge surveys. It takes scrupulous and expensive controls to prevent roughly a third of the surveys from being randomly answered for the lulz. Even with the most scrupulous controls and careful interview technique, there is still roughly 5% noise.

In other words, any newspaper headline of a newspaper article that is a restatement of the abstract of some random paper from some random academic journal based on some survey, especially if its a prepub paper or open access journal, that is of the form of "ONE THIRD OF LILPUTIANS ARE STUPID, ACCORDING TO SURVEY", is rank bullshit, and is anti-knowledge, as in anyone who reads it is less informed afterwards than before.

Show me the survey questions, the interview technique, the responder selection process, the population size, the population demographics, the pre-survey stats oversight board approval, the post-survey stats oversight board signoff, and the raw data, and THEN we will talk.

(My lead researcher when I was a RA sat on several of those Stats Oversight Approval Boards. I got well schooled in several of the ways that a researcher could lie to themselves, knowingly and unknowingly.)
tag:blogger.com,1999:blog-4771631636333583067.post-1411392091323406079
Extensions
A theory: Bank of Apple
Show full content
I have a theory, about Apple.

Apple has a quarter of a trillion dollars.  In cash.

That is a ludicrous amount of money.  That is so much money, that it is too much money.  It is too much to deposit as passive cash, because a bank can no longer be a neutral unbiased 3rd party when there is an account that big.   Not even a large nation's central bank.  When you have that much cash, risks like counterparty risk, fiat currency problems, and government confiscation start becoming a significant amount of the risk profile.

The way that most very large companies waste very large amounts of cash is to buy other large companies.  This almost invariably is a terrible idea.  The buying company almost always overpays, especially when they start bidding against someone else.  And companies on sale for a discount, are for sale for a discount for a reason.  So called claimed "synergies" almost never are realized.  Costs are always higher than expected.  Big mergers and aquisitions almost always are a mechanism where the senior executives burn the investors money in order to make said executives seem or feel more important.

Steve Jobs never felt the need to do M&A to seem or feel more important, so Apple only did aquistitions to obtain specific skilled teams or specific technologies.  And in his passing, Apple has generally continued this pattern.  (I think the aquisition of Beats was very non-Apple, and was a mistake, and probably is seen as an expensive mistake and expensive lesson by Apple's current leadership.)

But so and still, Apple has Too Much Cash.  What to do with it?

When you have a pile of cash that is so large that it in itself starts turning into a local economic distortion, there really is only one profitable thing to do with it:  Wrap a banking license around it, and open a bank.

Think about it.  Apple could run a bank, a very different kind of bank, with a much lower risks of fraud and loss.  They already have secure cryptoprocessors... everywhere! They can use iOS devices as the secure terminals, both for customers and for merchants.  They can use ApplePay for retail transactions.  They can use what they know about your from user's iOS devices and their AppleID for KYC.  They can push secure finantial messages around via the iMessage framework.  

With all this in place they could undercut all of the existing payment networks and still make, well, bank.

tag:blogger.com,1999:blog-4771631636333583067.post-299721709627803809
Extensions
notes from an opinionated talk about running IPv6 in production
Show full content
A few years ago, I was at SCaLE, and attended an excellent talk by someone who operated several campus-wide internetworks, and their hard won experience with IPv6.  They were very opinionated.  I loved it.  Here are some of the notes from that talk:


QoS is a bad word.
Control freaks love QoS.
They can debug it themselves.
People who have are held to SLAs operating production networks have better things to waste their time on, and better ways to crash their switches.

"But I'm not running IPv6!"  That means you actually are, and are nor longer in control of your network.
"I will block IPv6!".  Say goodbye to all the grants that pay your salary.  And everyone's desktops and devices will just make tunnels anyway.

Say NAT one more time, I dare you.

If you think that NAT is protecting you, let me know who you are, so I can blackhole your address range and your IS.

Turning off v4 ICMP is just stupid.
There are lots of stupid people.

You cannot turn off icmp6.
There is no frag in v6.
Thus mtu detect must be on.
Thus icmp6 must be on.
Live with it.

dhcp6 is port 547 not 67

tag:blogger.com,1999:blog-4771631636333583067.post-4403786570003926590
Extensions
Regarding that article about gender bias in GitHub Pull Requests
Show full content
Regarding "Gender Bias In Open Source: Pull Request Acceptance Of Women Vs. Men", or even worse, regarding all the uncritical and breathless articles by the BBC, Vice, HuffPost, and so forth:

First of all, anyone who names their project "DeveloperLiberationFront" and uses an icon of a raised fist in woodcut style, has already predeclared their bias away from objective truth.

Second, the authors of the paper exhibit little knowledge of about the large differences in workflow between different projects, and no knowledge about all the different ways that PRs are used and all the different meanings of an "abandoned PR", and also their definition of "project insider" is broken, as for many projects, an "insider" has write access, and may never use PRs at all.

Third, despite GitHub's growing influence, just grabbing tens of thousands of GH PRs is not in the slightest bit representative.

Fourth, their process for computing the gender of PR authors is laughably bad, for reasons that went on for 3 paragraphs before I edited down this text.

Fifth, how many have heard of "p-hacking"? or even have ever actually computed a p value since you took that really annoying stats class in college?  Did you even notice that the this paper both obviously did p-hacking, and then didn't even report the p values?

Finally, allow me to present the following disruption to the breathless and self-reinforcing narrative:

"So, let’s review. A non-peer-reviewed paper shows that women get more requests accepted than men. In one subgroup, unblinding gender gives women a bigger advantage; in another subgroup, unblinding gender gives men a bigger advantage. When gender is unblinded, both men and women do worse; it’s unclear if there are statistically significant differences in this regard. Only one of the study’s subgroups showed lower acceptance for women than men, and the size of the difference was 63% vs. 64%, which may or may not be statistically significant. This may or may not be related to the fact, demonstrated in the study, that women propose bigger and less useful changes on average; no attempt was made to control for this. This tiny amount of discrimination against women seems to be mostly from other women, not from men."
// ScottAlexander

If this was a real paper, submitted for real peer review, a good peer review would be:

"1. Report gender-unblinding results for the entire population before you get into the insiders-vs.-outsiders dichotomy.
2. Give all numbers represented on graphs as actual numbers too.
3. Declare how many different subgroup groupings you tried, and do appropriate Bonferroni corrections.
4. Report the magnitude of the male drop vs. the female drop after gender-unblinding, test if they’re different, and report the test results.
5. Add the part about men being harder on men and vice versa, give numbers, and do significance tests.
6. Try to find an explanation for why both groups’ rates dropped with gender-unblinding. If you can’t, at least say so in the Discussion and propose some possibilities.
7. Fix the way you present “Women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable”, at the very least by adding the comparable numbers about the similar drop for men in the same sentence. Otherwise this will be the heading for every single news article about the study and nobody will acknowledge that the drop for men exists at all. This will happen anyway no matter what you do, but at least it won’t be your fault.
8. If possible, control for your finding that women’s changes are larger and less-needed and see how that affects results. If this sounds complicated, I bet you could find people here who are willing to help you.
9. Please release an anonymized version of the data."
// ScottAlexander

I am willing to bet money that doing real honest academic statistical analysis of their raw data will invalidate their implications and their claims.

tag:blogger.com,1999:blog-4771631636333583067.post-5146345672079935661
Extensions
Why SSH keys dont have metadata
Show full content
And other tech rant. It was recently asked, in a forum that I read, the following: "Why is it that SSH public keys don’t have an embedded expiration date, anyway? PKI certificates have them."
My response:
Because as soon as you start adding all sorts of metadata to a key, then everyone will start adding all sorts of metadata to keys, with all sorts of obscure rules about how metadata interact with the environment and various implementations whether a key works or not.
And then the lawyers will show up and insist that you imbed 30 page PDFs of Word docs of someone’s T&Cs and their contracts of adhesion and their “don't hold anyone with money responsible for anything” disclaimers into metadata (you think I joke, I do not at all, this literally regularly happens with “standards based” PKI certs).
And then your keys are going to be huge weirdly encoded binary blobs of shit that you don’t have good tools to manipulate. And you will need to keep special indexes of them, and “bundles” of them, in multiple conflicting filesystem paths and “key stores”.
Part of why SSH took off at all in the first is because it doesn’t have this complex garbage wankery . An SSH public key is a SINGLE LINE, of printable ASCII7. You can edit and clean up your ~/.ssh/authorized_keys file with a textmode text editor.
The lack of metadata in SSH is a feature, not a problem.
tag:blogger.com,1999:blog-4771631636333583067.post-6083265202211696010
Extensions
This is how to do it, or waving my cane.
Show full content
1. Design a data abstraction that solves a class of problems.
2. Design a good wire protocol for that abstraction.
3. Better yet, design 2 protocols: one server-to-server and one client-to-server. Federation is the only model that has ever scaled large enough.
4. Implement a simple as possible server. Do not try too hard to make it performant, just very easy to install and very easy to understand. This is the protocol reference implementation.
5. Implement an open source client library, that completely covers the entire data model and the entire wire protocol.
6. Implement another open source client library, in a very different programming language. If this is difficult, you let your knowledge of your favorite language overconstrain the wire protocol. Go back to step 2 and fix it.
7. Implement a command line client on one of those libraries. Again, it must completely cover the entire data model.
8. Implement an ok GUI app.
9. Implement a very high performance highly scalable server. If you are tempted to change the wire protocol to do this, you screwed up.
10. Now, and only now, you can implement a very nice easy to use GUI. At this point, and at this point only, do you bring in any "designers", "UX" people, or anyone who uses Photoshop as working tool.

Of course, for the past 15 years, everyone has been doing this backwards, with disastrous results. It takes huge amounts of wasted CPU and wasted money by the millions and billions to make all the resulting garbage work at all.
tag:blogger.com,1999:blog-4771631636333583067.post-7633245108592876683
Extensions
Idea: RedFish aggregators, and running them on OpenSwitch
ideaopen sourcepatents
Show full content
Once upon a time, when you needed to "do stuff" to take care of a computer, you had to go there in person.  By "do stuff", that means things like: turning it off and on, looking to see if the AC was working, were the tape or disk motors broken, were any of the red warning lights on, had the UPS tripped, and so forth.   But, for many and obvious reasons, it was useful to do all this kind of stuff from a distance.

This led to the creation of "IPMI", which was built into most computers that were designed to be used in racks and datacenters.  With IMPI, a team of sysadmins could remotely turn computers on and off, check temperature, fans, power, network carrier, installed cards and devices, and read off model numbers, part numbers, and serial numbers.

IPMI is currently being improved/replaced with a thing called "RedFish".  RedFish does all the same sort of things, but it is designed in a way that is called "RESTful", which means it works the same way that web applications work, which makes it a lot easier to write tools that speak it.  Another cool thing about RedFish is that it accidentally also looks like a complete database of a "computer like thing", and does it in a way that "things" can be inside "things" and connected to other "things" all within how the protocol works.

And then I had an idea...

Write a web application that scans the local network looking for RedFish servers, and then itself acts like a RedFish server that integrates all these other smaller RedFish servers.

You can even stack this, making it so at a higher level, one of these "RedFish aggregators" discovers and integrates the lower level ones, and so on up.   Eventually you would have a top level one that would give you all the data and all the control over an entire datacenter or even larger set of data centers.

It wouldn't even be that terribly hard to write a small demonstration implementation.  It would be a challenge to make it fast and efficient, and to properly handle caches and avoid accidental recursion loops, but it doesn't look like a really difficult one.


To use something like this for real, the logical place to put it would be in the network switches.   But that used to be difficult, because production level network switches have been very closed and proprietary.  However, that's changing.   There is a new open source project spinning up right now, called "OpenSwitch".  If I was to push this RedFish aggregator so that it would be real world useful, I would make it a be a module that runs in the reference OpenSwitch box.


How hard could it be?

tag:blogger.com,1999:blog-4771631636333583067.post-3721279586481904572
Extensions
About media leakers
Show full content

I wonder about media leakers.

I'm not talking about whistleblowers, who reveal coverups by governments and corporations that are keeping secrets of bad or illegal actions.

I'm talking about people who "confidentially source" to the media details of business negotiations, media productions, and gossip of private heartache.   Things that are private and confidential for a reason, will be revealed when they are properly baked, and that do nobody any good for being revealed early, except maybe for a burst of clickstream traffic for the "news" source that "scooped" it.

I know a fair number of secrets.  Some of them are close friends' private heartaches, which are theirs to reveal, if ever.  And some of them are business negotiation secrets incidental to my job, and a few of them part of my job to know.   I actually go out of my way to avoid learning things I shouldn't need to know at my employer, just so as to firewall myself from even the appearance of impropriety.

Any of them, if I "confidentially sourced" them to the tech press, would do nothing but cost money that is not mine for no honest gain to anybody, possibly prevent good things that I would like to have happen from happening, and would betray my own principles I try to hold myself to

So, why do other people do it?

tag:blogger.com,1999:blog-4771631636333583067.post-945742002390033899
Extensions
A temporary mistake
Show full content

I don't care about business models, I care about applications, and at true billion user trillion device scale the only scaling pattern that succeeds is user visible federation.

Email, the DNS, the HTML/HTTP hyperlink, XMPP, and the blockchain have no rent seeking gatekeeper business model, and do not require billion dollar data centers.

The past 15 year drive to unitary silo'ed apps with a rent-seeking gatekeeper has been a mistake that has diverted too much engineering effort towards just keeping them running instead delivering user-desired features and value, and has been driven by the corrupting need of VCs for their mythical billion dollar exits, and the telco encouraged temporary exhaustion of global address space and thus a temporary breaking of the end to end principle.

This is temporary, unsustainable, and not scalable to 10 billion users.


tag:blogger.com,1999:blog-4771631636333583067.post-5355850411208180242
Extensions