Encoding

Walker Griggs is a distributed systems engineer, focused on developer-first platforms. He is based in San Francisco, California.

Pyramids of map tiles in a single file on static storage - protomaps/PMTiles

Pyramids of map tiles in a single file on static storage - protomaps/PMTiles

With 2024 officially being the year of OpenStreetMap vector maps lets do a deep-dive into vector maps: their history and how the underlying vector tiles work in detail.

Introduction Link to heading Varint is a widely recognized technique used for compressing integer streams. Essentially, it suggests that it can be more efficient to encode a number using a variable-length representation instead of a fixed-size binary representation. By removing leading zeros from the binary number, the overall representation size can be reduced. This technique works particularly well for encoding smaller numbers. In this article, I provide a brief introduction and rationale for varint encoding. Additionally, I describe the Stream VByte format, which enables fully vectorized decoding through SSSE3 instructions. I also share my findings from implementing this algorithm in Rust, which includes both encoding and decoding primitives and the ability to read data from both RAM and disk.

gRPC is an incredibly popular RPC framework that efficiently connects services. But how does it work? Let's dive in!

Protocol Buffers is an amazing message format. It’s incredibly compact and performant. However, these advantages come at a cost. Since Protobuf is a binary format it lacks a lot in readability compared to text-based formats like JSON or XML. If you look at encoded protobuf data it just looks like meaningless ones and zeros. However, all hope is not lost. Even if you just have a binary protobuf file with no knowledge of the corresponding protobuf file we can still get some information out of it. Let me introduce a tool called Protoscope. Protoscope is a tool for inspecting protobuf binary. It can do this with or without the protobuf files or the equivalent descriptor set (but it can do a better job with the protobuf data).

At my current employer, we use Kubernetes to run hundreds of thousands of bare metal servers, spread over hundreds of Kubernetes clusters. We use Kubernetes beyond officially supported/tested scale limits by running more than 5,000 nodes and over a hundred thousand of pods in a single cluster.1 In these large scale setups, expensive “list” calls on the Kubernetes API are the achilles heel of the control plane reliability and scalability. In this article, I’ll explain which list call patterns pose the most risk, and how recent and upcoming Kubernetes versions are improving the list API performance.

How to extract raw source protobuf definitions from compiled binaries, regardless of the target architecture

How one million chessboards works

I worked with the Apache Kafka protocol on the low level quite a bit. It wasn’t easy to start doing this following the official guide only and I read the code a lot. With this post, I want to give you a head start by guiding you step by step from primitive values to meaningful requests. In this post: Explore the Kafka protocol code and the protocol in action with Wireshark. Learn how to read and write primitive values. Combine primitives to perform meaningful requests. We will use Python as the programming language. However, the code will be zero-dependency and easily portable to the language of your choice.

A deep dive on how Convex's reactive database works internally.

Battle-tested, Mom-approved

How one million chessboards works