A Practitioner's Guide to Wide Events | Jeremy Morrell

charity wtf's about technology, databases, startups, engineering management, and whiskey.

How do you correlate logs, traces and metrics in OpenTelemetry? Learn the Pivot Workflow to drill from a Grafana dashboard spike to the exact log line that caused it.

charity wtf's about technology, databases, startups, engineering management, and whiskey.

Lambros' curated list of recommended material to read, watch, and listen.

A tragedy in four acts

I spent some time recently catching up on my #to-read saves in Obsidian. More than a few of these were blog posts from 2024 about software observability. Talk of "redefining observability", "observability 2.0", and "try Honeycomb" had caught my eye in a few spaces, and so I had been hoarding links on the topic. After spending a few days immersing myself in those articles and branching out to others, I decided to write this bullet-form roundup.

How do you add custom metrics with OpenTelemetry in Python? Learn to use Counters, Histograms and Gauges with Prometheus and Grafana for production monitoring.

How do you record errors properly with OpenTelemetry in Python? Learn why try/except blocks hide failures from dashboards and how to make every error visible in traces and metrics.

The “old” internet: https://projects.kwon.nyc/internet-is-fun/ Networking: https://twitter.com/iximiuz/status/1859339483390296354 https://fabiensanglard.net/quake3/network.php AI: https://www.cs.toronto.edu/~duvenaud/distill_bayes_net/public/ WebRTC: https://twitter.com/arcinston/status/1858603234455126441 DL: https://fleuret.org/francois/lbdl.html https://fleuret.org/dlc/ https://spinningup.openai.com/ s3 https://simonwillison.net/2024/Nov/26/s3-conditional-writes/ https://docs.google.com/document/d/1yzLXSOtzBXyaWHfeVsWsMu4xmOH8rV6QyM5ZAnJZjMQ/edit?tab=t.0#heading=h.yhvnt89pggpj data structures https://x.com/debasishg/status/1858560338456952998 Observability https://news.ycombinator.com/item?id=39529775 https://www.honeycomb.io/blog/how-are-structured-logs-different-from-events https://x.com/mipsytipsy/status/1858341191932338354 https://www.honeycomb.io/blog/get-infinite-custom-metrics-for-free https://x.com/mipsytipsy/status/1857626227634524541 https://x.com/walktothesun/status/1837657459869339834 https://x.com/mipsytipsy/status/1857599831470125461 https://bsky.app/profile/mipsytipsy.bsky.social/post/3l77h7eyfwe2b https://jeremymorrell.dev/blog/a-practitioners-guide-to-wide-events/ https://bsky.app/profile/jeremymorrell.dev/post/3lasvhflczb2u Bluesky https://x.com/Werner/status/1859611503537594880 https://newsletter.pragmaticengineer.com/p/bluesky-engineering-culture Access tokens https://twitter.com/gunnarmorling/status/1860961333409104353 Caching https://x.com/hnasr/status/1861197188371751021 Design/architecture https://thefrugalarchitect.com/ Testing https://til.simonwillison.net/pytest/pytest-recording-vcr LLM/genai https://elk.zone/techhub.social/@simon@simonwillison.net/113539371054898619 https://elk.zone/techhub.social/@simon@simonwillison.net/113541020246317028 https://blog.dottxt.co/coalescence.html Databases https://www.scylladb.com/2024/11/25/database-internals-working-with-io/ Formal methods https://cacm.acm.org/research/how-amazon-web-services-uses-fo

Originally posted on the Honeycomb blog on November 19th, 2024 We’ve been talking about observability 2.0 a lot lately; what it means for telemetry and instrumentation, its practices and sociotechn…

My day started off with an innocent question, from an innocent soul. “Hey Charity, is profiling a pillar?” I hadn’t even had my coffee yet. “Someone was just telling me that profiling is the fourth…

Hazel Weakly, you little troublemaker. As I whined to Hazel over text, after she sweetly sent me a preview draft of her post: “PLEASE don’t post this! I feel like I spend all my time trying to hel…

In this article, Charity Majors goes over the simple, technical distinction between observability 1.0 and observability 2.0.

charity wtf's about technology, databases, startups, engineering management, and whiskey.

The "three pillars" are a lie that keep good engineers trapped inside a mental model from the 1980s, paying outrageous sums of money for tooling that can't keep up with the complexity of modern systems.