https://blogger.com/feeds/8260074503107229699/posts/default

I sat down on my couch one morning, opening my tablet to start my pre-work ritual of Wordle, study, and meditation. I was working from home on a day usually devoid of meetings. Before starting, I quickly checked the calendar on my phone to make sure I didn’t have any unexpected early meetings.
Huh. I had a 15-minute meeting scheduled for 10:10 AM. That was odd. I read more. It talked about changes at the company. I was affected.

Shit.

Again? 
During the meeting I got confirmation that I (along with many others) was being laid off, effective immediately. I suddenly found myself with no job and all the time in the world, for the second time in two years. The first time this happened I was unprepared for the disruption, ultimately leading to a long journey of reflection and learning.

This time, I was in a much better position. I had done the work to know what is important to me and I immediately put those learnings to use. For me, that included checking on colleagues who had been laid off, doing a lot of cooking, and ramping up my running.

This post covers how I built a sense of identity that was strong enough to take a layoff in stride. The process included framing questions, brainstorming on answers, organizing the ideas, and synthesizing something useful for myself. In the end, I had built a personal guide for living my life.

This is the concluding post in a series on job loss and identity and a follow up to Relearning My Identity – Step One.
On a hike with a colleague and my wife a few weeks after being laid off.  

---

Continuing the JourneyIn Relearning My Identity – Step One I started a journey of self-redefinition. In that post I listed four draft pillars of my identity:

1. I make things that other people use.
2. I make the people around me better.
3. I’m a learner who pulls together disparate ideas to create cool things.
4. I write in service of the first three goals.

That list was a work in progress as I began making sense of my world after losing my job. It quickly became clear that the list is very focused on my professional life and misses important aspects of my non-work life. I have been thinking about that ever since. I have tried to expand and generalize my self-definition to include all parts of myself. I am happy to say that those four pillars still ring true to me and are captured in my updated sense of self, but they are no longer the top-level themes. BrainstormingI thought and brainstormed on the topic for five months, even as I started a new job. Along the way, I had false starts, backtracked, and reframed my effort. I shifted from trying to answer the vague question “Who am I?” to attempting a focused task: building a guide for living my life aligned with my values. The guide should answer the question of who I am, but it should also help me make decisions and accomplish my goals.

I focused my brainstorming around building that guide. I tried to capture everything about myself, whether it was good, bad, or indifferent. Then I made a value judgment about each of those aspects of myself: Was this something I wanted to embrace? Was this something I was willing to change? Was each framing or description helpful to me?

That last question was particularly interesting to me. After some reflection, I found that a number of the details I had captured were not useful to me. For example, I’m smart. I’ve known that I am smart since I was a small child. But smart is usually defined in comparison to other people. I don’t want my identity threatened when I interact with people who are smarter than me. I often work with people who are smarter than me and I want to continue doing so, because I learn and accomplish so much when I do!

For each not-useful description, I tried to find related descriptions that were useful. For instance, part of what makes me smart is my love of learning. My love of learning will never be threatened by someone else’s love of learning. My love of learning is a helpful self-description. Explaining It AllMy collected brainstormings had a lot of great things, but they didn’t add up to a guide for my life. It was now clear that the four draft themes from earlier were not that guide, either, as they missed many important parts of my life. I still needed some short collection of top-level themes to organize everything into a guide.

The correct top-level themes did not just jump out at me. To find the correct themes, I committed to making several drafts, with each iteration as different as possible from the previous. I then reviewed my attempts, creating a final list of themes that felt right to me. Those final themes, combined with all the supporting details arranged underneath them, is my life guide.My ThemesI finished with four top-level themes:

1. I search for and find awe and wonder in this world.
   
2. I take care of myself and others.
3. I create things of use for myself and others.
4. I deeply enjoy myself.

Every time I read that list, I get a warm feeling. There is much more detail behind each of these four, but they capture the biggest ideals and give me something to both talk and think about.

You might notice that those four do not say “I am an engineer” or say anything explicitly about my profession. I could retire tomorrow and still satisfy all of them. Still, my professional identity fits comfortably within them, as do my original four themes.
For instance, one reason I became an engineer was because I loved (and still love) to understand how things work and how I could build them. I experience a sense of awe and wonder when learning how things are built. Being an engineer is an inherently creative endeavor, constantly creating new things. Working as a staff engineer, a large amount of my time is focused on mentoring junior colleagues and helping build a strong team culture. In other words, I’m taking care of others and myself. Finally, I regularly find opportunities to deeply enjoy myself in the role. 
That said, each of those categories is much wider than just my professional experiences. It includes a lot of art, food, gratitude, relationships, community, learning, exercise, hobbies, volunteering, and more.
Living My IdentityI applied these learnings the day I was laid off and every day since. I started by checking in on my colleagues to see how they were doing and how I might help them. That helped me take care of others, and in the process, take care of myself. I did a lot of cooking. Cooking helped satisfy my need to create, and also allowed me to take care of my family. I ramped up my running to take care of my body during this time of stress. I also got out on the trail, filling myself with awe and wonder, and deeply enjoying myself. In fact, I took my first post-layoff hike with my wife an hour after being laid off. And I’ve kept writing, which may hit all four of my themes.

I am very proud of and happy with my updated sense of identity. It has taken a large investment of time, but it has already returned value to me. I regularly review my themes and supporting details, reflecting on my identity, making sure I am living according to my values. Like everyone else, I am continually evolving. I will make sure to regularly update my sense of identity to match that evolution.

I would encourage you to spend some time thinking about what makes you who you are. If you do, I would love to hear about what you learn along the way. I’ve included some additional resources below to help you on your way.
Thank you to Heather Beasley Doyle for her feedback on this post and her support through this entire period of my life. Heather is a gifted writer. You should check out her homepage and her writing. Resources

• Designing Your Life (this was particularly useful for me)

• You Have a Choice

• Thanks for the Feedback  chapter 9 on Growth Identity and identity stories, and more

• Atomic Habits: Tiny Changes, Remarkable Results  Habit change is identity change. See Chapters 2 and 20.

• The Scout Mindset Part V Rethinking Identity

• Basics of Identity | Psychology Today

• Why we define ourselves by our jobs

• Who You Are is Not What You Do

• Psychology: How to Build Your Personal Identity

• Defining your identity & reconnecting with yourself and the world

• Identity: Definition, Types, & Examples (see mention of identity crisis and midlife crisis)

• Identity formation - Wikipedia

• How to Define Yourself: 6 Steps (with Pictures) - wikiHow

• Learning to live slowly

Some ideas rush into our heads with urgency and fanfare. Others take their time, showing up unannounced, pretending that they have always been there. One idea quietly snuck into my head on a Friday morning, eight months after I had abruptly lost my job. The job loss had shaken me. In all previous job changes I had been driving the change — not in this one. I had found and started a new job, but I had also spent a lot of those previous eight months re-evaluating my life, particularly the stories I tell myself and others about who I am and who I want to be.

This intrusive idea gently pointed out that I also needed to decide how I wanted to tell the story of my past. The idea was right: I no longer knew how I wanted to tell the story of my previous decade. If I couldn't tell the story of my past, how could I reasonably tell the story of my present and future?

In this post I explore that idea by diving into the disruption created by my job loss. I also review what I’m doing to tell my story with a narrative I both believe and feel good about. It is part of a series of posts reflecting on losing my job.

My daughter reading a story to her second-grade class on her birthday, many years ago. 

 

---

My Story Through 2014

I can easily tell my life story through 2014. Here is the condensed version:

I grew up a smart, if awkward, kid in the suburbs with supportive parents who loved me. I did well in school and was confident that I would succeed in life. I went to college to study computer engineering. Over five great years I produced a strong academic record and a year’s worth of professional work experience. Towards the end of those five years, I met and started dating the woman I eventually married.

High school yearbook picture of me

After graduating, I went to a top graduate school and pursued a Ph.D. I was still that same smart kid, but graduate school and research didn’t suit me as well as undergraduate classes had. I did well in my classes but struggled with the open-endedness and abstractness of the research. I also struggled with living in the midwest far away from my family. I fought through those struggles and (eventually) graduated with my Ph.D. 

Major personal life events unfolded during my grad school years. In chronological order: I married my wife, my father unexpectedly died, I attended the Final Four and watched my alma mater win (go Cuse!), and my daughter was born. 

With my wife at our wedding


My father with a map on a hiking trip
Me standing in the last row of the SuperDome watching the Final Four
My daughter at one day old

After finishing grad school, I got a job as a researcher at IBM in New York. I built a role for myself and did well. Among other things I worked on the architecture and performance evaluation of IBM POWER8 processor and co-created and ran a peer group for new hires. My wife and I continued to raise our family (now four of us after the birth of our son) and we put down roots in our new community. I joined a hiking group and hiked a lot. After nine productive years at IBM, I moved on to the next step in my professional journey.

My son at a few days old

 

Disruption to My Professional Story

I joined a small but rapidly growing company. I spent ten years with this company before I was abruptly let go. I’m still figuring out how I want to tell the story of those ten years of my professional life.

A year before I lost my job, I would have told the following story: I joined a small but growing database company that was full of excitement, optimism, and smart people. I joined a team working to understand and improve the performance of the database. We had no idea what we were doing at the start, but over time, we figured it out, building tools and processes to help ourselves. As we kept improving that infrastructure, we pushed the state of the art forward for our industry, including publishing articles about it. I took increasing responsibility for that infrastructure, first stepping into a management role, and then building a new team to cover the areas of performance testing that were still missing.

I discovered I didn’t enjoy being a manager and I transitioned into a staff engineer role, with an even stronger sense of ownership and responsibility for the performance of our software. I worked to build the best role for myself and to do right by the company. Along the way, I also helped build communities within the company and develop my junior colleagues. I celebrated that I got to work at a special place.

This is a story full of challenges, growth, and positive memories.

In my final year I would tell that story, with the addition of a big jump forward in defining that role for myself, changing jobs and managers, with all the stress that comes with that.

In the days after losing my job, I no longer knew how to tell my story. It felt like a bad breakup. I had left jobs and roles before, but I had always been the one in control, driving the change — I was not in control of this change.

Rewriting My Story

Today I am trying to figure out what story I want to tell to myself and others about my time at that job. I'm trying to reconcile my many fond memories with the very non-fond memories of the end. As described above, the first nine years are easy for me to talk about, full of challenges, growth, success, and joy. The challenge is ending the story.

I can tell a story ending with anger at the forces that resisted my intentional efforts and ultimately led to my dismissal. I always try to acknowledge my emotions, because they are my emotions. But I don't want to dedicate much of my time to anger. Anger seldom helps me advance my life goals.

I can tell a story ending with self-recrimination and self-doubt about missed signals and opportunities. I do not like this story, but I'm willing to stay with it long enough to see what I can learn from it.

I can tell a story of a company changing as it rapidly grows during times of market changes (Covid pandemic, increasing interest rates, ...) — a company changing into something that I found less exciting and welcoming. As I discussed in Special Places, change is inevitable and it’s not worth getting upset about it. Rather, we/I should acknowledge the change and celebrate that the great parts existed for a time.

Finally, I can tell a story of intentionally setting my path, taking chances, and living with whatever comes, even though that outcome wasn’t what I had wanted. That is the heart of the Scariness of Getting What You Want.

All these stories are true. I suspect the story I will eventually want to tell will include parts of all of them. It’s a messy story, including plans, intentions, emotions, and external forces. I’m working on developing that integrated story that acknowledges the bad parts so I can learn from them, while embracing and celebrating the good parts. I think it’s going to take me some time to get it right — writing this post helps.

Thank you to Heather Beasley Doyle for her feedback on this post and her support through this entire period of my life. This post has particularly benefited from Heather's help, with multiple rounds of feedback and edits. Heather is a gifted writer. You should check out her homepage and her writing. 

(Author’s Note – This is part of a series focused on job loss and is the first of three posts focused on identity. I wrote this post in December 2024.)

This past summer I suddenly and unexpectedly lost my job. I had been contemplating leaving this job for a while, but being asked to leave still came as a horrible shock. It was scary. Losing my salary and health benefits was scary. Losing my work social network was scary. But, by far and away the scariest part was the threat to how I saw myself — my identity.

Since that day, I have focused on how I define myself. I am trying to define myself in a way that is not dependent on my job and that is robust to changes in my job. While I have just started on this journey, I wanted to share some early learnings and experiences. 
My plaque for reaching 10 years of employment at my last job. I lost that job a couple of months after receiving the plaque. 

---

Backstory

For over 10 years I had worked at the same company. I had recently received a plaque from the company celebrating those 10 years, along with a nice gift, and the promise of an eight-week paid sabbatical. Over those 10 years, I had built myself into the performance person there. I knew more about testing the performance of our software than anyone else in the company. I was involved in designing and building all of our performance-testing infrastructure. Additionally, I was known in the performance engineering community as the performance guy at that company. I had published blog posts and papers about how we tested performance, and given many talks on the same subject.

Suddenly I wasn’t that person. I didn’t work there, so clearly I wasn’t the performance guy at that company. If I wasn’t that person, who was I? I was still the same husband, parent, friend, who loved to hike, cook, and read, but my professional identity was a large part of my total identity. It was gone and I felt incomplete and adrift without it.

I vowed to never let this happen again. I will no longer define myself (even just my professional self) in terms of my job. That starts by talking about my work, rather than about my job, as my work may be a job, but it may also be volunteering or a hobby. My work will be an expression of my identity, but will not be my identity itself. To meet that vow, I needed to develop a new self-definition. That is easier said than done. I’ve spent the last several months working on this challenge, and I expect to continue working on it for the rest of my life.

Learnings So Far

While I don’t have complete answers yet, I have learned a few things. I started this effort by seeing what I could learn from others: by talking to people and by reading. There’s a lot of literature around identity formation. While most of that literature focuses on adolescents, some, such as Designing Your Life, focuses on adults. The literature has recurring themes around identifying both your values and what you find rewarding. Experimentation can help confirm those learnings and test out potential identities.

I am doing that work. Thankfully, I have been doing some of that work for years, leading a very reflective life. I’m continuing to do that while keeping these new ideas in my head.

Pillars of My New Identity

I’ve started by focusing on the parts of my identity disrupted by losing my job. I intend to extend this work to have a complete sense of my identity. So far, I know that at my best:

1. I make things that other people use.
2. I make the people around me better.
3. I’m a learner who pulls together disparate ideas to create cool things.
4. I write in service of the first three goals.

Through it all, I want the world to be a better place for me having been here. The specifics of how I make the world better will change over time, but I always want to be a positive force.

At my previous job, I built tools that others used to make our product faster, and our many customers used that product to do incredible things. At my next job I will help improve software that helps small and midsized businesses focus on their key business, rather than human resources processes. I have mentored people in the past and will continue to do so, both formally and informally. I listen and give them advice to help them live their best lives. Writing lets me help more people than I could by just talking to individuals. And my learning mindset helps me do a better job at all of this.

Into The Future

I’m starting a new job in a few days. I will dive in and make the best contribution that I can. I will make sure that I express those four pillars of my identity, ultimately making the world a better place. I will also continue to explore who I want to be and work to become that person. At some point, my new job may stop being the best place for me to express and grow that identity. When that happens, I will celebrate my personal progress and shared work experiences, I will find my next, next thing, and jump forward with both feet.

Along the way, I will write about my learnings and experiences so that we can continue to grow together. I encourage you to reflect on how you define your identity and to update it, especially if you define yourself in terms of your job. Additionally, if you have experiences or perspectives that would help me on my journey, I would love to hear from you. I always love learning, and learning from others is usually the fastest path forward for me.

Thank you to Heather Beasley Doyle for her feedback on this post and her support through this entire period of my life. Heather is a gifted writer. You should check out her homepage and her writing. 

It was a beautiful Monday morning last summer as I sat down on a lounge chair to relax. I had lost my job five days earlier, but was in no rush to find the next job – or even to look. So I was taking some time off: My plan was to do whatever I wanted to do. I had spent the previous four days backpacking with friends on an already-planned (and wonderful) trip. This was the first day I would have been going to work if I still had my job. I did my morning meditation, solved the Wordle, and started flipping through the newspaper. Soon, I noticed a sinking feeling spreading through the pit of my stomach and a growing sense of dread.

This dread wasn’t about money or health care — I had planned for those things. It wasn’t about any specific need. Instead, it was about my complete freedom, and the lack of structure suddenly staring me in the face. I am a person of habit and routine whose routines were gone, and whose habits no longer felt relevant.

This post is the second in a series of posts reflecting on losing my job.

My lounge chair, newspaper, and games waiting for me.

Many months earlier I had started to plan for the paid eight-week sabbatical that my previous employer granted to employees on their 10-year anniversary. Initially, the idea of eight weeks off overwhelmed me. I had never taken more than two consecutive weeks off since college, including for the births of my children and previous job changes. I spent a long time conceiving of how to productively use that much time, and I had come up with a plan. Now I had an indeterminate amount of free time, and a plan, including travel, skiing, and a class, for eight weeks at the end of the year — not now.

Because I am a creature of habit, I create structure around me. That structure benefits me. My job had provided me, like so many people, with a lot of that structure, giving a rhythm to my weeks and years. That was gone, and I didn’t know when I would have it back. I also had intentionally tossed most of the other structure away in service to giving myself time and space to do whatever I wanted.

A Plan

When that sense of dread overcame me, I realized I needed some structure in my day. I needed to know there were some predictable parts to my days, every day. I also really did want to give myself the time and space to relax, regroup, and recover — I was tired and worn out from the shock and disruption of my job loss.

I set up a simple schedule for myself: two structured hours in the morning and the same in the afternoon. The morning hours were for doing useful things. I've always had a to-do list, and this is when I would work on it. That included keeping up with colleagues and updating my resume. The afternoon time was for reading and learning. I love learning and had a backlog of things I wanted to learn. The time might be reading non-fiction. It might be taking an online course. But it would be something.

And that was it. Four hours total in the day. Plenty of time to exercise as much as I wanted, to go for long walks, to take an afternoon nap, or to do the crossword. Plenty of time for just relaxing and enjoying myself, now (hopefully) free of the overwhelming existential terror of complete freedom with my time.

Executing The Plan

I started my plan on Tuesday — day two of my new normal week days. I started to feel better as soon as I put the plan in place: not perfect, but better. My journal for Tuesday includes:

I started building some structure for myself today. I needed it to not feel like I was drowning. I went through my well wishes and captured and organized them. That was really nice.  And I started to push forward on doing things. The progress is good for me. 

After losing my job I got a lot of notes from colleagues expressing sympathy and wishing me well. These came through email, LinkedIn, Slack, and a very nice virtual card. These messages really heartened me. I wanted to acknowledge them and make sure I didn’t forget them (I haven’t). So I collected and organized all these notes and responded to them expressing my gratitude.

Over the next couple of months, my two hours of useful time covered writing, dealing with health care, planning trips, and preparing for a job search. The writing was a combination of correspondences and early drafts of potential blog posts. The writing helped me make sense of my world. It helped me figure out how I wanted to think about everything going on in my life. That writing was just for me, although some formed the kernel of this blog post and series.

In my two hours of learning time, I finished a Coursera class on machine learning. I invested in my knowledge organization processes (i.e., note taking) greatly improving my Obsidian skills. And I learned about the latest on technical resumes and interviews, particularly design interviews. It felt good to do all of these things.

Outside those four hours, I did a lot of hiking, including planning and going on a wonderful backpacking trip with my children. I did a lot of cooking, including taking a week-long cooking class at the Culinary Institute of America. And I ran a lot, getting into great shape.

With my kids on the summit of Nippletop mountain in the Adirondacks during our backpacking trip. 

I did a lot of great things in this period. I still regularly faced worry: worry about the present and worry about the future. But that worry didn’t keep me from making the most of my time. And over time, that worry continued to get smaller. In short, I was living my life to its fullest, making the most of all the time I suddenly had.

After Accepting a Job

Last October, I accepted a job offer and the rest of my worry went away. I negotiated a start date of January 2nd so I could still enjoy all the things I had originally planned for my sabbatical. It also let me fully relax for a period of time, knowing that I had the next thing already lined up.

I took a number of trips in that remaining time, including two ski trips, and an amazing two-week-long 25th wedding anniversary trip to Bhutan, Singapore, and Bangkok. It was one of the best periods of my life.

When I wasn’t traveling, I stuck to my schedule of two hours in the morning and two hours in the afternoon. My afternoon focus shifted from interviewing towards starting a new job. I reviewed The First 90 Days. I brushed up and/or learned about the tech stack of my new employer. I still enjoyed and appreciated that structure to my day. It felt good.

Back Into Work

Towards the end of my time off, I started to feel excitement for that next adventure. I also started to feel a tinge of nervousness not unlike what I’d felt as I sat down in my lounge chair on that summer Monday. I was now used to four productive hours a day. What would it be like to go back to a full eight hour day? Would it be too much? Would I have time for the things I had come to appreciate more? Time would tell. It would definitely be an adjustment, but I would take it in stride, just as I had when presented with unlimited free time last year.

Thank you to Heather Beasley Doyle for her feedback on this post and her support through this entire period of my life. Heather is a gifted writer. You should check out her homepage and her writing. 

I waved goodbye to the smiling faces arrayed before me on my monitor and prepared to join my next call. It was late Wednesday morning, and these were my last two calls before starting a four-day weekend. I looked forward to finishing up some tasks, cleaning up loose ends, and heading out on my annual backpacking trip. The call had been a going-away for my friend Amy. Her last day would be Friday, but this call had been scheduled for Wednesday so I could attend. My next call was my regular one-on-one with my manager, also moved up because of my time off. Things had been strained between us, so I wasn't exactly looking forward to it, but I was looking forward to getting it over with and heading out on my vacation.

I clicked the link and the video call opened. Instead of one, there were two people on this call: my manager and someone I did not know. That could not be good. Was I being put on a performance improvement plan? My boss quickly introduced the second person — she was from HR — and then continued. "David, you are not meeting my performance expectations for a staff engineer. Further, I do not see a path for you to meet those expectations. Therefore we are letting you go, effective today.”

He told me our colleague from HR would walk me through the process.

“Do you have any questions for me?" he asked.

I did have questions. I had questions for him, for her, and later, I had many questions for myself.

That Wednesday was over a year ago. Many things changed for me in that moment, and I have spent a lot of time thinking about it since then. This post introduces a series of posts about my thinking, learning, and life over the past year. The first post is out now, with the remainder to follow.

---

Before continuing, I would like to briefly address “You are not meeting my performance expectations.” It’s true, I wasn’t meeting his expectations. It’s also true that I was good at my job. All my formal reviews had been good to great in my 10 years in that job. I usually received glowing reviews and was promoted twice. After being let go, I received many touching notes from colleagues who were shocked and upset that I had been let go. Many of the notes directly contradicted my manager’s story.

While it is scary to say publicly that I was fired, I think it is important to the story I want to tell. I also think sharing it may help others in similar tough times.

That said, this series is not about my dismissal per se, but rather my reaction to it and my continual attempts to make sense of it so I could move on and make the most of my life, experiencing and creating as much goodness (joy, wonder, love, satisfaction, …) as possible. This story starts well before my firing, and continues well afterwards. I have tried to intentionally live my life, regularly reflecting on what makes me happy and fills me with energy, and what does the reverse. Those reflections led me to both become a manager (lead engineer), and then eventually step away from that job. After transitioning from management to a staff engineer role, I continually tried to define my own job to be the best it could for myself and my company. Those efforts set the scene for this series.

• This series starts with me getting the role I had long worked to develop and sell. Starting that role was scary, as I then needed to deliver in that high stakes role. In this first post, I talk about framing that pressure and the nervousness of having to deliver in that new position.
• The series continues after my dismissal as I tried to give myself the space I needed to adjust. It turns out that too much free space can be terrifying, at least for me.
• Losing my job was a large shock to my sense of identity, and I have been actively reshaping my sense of self. Several posts in the series cover my thinking and shaping of my identity:
• Early in my time off, I started to grapple with the question of “Who am I?” I had let too much of my identity be tied up in my job. Losing my job disrupted how I thought of myself.
• After spending a lot of time grappling with who I am, I realized I didn’t know who I was. At least I didn’t know how to tell the story of my personal journey over the past 10 years, including losing my job.
• After continued reflection, I returned to who I wanted to be, as a unified human being. I didn’t want a professional identity and a personal identity. I wanted one unified identity, with my professional life being one expression of that identity. I expect this will be a continuing process for the rest of my life. I feel really good about my current framing of my identity.
• Finally, the period of reflection covered by this series ends with me starting my next job and moving on to the next chapter of my life.

These posts, including the reflection, drafts, editing, and discussion that went into them, are part of the larger process I’ve been going through as I’ve made sense of this part of my life — and my life going forward. Writing them has been a healing, growing, and positive part of my past year. I share it all with you in the hope that it might help you or otherwise resonate with you. If any of it does help or resonate, I would love to hear from you about it.

Thank you to Heather Beasley Doyle for her feedback on this post and her support through this entire period of my life. Heather is a gifted writer. You should check out her homepage and her writing. 

I was sitting on my mother's couch after Christmas, crossword puzzle in hand, when my stomach suddenly dropped. The puzzle hadn’t suddenly frightened me, nor had anything happened around me. Rather, my mind had jumped forward to the end of my vacation and returning to work. I liked my job. I had successfully changed it to better align with my interests and abilities over the previous year. What scared me was thinking about the stakes of those changes: I had spent a lot of political capital crafting that role for myself and I needed to deliver the expected results.

This post explores the expectations and anxiety of getting an opportunity you want and then trying to deliver on its promise. We never know how things will turn out or what challenges will arise. The most we can do is put ourselves in good situations with good chances for success, do our best, and then accept whatever comes — good or bad.

---

Desire for ChangeIn the summer of 2020, I left my role as a manager and shifted to a staff engineering role. I spent the next two years working with a new manager to rebuild the team, with him as its lead. We grew the team from two of us to 10, and all 10 of us continuously supported and learned from each other. I was proud of what we had built together, but with that done, I felt unsatisfied. I was not achieving the goals I had laid out for myself when I stepped away from being a manager: 

Now I’m working to keep the parts of my role that did bring me energy, and remove the parts that did not. I absolutely love the impact I've had on how we test performance. I love all the things that we built, including our structure and processes. I love having insight into so many parts of the engineering organization and helping drive the big picture on performance testing. I love helping junior colleagues learn and grow, sharing learnings with them (so long as I don't also have to evaluate them).

Specifically, I wasn’t happy with my progress on having insight into so many parts of the engineering organization and helping drive the big picture on performance testing. So in late 2022, I set out to change my role once again.Advocating for Change

I worked to define a new role that would make me happier. The role should have the insight and big picture impact I was missing. It should include reporting to the right level of management so I could have the reach, influence, and support to do the things I wanted. And it should include many of the things I knew filled me with energy, such as writing and sharing my learnings (blogs, papers, talks), reading and learning (research papers, blogs), and interacting with passionate people. Around this time I wrote a blog post on defining your own job. At the end of it I said:

I know what I want to do: I want to advance the state of the art of performance testing and software engineering at MongoDB, ideally through collecting, curating, and demonstrating the best ideas from the performance community and academia.

I started to pitch a role based on those things. A role in which I could interact with multiple teams, leverage the best the research community could provide us, and turn that into something real with big impact. I wrote a proposal capturing the role and talked to people about it – a lot of people.

There wasn't much appetite in my organization for a research-focused role. Everyone loved the results I had delivered using academic research, but we suffered from smaller, pressing problems that we needed to solve at that moment. Before we could invest in the larger and more interesting things, we needed to do the simpler things to solve problems today.
An Updated Proposal

I took the feedback, learned from it, and reflected. The perfect job should fit my needs and the company's needs. These conversations made the company’s needs clearer to me. I updated my role proposal to better align with both sets of needs: making our performance testing infrastructure stronger in the near term, while enabling greater things in the future. The proposal now focused on straightforward engineering work instead of research, and leaned more heavily into planning and coordination. It met my need for broad impact and the company's need for immediate results. The role would eventually enable the more advanced work I wanted to do.

I didn’t share the proposal at this point; instead, I started on the work it described. I teamed up with my product manager to write and submit a formal project proposal based on my role proposal. The project proposal laid out work for several teams for the next two years. 

Living the Change

The project was approved, with me as its technical lead. With that, I shared my updated role proposal with key people. It now described a role I was already doing.

I found a sponsor. With their help I changed teams and started reporting higher in the org chart. I was in a better place to achieve my goals for myself and for my company! Success!

Scariness

This is when things got scary. I had invested a lot of time, effort, and political capital to build this role for myself. If I didn’t succeed in the role, it would all be for naught.

I could see what I needed to do to succeed. However, there were many things I could not control. If this new role didn’t work out, I couldn’t go back to my old role – I didn’t want to, and most likely it wouldn’t be available to me if I did. No other role would be a better fit for me within the company. And, having spent most of my political capital, I couldn’t expect much help changing roles again. Essentially, my only options were to succeed or to leave. I was operating without a safety net.

Lowering the Stakes

While the differences between success and failure were stark, I didn’t want to live in fear. I worked to reframe how I looked at the situation. We never know the future with certainty, so every choice we make is a gamble. I asked myself questions about this bet: Is this a good bet? Would I make this bet again knowing what I know now? Can I live with the consequences if I lose this bet?    

As I sat on my mother's couch with my crossword puzzle, I answered these questions: Yes it is a good bet. Yes, I would make this bet again. Yes, I can live with the consequences of losing this bet.

I began to relax. The fear did not completely go away, but I could put it in perspective. Since then, I have worked hard to remember this perspective whenever that kind of fear comes back.

Similarly, I encourage you to choose your best opportunities. Whenever the future seems scary, reflect on whether you have given yourself your best chance for success and if you can accept the consequences if these chances do not pan out. If the answers are no, go make changes. If the answers are yes (I hope they are), try to keep that perspective and let go of your fear.

Thank you to Heather Beasley Doyle for her feedback on this post. I am a better writer and this is a better post due to her efforts. Heather is a gifted writer. You should check out her homepage and her writing. 

Tomorrow (January 2nd, 2024) I start a new job. Five months ago I left my previous job with no idea of what would come next, beyond taking a long break to relax and recharge. 

Me, at an early viewpoint on Wildcat Mountain towards the start of a backpacking trip and the beginning of my 5 months off. Photo courtesy of Tom Lehmann.

Over the past five months, I’ve done exactly that, taking a long break. I have done a lot, both physically and intellectually. I have literally traveled around the world, visiting Bangkok, Bhutan, and Singapore. I’m in great physical shape. I’ve gone on backpacking and hiking trips, climbing many mountains in the Adirondacks and White Mountains. I took a week-long cooking class. In December, I skied a lot, including trips to resorts in Vermont, Maine, and New York. I have read, learned, and relaxed. I’ve met many people and enjoyed sharing thoughts, ideas, and support with them. I’ve also spent time writing, which helps me think, although not much of that writing has found its way to my blog so far.

With my wife Orit on top of Khamsum Yulley Namgyal Choeten in Bhutan.
Now it is time to get back to creating and contributing directly to this world. I am excited to join a young company (Personio) that helps other companies succeed by automating much of their HR needs. Personio enables those companies to focus on what they do best and “empower their people to do their best work.” HR functions may not be sexy, but every company needs them.

With my son on the summit of Mt Marcy, the tallest mountain in New York State.
I am returning to a staff engineer role. I expect to spend most of my time working on our toughest technical problems, improving our technology and technical processes, and helping my colleagues grow and be the best they can be. After I get settled into my new role, I will also see how I can best continue to participate in my professional communities.  

With my friends Jay and Tom on the summit of Middle Carter Mountain in the White Mountains. As with all of life, I expect there to be triumphs, setbacks, challenges, and help from others. I look forward to all of it. I especially look forward to learning with my new colleagues. And as I learn, I will continue to share my learnings and experiences with you, here in my blog. 

View of Afghanistan out the plane window while flying to Singapore.

 

The tech world is obsessed with speed – the faster the better. Books are written about it (If You're Not First, You're Last, Fail Fast, Learn Faster), Facebook used to have the motto "Move fast and break things," and people tout ideas like "Act Before You're Ready.” They aren't wrong, but the focus on speed can be misleading. Sometimes going fast first requires going slow.

There are indviduals who seem to work slowly, but actually get things done very quickly. I call them Slow Thinkers. They are easily overlooked, but they do amazing things. We all lose out when we ignore them. 

I’d like to introduce you to them and to their more visible cousins, whom I call Fast Thinkers. I think both groups are rare – few people can compare to them. I have great respect for both and have been blessed to work with multiple Fast and Slow Thinkers; Two such individuals particularly inspire the composite characters Claire and Ed described below. While I may picture those two, Claire and Ed could be anyone possessing their skills.

I do not put myself in either of their leagues, but my habits are more aligned with the Slow Thinker’s than the Fast Thinker’s. I’ve worked with enough Fast Thinkers to know deep in my bones that I cannot and will never be able to compete directly with them. If I have to work to the standard of the Fast Thinker, I cannot give my best work. I will make mistakes. I won’t be happy, my colleagues won’t be happy, and my boss won’t be happy. I suspect that is true for many of you as well.

 
Image created using OpenAI's DALL·E tool.

---

A Fast Thinker

Paul looks around the conference table at his assembled team. He calls the group to order, asks them, “How do we solve hard problem?” and provides some background on hard problem. Sally offers her opinion on what they should do, followed by Sam, both to mixed reactions. Then Claire says, “We should do good thing,” and explains what good thing is. She continues, “We should do it because of A, B, and C. It will address the worst of hard problem quickly, and then solve it completely in a couple of months.” Good thing includes some aspects of Sally’s and Sam’s suggestions, but it is all-around better. Claire is right – they should do good thing now. Everyone agrees to her plan, they do it, and they fix the problem.

Claire is a Fast Thinker. She is very smart, comes up with correct solutions frighteningly fast, can clearly explain those solutions, and knows she is right. It is both a pleasure and intimidating to work with a Fast Thinker. Everyone listens to Fast Thinkers and they can make short work of big problems. While they are always valuable, they are extra valuable during a crisis when every minute counts. Everyone can take comfort from their confident leadership in such situations.

A Slow Thinker

A few weeks later Paul calls the same group to order again. Paul presents new hard probem. The members of the team offer their thoughts once again. There are a lot of thoughts, including some from Claire. However, this time Claire isn’t confident of her proposal, recognizing limitations to her solution. There is a lot of discussion, but there are no strong conclusions. There are a couple of promising options, but each has significant trade-offs. Towards the end of the meeting, Paul turns to Ed, “Ed, what do you think?” Many people in the room are surprised because they had forgotten that Ed was in the room. He hasn’t said anything the entire meeting. Instead, he quietly listened as each person spoke.

He starts to speak. He speaks slower than everyone else has, with a contemplative tone. Ed restates new hard problem but in a slightly different and more interesting way. He’s not sure what should be done. He calls out some of the good points made by the others but then ties the problem to a completely new idea. Ed says he would like to take a couple of days to look into this new idea and see if there’s something there. He hedges that there might not be. Paul agrees to give Ed a few days and everyone leaves. When they return a few days later, Ed shares an incredible new idea. This new idea has grown from the seed of his original thoughts. Ed has grown this seed into something beautiful. It is the correct solution and it is amazing.

Ed is a Slow Thinker. Unlike Claire, Ed does not know the correct answer right away. He has thoughts and suspicions, but he needs to think through all those thoughts and suspicions before coming to a conclusion. Those who know Ed well, know he is brilliant – just as smart as Claire, but smart in a different way. Those who don’t, see a quiet, polite man. They overlook him. Thankfully, Paul knows Ed well and makes sure to regularly ask Ed what he is thinking. Importantly, Paul listens closely when Ed answers.

The Value of Speed

Speed is treasured in the tech world. The faster the company, the team, or the individual, the more chances they have to try things, to learn, and to win big. Fortunes are made by having a good idea and being faster than the competition. This is why so many books and phrases exalt speed.

However, we need to be careful how we measure speed. The time we take to do any given task is not important. What is important is the time to get something new (a product, a feature, or a fix) to market. The Fast Thinker appears to be much faster than the Slow Thinker because they do tasks faster. They have answers immediately, not in a few days.

When measured in time to delivered value (e.g., a new product …) and delivered value per time, I think the Slow Thinker is as fast as the Fast Thinker. Yes, they may take a little longer at the start of the process, but larger problems and innovations take weeks or months. A few days of contemplation at the start can make the remaining time much more productive and shorter.

Responsibility and Rewards

Everyone recognizes the brilliance of Fast Thinkers. People listen to Fast Thinkers and do what they say (and they should). The Fast Thinkers get promoted.

It’s easy to miss the brilliance and ultimate speed of Slow Thinkers. If Paul doesn’t specifically ask Ed what he thinks in that meeting, the world could easily miss out on a great idea. That would be a shame.

The success of Fast Thinkers can lead to the marginalization of Slow Thinkers. As Fast Thinkers are promoted, they take on more responsibility, such as deciding who they will promote and whose ideas they will listen to and implement. They know the value of their quick decisions, making it easy for them to value others’ quick decisions. It’s harder for them to see the Slow Thinkers’ value, because it is different than the value they provide as a Fast Thinkers.

Don’t Ignore The Slow Thinkers

It’s easy for Fast Thinkers to ignore Slow Thinkers. It’s easy for us mere mortals to do the same. Please don’t ignore the Slow Thinkers. They are a force waiting to be unleashed. If we ignore the Eds of the world, we lose out on their ideas and all that we could learn from them. So don’t just listen to the people who speak up forcibly. Listen also for the quieter voices. See which of those quieter voices provide wonderful and impactful ideas when given the space. Then bring them your hardest problems, give them some space, and listen to what they say. Then we can really go fast.

And if you identify with the Slow Thinkers, don’t sell yourself short. Don’t get competitive and try to be faster than the Fast Thinkers around you. Do give yourself the space and time to do your best work, make sure to work on important problems, and make sure others know about your great work.

Special thank you to Heather Beasley Doyle for her feedback on this post. Heather is a gifted writer and you should read some of her writing.

 Note: This was originally published on the MongoDB Engineering Blog on April 30, 2019 here by Henrik Ingo and myself. Please read it there assuming the link works. I have copied it here to ensure the content does not disappear. The links in the article are the original links. 

On the MongoDB Performance team, we use EC2 to run daily system performance tests. After building a continuous integration system for performance testing, we realized that there were sources of random variation in our platform and system configuration which made a lot of our results non-reproducible. The run to run variation from the platform was bigger than the changes in MongoDB performance that we wanted to capture. To reduce such variation - environmental noise - from our test configuration, we set out on a project to measure and control for the EC2 environments on which we run our tests.

At the outset of the project there was a lot of doubt and uncertainty. Maybe using a public cloud for performance tests is a bad idea and we should just give up and buy more hardware to run them ourselves? We were open to that possibility, however we wanted to do our due diligence before taking on the cost and complexity of owning and managing our own test cluster.

Performance benchmarks in continuous integration

MongoDB uses a CI platform called Evergreen to run tests on incoming commits. We also use Evergreen for running multiple classes of daily performance tests. In this project we are focused on our highest level tests, meant to represent actual end-user performance. We call these tests System Performance tests.

For _System Performance_tests, we use EC2 to deploy real and relatively beefy clusters of c3.8xlarge nodes for various MongoDB clusters: standalone servers, 3 Node Replica Sets, and Sharded Clusters. These are intended to be representative of how customers run MongoDB. Using EC2 allows us to flexibly and efficiently deploy such large clusters as needed. Each MongoDB node in the cluster is run on its own EC2 node, and the workload is driven from another EC2 node.

Repeatability

There's an aspect of performance testing that is not obvious and often ignored. Most benchmarking blogs and reports are focusing on the maximum performance of a system, or whether it is faster than some competitor system. For our CI testing purposes, we primarily care about repeatability of the benchmarks. This means, the same set of tests for the same version of MongoDB on the same hardware should produce the same results whether run today or in a few months. We want to be able to detect small changes in performance due to our ongoing development of MongoDB. A customer might not get very upset about a 5% change in performance, but they will get upset about multiple 5% regressions adding up to a 20% regression.

The easiest way to avoid the large regressions is to identify and address the small regressions promptly as they happen, and stop the regressions getting to releases or release candidates. We do want to stress MongoDB with a heavy load, but, achieving some kind of maximum performance is completely secondary to this test suite’s goal of detecting changes.

For some of our tests, repeatability wasn't looking so good. In the below graph, each dot represents a daily build (spoiler -- you’ll see this graph again):

Eyeballing the range from highest to lowest result, the difference is over 100,000 documents / second from day to day. Or, as a percentage, a 20-30% range.

Investigation

To reduce such variation from our test configuration, we set out on a project to reduce any environmental noise. Instead of focusing on the difference between daily MongoDB builds, we ran tests to focus on EC2 itself.

Process: Test and Analyze

Benchmarking is really an exercise of the basic scientific process:

1. Try to understand a real world phenomenon, such as an application that uses MongoDB

2. Create a model (aka benchmark) of that phenomenon (this may include setting a goal, like "more updates/sec")

3. Measure

4. Analyze and learn from the results

5. Repeat: do you get the same result when running the benchmark / measuring again?

6. Change one variable (based on analysis) and repeat from above

We applied this benchmarking process to evaluate the noise in our system. Our tests produce metrics measuring the average operations per second (ops/sec). Occasionally, we also record other values but generally we use ops/sec as our result.

To limit other variables, we locked the mongod binary to a stable release (3.4.1) and repeated each test 5 times on 5 different EC2 clusters, thus producing 25 data points.

We used this system to run repeated experiments. We started with the existing system and considered our assumptions to create a list of potential tests that could help us determine what to do to decrease the variability in the system. As long as we weren’t happy with the results we returned to this list and picked the most promising feature to test. We created focused tests to isolate the specific feature, run the tests and analyze our findings. Any workable solutions we found were then put into production.

For each test, we analyzed the 25 data points, with a goal of finding a configuration that minimizes this single metric:

range = (max - min) / median

Being able to state your goal as a single variable such as above is very powerful. Our project now becomes a straightforward optimization process of trying different configurations, in order to arrive at the minimum value for this variable. It's also useful that the metric is a percentage, rather than an absolute value. In practice, we wanted to be able to run all our tests so that the range would always stay below 10%.

Note that the metric we chose to focus on is more ambitious than, for example, focusing on reducing variance. Variance would help minimize the spread of most test results, while being fairly forgiving about one or two outliers. For our use case, an outlier represents a false regression alert, so we wanted to find a solution without any outliers at all, if possible.

Any experiment of this form has a tension between the accuracy of the statistics, and the expense (time and money) of running the trials. We would have loved to collect many more trials per cluster, and more distinct clusters per experiment giving us higher confidence in our results and enabling more advanced statistics. However, we also work for a company that needed the business impact of this project (lower noise) as soon as possible. We felt that the 5 trials per cluster times 5 clusters per experiment gave us sufficient data fidelity with a reasonable cost.

Assume nothing. Measure everything.

The experimental framework described above can be summarized in the credo of: Assume nothing. Measure everything.

In the spirit of intellectual honesty, we admit that we have not always followed the credo of Assume nothing. Measure everything, usually to our detriment. We definitely did not follow it when we initially built the System Performance test suite. We needed the test suite up as soon as possible (preferably yesterday). Instead of testing everything, we made a best effort to stitch together a useful system based on intuition and previous experience, and put it into production. It’s not unreasonable to throw things together quickly in time of need (or as a prototype). However, when you (or we) do so, you should check if the end results are meeting your needs, and take the results with a large grain of salt until thoroughly verified. Our system gave us results. Sometimes those results pointed us at useful things, and other times they sent us off on wild goose chases.

Existing Assumptions

We made a lot of assumptions when getting the first version of the System Performance test suite up and running. We will look into each of these in more detail later, but here is the list of assumptions that were built into the first version of our System Performance environment:

Assumptions:

• A dedicated instance means more stable performance

• Placement groups minimize network latency & variance

• Different availability zones have different hardware

• For write heavy tests, noise predominantly comes from disk

• Ephemeral (SSD) disks have least variance

• Remote EBS disks have unreliable performance

• There are good and bad EC2 instances

In addition, the following suggestions were proposed as solutions to reducing noise in the system:

• Just use i2 instances (better SSD) and be done with it

• Migrate everything to Google Cloud

• Run on prem -- you’ll never get acceptable results in the cloud

Results

After weeks of diligently executing the scientific process of hypothesize - measure - analyze - repeat we found a configuration where the range of variation when repeating the same test was less than 5%. Most of the configuration changes were normal Linux and hardware configurations that would be needed on on-premise hardware just the same as on EC2. We thus proved one of the biggest hypotheses wrong:

You can't use cloud for performance testing

With our first experiment, we found that there was no correlation between test runs and the EC2 instances they were run on. Please note that these results could be based on our usage of the instance type; you should measure your own systems to figure out the best configuration for your own system. You can read more about the specific experiment and its analysis in our blog post EC2 instances are neither good nor bad.

There are good and bad EC2 instances

After running the first baseline tests, we decided to investigate IO performance. Using EC2, we found that by using Provisioned IOPS we get a very stable rate of disk I/O per second. To us, it was surprising that ephemeral (SSD) disks were essentially the worst choice. After switching our production configuration from ephemeral SSD to EBS disks, the variation of our test results decreased dramatically. You can read more about our specific findings and how different instance types performed in our dedicated blog post EBS instances are the stable option.

Ephemeral (SSD) disks have least variance

Remote EBS disks have unreliable performance -> PIOPS

Just use i2 instances (better SSD) and be done with it (True in theory)

Next, we turned our attention to CPU tuning. We learned that disabling CPU options does not only stabilize CPU-bound performance results. In fact, noise in IO-heavy tests also seems to go down significantly with CPU tuning.

For write heavy tests, noise predominantly comes from disk

After we disabled CPU options, the variance in performance decreased again. In the below graph you can see how changing from SSD to EBS and disabling CPU options reduced the performance variability of our test suite. You can read more about the CPU options we tuned in our blog post Disable CPU options.

At the end of the project we hadn’t tested all of our original assumptions, but we had tested many of them. We still plan to test the remaining ones when time and priority allow:

• A dedicated instance means more stable performance

• Placement groups minimize network latency & variance

• Different availability zones have different hardware

Through this process we also found that previously suggested solutions would not have solved our pains either:

• Just use i2 instances (better SSD) and be done with it (True in theory)

• Migrate everything to Google Cloud: Not tested!

Conclusion of the tests

In the end, there was still noise in the system, but we had reduced it sufficiently that our System Performance tests were now delivering real business value to the company. Every bit of noise bothers us, but at the end of the day we got to a level of repeatability in which test noise was no longer our most important performance related problem. As such, we stopped the all out effort on reducing system noise at this point.

Adding in safeguards

Before we fully moved on to other projects, we wanted to make sure to put up some safeguards for the future. We invested a lot of effort into reducing the noise, and we didn’t want to discover some day in the future that things had changed and our system was noisy again. Just like we want to detect changes in the performance of MongoDB software, we also want to detect changes in the reliability of our test platform.

As part of our experiments, we built several canary benchmarks which give us insights into EC2 performance itself based on non-MongoDB performance tests. We decided to keep these tests and run them as part of every Evergreen task, together with the actual MongoDB benchmark that the task is running. If a MongoDB benchmark shows a regression, we can check whether a similar regression can be seen in any of the canary benchmarks. If yes, then we can just rerun the task and check again. If not, it's probably an actual MongoDB regression.

If the canary benchmarks do show a performance drop, it is possible that the vendor may have deployed upgrades or configuration changes. Of course in the public cloud this can happen at arbitrary times, and possibly without the customers ever knowing. In our experience such changes are infrequently the cause for performance changes, but running a suite of "canary tests" gives us visibility into the day to day performance of the EC2 system components themselves, and thus increases confidence in our benchmark results.

The canary tests give us an indication of whether we can trust a given set of test results, and enables us to clean up our data. Most importantly, we no longer need to debate whether it is possible to run performance benchmarks in a public cloud because we measure EC2 itself!

Looking forward

This work was completed over 1.5 years ago. Since that time it has provided the foundation that all our subsequent and future work has been built upon. It has led to 3 major trends:

We use the results. Because we lowered the noise enough, we are able to regularly detect performance changes, diagnose them, and address them promptly. Additionally, developers are also "patch testing" their changes against System Performance now. That is, they are using System Performance to test the performance of their changes before they commit them, and address any performance changes before committing their code. Not only have we avoided regressions entering into our stable releases, in these cases we’ve avoided performance regressions ever making it into the code base (master branch).

We’ve added more tests. Since we find our performance tests more useful, we naturally want more such tests and we have been adding more to our system. In addition to our core performance team, the core database developers also have been steadily adding more tests. As our system became more reliable and therefore more useful, the motivation to create tests across the entire organization has increased. We now have the entire organization contributing to the performance coverage.

We’ve been able to extend the system. Given the value the company gets from the system, we’ve invested in extending the system. This includes adding more automation, new workload tools, and more logic for detecting when performance changes. None of that would have been feasible or worthwhile without lowering the noise of the System Performance tests to a reasonable level. We look forward to sharing more about these extensions in the future.

Coda: Spectre/Meltdown

As we came back from the 2018 New Years holidays, just like everyone else we got to read the news about the Meltdown and Spectre security vulnerabilities. Then, on January 4, all of our tests went red! Did someone make a bad commit into MongoDB, or is it possible that Amazon had deployed a security update with a performance impact? I turned out that one of our canary tests - the one sensitive to cpu and networking overhead - had caught the 30% drop too! Later, on Jan 13, performance recovered. Did Amazon undo the fixes? We believe so, but have not heard it confirmed.

The single spike just before Jan 13 is a rerun of an old commit. This confirms the conclusion that the change in performance comes from the system, as running a Jan 11 build of MongoDB after Jan 13, will result in higher performance. Therefore the results depend on the date the test was run, rather than which commit was tested.

As the world was scrambling to assess the performance implications of the necessary fixes, we could just sit back and watch them in our graphs. Getting on top of EC2 performance variations has truly paid off.

Update: @msw pointed us to this security bulletin, confirming that indeed one of the Intel microcode updates were reverted on January 13.

Note: This was originally published on the MongoDB Engineering Blog on April 30, 2019 here by Henrik Ingo and myself. Please read it there assuming the link works. I have copied it here to ensure the content does not disappear. The links in the article are the original links.

In an effort to improve repeatability, the MongoDB Performance team set out to reduce noise on several performance test suites run on EC2 instances. At the beginning of the project, it was unclear whether our goal of running repeatable performance tests in a public cloud was achievable. Instead of debating the issue based on assumptions and beliefs, we decided to measure noise itself and see if we could make configuration changes to minimize it.

After thinking about our assumptions and the experiment setup, we began by recording data about our current setup and found no evidence of particularly good or bad EC2 instances. In the next step, we investigated IO and found that EBS instances are the stable option for us. Having found a very stable behavior as far as disks were concerned, this third and final experiment turns to tuning CPU related knobs to minimize noise from this part of the system.

Investigate CPU Options

We already built up knowledge around fine-tuning CPU options when setting up another class of performance benchmarks (single node benchmarks). That work had shown us that CPU options could also have a large impact on performance. Additionally, it left us familiar with a number of knobs and options we could adjust.

KnobWhere to setSettingWhat it does Idle Strategy Kernel Boot idle=poll Puts linux into a loop when idle,  checking for work. Max sleep state    (c4 only) Kernel Boot intel_idle.max_cstate=1  intel_pstate=disable Disables the use of advanced  processor sleep states. CPU Frequency Command Line sudo cpupower frequency-set -d  2.901GHz Sets a fixed frequency. Doesn't  allow the CPU to vary the  frequency for power saving. Hyperthreading Command Line echo 0 > /sys/devices/system/  cpu/cpu$i/online Disables hyperthreading.  Hyperthreading allows two  software threads of execution to  share one physical CPU. They  compete against each other for  resources.

We added some CPU specific tests to measure CPU variability. These tests allow us to see if the CPU performance is noisy, independently of whether that noise makes MongoDB performance noisy. For our previous work on CPU options, we wrote some simple tests in our C++ harness that would, for example:

• multiply numbers in a loop (cpu bound)
• sleep 1 or 10 ms in a loop
• Do nothing (no-op) in the basic test loop

We added these tests to our System Performance project. We were able to run the tests on the client only, and going across the network.

We ran our tests 5x5 times, changing one configuration at a time, and compared the results. The first two graphs below contain results for the CPU-focused benchmarks, the third contains the MongoDB-focused benchmarks. In all the below graphs, we are graphing the "noise" metric as a percentage computed from (max-min)/median and lower is better.

We start with our focused CPU tests, first on the client only, and then connecting to the server. We’ve omitted the sleep tests from the client graphs for readability, as they were essentially 0.

The nop test is the noisiest test all around, which is reasonable because it’s doing nothing in the inner loop. The cpu-bound loop is more interesting. It is low on noise for many cases, but has occasional spikes for each case, except for the case of the c3.8xlarge with all the controls on (pinned to one socket, hyperthreading off, no frequency scaling, idle=poll).

When we connect to an actual server, the tests become more realistic, but also introduce the network as a possible source of noise. In the cases in which we multiply numbers in a loop (cpuloop) or sleep in a loop (sleep), the final c3.8xlarge with all controls enabled is consistently among the lowest noise and doesn’t do badly on the ping case (no-op on the server). Do those results hold when we run our actual tests?

Yes, they do. The right-most blue bar is consistently around 5%, which is a great result! Perhaps unsurprisingly, this is the configuration where we used all of the tuning options: idle=poll, disabled hyperthreading and using only a single socket.

We continued to compare c4 and c3 instances against each other for these tests. We expected that with the c4 being a newer architecture and having more tuning options, it would achieve better results. But this was not the case, rather the c3.8xlarge continued to have the smallest range of noise. Another assumption that was wrong!

We expected that write heavy tests, such as batched inserts, would mostly benefit from the more stable IOPS on our new EBS disks, and the CPU tuning would mostly affect cpu-bound benchmarks such as map-reduce or index build. Turns out this was wrong too - for our write heavy tests, noise did not in fact predominantly come from disk.

The tuning available for CPUs has a huge effect on threads that are waiting or sleeping. The performance of threads that are actually running full speed is less affected - in those cases the CPU runs at full speed as well. Therefore, IO-heavy tests are affected a lot by CPU-tuning!

Disabling CPU options in production

Deploying these configurations into production made insert tests even more stable from day to day:

Note that the absolute performance of some tests actually dropped, because the number of available physical CPUs dropped by ½ due to only using a single socket, and disabling hyperthreading causes a further drop, though not quite a full half, of course.

Conclusion

Drawing upon prior knowledge, we decided to fine tune CPU options. We had previously assumed that IO-heavy tests would have a lot of noise coming from disk and that CPU tuning would mostly affect CPU-bound tests. As it turns out, the tuning available for CPUs actually has a huge effect on threads that are waiting or sleeping and therefore has a huge effect on IO-heavy tests. Through CPU tuning, we achieved very repeatable results. The overall measured performance in the tests decreases but this is less important to us. We care about stable, repeatable results more than maximum performance.

This is the third and last of three bigger experiments we performed in our quest to reduce variability in performance tests on EC2 instances. You can read more about the top level setup and results as well as how we found out that EC2 instances are neither good nor bad and that EBS instances are the stable option.

Note: This was originally published on the MongoDB Engineering Blog on January 30, 2023 here. Please read it there assuming the link works. I have copied it here to ensure the content does not disappear. 

 At MongoDB, we want to (honestly) tell our users that each new version of our software is faster than the previous version. We also want to be able to explain why. We definitely do not want to learn that a release is slower (we have a performance regression) from our customers telling us after discovering it for themselves. In order to do this, we need to understand the performance of our software, detect performance changes early, and aggressively redress the root cause.

We have invested significantly into building a performance testing system to achieve these goals. This includes creating a large number of performance tests, automating the running of those tests, and building tools to diagnose performance regressions when we find them. Those tools and tests are not enough by themselves: they produce an overwhelming amount of data. That data needs to be analyzed to determine if the performance changed. We could not process it all. We have developed new tools to process the data, using advanced statistical techniques to detect real performance regressions and identify the causes of those regressions.

Where we started

We built our original performance testing system in 2015. It ran a collection of performance tests directly in our CI system (Evergreen). We automated every step of running a test and collecting the results. That left the hard part: making sense of the results.

Computers are fascinating things, built up from a huge number of simple and deterministic components. However, the interactions between those simple components lead to the emergence of non-deterministic behavior. As computers get more complex, the emergent behavior becomes more pronounced. The net effect is that when you run a program twice, the two executions will differ (i.e. one may take longer), even when run on the same machine.

The problem gets even harder when you go from running on a single computer, to multiple computers in a distributed system. Network latencies will vary depending on the state of the network switches and other traffic on the network. The combination of each computer's variability combined with the variability of the network leads to more variability. MongoDB is a distributed system. When we test the performance of MongoDB, we have to address all of these issues.

For performance tests, these differences show up as different measurements of performance. Your program may take more or less time to run. It may execute more or fewer operations within a period of time. You may see more or fewer slow operations. We call this phenomenon run to run variation or measurement noise. Run to run variation makes it harder to determine if changes to the software made the software intrinsically faster or slower. Thus, we did an enormous amount of work to limit the measurement noise in our tests, both in the original project, and in subsequent projects.

Still, no matter how hard anyone tries, there will always be run to run variation. This presents a challenge when we want to interpret our performance results (or if you want to interpret your performance results). Maybe we are comparing two versions of our software and want to know which one is faster. If we have results that are 5% faster on the new version, is that due to our software being 5% faster? Or is the 5% due to run to run variation? Or worse, is the 5% change due to 10% run to run variation combined with our software actually being 5% slower?

When we started, we only had a few performance tests. We manually inspected the results and could understand if and when the performance changed. However, as we added more tests, and more results per test, human inspection became less effective: we missed things and it was hard and unsatisfying work.

We automated comparing the performance of one version of the software to another very early in the development of our system. We wrote software to compare the new performance results to older performance results. If the results changed more than 10%, we flagged it and had a human look at it.

Using a direct comparison was common practice in the industry. It was also awful. The comparisons missed small regressions, they flagged a lot of false positives on noisier tests, and sometimes they flagged real things, but at the wrong time. The automated comparisons were much better than manual inspection, but still awful.

We continually built improvements to make the system less awful. We had a system to increase the comparison threshold (from 10%) for noisier tests, and a system to reset the comparison when there was a change in performance (i.e., compare to the new normal). These changes improved the system, but they did not fundamentally overcome the challenges we faced.

Solving the right problem

Along the way, we realized we were trying to solve the wrong problem. Our automated comparison was answering the question: “Has measured performance changed more than 10% between these two versions of software”. What we really wanted to answer was “Which software changes altered performance (for better or worse)”. Those two questions overlap for large performance changes in low noise environments, but they differ on noisy tests or for small changes in performance.

The second question (“which software changes altered performance?”) focuses on detecting changes in a measured value over time. This question maps to a known problem called change point detection. Change point detection is the problem of finding when changes in values occurred in time (time-series) in the presence of noise or other confounding variables. For example, it’s used to detect changes in behavior on such things as electricity consumption, population totals, local weather, and stock prices. There’s a lot of existing work on change point detection, so we just needed to pick the best existing work, implement it, and put it into production. Simple, right?

Well, maybe not. We did not know what was the best existing work, and we did not know if it would fix our problems. So, we did some research, identifying likely techniques and collecting papers on them. The papers accumulated and stayed on my desk, because I didn’t have time to dive into a speculative project when there were plenty of things that needed to be done NOW.

Enter an intern

During the summer of 2017, two interns joined us on the performance team. They spent the summer working with us on our performance testing infrastructure. Both of them were great, giving our work an extra push forward.

We encourage our interns to learn and grow. One way we do this is by explaining what we are doing and why we are doing it. We explain the larger context of the work. This naturally leads to discussing open challenges. One of our interns asked if they could read that stack of papers sitting on my desk (of course they could). Towards the end of the summer, he had completed his summer project early. Further, he had read the papers, understood them, and asked if he could make a prototype! In particular, he had gone through the complex math of the papers, and figured out how that math could be implemented in software.

He built a prototype. It was limited, but it proved that the concept could work. The algorithm clearly found the changes in the sample traces we created, and did not get confused when run on sample data containing random background noise. Based on this initial success, we scheduled a larger proof of concept project to integrate the algorithm with our production system. We compared this second proof of concept with the existing comparison code, and determined it was MUCH better. We then did the work to get the algorithm in production and update our processes to use it.

Our production system today

When we started in 2015, we ran only a handful of tests and only a handful of people used the performance infrastructure directly. Today we run hundreds of distinct performance tests, generating over 100k distinct results per software commit. Today, everyone who develops MongoDB interacts with our performance testing infrastructure.

When a developer commits a change to MongoDB, tests are run. Upon completion, change point detection is used to detect performance changes (improvements and regressions). A dedicated team triages these changes, isolates them to specific commits, and assigns these changes to developers to investigate. In the case of improvements, the developers confirm that the change was expected, or investigate the change to understand why the performance got better. Sometimes things get faster because of bugs – we have found bugs this way.

Our system is good at detecting regressions and our engineers are good at fixing them. Even better than fixing a regression, is preventing a performance regression from ever being committed to our development branch. Developers can test their proposed changes before committing the changes, using something called a patch build. In this way, the developers can make sure they are not introducing new performance regressions, verify a fix, or confirm an optimization before committing their code.

Advancing science!

At MongoDB we take pride in developing a database and a database platform that empowers developers to make applications that change the world. We depend on our performance testing infrastructure to ensure we ship a performant database. We are proud of the performance infrastructure we have built and the impact it has had on the software we ship to our users.

We do not do any of this work in a void. At MongoDB we benefit from being part of several communities, and we want to support these communities. It is for this reason that most of our database source code is publicly available and our JIRA project for database development is also public.

When we developed a new way of finding performance regressions in our software, we didn’t hide it away. Instead, we shared it with the community, and will continue to do so as we learn and progress. This started with submitting a paper called “The Use of Change Point Detection to Identify Software Performance Regressions in a Continuous Integration System” to the International Conference on Performance Engineering (ICPE). It has continued with more papers (Creating a Virtuous Cycle in Performance Testing at MongoDB, Automated system performance testing at MongoDB) and presentations.

These talks and presentations have helped the community, but they have also helped us. By sharing and participating in the community, we have more people thinking about our problems. We’ve had the best minds in performance engineering in academia sharing ideas and suggestions with us on how to improve our technology!

Often the ideas build on each other. One such idea led to the creation of the Data Challenge Track at ICPE in 2022. Building on our papers, we were able to open up our performance test results as a shareable artifact. The data challenge itself was simple: do something interesting with our performance test data. Researchers were thrilled to have industry data to evaluate and demonstrate their ideas. We were thrilled to have researchers working on our problems. In the end, it led to four strong papers which have impacted how we test performance at MongoDB.

We continue to work on sharing our data and learnings. We have an ongoing collaboration within the SPEC Research Group to create better datasets and algorithms for detecting performance regressions. The group is combining our data with other industry datasets and curating the data. The results will enable researchers to understand the performance and accuracy of current algorithms, test new algorithms, and clearly show any improvements. All using real industry data from us and other companies.

In each of these interactions, the community wins and we win. By sharing our data we enable better research, we get to take advantage of that research, and the research is better aligned with our needs.

Investing in the future

Of the two interns mentioned in this post, one is now a full time employee of MongoDB, and the other is pursuing a Ph.D. in computer science at Columbia. One is directly improving our software, and the other one is improving the theory and tools we use to build our software. We are very proud of both of them.

The MongoDB database is faster today because of their work on our performance testing infrastructure. Thanks to that infrastructure we better understand why the database performs the way it does, why that performance changes, and when that performance changes.

We continue to invest in and improve this critical piece of our infrastructure. We have teams dedicated to extending and improving it. We lean into our academic interactions to improve the state of the art for everyone. And we invest in the people who work on these systems (interns included).

We hope you consider using these techniques yourself and letting us and the community know how it goes for you. If you are an academic, please improve the theoretical underpinnings of this entire space – we are happy to talk to you about it. And if the problems and software described in this post sounded interesting to you, we are hiring! Come join us and help us solve these problems.

Note: This was originally published on the MongoDB Engineering Blog on April 30, 2019 here by Henrik Ingo and myself. Please read it there assuming the link works. I have copied it here to ensure the content does not disappear. The links in the article are the original links.

In an effort to improve repeatability, the MongoDB Performance team set out to reduce noise on several performance test suites run on EC2 instances. At the beginning of the project, it was unclear whether our goal of running repeatable performance tests in a public cloud was achievable. Instead of debating the issue based on assumptions and beliefs, we decided to measure noise itself and see if we could make configuration changes to minimize it.

After thinking about our assumptions and the experiment setup, we began by recording data about our current setup.

Investigate the status quo

Our first experiment created a lot of data that we sifted through with many graphs. We started with graphs of simple statistics from repeated experiments: the minimum, median, and maximum result for each of our existing tests. The graphs allowed us to concisely see the range of variation per test, and which tests were noisy. Here is a representative graph for batched insert tests:

In this graph you have two tests, each run three times at different thread levels (this is the integer at the end of the test name). The whiskers around the median, denote the minimum and maximum results (from the 25 point sample).

Looking at this graph we observe that thread levels for these two tests aren't optimally configured. When running these two tests with 16 and 32 parallel threads, the threads have already saturated MongoDB, and the additional concurrency merely adds noise to the results. We noticed other configuration problems in other tests. We didn't touch the test configurations during this project, but later, after we had found a good EC2 configuration, we did revisit this issue and reviewed all test configurations to further minimize noise. Lesson learned: When you don't follow the disciplined approach of "Measure everything. Assume nothing." from the beginning, probably more than one thing has gone wrong.

EC2 instances are neither good nor bad

We looked at the first results in a number of different ways. One way showed us the results from all 25 trials in one view:

As we examined the results, one very surprising conclusion immediately stood out from the above graph: There are neither good nor bad EC2 instances.

When we originally built the system, someone had read somewhere on the internet that on EC2 you can get good and bad instances, noisy neighbours, and so on. There are even tools and scripts you can use to deploy a couple instances, run some smoke tests, and if performance results are poor, you shut them down and try again. Our system was in fact doing exactly that, and on a bad day would shut down and restart instances for 30 minutes before eventually giving up. (For a cluster with 15 expensive nodes, that can be an expensive 30 minutes!)

Until now, this assumption had never been tested. If the assumption had been true, then on the above graph you would expect to see points 1-5 have roughly the same value, followed by a jump to point 6, after which points 6-10 again would have roughly the same value, and so on. However, this is not what we observe. There's a lot of variation in the test results, but ANOVA tests confirm that this variation does not correlate with the clusters the tests are run on. From this data, it appears that there is no evidence to support that there are good and bad EC2 instances.

Note that it is completely possible that this result is specific to our system. For example, we use (and pay extra for) dedicated instances to reduce sources of noise in our benchmarks. It's quite possible that the issue with noisy neighbours is real, and doesn't happen to us because we don't have neighbours. The point is: measure your own systems; don't blindly believe stuff you read on the internet.

Conclusion

By measuring our system and analyzing the data in different ways we were able to disprove one of the assumptions we had made when building our system, namely that there are good and bad EC2 instances. As it turns out the variance in performance between different test runs does not correlate with the clusters the tests are run on.

This is only one of three bigger experiments we performed in our quest to reduce variability in performance tests on EC2 instances. You can read more about the top level setup and results as well as how we found out that EBS instances are the stable option and CPU options are best disabled.

 Note: This was originally published on the MongoDB Engineering Blog on April 30, 2019 here by Henrik Ingo and myself. Please read it there assuming the link works. I have copied it here to ensure the content does not disappear. The links in the article are the original links.

In an effort to improve repeatability, the MongoDB Performance team set out to reduce noise on several performance test suites run on EC2 instances. At the beginning of the project, it was unclear whether our goal of running repeatable performance tests in a public cloud was achievable. Instead of debating the issue based on assumptions and beliefs, we decided to measure noise itself and see if we could make configuration changes to minimize it.

After thinking about our assumptions and the experiment setup, we began by recording data about our current setup and found no evidence of particularly good or bad EC2 instances. However, we found that the results of repeated tests had a high variance. Given our test data and our knowledge of the production system, we had observed that many of the noisiest tests did the most IO (being either sensitive to IO latency, or bandwidth). After performing the first baseline tests, we therefore decided to focus on IO performance through testing both AWS instance types and IO configuration on those instances.

Investigate IO

As we are explicitly focusing on IO in this step, we added an IO specific test (fio) to our system. This allows us to isolate the impact of IO noise to our existing benchmarks. The IO specific tests focus on:

• Operation latency

• Streaming bandwidth

• Random IOPs (IO per second)

We look first at the IO specific results, and then our general MongoDB benchmarks. In the below graph, we are graphing the "noise" metric as a percentage computed from (max-min)/median and lower is better. c3.8xlarge with ephemeral storage is our baseline configuration which we were using in our production environment.

The IO tests show some very interesting results.

1. The c3.8xlarge with EBS PIOPS shows less noise than the c3.8xlarge with its ephemeral disks. This was quite unexpected. In fact the c3.8xlarge with ephemeral storage (our existing configuration) is just about the worst choice.

2. The i2.8xlarge looks best all around with low noise on throughput and latency.

3. The c4.8xlarge shows higher latency noise than the c3.8xlarge. We would have expected any difference to favor the c4.8xlarge instances, as they are EBS optimized.

After these promising results, we examined the results of our MongoDB benchmarks next. At the time that we did this work, MongoDB had two storage engines (wiredTiger and MMAPv1), with MMAPv1 being the default, but now deprecated, option. There were differences in the results between the two storage engines, but they shared a common trend.

There were no configurations that were best across the board. That said, there was a configuration with below 10% noise for all but 1 test: c3.8xlarge with EBS PIOPS. Interestingly, while i2 was the best for our focused IO tests, it was not for our actual tests.

Valuable lessons learned:

• As far as repeatable results are concerned, the "local" SSDs we had been using performed worse compared to any other alternative we could have possibly chosen!

• Contrary to popular belief, when using Provisioned IOPS with EBS, the performance is both good in absolute terms, and very very stable! This is true for our IO tests and our general tests. The latency of disk requests does have more variability than the SSD alternatives, but the IOPS performance was super stable. For most of our tests, the latter is the important characteristic.

• The i2 instance family has a much higher performance SSD, and in fio tests showed almost zero variability. It also happens to be a very expensive instance type. However, while this instance type was indeed a great choice in theory, it turns out that our MongoDB test results were quite noisy. Upon further investigation, we learned that the noisy results were due to unstable performance of MongoDB itself. As i2.8xlarge has more RAM than c3.8xlarge, MongoDB on i2.8xlarge is able to hold much more dirty data in RAM. Flushing that much dirty data to disk was causing issues.

Switching from ephemeral to EBS disks in production

Based on the above results, we changed our production configuration to run on EBS disks instead of ephemeral SSD. (We were already running on c3.8xlarge instance types, which turned out to have the lowest noise in the above comparison, so decided to keep using those.)

After running with the changes for a couple of weeks, you could clearly see how the day-to-day variation of test results decreased dramatically. This instantly made the entire System Performance project more useful to the development team and MongoDB as a whole.

Conclusion

Focusing on IO performance proved useful. As it turns out using Ephemeral (SSD) disks was just about the worst choice for our performance test. Instead, using Provisioned IOPS showed the most stable rate results. While i2 instances were the best in our non-MongoDB benchmark tests, they proved less than ideal in practice. This highlights quite clearly that you need to measure your actual system and assume nothing to get the best results.

This is the second of three bigger experiments we performed in our quest to reduce variability in performance tests on EC2 instances. You can read more about the top level setup and results as well as how we found out that EC2 instances are neither good nor bad and that CPU options are best disabled.

Putting down my book, I stood up from the bench and walked over to my 10-year-old son by the water. His fishing line was tangled. Again. This was the third time he had needed me to fix something. He hadn't caught a single fish and was getting very upset. He loved the idea of fishing and had visions of catching impressive fish like those caught in the videos he watched. Sadly, he was not good at fishing and neither was I.
I recently wrote about how to read academic papers in volume. When I researched how to read academic papers, I also learned how to read more effectively in general. Those skills have helped me better achieve my goals, such as helping my son catch fish.

This post covers how I use books to answer my bigger questions, and is largely based on ideas from the book How to Read a Book by Mortimer Adler.
My son holding a small fish he caught.

---

Learning Through Books
To help my son catch fish, I had to learn both facts and skills. I needed to learn about a subject (fishing) and a skill (how to catch fish). It was one of my larger questions, requiring time and effort. I’ve investigated other larger questions focused on work problems such as working with a challenging colleague, personal goals such as remembering things better, and personal challenges such as supporting my family with a medical issue. The best way to learn facts and skills quickly is to learn from those who have already learned the topic. Books are a treasure trove of other people’s learnings.

I follow three main steps on a learning project:

1. Find many potential resources

2. Filter those results to a manageable number of the best resouces

3. Extract what I need from those resources. 

In my life I am constantly asking questions and searching for answers. Only a few of those questions merit the time and effort required for this process. For those that do, I adjust my effort to my question and stop when I have what I need. Find Many Potential ResourcesI start with a wide search using the internet and my local library. I also ask friends, colleagues, and social media for suggestions. When I do this right, I turn up a lot of resources. For each resource, I’ll quickly check what I can about it, such as summaries and reviews, to see whether I think this book might be on topic. Key word searches often turn up things that are clearly unrelated to my question. For example, Incredible--and True!--Fishing Stories may or may not be entertaining, but it’s not going to help my son catch fish. I can immediately reject that book. 
My search extends beyond books to include articles, blogs, podcasts, and videos. Books tend to be higher quality and I have a personal bias to the written word, but I want the best available resources regardless of its medium. 

Then I do a second search based on the results from the first search. Did I find other search terms? What other resources are related to these? Do any of these books reference other resources?
Filter For The Best Resources

Now I have a large list of books and other resources. I request as many of the books as I can from my local library. I download or bookmark the online resources. Then I scan all of these together quickly so that I can think about the ideas from multiple books at the same time. Speed is essential for that mixing of ideas. For each book I want to know: 

1. What is this book about?
2. How does it address my question? What techniques does it propose? 
3. What words does it use for my question? What do they mean? 
4. Do I believe the answers this book proposes? Alternatively, is my bullshit detector going off? 

I quickly inspect each book, scanning the table of contents, reading the publisher's blurb, skimming the end of the last chapter, and checking for summaries at the end of key chapters. I take notes on anything that addresses my prompts, anything that catches my attention, and any new questions that arise as I read. By the time I’m done, the books are usually covered in sticky notes.
I look for common words and ideas, and I usually see patterns across the books. Sometimes several of the books have the same "revolutionary" or "game-changing" ideas. This used to annoy me, but I’ve since learned this is a great result. This common “revolutionary” idea is likely what I'm looking for. Other times I learn from overlapping words and ideas in the books, even if the books don't agree on everything. When they disagree I can see which ideas stand up to criticism and which fall over.
At this point, I have two follow-up questions:

1. Does one book capture a common idea better than the others? 
2. Are there interesting differences between the books? 

If one book does capture the key idea better than the others, I focus on that one. When there are interesting differences, I learn more from reading the books together than by reading them separately, just as I learned more from scanning them together. Extract Answers I now have at least one book to read deeply. First, I revisit my existing notes to answer questions: What do I expect to get out of each book? What is common between them? What words do they use for important items? Where do they differ? 
From these answers I form focused questions for each book: Book A says X, what does Book B say about that? Book C and D seem to differ on point C. Is one of them right? Is there something more complicated going on? Book E claims point D. What’s the evidence for it?
Next, I read each book, trying to answer my questions, still taking copious notes. I listen for the conversations among the books. The differences among books may be true disagreements requiring real thought and evaluation on my part. Or they may merely be different framings of the same idea. The common themes become clearer, and the differences enlighten me. In both cases I end up with a deeper understanding of the underlying issue. 
When I’ve finished reading everything, I review and organize my notes. I often write a summary for myself. I hopefully have answers to my questions. If I don’t, I restart the process by looking for more sources focused on whatever is missing. If I’m really excited about the result, I may write a blog post or share my learnings with others. Catching FishThat day that I took my son fishing, I asked him two questions when we were back home: Would you like my help figuring out how to catch fish? Yes, he did. I told him if he wanted my help, we had to use the goal of "catch fish — any kind of fish." We couldn't only aim for the big fish he'd seen in videos. Would that still be worthwhile? He thought briefly: Yes, it would be. 
I then got every book about fishing I could from my local library. I scanned each of them and noticed trends. It was clear that we should start by targetting small fish such as sunfish, with small hooks and bobbers, and we should fish in smaller bodies of water. I bought a couple of the more promising books, as well as the appropriate fishing gear, including the small hooks and bobbers. We then went to a local pond and he caught fish! That summer he caught a lot of fish. It was wonderful for him, and wonderful for me to help my son succeed at something that he loved so much. 
I hope this post helps you achieve similar success with your large questions. I would love to hear about it when it does.
Special thank you to Heather Beasley Doyle for her feedback on this post. Heather is a gifted writer and you should read some of her writing here, here, or here. 

I sat at my desk with a pile of academic papers next to me waiting to be read. I worked at a research lab and reading papers was part of my job. My colleagues amazed me by reading scores of academic papers, calling up the right paper at the right time. I couldn’t compete. I couldn’t even keep up. I had received my Ph.D. a few years prior and felt I should have been able to. I wondered if I just didn’t belong at a research lab. 
Then I wondered if I was missing something. I searched the internet and quickly found I was. I found Steve McConnel’s How to Read a Technical Article and Greg Phillips How to read technical papers—in quantity! Steve McConnel’s article was itself based upon (and acknowledged) How to Read a Book, the classic by Mortimer Adler. I learned that efficient reading is a learnable skill, just like any other. Here is what I wish I had been taught when I started grad school.
First, a caveat. If you are in school, you need to learn to fully read individual technical papers, in detail and considering every aspect, before you can read them in volume.

Image by Kindel Media

---

I am not an academic. While I have a Ph.D. and used to work in a research lab, I am not paid to do research or to publish papers. Yet I still find it valuable to read papers such as conference proceedings and journals, as well as blog posts and white papers. Reading helps me to stay up to date in my field and to do my job better. Adding other people’s ideas to my experiences helps me to come up with better solutions to my technical problems.

I am paid to solve problems, not to read. Therefore, I must make the most of my reading time. I cannot read everything. Instead, I read papers through the lens of my problems.

 
Selecting Papers to Read

I read papers in three different situations, all in service of doing my job better:

1. I’m trying to solve a specific technical problem.
2. I’m keeping up on a general professional interest.
3. A colleague sends me something interesting.

When I try to solve a technical problem, it often involves a lot of very focused reading. I cast a wide net for papers, using keyword searches in Google Scholar, tracing references from other papers, and asking colleagues in person and beyond. I generate a large pile of papers that need to be thinned into a manageable collection.
For example, I collected and summarized papers on flaky tests for my colleagues. We run a very large number of correctness tests to ensure our software works properly and without bugs. Flaky tests—tests that sproadically fail—are a distraction. They slow development and they increase the risk of bugs slipping into our code base. Based on my reading, we reduced the pain of flaky tests.
Keeping up on general professional interests is a more continual background activity. I am an expert on performance testing of software. I follow academic literature on the subject, keep an eye out for relevant industrial papers, and look out for other resources, all so that I can remain an expert in the field. Important conferences are a rich source of academic papers in one place.

Finally, I enjoy new and interesting ideas. Colleagues forward things to me since they know I appreciate when they do so. These papers may be performance related, performance adjacent, or touch on other interests of mine such as peer groups, technical education, and DEI, among others.
I suspect that I could fill all my time reading the papers I accumulate and still not read all of them. I must be efficient if I want to get the most out of them. How I Read PapersWhen I have a set of papers to read I follow a specific process. I take the first paper and set a timer for 5 minutes. Within my five-minute limit, I proceed in the following order: I read the abstract, the conclusion, the introduction, and scan the figures in the paper. Doing so gives me the best opportunity to get a sense for each paper. I actively look for and record the questions that come to my mind as I scan the papers: questions to dig deeper on, questions about the parts that confuse me. “How do they do X?” “How do they prove Y?” “Could we use Z?”

At the end of the 5 minutes I review my notes. If I have no questions, I am done with this paper. If I have questions, I put the paper aside for a deeper read specifically to answer those questions. Then I pick up the next paper to scan. 
With this process I can scan 12 papers in an hour. That might not sound like much, but without my process in place, I could easily spend an hour or more fully understanding one paper. A Deeper ReadAfter triaging the collection of papers, I have a smaller collection of papers, each with a list of questions to answer. I’ll pick up a promising paper. I review my notes and questions for it, then read the paper solely to answer those questions. The questions provide a focus, allowing me to read faster. If the introduction is not relevant to my question, I skip it. If I only have a question about the implementation, I go straight to the implementation section.

This all felt unnatural when I started doing it. I was not reading the paper as the author intended. That is okay. I (and you) do not owe it to the author to read their paper, nor to read it as it was “meant to be read.” Good readers and writers know this. We owe it to ourselves to get the most out of the papers we read. We only owe the authors our gratitude and an acknowledgement when we use their ideas. Sometimes I send a thank-you note to acknowledge ideas I find particularly helpful. Remembering Papers
My wonder at my colleagues’ reading ability was not just because they read so much, but also because they remembered details and made use of those details. I must take notes if I want to do that.

I use two tools to take, organize, and remember my notes from technical papers: Zotero and Anki.

I use Zotero to organize all my academic papers. New papers go directly into a triage folder in Zotero. Zotero supports highlighting PDFs and extracting highlighted text. It also automatically collects the bibliographical data for each paper.

In Zotero, I use folders to organize papers by subject. When I want to find details on a given topic, I look through the appropriate folder and review my notes on the papers. Sometimes I rescan papers. Each part of this process is so much faster and more efficient than when I photocopied papers and stored them in a filing cabinet in graduate school.

While I can find things quickly in Zotero, simply using the tool does not put the information at the tip of my tongue. For ideas and details I want to talk about or otherwise have available in my mind, I create digital flashcards in Anki. See The Nerdiest Thing I Do for more on my Anki usage. Reading to Learn to ReadI am able to read much more efficiently since that day sitting at my desk overwhelmed by papers. There are still people who can read and retain much more than I can, but my reading is no longer a weakness. It’s a strength.

Special thank you to Heather Beasley Doyle for her feedback on this post. Heather is a gifted writer and you should read some of her writing here, here, or here. 

Sitting at my computer recently, I looked back across my notes for 2023. I was following my annual review process, reviewing the goals I had set for myself last January and the notes accumulated since then. I had a good 2023, accomplishing many of the things I had set out for myself. Those accomplishments spanned changes to my job (new role), my writing, and how I take and organize my notes. 
Several resources helped me achieve those goals in 2023. These are the books, posts, videos, and tools that helped me change my world for the better. Maybe some of them will be useful to you.

The Staff Engineer's Path by Tanya Reilly was one of the books particularly helpful to me in 2023. 

CareerI have continually tried to define my own job since I stopped being a manager and transitioned into a staff engineer role. I have tried to keep the impact and influence I enjoyed as a manager while removing the parts of that role that drained me. Over this past year, as I worked to change my role and my reporting structure, I searched for resources to help me in this goal.
The role of staff engineer (a senior individual contributor) in software engineering is not well established. There are only two books on the topic, and both were useful to me. The first, Staff Engineer by Will Larson, discusses the idea that there is no one ideal staff engineer. Instead, staff engineers may excel in very different ways. For instance, one staff engineer may have broad impact across several teams, while another may go very deep in one technical domain. He introduces four staff engineer archetypes describing four of the more common paths to staff engineer excellence. The book provided me with the language to describe what I was doing and what I wanted to do. 
The second book is The Staff Engineer’s Path by Tanya Reilly. Tanya is an active member of the staff engineering community, organizing events and answering questions from people like me. She wrote her book from the staff engineer’s perspective, offering maps for navigating and thinking about your role as a staff engineer. I found two of Tanya’s insights particularly relevant: reporting to the correct level and evaluating your job. 
As a staff engineer, you face the potential challenge of reporting to a manager at the wrong level (too high or too low) in your organization. Tanya explores the issue and how to handle it. I started 2023 reporting to a first-line manager. I ended the year reporting to a manager of managers, with better access to the information, perspective, and discussions I need to do my job well. 
The Staff Engineer’s Path also suggests a tool for evaluating your current job, extending this blog post. The idea is to grade yourself on a scale from 0-5 on five criteria. I made a spreadsheet to track my progress, so that I could make informed decisions rather than decisions based on short-term swings in mood and apparent job status. It helped me put bad days in perspective, and to better spot the improvements when they came. 
It can be challenging to clearly explain what you do as a staff engineer, since staff engineering roles differ so much. I used the ideas in two blog posts by Cian Synnott to help me explain what I do. I started writing weekly snippets at work, so anyone interested can see what I am working on and thinking about. More importantly, I wrote my own job description, got buy-in from my management, and shared it internally. The process of writing my job description was a key step in changing my role at work. 
Finally, I greatly appreciated the encouragement from Nine Lies About Work to lean into what I’m good at, so I can be great at it, as opposed to focusing on improving my non-strengths and being uniformly okay. This message aligned nicely with the message from Staff Engineer that there are many ways to be a great staff engineer. This book had a large enough impact on me that I wrote a dedicated blog post about it last year. Information OrganizationFor over a decade I used Evernote to collect, store, and organize my notes. In 2023 I stopped using Evernote and switched to a new tool called Obsidian. Two factors prompted this change: the development of more advanced note-taking techniques and Evernote updates disrupting my note taking. This video by Jorge Arrango (since expanded into the book Duly Noted) opened my eyes to the new possibilities of developing relationships between notes and using those relationships to create something greater than the parts. I have long used note-taking to remember things. Actively creating something new from my notes is an exciting step forward for me. Based on the video, I tried out and subsequently completely migrated to Obsidian. 
Obsidian is a powerful and extensible tool. Even with my technical background and note-taking experience, I was intimidated by all the features, options, and possibilities. There are several helpful guides. I found the videos and resources from Nicole van der Hoeven particularly useful. I have copied several techniques from her videos and employed the sample Obsidian vault available to her Patreon members. The vault made it easier to use her techniques. 
Jorge’s video also introduced me to Readwise, which helps organize highlights from digital media such as Kindle books. It pulls all my highlights into one system and automatically imports them into my Obsidian vault. I am an avid reader. Now I can find and review key highlights from books effortlessly. I am getting much more out of my reading with Readwise. Writing
I started 2023 intent on improving my writing. Writing my blog has been incredibly satisfying and I wanted to take it to the next level. Several people helped me in this effort (thank you Heather, Eoin, Cian, Alex). I also found two books particularly helpful to me, Bird by Bird and Storyworthy. 
Bird by Bird by Anne Lamont is a classic guide on writing. While nominally created for writers of fiction, I found that its lessons apply easily to my writing: the importance of vulnerability, honesty, showing up every day, the “shitty first draft,” and finding supportive people to review your writing. Bird by Bird has changed how I think about writing. It is also (of course) delightful to read. This quote on writing every day has particularly stayed with me. 

"Do it every day for a while,” my father kept saying. “Do it as you would do scales on the piano. Do it by prearrangement with yourself. Do it as a debt of honor. And make a commitment to finishing things." 

The second book, StoryWorthy by Matthew Dicks, isn't even about writing. It is about how to tell a story that resonates with your listener. I hope that my writing will resonate with you, my reader. I illustrate many of my blog posts with stories from my past to reinforce their point and to make the writing more engaging. This book is full of ways to mine your experiences for interesting stories—techniques that I have included in my writing. And, similar to Bird By Bird, it has changed the way I think about my writing. ConclusionJust as I hope my writing will resonate with you, my reader, I hope that some of these resources will resonate with you as well. Whether you are looking to advance your career as an individual contributor in tech, improve your ability to capture, organize, and make use of information, or have more impact through your writing, there is something here for you. These resources have helped me grow and learn in 2023. I hope that they may help you in 2024 and beyond. When they do, I would love to hear about your resulting victories.
Special thank you to Heather Beasley Doyle for her feedback on this post. Heather is a gifted writer and you should read some of her writing here, here, or here. 

I stepped up to the end of a growing line of people at work. We were having an end-of-summer event and the line was for ice cream! To make the most of my wait, I introduced myself to the person in front of me. To my horror, he knew me and we had talked before. With prompting, I vaguely remembered our conversation. I wanted to crawl into a small cave and hide. 
I have always been impressed with people with great memories: The people who remember others’ names, along with supporting details, after meeting them once. Those people are so dang thoughtful. To my continual disappointment, I cannot do that. I just can’t. Since I cannot remember those things naturally, I try to achieve the same effect through brute force, using flash cards.
Screen shot of my Anki decks before studying one morning. 

Over the years I have built a support system to fill in the details I cannot naturally remember. My system has evolved over the years and has led me to be an avid notetaker and information organizer. When I vaguely remember something, I can find the details in my notes. My notes are external storage for my brain, similar to a hard drive in a computer, or records in a filing cabinet.
FasterUnfortunately, my external storage is slow. It's not fast enough to help me in conversation (“Hi Paul! Is Sue still playing soccer?”). It's not fast enough for me to check a calculation or make an estimate. 
Thankfully, shoving information into your brain can reduce recall time. Shoving information into your brain is really memorization. While memorization requires hard work and consistency, it does not require any special innate ability. The word “memorization” may trigger painful flashbacks to memorizing arbitrary things in school (why did I have to memorize all the English prepositions?). However, some of that memorization was useful — you cannot do arithmetic without memorizing addition and multiplication tables. 
Humans have improved at memorization over thousands of years of practice and learning. One powerful memorization technique is flashcards. However, physical flashcards can become overwhelming — many easy cards can keep you from reviewing the harder cards, which need more attention. 
Digital flashcards and the concept of spaced repetition have solved this problem. Spaced repetition adjusts when you see each card: Cards you know well show up infrequently, while cards needing reinforcement show up frequently. This means there are fewer cards overall to review, and you are reviewing the most important cards.
Using FlashcardsEvery morning I perform three rituals: I meditate, I play Wordle, and I study my flashcards — every morning. I currently have 8,482 active flash cards. I use flashcards to study everything I would like to know, or anything I don’t know but feel that I should. Examples include people, things I’m learning, and details about my work. 
I started using flashcards when my daughter was in first grade. I had trouble keeping her classmates straight, so memorizing her classmastes’ names and faces seemed like a good trial for the technique. With the aid of her class picture I did so and it worked. I remembered her classmates through several years of schooling. We would have dinner conversations and I could ask follow-up questions about her stories from the day. And based on this scaffolding of knowledge, I could understand the new information and remember it. “Wait, you worked with Sally today on that? I thought you and Sally fought over that book?” “Dad, the library had a second copy. Now we’re both three books into the series. It’s so cool.”
Today when I meet someone I would like to remember, l sheepishly explain my challenges remembering names and faces, and ask if I can take their picture. Later, I will add a flash card with their name on one side and their picture on the other. I will also add a separate card for each detail I would like to remember about them so I can appear thoughtful the next time I talk to them.
At work, colleagues frequently come to me with questions about performance testing and performance engineering. Answering questions like these is part of my job. Often, but not always, I can answer their questions immediately. When I have to consult my notes but wish that I hadn’t, I add flashcards for the relevant information. 
When I am organizing a project, I feel there are certain facts I should always have available at the tip of my tongue. I add flashcards for them. And when I am learning something new, I add flashcards for my learnings. For example, when I finish reading a book, I review my notes for points to memorize.

This morning I studied 61 flashcards in 5 minutes. The combination of spaced repetition and my consistency means that I only have to review around 40-50 cards most mornings, requiring 5-10 minutes. When I miss a day(s), there are more cards to review and I start forgetting things. That feedback loop is bad. I try hard not to miss days.
I have missed studying four days so far this year. One of those days was while visiting family, and the remaining three were during camping trips. I do my study as Anne Lamott does her writing, as a debt of honor. It is a commitment to show up each and every day. And because I show up each and every day, I know so much more than I did before. Using My KnowledgeMy daily flashcard review lets me do several things that I otherwise could not. I can address people by name when I meet them. I can give a detailed answer when someone asks about our performance testing system. Even when I can’t answer a question immediately, I can think of the answer much faster than before. I no longer feel embarrassed when I don’t know something, because I know I know so many things.

I can estimate better and faster, because I have so much reference information (e.g., the circumference of the earth is ~24,900 miles). As an engineer, being able to make quick and accurate estimates is a requirement. 
None of this works perfectly, but it works much, much better than it did before I started using my flashcards. Being SmarterThere is a still larger benefit beyond recall: I can think better. Thinking involves developing connections between ideas. When we synthesize new ideas, we build from existing ones. When we deduce, we deduce from rules and facts. Having more ideas in my head allows me to draw more connections and to synthesize more ideas. Having more rules and facts in my head allows me to deduce more. Starting with more knowledge makes it easier to learn and integrate new knowledge. 
I think of creativity as the exploration of the adjacent possible. Stuart Kaufmann developed the concept of the adjacent possible to describe the development of biological systems, while Stephen Johnson built on that idea to describe all forms of innovation in his book Where Good Ideas Come From. The possible is everything that exists (or we know) today. The adjacent possible is everything that does not exist today, but can be built from those things that do. My possible is much larger due to my memorization, making my adjacent possible larger still. In the end, I am more creative because of my memorization. ConclusionWhile waiting for my ice cream I had a nice conversation with my colleague. I focused so that I could remember his name and some details. Later, I wrote down these details and I created flashcards, especially one with his name and face on it. 
These days, when I see him in the hallway, his name pops into my head instantly. I make sure to say hello.

Special thank you to Heather Beasley Doyle for her feedback on this post. Heather is a gifted writer and you should read some of her writing here, here, or here. 

One of my favorite things to do is to formally and publicly address newly minted Ph.D.s with their title. “Congratulations Dr. Smith!” I’ve done it in person, on LinkedIn, and Twitter. Getting a Ph.D. is hard. It requires time, luck, and personal sacrifice. It makes me happy to acknowledge that person and their accomplishment.

I get to work with many extraordinary college students through my company's internship program. When I talk to them, some are deciding between going to graduate school or getting a job after graduation, but aren’t sure what to do. This post is my advice to those students.

Graduation day many years ago, holding my daughter after receiving my Ph.D., with my mother to the left. 

Options After Undergraduate GraduationWhen a top student finishes an undergraduate degree in computer science, engineering, or related fields, they have their pick of job opportunities as well as the option of going to grad school to work towards a Ph.D. on a paid assistantship or fellowship. They can either start growing their career and getting paid now, or they can keep studying, which will open up alternatives such as becoming a professor. Having a Ph.D. is a requirement for some jobs and may speed up access to others. However, there are very few technology jobs with a strict requirement for a Ph.D. and you can continue to learn and grow while in industry.

I went to grad school and got my Ph.D. because I never wanted to be barred from an interesting job for lack of the Ph.D. credential. As we shall see below, I would discourage a student from pursuing a PhD for this reason.

 

Two StudentsI have standard advice for students considering grad school, but it needs to be tailored to each individual. The main adjustment accounts for the background of the student. Let me present two fictional students to demonstrate the range of adjustment. I’ll call them Adam and Sheila.

Adam is crushing it at an elite university. He grew up in a well-to-do school district and took AP computer science in high school. He has spent each of his college summers at prestigious programs and internships. Adam has an uncle who is a doctor and a cousin who recently finished a Ph.D. His family is used to turning out degrees.

Sheila is similarly crushing it in college, but she got here by a very different road. Sheila was raised by a single mother and has three younger siblings. She went to a underfunded school district and took a lot of responsibility for her younger siblings. Sheila got a part-time job while in high school to help pay the bills, and continues to work through college.

Although opportunities weren’t handed to her as readily as they were to Adam, she discovered she loved technical work, and will be the first college graduate in her family. However, Sheila’s resume isn’t as impressive to recruiters as Adam's because she didn’t have the time to hunt for the opportunities or to take advantage of them. She also did not have the connections to tell her where to start looking for those opportunities.

Should They Go? 
For Adam, deciding to go to grad school is an encouraged option. His family will be thrilled to have another degree and can easily support him through any challenges. For Sheila, grad school is unknown and not an encouraged option. She could help her family and herself today by getting a job and she doesn’t know anyone who has gone to grad school. When talking to each of them, I would focus on the less explored side of the issue. That is likely to be why not to go for Adam, and why to go for Sheila. Advice for AdamI would ask Adam, “Why do you want to go to grad school?” Adam may feel that it’s expected of him, he may want the Ph.D. credential, or he may generally want to put off starting his professional career with all of its implications and responsibilities. If those are Adam’s reasons, I would push him to re-evaluate his plans. Alternatively, Adam might say, “I love digging into the unknown. I am fascinated by area X, and would love to spend the next several years trying to solve a problem in X”. If this is Adam’s answer, I would encourage him to go to grad school.

In either case, I would explain to Adam that grad school is hard work. It takes a long time. A person should only go to grad school and pursue a Ph.D. if they want to work on a problem and do academic research for several years independent of getting a Ph.D. as a result. The students who do so have positive experiences and graduate on time, while students who go to grad school to get a degree end up suffering and taking longer to graduate if they do graduate. Advice for SheilaMy advice for Sheila is the same as it is for Adam, but it may sound different. I would ask Sheila why she wants to go to grad school. Sheila may give any of the same answers as Adam, and I would give the same explanations to Sheila that I would give to Adam. However, considering the obstacles she’s had in her path, I would want to understand if Sheila has something personal and strong pulling her toward grad school. I would want to know what that is, acknowledge it, and honor it. I would tailor my advice to that force.

The challenges in Sheila’s path also mean there are fewer people like Sheila in academia. This is a loss for all of us, as bringing together people from different backgrounds and experiences leads to higher creativity and innovation for everyone. Therefore, I hope that Sheila will choose to go to grad school and ultimately succeed there. I would encourage her a bit more because of this. However, I would still do my best to give unvarnished advice and set Sheila up for future success. 

Advice if Going to Grad SchoolIf Sheila were to go to grad school, she would face more challenges than Adam, mostly from an assortment of institutional biases. They are real and will make her path harder. I want to make sure that Sheila sets herself up for success, and gets all the support she can get (I want Adam to get support also). Therefore, I have some additional advice for Sheila.

I would strongly encourage Sheila to build a strong support system for herself, composed of three parts: peers, mentors, and school. She should seek out peers with whom she identifies who will be going through the same things as her. Similarly, she should seek out mentors with whom she identifies; people who have gone through grad school and can give her great advice. This may include mulitple mentors who match different aspects of her background (as opposed to one person who matches everything). These mentors will be able to give her support and advice from having gone through similar experiences. There is good science on the value of mentors and role models with whom you identify. Sheila may also benefit from other mentors with whom she might not identify, but she should prioritize getting mentors with whom she identifies.

Finally, I would encourage Sheila to apply to schools and programs that would support her, with strong track records of graduating people with similar backgrounds to hers. Within those options, pay attention to potential advisors and their track records as well. Grad school is hard—don’t add extra challenges by going to a school that doesn’t help, or pretends that institutional biases don’t exist. Good LuckIf you are considering going to grad school, I hope you will consider my advice. If you decide to go directly into industry, congratulations, and good luck getting started with your career! If you do want to spend the next several years doing research for research's sake, good luck, and Godspeed in grad school. Whoever you are, set yourself up for success by building a strong support system including mentors and peers.

And, after you have defended your dissertation, let me know so that I can congratulate you, doctor!

---

Special thank yous to Heather Beasley Doyle and Danielle James for their feedback on this post, including making sure that the fictional Sheila was a believable person and not a caricature. Heather is a gifted writer and you should read some of her writing here, here, or here. Danielle is also a gifted writer who is making positive changes in the world. You should read her essay in We Rise To Resist.

I got my current job because I hike. Rather, I wouldn’t have gotten my current job if I didn’t regularly hike. 
Many people think that their success is solely due to their hard work. They planned, put in the effort, and achieved what they set out to do. They deserve credit for what they have done. However, I also think they benefited from a large helping of luck. More importantly, I think we do everyone a disservice when we claim complete control over and credit for our own success.
Me on a recent hiking group outing with the Hudson River behind me.

---

A Story of Success
The short version: I was a smart kid. I did well in school, and got into a good college and a better grad program. I earned a Ph.D., got a great job after graduating, and then an even better job several years later. Along the way, I published papers and patents and grew my network. I worked hard, charted my path, and succeeded. This is the story that teenage David would tell, of completing my destiny to have a great career in computing.
That story is short because it leaves out a lot of details. Let’s start with the easy omissions: I was born to two loving parents who raised me in a well-to-do neighborhood with good schools. I may have been an awkward child, but all the adults (and many of the children) expected and encouraged me to succeed. Unfortunately, many people do not grow up with that support. I was very lucky.A Story of ChanceThe long version includes a number of lucky breaks. I ended up at graduate school at the University of Illinois at Urbana-Champaign (UIUC) in part because my friend and classmate went there a year earlier. Seven years later I was hunting for a job as I finished my doctorate. The company and research lab I ended up with had a strong pipeline from UIUC. If I had gone to another school, I would have been less likely to end up where I did. Further, I was hired right before a hiring freeze. My hiring manager sped up my paperwork to make sure I got in. So, if I had been on the market a few months later, I would not have ended up with the same job. I loved working in that lab and got to know many amazing people, including IEEE and ACM Fellows, National Academy Members, and a Turing award winner.

Soon after joining the company, I joined a weekly hiking group at work. Many fellow hikers became friends. I know them solely because they also like hiking and worked at the same company.
A Second Chance

Moving forward several years, I had built a nice role for myself, in which I got to work on interesting projects and was well rewarded. However, my company was not growing and had gone through periodic layoffs. I didn’t worry about my job, but I did worry about how much I could accomplish in that environment. Then, across a year, several unrelated but important things happened.

Several members of my team had a side project that suddenly became successful. They and their project were moved to a new division. I was excited for my colleagues, but it also meant I needed to find a new team. Around the same time, my youngest child was finishing daycare and starting kindergarten. His daycare was located at the site of my office, and I had been dropping him off and picking him up every day. Finally, one of my hiking buddies had been laid off and started a new job at a startup I had never heard of before.
Because of the reorganization, I had to re-evaluate my job. Because of the health of the company, I was willing to look elsewhere. Because my son was done with daycare, I was no longer tied to my current office. Because of all these things, I decided to look far and wide for my new job.

One of the places I looked into was my friend’s startup. The one I only knew about because of him. The one he worked at because he had been laid off. I told him what I was looking for and described my skillset. It turned out that his company was looking for my exact skillset at that exact moment! I interviewed and was hired.
Let’s see all the small changes that could have happened and kept me from my current job. If my colleagues’ project hadn’t taken off, I’d have no need to look for a new job. If my son was younger, I’d still be tied to the daycare. If the company was healthier, I wouldn’t have looked outside it. If I didn’t hike or my friend wasn’t laid off, I’d be completely unaware of my current company. If all of this had happened a few months earlier or later, the company wouldn’t have needed me.


Hiking group poster from 2010. The man in the yellow shirt is the friend who introduced me to my current company.
Do the Random Parts Matter?
At this point you may object: David, there may have been a lot of chance involved, but things would have worked out well for you anyway – the particulars were random, but your being successful was not by chance.

You may be right. But there are two important points to consider. First, my life would have been very different in ways I couldn’t have planned if I hadn’t ended up where I did. I had an acceptable competing offer in hand – I would have ended up at a very large company, instead of a small one. All my experiences would have been very different and I strongly doubt I would have become as well known in my field as I am. I may have been successful, but it would have been a very different version of success. It would have looked like a very different plan.

Second, I have the privilege of having a lot of opportunities. Yes, I have worked hard to provide myself with opportunities, but as covered above, I have benefited from many factors outside my control. Many people do not have all those opportunities or never get the right one.

Increase Your Luck

Hard work and planning matter: they are necessary for success, but not sufficient. So, by all means, celebrate your victories. You earned them. But also acknowledge your luck and be grateful for it. I am.

Finally, consider what you can do to increase your luck. You can put yourself in good situations where things are likely to go your way. I did that by working hard in school. I did that by applying for many jobs so I could compare multiple offers. I also did it when I chose to do something I love – hiking – which happened to introduce me to many people who would become friends.

A special thank you to Cian for his feedback on an earlier version on this post.

A few years ago, I read a New York Times article that changed my life. That may sound overly dramatic, but the effects of it show up repeatedly in my blog posts, including My Yearly Reflection and Planning and Writing and Labeling in a Journal. It’s why I knew Cookie Club was so important to me and it’s why I knew I should stop being a manager.

While writing one of those posts, I discovered that the article was based on a book. I was more than a little embarrassed to realize that I wasn’t crediting the ultimate source, and worse, that I hadn’t read that source.

I have finally rectified my mistake by reading the book Nine Lies About Work, and I’m so happy that I have. In this post, I share some of my notes from the book. Nine Lies About Work is nominally aimed at managers, but I think it is also useful for anyone who has a manager. Most importantly, I encourage everyone to read chapter eight about love in work.

OverviewAt first glance, this book appears to be a list of disconnected lies, with each chapter covering one lie. A deeper reading reveals that there is much more to this book, with a structure that flows from one chapter to the next. While the first chapter is listed as “LIE #1 People care which company they work for,” it actually sets the foundation for the rest of the book. The chapter introduces the idea that the most important feature of a company is not its culture, goals, or processes, but its teams. The most a CEO can do to build a great company is to build great teams. The book then focuses on what a manager can do to make sure their team is high performing.

Very briefly, high-performing teams are teams that are strong, well-rounded, and composed of unique–spiky, not well rounded–individuals. The manager’s job is not to make each team member uniformly well rounded. Rather, their job is to help bring out the best in each of their people, helping them to become spikier by improving their best abilities. They then need to provide their people with the information needed to make informed decisions (as opposed to top-down plans), the shared company meaning to make sure everyone is pushing in the same direction (as opposed to cascading goals), and the support they deserve.

The book concludes with “LIE #9 Leadership is a thing.” It matches chapter one, serving as a proper conclusion, bringing the book’s many threads back together. Just as the best team members are spiky, the best leaders are spiky individuals who are very good at specific things. How they use their strengths is what makes them great leaders. Working with Strengths
Building up an employee's best abilities means building up their strengths. However, being good at something is not sufficient for it to be a strength. Instead, a strength is “an activity that makes you feel strong.” You look forward to using your strengths, you have a sense of flow while using them, and you are filled with satisfaction afterwards. You may be tired after using your strengths, but you look forward to doing more. Those are the things you will lean into and work to become great at. As employees lean into their strengths, they will become spikier rather than more well rounded.

Further, strengths play a strong role in employee engagement, which is highly predictive of productivity. When studying engagement, the authors found that agreement with the statement “I have the chance to use my strengths every day at work” was the most important factor for engagement. As such, they want to maximize the time employees are using their strengths, in addition to helping them grow their strengths.

To do this, managers need to know how to help their employees grow their strengths, and how to measure their employees’ progress. This touches on the lies “People need feedback” and “People can reliably rate other people.''
As for rating other people, we are all bad at it. Our ratings say more about ourselves than about the person being evaluated. The only things we can reliably rate are our own experiences. Therefore, the authors encourage asking questions solely about personal experience. Surprisingly, appropriately framed questions allow us to evaluate others. For example: “Do you always go to this team member when you need extraordinary results?” or “Do you choose to work with this team member as much as you possibly can?” are both questions about your experiences, not the team member. They are also questions that lead to constructive and actionable feedback for your team members.Love in WorkChapter eight is the chapter that changed my life. I encourage everyone to read it. It is the lie: “Work-life balance matters.'' Work-life balance is based on the idea that work is depleting and (non-work) “life” is restorative. The answer to stress is time off from work, to balance (bad) work with (good) life.

The authors take issue with the dichotomy that work is depleting and life is restorative. Instead, the right work can be invigorating–and sometimes life is draining. They call this “Love in work” and argue that it matters most, not work-life balance. They present evidence that if you can spend at least 20% of your time every day doing what you love, that you will be more engaged, less likely to burn out, and do better work.

“Love in work” differs from “Do what you love.” No one has complete freedom to choose their work. Instead, you have to find the parts of your work that you love, and find ways to do more of that. They encourage taking a piece of paper and splitting it into two columns: “Loved It,” and “Loathed it.” Then, spend a week filling in the table. Whenever you find yourself looking forward to something, enjoying it, and feeling satisfied afterwards–no matter how small–write it down in the “Loved It” column. Similarly, anything you dreaded or found yourself trying to avoid, write down in the “Loathed it” column.

Then figure out how you can do more of the things you loved, and less of the things you loathed. You will be happier and more productive for it. Community-Building Instead of Managing
Performing the exercise described in chapter eight has let me spend more time in love with my work. It has helped me realize how much I love community-building at work, and how draining I found managing others. I love learning and getting better at things–so much so that reading this book made me (momentarily) want to become a manager again.

I hope you will spend more time in love with your work. If you are a manager, I hope you will also help your people spend more time in love with their work. They will be happier and your team will be more productive.

A special thank you to Heather Beasley Doyle for her feedback on this post. Heather is a gifted writer (on a completely different and higher level than myself) and you should read some of her writing here, here, or here.

A few summers ago I got to work with a wonderful intern. It was a pleasure to work with him, as it always is with our interns. He was a rising senior, and at the end of the summer, we extended him an offer to return as a full-time employee after he graduated the following year. He was excited to receive the offer, but then, over the next couple of months, he tortured himself and everyone else over this offer. The problem was that he had too many good options, and he couldn’t choose among them.
This post contains my advice to that intern and to many interns since. However, I suspect (and hope) that it will be relevant to many people in addition to interns. Specifically, this post focuses on the importance of making good decisions, rather than trying to make perfect decisions.

Hypothetical comparison of three job offers

Very Good Decisions vs Perfect DecisionsMy intern didn’t get his internship merely by being good, but rather, by being great. He and our other interns are used to being the best. Good choices and hard work have gotten them where they are. That plus a large helping of good fortune (including being in a field full of opportunity). The combination of their efforts and fortune has provided our interns with many strong options. Deciding on a first full-time job is a big decision and they want to make the perfect decision. However, trying to make the perfect decision is the problem.

For any job offer, it is possible to know many things about the company: its growth paths and opportunities, the financial implications, and how other recent grads have done. But it is impossible to know what will happen to the company, to the team, and to dozens of other important things.

Because you cannot know these things, you cannot make a perfect decision. However, you can estimate likelihoods for these things. How likely is it that the company will do well? How likely is it that you will be exposed to something exciting? How likely is it that you will have a supportive team and ‘good for you’ growth opportunities? How likely is it that you will get that project that propels you forward, or you will develop that relationship that opens up future opportunities?

You cannot make a perfect decision or “the best” decision, but based on that information, you can make a very good decision. A decision that gives you a very good chance for success. So, How Should I Make A Decision?I suspect most of our interns already know how to make a good decision, and there are many guides on decision making. Consider what is important to you and how each of your options stack up against your criteria. So take out a piece of paper, make a column for each of your options, a row for each criterion, and write down what you think. Or use whatever decision-making framework you prefer. And if you don’t know what to fill in for a particular entry, do a little more homework to learn.

Also consider asking for advice from Future You. Future You is you, but in the future, after having picked each one of your options. Ask Future You about your criteria (how did option A do?), if your criteria actually mattered, what you aren’t considering, and, if they are satisfied with the option they chose.

Since we haven’t yet conquered travel through time and alternate universes, you cannot actually talk to Future You. You can come close, though. Instead of asking Future You, find someone with a similar background and interests to yours who has already decided on one of their job options and started working. Ask them all the questions you would ask Future You, and then listen closely to their answers. And if you can’t find anyone similar to Future You who’s chosen that particular job option, take that as a possible warning.

At this point you have hopefully invested time in learning about and evaluating your options, including talking to others. Maybe you have a decision framework in front of you. By all means, see what the decision framework says. Does it say to pick option B? How do you feel about that? Does it feel right, and do you start to get excited when you think about it? Great! You’ve made your decision.

However, if you review the selected option and find yourself uncomfortable, listen to that reaction. Your mind is trying to tell you something. Re-review your criteria, evaluations, and weights, and see what doesn’t feel right. One of the strengths of any decision framework is that it prompts you to think deeply about the options. That pushes a lot of information into your subconscious, and you should listen to your subconscious. Maybe you have over-weighted an option. Maybe option C is less exciting than what your framework answers indicated. Whatever comes up for you, listen. Just Choose Something
Some of you will have diligently done all of the above, and will still feel stuck. You have created options for yourself. You have evaluated them and reviewed them. You have listened to your subconscious. And yet, you are still stuck between two or more options. Consider the following thought experiment: If most of your options vanished, leaving you with only one, would you be comfortable picking it? If so, congratulations–you have wonderful options! If not, consider taking a step back and looking for more options.

Assuming you indeed have wonderful options, you’ve reached the point when you need to make an arbitrary choice. Would you be more excited to tell your family and friends about one option? Great–decision made. Do you like the color of the logos of one of them? Good, done. Find some (possibly arbitrary) reason to pick one over the others–including flipping a coin–make your decision, and celebrate your good fortune. It’s Okay to Be Scared
This can feel very scary. Choosing one option is saying no to other very good opportunities. As humans, we are loss averse, and saying no means losing an opportunity. This is okay. By making a very good decision, you have put yourself in a wonderful position. Ultimately, you could only take one of these opportunities anyway.

But what if your choice doesn’t work out? You’ve said no to these other opportunities. What will you do? I’m here to tell you it will be okay. Even if this choice doesn’t work out, you will have more opportunities, good opportunities. They may not be the opportunities you have turned down, but that’s okay.

So stop worrying, make a very good (but not perfect) decision, and start living again.

 

A special thank you to Heather Beasley Doyle for her feedback on this post. Heather is a gifted writer (on a completely different and higher level than myself) and you should read some of her writing here, here, or here.

Everyone is different, with their own unique interests and quirks. If you can tailor your interactions for each of your colleagues, you can get more out of them. If you make it easy for your colleagues to understand and adjust to you, they can get more out of you. In both cases, you end up with a more productive and enjoyable environment.

This post is my attempt to make it easier for my colleagues to understand and adjust to me. I wish to provide to anyone who works with me, the tools to get the most out of me. It builds upon (but does not depend upon) my previous post on learnings about myself. Additionally, I hope that this post may provide you with tools to work with anyone who happens to be like me, as well as the inspiration to share your own guide so that I may adjust to you.
My son standing on a wooden bridge on a hiking trail. He built the bridge with my help.


Brief Recap on Self-Reflection
I included an introduction for myself in this post (see “Let Me Introduce Myself”) and share learnings about myself in this post. I’m an introvert, with a strong sense of responsibility who can only focus on a limited number of things at any one time. I live on a schedule, with regular sleep and meals. Achieving goals, learning, creating, and positive feedback all fill me with energy. Conflict, negativity, and lack of control all steal my energy.

Understanding these aspects of who I am, above all others, is the key to getting the most out of me. If you can align your requests with the things that give me energy, I will happily do a lot for you. Figure out how to make your goal my goal, set me up for success, provide me positive feedback, and watch me go!How to Work With Me

There are several things you can do to set me up for success. Some flow directly from the description above, while others are less obvious, and still others represent interesting edge cases between the positive and the negative.

 
Direct Applications 

To get the most out of me, avoid the things that drain my energy, and lean into those that give me energy. Help me learn, help me achieve goals, and give me plenty of positive feedback along the way. Don’t put me in heated situations full of conflict. If you want my honest opinion, don’t push back when I share it.

Help me stay out of situations full of negativity, whether directed at me, at others, or just simmering in the background.

Finally, pay heed to the structure (or lack thereof) and demands of the situation you are asking me to enter.. I need the time and space to build structure for myself. For example, do not throw me into an important customer escalation. Do throw me into the contingency planning for future customer escalations. The latter is building structure for future use–I can do that. Less Obvious Implications

Please don’t interrupt me. Please do help me avoid interruptions. Recall, I can only focus on a few things at a time. When I am interrupted, I have to focus on two things at once: the thought that was in my head and the interruption (maybe your question). They don’t both fit in my head. Both get short shrift, and I get frustrated.

Sometimes requests come as interruptions, but they do not need to. If you have a question or request, sending an email is least disruptive. Receiving an email allows me to schedule my day and give your request my full attention when I get to it. Even if you need something now, please do not interrupt me. Come to me, make your presence known, and let me finish my current thought. Just wait. I will look up and then you will have my full attention.

If you are physically remote from me and need something soon, you can use Slack. I check it regularly and I will answer when I can. If it can wait, please send an email.

Notes on Email Requests

Focused and actionable emails are easier for me to respond to. If you are asking two things of me, consider sending two separate emails. If you send me multiple emails, I may answer an easier one before a harder one. If they were combined in one email, everything would wait, even the easier request.

Edge Cases

Some interactions ride the edge between things I find energizing and things I find de-energizing. It may not be obvious whether a given interaction will drain or energize me. Two main examples are feedback and getting things done. Constructive vs. Negative Feedback
I love getting better at things. Constructive feedback and advice are great because they speed up my improvement. However, I don’t like negative criticism–it steals my energy. Yet all constructive feedback contains some negativity, in the form of something that isn’t good and can be improved.

I believe the difference between constructive and negative feedback is largely one of degree and my perception. If the feedback is operable (and I can see that it is) or exposes me to a new idea, it’s constructive. Constructive feedback opens up possibilities for me and gets me excited. However, if the feedback only shows me my weaknesses, it’s negative.

It’s the difference between: “David, it looks like you might struggle with X. Does it feel that way to you? Have you thought about doing Y to address X?” and “David, did you realize you have a problem with X? Yeah, that wasn’t good. You should focus on that.” The first one is confirming a challenge and helping me get better, while the second is pointing out I’m not good (at something). For the record, I probably know that X didn’t go well and have thought about it already. You telling me to think about it more is not helpful. However, you telling me a new way to think about it is helpful.

Since the difference between constructive and negative feedback comes down to my perception, you may have to pay attention to my reactions to see which case is happening. Do I seem to be deflating or am I perking up with your feedback? You can also (gently) ask me if your feedback is feeling more constructive or negative, but please don’t push if you want an honest answer. Doing vs. Being Overwhelmed
I love doing things so that I can accomplish more and make more. At the same time, having too much to do (feeling out of control) drains my energy. Once again, the difference is one of degree. I have some limit to what I can do at a given time. It feels wonderful to operate near, but below that limit. However, when I try (or am pushed) to exceed that limit, I start to degrade. I get less done and my results are of lower quality.

Being overwhelmed feels bad because I am no longer in control. I can no longer meet all of my commitments. I cannot plan out my day. I cannot focus on one thing at a time. All of those hurt and lower my productivity.

So, please help me fill (but not overfill) my day with interesting and challenging work. But also respect my limits. Respect my answer when I say I don’t have enough capacity to do what you want. Let’s Work Together and Do Something AwesomeHopefully this post will help you better understand and work with me. I am a human with human foibles and peculiarities. I work best when collaborating and learning with others. Give me some space to focus, a sense of control over my environment, share some positive energy, and let’s build something awesome together! And please share with me, how I can best work with you.

A special thank you to Heather Beasley Doyle for her constructive feedback on this post and her supportive interest in my writing journey. Heather is a gifted writer (on a completely different and higher level than myself) and you should read some of her writing here, here, or here.

Two years ago I was hiking with two friends in the pouring rain on the side of a mountain in New Hampshire. We were a few hours into a two night backpacking trip and we were soaked through. For many people this experience would be hellish. It wasn’t for me. I knew we were well prepared and we were doing something that I love. It was the start of a great trip. 
I spend a lot of time reflecting. I do this to work towards being my best self. I structure my life based on these learnings, making sure I satisfy my basic needs, do more of the things that give me energy, and limit my exposure to the things that steal my energy. Today, I would like to share what I have learned about myself. Maybe it will help you understand me a little bit better. Maybe it will inspire you to reflect on how to work towards your best self.
Our view from Zealand Cliff in the White Mountains, several hours after hiking in the rain. Photo by Jay Doyle.The Basics

At the most basic level, I have learned that I am an introvert, with a strong sense of responsibility, and an ability to focus deeply. I have also learned I need some basic self care and structure to function well.

IntrovertWhile I really like talking to people, I can only do so for so long before I need some quiet time. The quiet time recharges me. My commute home often gives me that quiet time. When I don’t get some quiet time in my day, I end up exhausted and overwhelmed.

I recently attended an international conference, full of great ideas and better people. I spent each day soaking in the presentations and talking to people. At the end of the third day I was invited to dinner at a beer garden. It was tempting, however, I politely declined. Instead, I went back to the hotel, organized my thoughts from the day and decompressed. That quiet night left me ready to fully engage the next day. It was the right thing to do – for me. Responsibility When I say I will do something, I try very hard to do it. It causes me pain if I am unable to meet my commitments. At the same time, it is easy to commit to too many things. Knowing this, I try to commit only to things that are both important and that I have time to complete. I often turn down otherwise interesting opportunities because of this, but it is worthwhile. FocusI can focus deeply on a topic for an extended period of time and I need some deep focus in my day. However, I can only focus on a small number of things at once. Very likely, you can hold more things in your head at one time than I can. As a result, I appreciate simple models, systems, and (programming) code. My hiking trip is an example of focusing on one activity for three days. All I had to do was put one foot in front of the other and enjoy the view.Care and FeedingI live on a schedule with regular sleep and meals. If I miss either, I degrade rapidly. I also require regular quiet time for reflection, focus, and learning. I work very hard to protect my schedule, making sure I get good sleep, regular meals, and focused time.

My combination of introversion, responsibility, and focus helps me tackle hard problems and drive them to completion, provided I get my basic care.
Things that Give Me EnergyMy reflections also help me understand which things in my life fill me with energy, so that I’m ready to take on the world.
I like to accomplish things. This can be setting a goal for myself and achieving it, making something (baking, woodworking, writing, …), or learning something new.

Even better than accomplishing something is getting better at something. I can do so much more afterwards! Help from others is great, since it speeds up my improvements. It can be direct advice, a tool (ask me about my ski gadget), a teacher, a strong example, or someone’s writing. In turn, I love sharing my learnings with others, helping them get better.

I also love (earnest) positive feedback on any of those things. The feedback shows that I’ve made someone happy and, maybe, they like me a little bit more. Even small bits of feedback have a disproportionate impact on my day. They also make me want to do more. Things that Steal My EnergyThings that drain my energy make me want to curl up in the corner and take a nap, or go outside to rage walk. They fall into two broad categories: negativity/conflict and lack of control. Negativity and Conflict
I don’t like conflict. It drains me. I want to make people happy and for them to like me.

I know people who live for the outrage and arguments of Twitter. I don’t and can’t. Due to my dislike of conflict, I shouldn’t be the ultimate quality gate on anything such as releasing software or launching rockets. I’ll tell you what I think, but if you then push on me…

General workplace negativity contributed to me leaving a job. Additionally, I have a thin skin for personal criticism. I do not like being told (directly or indirectly) that I’m not good at something. Yes, there are many things I am not good at, but it’s easy for me to feel attacked and end up defensive. I’m working on it.Lack of ControlDue to my focus, responsibility, and need for schedule, I tend to plan. I can get amazing things done when given the space and time. My perceived level of control is very important. When I don’t feel in control, I get frustrated, annoyed, and tired.

Direct lack of control and overcommitment are two common cases. Lack of control happens any time something keeps me from shaping my day to day environment. For example, I’m dragged into a meeting when I thought I had a block of focused work time, or priorities keep changing (no, you need to work on X now. Wait, no, Y, Y has to be done now!)

Overcommitment is more subtle. It would still seem like I have control over what I am doing, but I don’t. I am now reactive, overloaded, and set up for failure. It may be several people in aggregate asking me to do more than fits into my day, or a single person. Or maybe I am just pushed to go too fast. When asked to go 10% faster than I can sustain, I don’t have time to focus and reflect, so I’m not doing my best work. I’m less productive and more frustrated. Building a Better MeWhen I am backpacking, there is plenty of time for my own thoughts. I am responsible for bringing everything I need (and not more), and there’s an extended opportunity to focus on one thing. Further, there is an incredible sense of accomplishment for covering the distance, summiting the peaks, getting to the views, camping in the backcountry, and getting back out. It is a hobby that fits me very well.

I would encourage you to learn more about yourself, so you can find work and hobbies that fit you very well. I hope my example inspires you in this direction. I also hope that it gives you more insight into me and people like me. Additionally, if you are interested in working with me, I explore the implications of this post on working with me in How To Work With David.
  A special thank you to Cian and Alex for their feedback on an earlier version on this post.