Jade Rubick - Engineering Leadership

Organizations that write things down have a larger advantage than they used to.

Let’s take two companies:

• Speakeasy Corp
• WriteItDownCorp

At Speakeasy, leaders share information verbally. They have spoken updates, status meetings, and get everone on the same page verbally.

At WriteItDownCorp, the company has a written culture. Everything is documented and organized. It's not official until it's written down in the wiki™.

There are tradeoffs to both types of companies. But I believe that over the next few years, written cultures will have higher leverage than spoken ones.

What is written culture

A written culture doesn't mean that people don't talk to each other. It means that there is a strong culture of writing within the company.

I've worked at a lot of companies. More than most people. And company culture varies significantly, mostly based on the biases of the founders and executive team.

Signs of a written culture

• You have a company handbook or departmental handbooks that are up to date. Usually this is because it's used as the source of truth. (I've written about this in your process should be open source).
• You have a written company strategy and product strategy.
• Product plans are written down. Engineering plans are written down.
• Collaborating on a document is actually a high leverage activity, instead of a waste of time.

Why is a written culture high leverage?

Whatever your feelings about artificial intelligence (AI) (and I have complex and conflicting feelings about it myself), it is a technology that works better with good context. And AI is orders of magnitude more useful when you have written context.

If you believe that AI is a high-leverage technology, then writing is something that allows AI changes to happen.

Here are a few example prompts:

• "Look at our engineering standards, and my plan for this project. Highlight anything I might be missing or should be thinking more carefully about. Also read the security checklist and make sure my plan incorporates all of the points from the security checklist."
• "Look at our product strategy, and my team's roadmap, and highlight anything I should make sure is on strategy or note that it will need a carve-out. For items that aren't on strategy, read the pitch for the project and give me the best reason to keep the project on the roadmap, and the best reason to delay or cancel it."
• "I'd like to gradually automate Process X. Look at this documentation for the process, and convert it into a command line checklist. Each time we run the checklist, suggest something we might be able to automate. Update the process page to refer to the command line tool, and interview me to add some historical context for people so that if they run into problems they'll understand how the rationale for this automation"
• "These three projects from last year were considered some of our most successful projects. Look at the execution plans for them and highlight anything they did differently than the other plans on the project list. Also highlight any processes from the engineering project template they might have skipped, in case there is some process we might be able to prune or modify."

None of these should substitute for real, thoughtful work. And you can't blindly accept anything the AI tells you. But they can surface things that you weren't considering, and can help you do your work better.

For example, reviewing your team's roadmap against the product strategy can highlight projects that you may need to make a good case for them remaining on your roadmap. Or help you cancel them early, because they're off strategy. The decision is up to you, but the fact that it's off strategy is easier to highlight if it's all written down.

A written culture has benefits for humans too

• Writing forces people to think more clearly.
• Writing can be a more inclusive way of critique and decision-making.
• People also benefit from being able to look things up.

Technology progresses from text to 3d

When I've brought this up with other leaders, one reply I heard was that they didn't think that a written culture would be important in a few years.

We have seen this type of progression before in the past. Many technologies start with text, then move to audio, then photos, then video, and then to 3D.

For example, blogging came before photo sharing, which came before video sharing. The reason for this is that each is a progressively more demanding technology.

You do see examples of AI today engaging natively with audio and photos. But it is so much more demanding that I expect text based communication to be more effective for a while yet.

Writing considered harmful?

One potential counter-argument to this might be that people are engaging in writing in a sort of AI slop way nowadays, and generating so much slop could result in poor context setting for companies.

Curating and managing the writing is becoming more expensive, so that could blunt some of the benefits of a written culture. That seems plausible to me.

How to move to a more written culture?

• Read my post on open sourcing your process. I especially think using a wiki as the source of truth is an effective lever.
• Reinforce it from the top.
• Typical change management: describe what you're doing and why, explain, and model it.
• Chat me up if you're interested.

Thank you

Image by Nile from Pixabay

Teams often don’t handle reliability risks well. Reliability is tough to reason about.

To have a reasonable response to reliability, we share an approach called a risk matrix, with a process to create plans to manage your reliability risks.

A risk matrix is an approach teams and groups can use to help them reason about risk. They develop a list of the things they own, the dependencies they rely on, and a list of who relies on them. Then they systematically evaluate the amount of risk and impact of likely failures. And create plans to transfer, avoid, reduce, or accept each risk.

This has been used across many organizations as a high leverage approach to improving reliability.

A risk assessment framework

This document presents one approach to risk assessment as part of an overall risk mitigation strategy. The process described here is intended to be used at the team level, and the examples reflect that. But it is readily adaptable to a group and organizational level when you begin to think of teams and groups as areas of responsibility or functionality. In order to work well, this process should be recurring and required, meaning that every team should perform a risk assessment for their own services and dependencies, and that the assessment should be revisited and updated regularly (quarterly or half-yearly).

Why should we do this?

Software systems readily grow to a level of complexity that makes it difficult, if not impossible, for one person or team to hold the entire system in their head and understand how it all works together. This also means that it is hard for any one person, or team, to understand the ways that parts, or all, of the system can fail and how those failures in sub-systems will impact up and downstream components. By engaging in a recurring process of risk assessment, and mitigation, we deepen our understanding of our sub-systems, as well as the behaviors of our up and downstream dependencies. This increases the reliability of the whole system and allows us to reduce the Mean Time To Resolution (MTTR) when we do have service interruptions.

What is a risk?

In our real lives, and in software, we are faced with hazards every day. Water, gasoline, and driving are all hazards we encounter in our daily lives. They are circumstances and situations that present an opportunity for harm. A Risk is the likelihood that a hazard will cause harm. Water is the hazard, falling in and drowning is the risk.

What is a risk assessment?

The process of enumerating the hazards for a system, or subsystem, determining their likelihood of occurring (their Risk), and also the impact of their occurrence is, in broad strokes, a Risk Assessment.

A common approach is:

1. Have the team sit down together (or asynchronously with whatever tools you prefer)
2. Get the team to think about everything a service or component relies on, touches, uses, runs on, etc. Note these down in a shared document (often a spreadsheet is a good place)
3. Sort through these to ensure that the risks identified are relevant. We could all be killed in an unexpected meteor impact but we can leave this out of the risk assessment.
4. Once you are confident that you have identified the risks to your service/team/platform, take the group through each risk and reach a consensus on the Likelihood of a risk and the Impact of an occurrence of it.
5. Use these evaluations to determine what, if anything, you can do about this risk.
6. Create a plan to do what you can to reduce the identified risks.
7. Finally, revisit your assessment, and the plans to reduce risk, regularly to ensure you are working to reduce risk and increase reliability.

How to?

There are a few tools we can bring to bear to help us with this thought exercise. I will introduce the tools and techniques before walking through the process of using them in the process outlined above.

Tools of the trade Service catalog

This is the list or collection of services/functionality owned and operated by the business unit that is performing the risk assessment. For teams, this is commonly the list of services, libraries, tools, etc owned and operated by that team. For groups, this is often expressed as functional capabilities of a platform, or components of the platform, owned by the group.

Dependency tree

The interconnected web that forms the context your service or functionality exists within. This includes services/APIs that your service interacts with (both up and down stream), the infrastructure your components run on, the libraries imported by a project, the monitoring tools you use to operationalize your service, etc. Often, building the dependency tree is one of the more labor intensive stages but, once done and when kept up to date, it provides the jumping off point for identifying and classifying risks.

Risk matrix

The grid of Likelihood vs Impact that helps illustrate which risks present the greatest danger to your service/system. Risks with the highest likelihood and greatest impact are good areas of focus, but even less likely or less impactful risks can provide significant wins for reliability and a reduction in MTTR.

T.A.R.A

A framework for deciding what, if anything, can be done about a risk. Transfer, Avoid, Reduce, Accept or TARA outlines general categories of action for risk mitigation.

Transfer

When you transfer a risk, you are attempting to shift responsibility to some third party. This can be a vendor, another team, etc. When deciding to transfer a risk, you are not eliminating the risk, necessarily. Instead, you are saying that the best mitigation for that risk is to have some other entity be responsible for it. Running your infrastructure on AWS is a very common example of transferring the risk of running your own infrastructure to a third party.

Avoid

Eliminate the risk of encountering a hazard entirely. For example, if you rely on an external API for your service, you could avoid the risk of that API becoming unavailable by removing that dependency entirely. Avoiding a risk will often introduce other risks or dependencies, though, so be certain you understand the tradeoffs when choosing this mitigation strategy.

Reduce

This is where a lot of teams will spend their time. When you choose Reduce as your mitigation strategy, you are attempting to lower the impact or likelihood (or even both) of encountering a hazard. Increased monitoring to reduce the MTTR can reduce the impact and likelihood of an outage. Introducing redundancy to dependencies that have inconsistent reliability is another common mitigation strategy. What this looks like will vary greatly but you should always have a specific goal in mind when making attempts to Reduce a risk.

Accept

Some things you cannot do much to change and you have to Accept that. This does not absolve your team from responsibility to have a plan in place for when a hazard is encountered, but you are Accepting that the best you can do is ride out whatever storm you encounter. Again, AWS is a good example of a risk that is commonly accepted but strategies for communication and recovery after an outage occurs should be in place.

Process

The steps outlined below assume that this exercise is being done by a team. I will speak in terms of “services” and “tools” which are colloquially the sorts of things individual contributors are used to being responsible for. You can readily move up the organizational chain to perform this exercise at higher levels by substituting systems or capabilities of a platform in for services and tools.

1. Agree on Definitions

Before you can start this process you will want to come to a consensus on at least a couple definitions: Impact and Likelihood. These will be used throughout the assessment to communicate about risks, and to decide where to direct your efforts at risk mitigation. A common approach focuses on Red/Yellow/Green or Low/Medium/High for terminology.

Impact

• All of our services are unavailable for all of our customers
• Some features are unavailable or have significantly degraded performance
• All features are available but they are slower in a noticeable but minimally impactful way

Likelihood

• This happens often when we deploy
• If several other criteria are met, this will happen
• This almost never happens but could

2. Define scope and catalog services

The first step in performing a risk assessment is to understand the boundaries you are considering. Using whatever collaboration methods work best for your team, try and catalog everything your team is responsible for. A Google Sheet is often a good default. This will commonly be services, libraries, and tooling used by yourselves, and possibly other teams. Do not decompose these yet (eg service, database, infrastructure all make up some piece of functionality) as that will be done later. What is out of scope is just as important as what is in scope, in order to define and understand these boundaries, so do not be afraid to bring anything up for discussion. If your team already has an existing service catalog, go over it and ensure it is up to date. There may be things that should be added or removed since the last time you did this exercise. Finally, write the results of this down. You will need the output of this step for the next stages and having this somewhere that is discoverable by other teams can help build a shared understanding of how our sub-systems interact.

3. Identify impact

In order to rate and assess the impact of a risk, we have to be certain we understand who is impacted, and how. The first time your team goes through this exercise, you may only have theoretical ideas about this. If you have never encountered this risk before, you may only be able to guess about the impact, but do your best to think of who would be impacted and what their experience would be. This can be internal stakeholders (a team that consumes an API you support or an upstream team that sends you data), customers, or even just your team. The point is to understand “Who” and “How” not to decide whether that group or their impact “matters” yet.

4. Gauge likelihood

Similarly to impact, we have to understand the likelihood of encountering a risk in order to assess it and decide on what, if anything, to do about it. Using the definitions agreed upon earlier, come to a consensus on how likely your team thinks a given risk is to be encountered.

5. T.A.R.A.

Once you have agreed on impact and likelihood, you can use the TARA framework to decide how to classify your risk.

One thing you can do is have some heuristics that you agree upon as a team. For example:

• High-High: we work on it before anything else.
• High Impact, Low Likelihood: might be a good candidate for improved monitoring or early detection.

6. Make a plan

The most important output of this process is a plan for how to mitigate and manage risk. Each of the four classifications has a possible output and you should decide what that should be. Creating issue tracking tickets for each risk that outlines the decision about what to do can help you stay accountable for risk mitigation, and speed the process up for your future selves. For any work that will get done, it can be helpful to create separate tickets and link them to the risk tickets as risks are often persistent over time, whereas the work to mitigate them is discrete.

7. Follow up and repeat regularly

Finally, this process is most effective when it is done regularly. By going through this exercise on a recurring cadence you are able to track the evolution of risks over time, and gauge the efficacy of your mitigation strategies. When this becomes a habit for a majority of teams in an organization, you are able to track trends and evolve your risk mitigation strategies to, hopefully, continue to increase reliability and decrease Mean Time To Resolution.

Summary

A risk matrix is an approach teams and groups can use to help them reason about risk. They develop a list of the things they own, the dependencies they rely on, and a list of who relies on them. Then they systematically evaluate the amount of risk and impact of likely failures. And create plans to transfer, avoid, reduce, or accept each risk.

Thank you

Thank you to David Parrott for being willing to publish this. This document wouldn’t exist without the work of Nicholas Valler. It’s in some ways an expression of much of the work he did. So thank you Nicholas Valler! Jade contributed some edits (intro, summary, heuristics for TARA) and wanted to see it published.

Image by wastedgeneration from Pixabay

Should engineering organizations have quality assurance (QA)? A lot of engineering leaders say they should not. I review the topic, and provide some suggestions for QA leaders, and some nuance on the problems with the way QA and engineering work together. 🌶️ ahead!

QA should not exist

In some engineering leadership circles, the answer is “of course not”. QA is largely seen as an old fashioned practice: “Engineering should own quality”.

I tend to be skeptical of this type of engineering hubris. But there are some good arguments for this:

• QA slows things down. There is too much back and forth when you’re passing work in a gated process. QA finds an issue, dev fixes it. Then QA rechecks it and finds other issues. This continues forever, greatly slowing down engineering velocity. Note this isn't because QA is finding issues -- it's because of the back and forth.
• Throw it over the wall. There is a moral hazard in having other people responsible for whether your work is correct. You’re creating bad behavior for your engineers by not having them responsible for their quality.

QA is often compared to Operations. Having engineers write code and operators put it in production and care for it has similar problems: handoffs and incentive problems.

In a lot of modern and startup environments, engineering leaders view that as a harmful practice. And they’re often dismissive of the value of QA as a result.

QA should definitely exist

Other leaders are firmly in support of QA.

They argue:

• Testing is a skill. Looking for issues is a real skill, and having an adversarial approach does result in finding deeper issues. Engineers sometimes exhibit an arrogance that they can do everyone else’s job, but the truth is this is a real skill, and not one that is easy for developers to pick up. And there can be a value in having independence from the engineering organization.
• Automed tests are valuable. Automated tests, when embedded in your workflows, are incredibly valuable. And they can be done by people who are less expensive than engineers, making their work high leverage.
• Some situations absolutely require QA. There are some high stakes situations where a skilled expert can be worth their weight in gold. They can save the company from considerable risk and ensure a better product experience.

I find this attitude more often in traditional companies, but I see it in a lot of places.

Perhaps the strongest argument nowadays for QA is that with AI, automated verification is a leverage maximizer.

The testing pyramid

Before I weigh in further, I’d like to make sure you’re familiar with the testing pyramid. The basic ideas is that there is a natural pyramid of value with testing. The base is unit tests, which are fast and deterministic. Then above that you have integration tests, and then above that, slower UI tests. You want to have lots of fast unit tests. And fewer integration tests, and fewer still UI tests.

Everyone agrees that the engineers should be responsible for unit tests and integration tests. And the UI tests are up for debate. In a lot of organizations, that’s where the split is: QA owns end to end UI testing, and is testing it from the end user perspective. Other organizations share ownership of the end to end tests.

My (slightly) more nuanced take?

I polled my engineering leadership peers, and 100% of a sample of 10 people said, “You mostly should not have QA on software teams, with some edge cases and exceptions.”

I mostly agree with that, but I do have some nuance I’d like to add, and some ideas on ways that QA might be made more high leverage. And I have some advice for QA leaders to make sure they’re as valuable as possible in their companies.

Don’t start with QA

If I am building out an organization from scratch, I wouldn’t start out with QA on a team. Everyone on the team would know that engineering owns quality. Engineering would do unit, integration, and UI tests as a regular part of their work. CI/CD and test automation would be built in from the beginning. Most teams should not start with QA unless they have some very specific reason for doing so.

Don’t have a handoff

I agree that structurally, you don’t want to have a handoff between QA and engineering, if you have QA. It’s much better to have QA “shift left” and be a part of the team itself, doing the work more continually with the engineering team. So my first suggestion is that if you have QA, you embed them on a team and aim to get their work into the same cadence as the engineering team.

Have engineering own quality

If you do have QA, I believe it’s best to still have engineering “own” quality. Metaphorically, I like to treat QA as the quality experts, similar to SREs who are reliability experts. SREs should not be doing all the reliability work, just like QA should not be doing all the quality work. They are experts who can help the team be better at quality, and make sure the team is doing quality work.

Note that quality includes more than just testing. Testing is a big part od it, but it is also making sure things are built correctly. So if you have engineers "own" quality, that means a lot more than engineers do testing. it impkies a much more customer focused engineering team.

Require automation

You almost always want to focus on automated testing. Any manual testing should be for highly unusual situations. I’ve seen it be useful for brittle architectures, or for basically probing for quality issues by a highly experienced QA person.

So if I was building a team from the ground up, and evolving it over time, I’d lean hard on automation and the team being customer focused, and try to solve those problems before reaching for QA.

When would I reach for QA?

There are a few times where QA is high leverage. I’ve seen brittle architectures that are hard to test in an automated way. (Again, I would try to automate first, but this could be a backup plan while we fix the architecture).

I’ve also seen very skilled QA people be deployed as a strategic roaming asset within the company, to focus on the most sensitive projects. I remember working with a couple of people in the past who were just amazing at making sure projects were successful and were shipping with quality. Having roaming QA can be a good pattern, because it can only operate within a larger engineering culture of engineering being responsible for the quality of their own work.

I also think there is a lot of untapped potential with QA. The space is changing rapidly with new AI based tools, and I’ll talk below about some potential ways you might tap QA for high leverage benefits.

What about when there is already a QA team?

I often join teams that have an existing QA team. What do I do when I join a situation like that?

Sometimes it’s not the top priority

First of all, it’s often not my top priority. If there are other higher priorities, I’ll focus on them first. But I can usually improve things a bit with just a few nudges, that don’t require a lot of extensive work. I’ll often meet with the Head of QA and talk these over as goals:

Shift QA left

• Get rid of handoff between dev and QA.
• Have QA members embedded on the team and attend meetings.
• Try to get the workflows of the QA people onto pull requests (PRs) rather than later in the process.
• Emphasize that engineering owns quality, but QA are quality experts. Engineers can do QA tasks as well. Often QA can help with test plans that engineers can be involved in doing. Engineers can also write test plans and have QA review. Less process gates and more working together.
• Share concept of testing pyramid, and make sure engineers are writing unit tests, and that PR review requires unit tests. Ironically, I find that most companies that have had QA for a while also don’t have a good unit test suite. I’m not sure why that is, but I think it’s because engineering thinks QA is there as a safety net.
• One challenge is with QA teams in different time zones, or where they are offsite and the team is on-site.

Focus on automated testing

• Emphasize that testing needs to be automated. Ask the leader to recommend training or replacing team members to move us to mostly automated tests.
• Make sure the test suite is fast. Make sure we have statistics on how quickly it runs, report on it frequently. Get the QA team to make it faster. Often this means separating parts of the test suite by criticality so that a portion of it can run fast enough, and other parts can run later.
• Make sure the tests are a part of the developer workflow, not only seen by QA.

Metrics

• Unit test coverage.
• Unit and UI test run times.
• Functional (requirements) coverage.

Evaluate the situation

Once the burning fires are put out, improving the way QA operates rises towards the top of the list. But it depends on the company. I’ve seen companies where the problem was so egregious the best solution was to let go of QA. I’ve seen situations where it was working fine, or fine enough that I didn’t care to intervene because I had other priorities.

So my actions at this point depend a lot on the situation. I might invest less in QA. I might pare down QA gradually. I might let QA go completely. Or I might explore ways to make QA more high leverage. I think the last point is the most interesting, because at least in my circles, I haven’t heard people talk about it much. So let’s talk about that.

Introducing the Automated Verification Engineer (AVE)

If I were a QA leader, here’s how I would see the world:

1. I’m in a role that is sometimes treated as old-fashioned.
2. AI is rapidly increasing the leverage of automated verification. Why? If you can improve a coding agent’s ability to detect failures automatically by 50%¹, you’re speeding up the company something like 2x, because a much larger percentage of its changes will be able to be accepted. Automated verification means your work is speeding up the company AND improving quality, instead of slowing it down.
3. Automated verification is something my team can help with.
4. I’m going to need to market this differently than QA. I would probably create a new term like “Automated Verification Engineer” (AVE) and a new set of responsibilities. That way I can redefine expectations of the work my team does.

Then, I'd embark on a somewhat experimental approach to redefine how we work together. Here's an example of how it might look, an Automated Verification Engineer.

¹ (This number is just a guestimate, not a real mathematical calculation. I think the value actually would increase nonlinearly as you got more automation, because of the increased confidence you haven't broken something.)

What is an Automated Verification Engineer?

• I’d view the role as a sort of cross between a Developer Experience Engineer and Quality Assurance.
• The AVE would be focused on making sure the developer pipeline gets great feedback early on to customer problems introduced by code.
• The AVE’s tests are mostly for AI agents. The aim is to help the agents realize when their hallucinations and bad changes aren’t valid.
• All tests are automated, deterministic, non-flaky, and as fast as possible.

Metrics for an AVE:

1. The goal is fast feedback for the engineering team. So I’d use “time to feedback” as one of the most important metrics for success.
2. You could instrument your test suite, and report on the number of tests that fail in an automated testing situation. Each one of those tests was highly valuable (with some caveats).

What is the role of an AVE Leader?

1. Part of your job is to help engineers understand they’re responsible for quality. But also help the organization understand how automated verification engineers work, and how it’s different than QA.

2. I think you need firsthand understanding of how coding tools work. So I’d encourage you to play with them every week.

3. You also need to level up your skills in the testing and deployment lifecycle. I’d partner with the engineering teams responsible for it, study how it works, and volunteer to pair and do work there. You should be able to change the set of things being testing in each PR, or move the tests to run in paralllel.

4. In the age of AI agent development, I believe you should probably be doing a lot more experimentation. Evaluating different tools, looking for things that change the way you do your work.

5. One idea, that I’m not sure is a good idea or terrible idea, is to get more involved in the signals coming back on quality. For example, an AVE Leader could be involved in the bug triage, and work with support on issues on things that have been released. This would give good signals on what is missing from the quality process. Move from being transactional (reviewing work), to try and make the quality better.

6. Another idea is to work with design, to get a sense of what types of design issues can be automatically tested for. Usability and quality are linked.

7. Every month, try to make your tests faster, employing patterns like 1) testing only what has changed, 2) testing locally as you work, 3) breaking tests in parallel, 4) automatically disabling or deleting or ticketing or breaking the build for flaky tests. You should always have projects in flight to be testing out various schemes of speeding up tests.

8. The company should have a “testing pyramid” approach, with unit tests at the bottom as a foundation, then integration and UI tests above that. There are other tests to consider as well, but generally you want more unit tests than UI tests.

Some of these responsibilities might already be covered well in your organization, and this is just a sketch of how it could work, not a job description. So modify it for your company.

Automated Verification Engineer is experimental

I’ll add that this is all experimental. It might even be a bad idea. But if I were a quality leader, I’d be doing some sort of experiments along these lines.

Disrespecting QA?

I’m more nervous about this post than most, because I think it risks making people feel like their role isn’t important. I’d like to emphasize that it isn’t about the people themselves, but the role that is the problem. In the same way that Operations was a problematic role in many organizations, I believe QA is not often set up well. I think a combination of respect for the people involved, and some experimentation in the role might yield some big results in the age of agent based AI development. But the bottom line is that working more continuously, more within the same workstreams, typically yields much better results than passing things on in a transactional, assembly line process.

It would be a lot easier to say "no, just don't do QA", and honestly for a lot of places that makes sense. Or to say, "yes, let's just keep QA the way it is". But a lot of the challenge is to experiment with the workflows, and see if you can make QA a high leverage point for the company. If you do manage to make automated verification exceptional, you're taking your role from a cost center to a point of differentiation for the company.

If you have any feedback on this post, let me know. I imagine my thinking will evolve, and I’d like to hear your perspective.

Also, if you do end up experimenting with these ideas, let me know. I'd like to see what worked and what didn't!

A few more thoughts

This went to the top page of Hacker News, and had the predictable type of responses you would expect. The title seems to have thrown a lot of people -- a lot of people react to the title instead of the body of the post. Predictable, I guess.

One comment I thought was quite interested was from Jenny Wanger where she said maybe engineers will be working more like QA in the future. I can also imagine the opposite: QA will be acting more like engineers. I can imagine the lines getting blurry between the roles.

Thank you

• Bruce Vaughn reviewed the post and provided feedback. George Lantzourakis suggested adding the metric of functional (requirements) coverage.

Image by John from Pixabay

Today we cover the topic of creating an Innovation Lab.

Companies suck at risky long term planning

Engineering organizations struggle to focus on anything that isn’t a project.

Yes, you can and should work to change this. But there is a lot of inertia behind projects being the unit of work.

So what do you do when there are things in your future that don’t easily become projects? When there are risky activities that need definition? New potential products you need to vet and validate?

Product and engineering have some part in this

Most of this is what Product does. Product management is all about exploring and understanding the terrain.

But sometimes this exploration is between Product and Engineering. For example, is this even technically possible? Or is there a product possibility here that can be built in an economical way? Are there novel approaches we can take that we don’t even know are possible right now?

This is why companies make Innovation Labs

Exploring the space between Product and Engineering is often why companies form Labs. The input is risky ideas that could be highly profitable. The output is things that can be made into projects.

The Three Horizon Model

This is something that companies have done for ages. McKinsey famously packaged up some previous work on this into the “Three Horizon Model”.

• Horizon 1 is the core business that provides profits and cash flow.
• Horizon 2 is emerging opportunities, likely to generate profits, but requiring investment.
• Horizon 3 is ideas for profitable growth that are risky, untested and may not generate profit for a while. “Zero to one”.

The Horizon Model helps you see how to structure the business for each type of Horizon.

Businesses are almost always optimized for Horizon 1. After all, that really is what the business IS. A machine for generating profit. All of the practices in the company are optimized for that. Horizon 1 focuses on performance and optimizations. It is operational in nature.

Some companies are good at Horizon 2 activities. But the important thing is that often they require investment. So there can be some competition between Horizon 1 and Horizon 2 investments. But Horizon 2 is often clear enough that the investment makes sense and the company can logically act on it. Horizon 2 is more about acquiring usage and customers, and anticipated revenue.

Horizon 3 is where things get tricky. They are ideas, and they need experimentation. Lots of experimentation! They are often risky and more often than not, they should be discarded. Horizon 3 activities require a totally different approach. Horizon 3 activities require protection from optimization. Their goals tend to be more about milestones and experiments, very market oriented but driven more by possibility than actuals. They often require protection from processes that the rest of the company takes for granted.

And Horizon 3 will always lose in a competition with Horizon 1. It needs serious protection.

Advice for putting together Labs

I’m not a world expert on this topic, but I have done it before, and I’ve seen about 5-10 examples of this at places I’ve worked. I reached out to my network for some other people I know who have done this well in the past. So here are some suggestions if you’re thinking about putting together a Lab:

Consider putting the Lab under Product

Product and Engineering both have a big stake in this, and it’s often best to have some of your most entrepreneurial, customer focused, and deep technical leaders in your Lab. And you usually want both a product and engineering counterpart, so the things you’re experimenting with are most aligned with the market.

If you put Labs under engineering, it will look like a place people go work on cool ideas without shipping anything. And it breeds deep resentment with the rest of engineering. If you put it under Product, you clarify that this is a scouting function – the work is to experiment and scout out the terrain. It also helps ensure Product and Engineering are aligned and not competing.

The message if you put it in Product is: the focus is on defining the future rather than building it.

You can put Lab elsewhere. In fact I’ve seen it done outside of Product most often. I’ve seen it in an Office of the CTO, which worked pretty well. And I’ve seen the head of the Labs report to the CEO, which also worked pretty well. I would consider all of these options, but default to putting it in Product.

Challenge: the “white page problem”

One challenge is that you may have too much freedom and not enough focus. Because of this, collaboration with Product or other people with deep market expertise is essential. The idea is to find the areas that you’re getting signals may be highly valuable, but require validation and technical exploration.

For example, I was at a startup when Kubernetes was starting to take off. Product knew that this represented a big opportunity. But there were a lot of unknowns about what a “Kubernetes version of our product” would look like. Engineering could have tackled this by having some exploratory projects. But we had other clearer projects that we knew would be good for the business, and that were easier for Product to wrap their heads around. So rather than prioritize an ambiguous project that may or may not succeed, Product would prioritize a more certain but lower potential value project, over and over. Exploring the Kubernetes project was a classic Horizon 3 activity: go explore this emerging development, and give us a better understanding of how our product can expand into that space. Validate the approach we’re taking. Figure out where the dangers are, and what a project would look like. Give us a potential project.

It's mostly rejecting ideas

"A lab is mostly about rejecting good ideas that aren't great ideas. Ideas are plentiful in companies. If they were all worthwhile, why experiment? Just build them all. So the lab should have a strong habit for rejecting ideas with evidence." -- Jim Ruppert

Challenge: interface with engineering

It’s pretty important for the Head of Engineering to be on board with the importance of the Labs. So work with the Head of Engineering on the role boundaries, on project ownership, and on how handoffs work.

The handoffs are especially important. One of the more successful patterns I’ve seen is that the team that did the exploration joins with the engineering team when the idea is productized. They work together with that larger, expanded team for a while. Often the engineering team is newly spun up to deliver the new thing. After a while, when the initial versions of the product have been built, and you have a team that can continue the work, you start to pull out the Labs members. Then the engineering team owns the work and future extensions of it.

Danger zone: Lab members working in other teams’ codebases

When Labs team members are working, the nature of their work is that they may be doing extensions or experiments in other teams’ areas. This is especially hard because often the people that are on Labs teams are people who may have been around the company for a while, so they’re often comfortable doing so.

Labs team members must act like guests in the codebase. Even if they were the ones who originally shipped that code, they should go out of their way to make the team owning that code comfortable. Code ownership tends to create territorial behavior as a side effect. The team owning that code is responsible for it and any problems you introduce. There is a moral hazard in working on something and not maintaining it. So you have to be extra extra careful about this.

One trick is to take on-call for anything you’ve worked on for a while. Another is to spend time up front with the manager and product manager of that team, and make sure your presence is welcomed. Show you understand it may be a burden for them, or additional work for them, and that you’re going out of your way to make it as easy for them as you can. Sometimes you can work out arrangements where you have a member of the team you’re working with or who is reviewing your work.

Challenge: morale issues

One of the more common issues with a Labs function is that it can be seen as the most “fun” or “interesting” work. Why do they get to do all this work?

Engineering leaders need to decide what kind of operating model they want for their teams. I generally have a bias towards teams that are as customer centric as possible, and that have some ability to influence the production direction. Ideally, they understand their space better than anybody, and mostly control their own work. They experiment and innovate in their area.

A Labs function can be seen as being in opposition to this. You have an innovative part of the organization, and everyone else is just a feature factory. Working on bugs and well defined features.

This is a pretty large leadership topic, and I’m not going to cover it completely here, but there are some real choices and tradeoffs to be had.

One suggestion for Labs is that they not solve the entire problem space. Just the most dangerous and validation focused ones. Leave some things to be solved that are interesting and engaging. You want to smooth the terrain, but you don’t want to lay out the entire project plan. Think of yourself as the team that dynamites the way through the mountains, not lays the roads.

Danger: don’t become alternate engineering team for urgent projects

One temptation will often be to use the Labs team to deliver projects. If you see that happening, intervene early. If you’re doing that, you’re not really a Labs function, you’re a Tiger Team, and you should structure things completely differently.

Challenge: Skill requirements

You often want your most startup-like and entrepreneurial types in your Labs. Often it is a good fit for founding engineers, a principal engineer with the right skillset, or a CTO that has a lot of technical skills.

You want people that won’t be discouraged when an idea is shown to be a bad idea.

You want people that explore ideas rapidly.

Challenge: metrics and risk management

Your team should probably operate differently than the rest of product development. Work with your leadership to carve out a space that has different requirements.

You’re not shipping product, so you shouldn’t be subject to a lot of the same requirements as product engineering.

You’re doing exploratory work, so you shouldn’t be focused on how much revenue you’re generating.

If you do have to come up with OKRs or metrics, they probably should be more around experiment delivery, and satisfaction of your stakeholders. You might consider surveys rather than financial measures.

Danger: be very careful with how you invest across Horizons

A couple of investment dangers are:

• You can treat something that is actually a reasonable investment as unproven, and leave it too long in Horizon 3. If you're unwilling to invest in it, that doesn't mean it's Horizon 3. It just means it is an unfunded Horizon 3 projects. Jim Ruppert: "At New Relic we had several false starts on the Logging project. It lived at Horizon 3 for a long time when it was actually Horizon 2 all along but we hesitated to invest. If we had invested earlier, it would have contributed revenue earlier." Instead, we kept it like a research project even though the reason we weren't investing it might have made sense at the time.

Challenge: the pressure to give exciting things to customers

One risk with Labs teams is that when you talk with a customer, they may get excited by your exploration. That’s excellent – excitement is an important signal. But you have to be careful that they don’t expect anything you’re exploring to become product right away.

You might talk about it as a design partnership, but set the expectation that it may or may not become part of the main product.

You might also find internal pressure to give things to customers. We're doing all these exciting, cool things. Let's make them into a product! Be sure to fully vet the customer need. If your Innovation Lab hasn't fully vetted the customer need or the quality before it goes out, you'll ship something that will fall flat. I've seen this a couple of times with Innovation Labs.

Challenge: avoid innovation theater

If nothing you explore becomes product, you’re not generating value for the company. If everything you do is validated and a good idea, then you’re generating more work than will ever get done.

Because of this, it’s often good to talk with Product about how you’ll decide which ideas get on to the roadmap. Having a person within product advocating for it is great. But are vetted ideas funded as new teams, or given to existing teams? How often do we want that to happen? There is a lot of room for figuring things out there.

Governance and leadership

You want to be showing lots of experiments you’ve run, and share what you’ve learned. You want there to be active debate on what that direction should be.

You probably want to have some monthly or biweekly meetings with the CEO, Head of Product, and Head of Engineering. And potentially some other parties as well.

It can be good to demo your work to the rest of engineering or product. But you have to be careful to contextualize things. And you definitely don’t want the company selling your experiments as product.

Product in a year prototypes

One thing I’ve seen is collaborations with design, product, and engineering to put together a hand-wavy but directional prototype of what the product is evolving into.

This is useful when there is a lot of confusion about the direction the product is going. And it can provide additional clarity.

But it can also confuse if it’s not fully agreed upon as the direction. Often its desirable to have a lot of these possible prototypes. They should be clearly communicated as possibilities until you get alignment that they are the way forward.

Whether or not this is a Labs function kind of depends on your Lab. It may be more of a Horizon 2 activity, but the more uncertainty there is, the more it could be a Labs collaboration.

Is it worth it?

To me the main question is whether you can achieve the same results without an Innovation Lab.

One question is whether innovation is really the most important thing for your organization. Sometimes execution is more important than innovation.

If you have a high need for innovation, then the question is whether you can do it within your existing engineering and product structure.

If the need for innovation is high, then your best bet may be to do it both places: both within your existing Engineering organization, and ALSO with an Innovation Lab.

As you can see, there are a lot of potential pitfalls with an Innovation Lab. But they can generate extremely high value. Or be a complete waste of time. Hopefully this post will help you avoid some of the failure modes!

Thank you

I believe the work McKinsey based the Three Horizon model on was The Alchemy of Growth, by Mehrdad Baghai, Stephen Coley, and David White. The Podcast for the Three Horizon Model is here. My brain trust on Innovation Lab folks is Jim Ruppert and Jacob Groundwater. Jacob reviewed the post, and Jim provided a ton of useful insight, several points which became sections in this article. Thanks to Sam Alexander for helping me find and digest the podcast on the Three Horizon Model. Tim Krajcar provided some valuable suggestions and feedback, especially around handsoffs and failure modes. Karl Schmidt emphasized the importance of throwing away your prototypes. Susie Dirks helped me fill in the section on the connection with engineering and morale issues.

Image by PublicDomainPictures from Pixabay

People disagree. It's something that you expect.

• Disagreements can be beneficial. They can help bring up areas that people aren’t considering.
• Disagreements can be destructive. People can get set in different directions, and the friction can prevent forward progress.

I'd like to share an approach that can get people with strong disagreements to get aligned:

“First, agree on the tradeoffs”

What do people typically do?

People generally have their own approach to deciding on things:

• Arguers: Some people think arguing about things is how you come up with the best ideas. They believe it’s a duel. The problem is, this relies on everyone else engaging in the same way as they do. So they tend to be evangelical about arguing.
• Askees: Some people need to be asked to volunteer their ideas. They won’t speak up unless they’re asked.
• First ones: Many people latch on to the first idea that feels good enough. They do this because there is a lot of discomfort in not having a good plan, so once the first good enough plan is developed, they latch on to it. It resolves the tension, so it is the best plan.
• Perfectionists: Other people are perfectionists about plans. They have difficulty choosing, because they are always trying to figure out what the perfect plan would look like. For them, it is risky to choose, because you're losing options.

There is a lot of variation in the approach people take. And some approaches are incompatible with others.

I generally find that groups don’t explore the solution space very well. They argue about the wrong part of the problem, often aren’t even on the same page about the problem they’re solving, and often the solutions fall short.

The problem-solving approach I recommend

This is an approach I’ve found useful:

1. Come up with at least three approaches.
2. Outline the tradeoffs, and make sure we agree on the tradeoffs.
3. Decide who should make the decision, if it’s not clear and we don’t agree.

Come up with at least three approaches.

Groups tend to not explore the problem space very thoroughly. Designers have a lot of experience with this, and you’ll find they often divide their activities into "divergent" and "convergent" phases.

The first phase is divergent: you come up with a lot of possible approaches. You explore what is possible, and think less about constraints. The goal is to come up with a lot of variations.

Then, you aim for convergence: you weigh the solutions to find the one that best matches your constraints.

In engineering teams, I find that a lot of our troubles come from poorly exploring the solution space. We jump to solutions without fully considering all the alternatives. We get attached to a solution and run with it. This is so prevalent! I believe it's one of the most fruitful areas for improving projects.

We can come up with dramatically better solutions if we think about alternative approaches and challenge our assumptions. We can deliver things in much less time if we can come up with shorter, more efficient approaches.

Or as I like to say, “if we can deliver projects that are 50% smaller, we can double our speed without doing anything else”.

So the first stage is to get the group to come up with a couple of approaches. I usually suggest a minimum of three. Often, teams will not come up with three choices unless I ask for it.

As a leader, I sometimes am not as involved in the details as the team. But I am often way more experienced with software than they are. Leaders in this position may have good instincts about potential solutions. I have a lot of history to pattern match against. But I also might not be close enough to the solution to know why some approach might be inappropriate. Adding my favorite idea to the pile can be a way to make sure it is tested against other ideas, especially if I make it clear I want it to be critiqued.

Outline the tradeoffs

After the team has come up with a couple of approaches, then as facilitator, I have the team outline the tradeoffs of the various approaches. This can be done asynchronously or in a meeting.

What you’ll often find is the tradeoffs are between time horizons: short-term vs long-term. Or they are tradeoffs of completeness. Get a list of the tradeoffs, and what would make a solution better under which circumstances.

Make sure we agree on the tradeoffs

The next phase is crucial: you want to get the team to agree on what the trade-offs are.

This is important because you need to get the group to operate under the same shared reality.

Often, there are people that understand things that others don't. Or people might be seeing the problem in an incomplete way. Or they may not have even heard the problem statement correctly.

Unless everybody agrees on the trade-offs, you haven't done enough discussion to actually make a decision. If you're not operating under a shared understanding of the situation, how could you make a decision?

Another advantage of getting the team to agree on the trade-offs is it shifts the thinking of the team. Instead of having a bunch of people with positions, you switch to analytical thinking. You get a set of tradeoffs, instead of a set of positions. People think better when they’re being analytical.

A final advantage of this is that because everybody understands the trade-offs, they will be more willing to go along with whatever decision is made. They will feel like their concerns have been listened to, and you will have greater alignment about whatever decision is made.

I can't emphasize enough how important this is. You get alignment for free when using this approach.

Decide who should make the decision

When I'm facilitating technical decision-making meeting, the next step is to make a decision.

Often, it will be clear who the decision-maker is.

In some companies, you might have agreed-upon decision-making frameworks. For example, you might have "disagree and commit." You might have a consent-based approach where, unless somebody substantially disagrees, a particular person decides. Some companies use a single responsible person approach where a person is identified using some sort of criteria.

And there might be roles that get to make the decision. For example, the chief architect or tech lead is the person making the decision.

Even if that is the case, it can be good to try to come to some sort of consensus. But if there is a clear decision-maker, after all of the trade-offs are heard and people's opinions have been solicited, that decision-maker should quickly make the decision and you can move on to other topics.

If it's not clear who the decision-maker is, then you have to decide on how the decision is going to be made.

If you're facilitating the meeting, you can often ask a question that guides who the decision-maker should be. For example, if you think the decision should be made by the person closest to the problem, you can ask, "Who is most knowledgeable about this problem and best able to make the right decision?"

If you think the decision should be made by someone with the most business context, you can suggest something like, "Okay, we have outlined the trade-offs, but ultimately this is a business decision, so it seems like so-and-so should make the decision."

Farm for discontent: “Does anybody have any substantial concerns with this decision?” Make sure you have alignment. Sometimes people seem like they’ve agreed to something, but they really are just keeping their mouth closed.

Once the decision is made, record the decision somewhere public, and share the tradeoffs and context.

Note that this is often best communicated at the beginning of the discussion. Outlining how the discussion will go can be useful and prevent surprises.

Conclusion

You might call this a “reality-first” approach. You get people to align on the tradeoffs as a way of making sure we’re all seeing the same problem. But you’re also engaging in divergent thinking, coming up with a couple of approaches. Then you use the structure of “tradeoffs” to see how the solutions you’ve come up with map against the constraints you care about. Then you make a decision.

Let me know what you think.

Thank you

This post was based on advice I’ve been giving for years in my coaching and advising practice. I remembered it due to a wonderful post by Nadya Duke Boone on alternatives to disagree and commit. Which was in turn inspired by Molly Graham’s great post on disagree and let’s see. Paul Tevis pointed out that it's often best to clarify how the decision will be made before the discussion starts.

Image by kp yamu Jayanath from Pixabay

I once asked an executive assistant to help me be better at calendaring. Of course, she was a complete expert and pro at this, and I learned a lot. I'd like to share some of the things she taught me.

Calendar rules I learned from an EA

Note this whole article is based on Google Calendar, because that's what I've been using professionally forever. But many of these principles apply to other calendaring tools.

Don’t let two meetings overlap

There are a lot of ways to handle your calendar, but I recommend handling it "as invites come in".

One of the most basic errors people make with their calendar is they don't handle the fact that there are conflicts in their calendar.

When you get an invite to a meeting, if there is a conflict, you should communicate your intentions. Otherwise, when you DO address the conflict, the impact of that decision will hit other people at inconvenient times. For example, you might do it the day of, and then all the people who expected you to show up now have to handle the fact you won't be at that meeting. Rude! Or more accurately, disruptive.

This is metaphorically similar to merge conflicts in code. It's way better to keep your calendar integrated with others, than to produce a bunch of surprises at the last minute.

Color code your meetings by what kind of meeting they are.

One thing I noticed when the EA showed me the calendar for the CPO she worked for was that the calendar was color coded. When I asked her about that she explained that there was a whole system to it. The colors represented what type of meeting it was.

For me, this was a relevation. Ever since then, I've played with a couple of different schemes for color-coding my calendar. They have varied as my work has varied, and you should probably create your own schemes. But they can represent things like:

• Level of focus: your staff level, stuff within your org, stuff above your level.
• Which clients it is for.
• What type of problem the meeting focuses on.
• Whether the meeting is to help others, or deliver your own goals.
• Type of meeting: 1-1, operational, technical, people matter, etc.

Do what you find useful, and evolve it over time.

Consider a calendar budget

One thing she talked about with me is the use of a calendar budget. Essentially, you look at how the executive wants to use their time, and try to limit the number of meetings of each type to that amount of time in the week.

The executive she modeled this on spent some time at the beginning of the year figuring out what they wanted an ideal week to look like. And they created a budget for that. Stick that budget on a sticky on your monitor, and then color code the budget categories. As things comes in, start saying no to things that exceed your budget.

Now this does imply a level of control of your calendar that some people will feel like they don't have. But even if you only control a couple of the categories, that can still be useful.

For me, I like to be helpful to other people, but I can tend to take on more of that type of meeting than I probably should. So I color code those meetings, and try to limit them to 1 or 2 per week. If someone requests my help with something, I'll then sometimes be willing to schedule it, but ask that it be further out.

Schedule and block out lunch.

I'm surprised how many people don't do this. If you're flexible about the time for your lunch, you can make it a "habit" with Reclaim, and it will make sure you get a lunch within a time period you set.

Don’t show declined events.

People often leave declined events on their calendar. Simplify your view of the world -- take them off, and only add them back on when you're looking for them.

Calendar rules I learned from Bjorn

The first VP of Engineering I served under was Bjorn Freeman-Benson. He was very insistent that his managers follow some rules with their calendar. He drilled all of these things into our mind, and it became "the way" we all did things. It was a great early education into how to use a calendar well.

Start meetings on time.

The motto I've developed over time is that if people are late to a meeting, you want them to feel a little bad about it. If it hasn't started already and they're late, you're probably encouraging lateness.

This has to be led by example from people high up in the organization. I try to set it as an expectation within my organization.

Remote work does make this a little more difficult, because people sometimes have technical challenges. And there also is a greater need for chitchat and human interaction. You might decide meetings start at 3 minutes after?

Make your calendar entries editable by default.

If you need to change the time for an appointment, it's a hassle to go through the "propose another time" workflow. If they have access to your calendar, it's much easier for them to just edit the invite.

So set your calendar entries to be editable. You can do it on a case by case basis in the invite:

Or do it in your preferences (calendar setting, general, Event settings, guest settings -> Modify event) and have it default to that for all future calendar invites. I recommend making the default editable, and then turn off editable for meetings that are so large you worry someone will screw up the invite.

If you go on vacation, people can still edit the invites for your meeting. They can change the conferencing options, or add agenda items, or add a document to the calendar invite. Treat it more like a wiki.

Of course, this works as far as you can trust the people you're making the invite for. But honestly it's a better default.

Use speedy meetings

One of the reasons meetings waste time is that people have biological needs. They need to use the restroom or get water.

If you run your meetings right up to the end of the hour, you're making meetings as a whole less effective. Start on time, and end at :25 or :50 or :55.

You can do this by default by turning on "speedy meetings".

The more hardcore way of doing this is starting meetings at :05 or :35. I kind of like that, although people sometimes loose track of time or forget if it's at a weird time like that.

Calendar rules I learned from Reclaim

For several years, my favorite calendar tool has been Reclaim. In the course of using Reclaim, I've learned a few things about calendaring.

Sync your personal and work calendars (Reclaim)

Unless you want to manually sync events between calendars all the time, having a sync between calendars is just going to make your life easier. Reclaim is quite good at this. I've also used Calm Calendar for this -- it's a better choice if you only want to sync calendars as it's less expensive than Reclaim.

Schedule your work on the calendar

There is a lot of tooling you can use to put the things you're working on onto your calendar so you can plan out your time.

Reclaim, with it's Todoist integration, is my favorite for this, but I've used other tools. SkedPal was fine (although right now as I write this, the site is down, which is ominous). Trello has some new functionality for this, although you have to do it manually. And Todoist does as well.

What I like about Reclaim is that you put in your todo items, say how long they will take (with quick abbreviations like [2h] for two hours), prioritize them (p1 p2 p3 p4), add deadlines, and you can just forget about them. It tells you if you have anything that won't get done on time. As long as you do what your calendar tells you to do, you're fine! And it automatically updates as you move things around, add new calendar items, etc.

It's become my favorite way to organize my time.

Take advantage of the busy/free bit on calendar items.

You can schedule things on your calendar that other people don't see. Just mark the calendar invite as Free. This can be useful if you want to annotate your calendar but not have it show up as busy.

I also sometimes will mark something as Free if I am setting aside that time to work on a particular thing, but I'm willing to be interrupted.

Calendar rules I learned from colleagues Schedule a day after vacation as reentry time.

One thing I noticed Nic Benders doing was after taking a vacation, he'd take an additional day of vacation on his calendar so he could get back up to speed and get situated at work. He'd catch up on everything he missed, plan out his work, and generally get ready to go.

Thank you

I believe it was ARay who showed me how to level up my calendar game (I have a terrible memory, so if I'm misapplying credit, let me know!). Bjorn Freeman-Benson and Nic Benders have obvious ways they taught me things (as shown above).

Image by Lubos Houska from Pixabay

Conflict of interest

I've been recommending Reclaim for years, and it's one of the products I'd most be unhappy about losing. I worried when they were purchased by Dropbox, because I thought the product might change (so far it hasn't). They started offering affiliate codes recently (Dec 2025), so I'm going to link Reclaim with those links. I'd like to say that won't influence what I recommend, but of course people aren't as unbiased as they think they are. So I try to disclose that fact so you can take my recommendation with suspicion! From my point of view, the likely very small financial upside for me is really not worth harming my reputation, so if I change my evaluation of Reclaim, I'll take it down.

Things are changing pretty rapidly, so I have hesitated to write down observations that I know will get out of date quickly. But I have observed a few things I haven't seen widely discussed, so I'll risk it.

Slack could be your new data lake

We use data lakes to drive charts and analytics. We dump all of our company data in these lakes, and find a lot of useful ways to make better decisions with that data.

I'm really not sure if this observation holds water (let me know what you think), but it's possible that Slack will be the data lake for LLMs. It has a million integrations that allow you to feed valuable context into channels in Slack. LLMS are language models, and can use text in a lot of interesting ways, especially when augmented by tooling that supports selecting the right content.

This is a much better user interface for a data lake, because you can ask your company questions, and get summaries and responses back that are in your own language. And you get references to the source.

So my suggestion here is to not sleep on the summarization and AI features of Slack, and to really think about the consequences of having all your meeting summaries piped to Slack. All the work being done piped to Slack. Status updates, and discussions, summaries of decisions. You might want to enable a lot more integrations than you do now.

It's too soon to know how real this will be, but anecdotally, I've heard of one company that eliminated a significant amount of meetings. Most were information sharing meetings.

This doesn't feel like the right solution, and it may be a bad approach. But I think it's interesting. It kind of shows a directional way that communication and companies might change in the short future.

Interestingly, I think remote companies might be better positioned to capitalize on this that in person companies.

Think of all of us as beings, engaged in communication. Making decisions, and creating things. LLMs become a part of that ecosystem of communication. If most everything that is said within a company is written down, then we get some really weird properties of being able to be summarize, process, and interact with a much wider pool of information than we did in the past. In the same way that augmented reality is a hybrid of the computer world and the real world, LLMs could weave their way into the fabric of our communication within companies.

Your ability to sense your own company goes up significantly. Product managers probably have a very different role, because their ability to sense and integrate information in their companies is augmented. It all becomes "on demand".

I'm sure this will be terrible in some ways, but it will also be quite powerful in other ways. I'm particularly curious about how we'll deal with privacy.

My guess is that this will ripple through our communication patterns in companies for the next couple of decades.

Targeted status updates

I view status updates as a form of "compression". You take a complex state of a situation, reduce the information to the minimum amount you can communicate that gives the most essential information, and reduce the amount of complexity that other parts of the company have to engage with.

Status updates have typically been a difficult problem. Humans are pretty bad at status updates. It is a skill that takes a while to teach, and a while to learn to do well.

And it turns out LLMs are pretty good at status updates nowadays. In my current interim role, I write a weekly update to the CEO and a few stakeholders. Typically what I do is make a list of information sources (including slack summaries, metrics I care about with a few sentences of commentary, and meeting summaries), dump them into an LLM, and give it a good prompt, and voila, I have a pretty decent first draft. I had someone tell me they found my weekly updates intimidatingly impressive. (When they found out my method, I think they were relieved). It was doing a much better job that I would have. Or rather, it gave me a better ability to do a good job with it, because the first draft wasn't something I could have sent, but it gave me a better starting place than I would have had on my own. And it takes less time.

What I like about this I can also craft the prompt to my audience. I might emphasize delivery and quality to the CEO. I might emphasis cost implications and efficiency to the CFO. If I wanted to I could pretty easily craft a different version targeted to what each of my stakeholders cares about.

I do have to edit these drafts. Sometimes they're inaccurate or don't mention what I care the most about. Or they can be overly bland. But to be honest, they usually mention a lot that I would forget, or they emphasize the content more concisely than I do.

I can see status updates becoming more automated over time. Ideally, that would lead to people spending more time on planning and execution.

Automated verification is your accelerator

Standard XP and CI/CD practices are becoming even more high leverage. Investing in automated verification seems like the smart bet in 2025.

If your agent can get better context, or test its results as it works, it can keep going on a problem, or self-correct. It seems like this will be the thing that drives a lot of the productivity gains over the next year.

Developer workflows are even more essential: reducing unnecessary code review steps, improved linting, unit tests, integration tests, end to end tests. Synthetic monitoring, observability tooling.

The cost of evergreen

One thing that has surprised me about coding agents is that they're often pretty good at upgrading things. I've seen a number of times where upgrades are just much less expensive to make than they were in the past. I've even seen some pretty surprising examples of LLMs doing some tricky things during upgrades that I didn't expect.

So the calculous of how expensive software is to maintain is changing. This could result in a lot of legacy software being revitalized.

Thank you

Image by Pippontis Giuseppe Rubens Rubino from Pixabay

If you're in a growing organization, you will experience thrash mode. Thrash mode is when you are underwater. When you are too busy to do anything.

This post is not about prioritization. It's about how to get unstuck when you don't have the capacity to get unstuck.

My first experience with thrashing

Thrashing is a term used in computer systems, where your resources are overcommitted. The computer system grinds to a halt, because it spends all of its time dealing with the fact that it is overcommitted.

Something analogous happens with humans.

I first noticed this when as a manager I took on more and more responsibilities. I had 12-13 direct reports, several teams, and six or seven projects. It all seemed like a lot of work, but manageable.

But all the sudden, it wasn't manageable any more. Just keeping track of everything that was going on consumed all my bandwidth. I had very little capacity to actually do very much work. Even worse, I didn't have the bandwidth to actually make my situation better. I was stuck.

I thought this was a rare thing, but it kept happening to me. Over and over in my career, I've hit thrash mode and had to deal with it.

It's now become so familiar I have a playbook I run. We'll talk about that in a bit. But first:

Why to expect thrash mode

I view thrashing as inevitable. Why?

• Organizations are never 100% resilient. Managers will leave (permanently or temporarily), so someone will need to cover their responsibilities. Someone will get promoted, so their old responsibilities will need to be covered. These things happen.
• Hiring has lead time. So even when people realize they need to hire, they have to get approval, post the job, and find someone.
• Companies optimize to reduce costs, so they'll try to get the most out of the team they have. The work will take all available capacity, and then go until things start to break down and there is a counterpressure.
• People are inefficient in new roles. When you move into a new role or position, you won't be as efficient as you will be after you're more experienced with it. This means you have less capacity and less time, and are more easily thrashed.

I expect it will happen to you many times in your career.

My playbook for thrash mode Clear up some bandwidth

1. Clear your schedule. you might clear next week. Be harsh. Cancel every meeting. Or at least start from the assumption you'll cancel every meeting.
2. Cancel your 1-1s. When I said clear your schedule, I meant it. You may feel bad about it, but you're not being useful right now. You need to clear up some bandwidth, or you'll never get better.
3. Pretend you're on vacation. These three top items are really emphasizing the same thing, you have to clear up some temporary bandwidth. So put up a vacation message in your email. Mark yourself out of office. Do what you need to to clear up some bandwidth.

Figure out what gives you more bandwidth

1. Make a list of anything that will structurally improve your situation. Usually this includes: delegation and giving away responsibilities, finishing up things that aren't done, restructuring meetings to run themselves, big pushes to get something vital done like hiring. Focus the list on things that free up future bandwidth.

2. Track less things. Decide on things you're going to let go poorly. Decide on things you're going to reset expectations for. All of these are things that give you more bandwidth to focus on your priorities.

3. Redefine big things into small things. Sometimes you can just decide not to do a big thing for now. Do a smaller version of it. Delay it. Sometimes you can decide to not complete something, but just make sure it's always getting better.

4. Look at trends and structure. Sometimes I find that people miss that the long-term trend is running against them. Or that the structure of the situation is just not going to work. Sometimes you can use this freed up bandwidth to figure out how to untangle that situation. Do you need a new architecture, or a new self-service approach? Sometimes you can figure out what the solution needs to look like, and then pitch a solution.

My toolchain for thinking about less

My current tooling of choice is Reclaim + Todoist. I don't particularly care for Todoist, but use it because of its Reclaim integration.

What I do is enter my todo items in Todoist, with details about priority and deadlines. And then Reclaim puts these items on my calendar, to tell me when to work on them. As my schedule shifts, the items move around. And I can move them manually if I like.

There are two advantages to this setup:

• Once I enter something on my todo list, as long as I have the deadline and priority set correctly, I don't have to worry about when it will happen.
• I can look at a page on Reclaim, and know any commitment I've made that I'm not able to fulfill, and reset expectations.

This is an enormous reduction in the number of things I have to think about every day.

I share this because I find it useful, but you may be able to find other tools that do the same thing. If you do look into Reclaim, please use my link for it: Reclaim

Thank you

Image by Alexa from Pixabay

Conflict of interest

I've been recommending Reclaim for years, and it's one of the products I'd most be unhappy about losing. I worried when they were purchased by Dropbox, because I thought the product might change (so far it hasn't). They started offering affiliate codes recently (Dec 2025), so I'm going to link Reclaim with those links. I'd like to say that won't influence what I recommend, but of course people aren't as unbiased as they think they are. So I try to disclose that fact so you can take my recommendation with suspicion! From my point of view, the likely very small financial upside for me is really not worth harming my reputation, so if I change my evaluation of Reclaim, I'll take it down.

When your company gets large enough, you'll start having problems with dependencies. Here's how to address it.

What do I mean by dependencies?

• Teams not able to deliver their work without other teams doing work for them.
• Projects that can't be successful unless a lot of teams do work.

Signs you have a problem

• Big projects are always challenging and problematic.
• Lots of efforts to solve this through project management and better tracking.
• Lots of efforts to solve this through heroism.
• Lots of efforts to solve this through process.

The problem is that you have a structural problem, so none of these will work consistently. I write more about how people try to work around these structural problems.

All the things you can do

• Reorganize the teams to reduce dependencies. Generally favor cross-functional organization. Team Topology style structures. Or FaST approaches if you have a higher tolerance for experimental but promising approaches. There is a lot of nuance in this -- it's a fairly advanced leadership topic. You might have different models in different parts of your org, for example.
• Make your projects smaller and more incremental. It's easier to ship smaller things.
• Focus on fewer things. If you halve the number of projects, that means each projects gets that much more attention, project management, attention to detail, etc. If a manager is project managing four things, they're dramatically less effective that project managing one thing. Consider constraints on the number of projects per team.
• Define engagement models for your teams. I write about a lot of these in Coordination models. Be very explicit about the way teams work with each other. You might have a team page in the wiki, with a section on the engagement model for each team. Examples of engagement models: self-service, independent executor, objective expert. Moving all your platform teams to self-service is an example of a very good improvement.
• Define how to work across team boundaries. Away teams, open source model, etc. I haven't completed some of those posts yet, but have some detail in the independent executor post.
• Look at your bottlenecks. You always will have bottleneck teams. Not only are these usually the most problematic places in engineering, they also are often where a fix can have outsized impact throughout the whole organization.
• Have someone who explicitly looks at dependencies and org structure. If you don't have enough systems thinkers in the organization, it might be useful to make this more explicitly something that the right person is thinking about a lot. Dependencies are one of the main decelerants for your org -- make someone responsible for taking care of the org structure.
• Bring in consultants or advisors. This can go very well, or very poorly. But it can be a good idea to bring in outside expertise. Be wary of SAFe or other one size fits all approaches. I can help, or refer you to others.

Thank you

Image by Trond Giæver Myhre from Pixabay

I've been having some conversations recently on strategy, and have been reading some strategies (product, corporate, and departmental). I'd like to share a few observations of what is essential for great strategy.

Strategy should feel uncomfortable

Strategy needs to make explicit choices. Unless those choices are uncomfortable, it's probably not sharp enough. And thus it's not a fully articulated strategy.

A personal example of this was when I was leading up an innovative new analytics product. We had poured a ton of work into making a completely new type of analytics product possible. We were excited about what it would do for customers, and the business.

And then, a new product strategy came out. It made it clear that the focus was on the core product, and explained why that was important.

We had to ask ourselves: do we cancel this work? Do we delay this other product line too, which was about to be released? We chose to do so, and it sucked, but was the right decision. That product never saw the light of day, but we chose an approach that was better for the company. This was good strategy work!

There are some rare exceptions to this, when you're the first to exploit a new opportunity. But usually strategies should hurt. Or at least be uncomfortable.

Strategy should outline "the diff" against now

A strategy document should outline where you are at today, and where you need to be. It should articulate what the gap looks like.

It's common for early versions of a strategy document to describe a set of goals. You benefit by going further: where are we at today? Where do we need to be, and why? And importantly, what does that gap look like? This paints a stark picture that then is filled in with your prescription for addressing the path to success. And that path to success is the heart of your strategy.

If you don't outline "the diff", you won't create an argument for your path forward.

Strategy should tell a story

Great strategy is storytelling. It should explain and make sense of the situation.

For example, I imagine most product strategies this year are about AI. But these strategies need story-telling. "Times like these are times of disruption. If you look at previous disruptions of the past, that has been when great companies have emerged. For us to take advantage of the moment, we need to respond to this disruption better than our competitors. We have to make sure we're best positioned in our response. Here's what that means..."

Don't just say what you're doing, but paint a picture of a narrative. And the narrative ideally should be exciting or scary, like a real story.

More on strategy

I have a longer piece on product strategy that is a good companion piece to this.

Let me know

If you find this useful, or you have feedback, I love to hear it. And I'm happy to help with strategy work!

Thank you

Strategy should be uncomfortable was a distillation of something I had been thinking about, but was driven home by Molly Graham in a LinkedIn post. She in turn was quoting Claire Hughes Johnson, of Stripe, who said "Strategy should hurt".

Image by chiến nguyễn bá from Pixabay

Many engineering teams have sprint commitments. You want accountability, right? So you have the team commit to what they’ll accomplish during the sprint.

Mostly, this is a terrible idea: commit based approaches trade slower velocity for higher predictability.

If you ask me what I’ll accomplish in a week, I do an internal calculation:

• Am I being asked for an estimate, or a commitment?
• If it’s a commitment, the penalty of overcommitting is much higher, so I will hedge.
• I’ll agree to do less, because I don’t want to be penalized for not meeting my commitment.

A better approach

A better approach is to reduce the cost of commitment. Make it safe to agree to do more.

You do that by setting clear goals, and focus all of our wonderful creativity on how to achieve those goals and learn how to be faster and better.

For sprints, I often will have someone on the team put together a draft goal for the week, and then have the team talk about whether they think that goal is a reasonable thing they can demo, and whether the goal should be bigger or smaller.

Emphasize that the sprint is about your best guess of what you can accomplish by the demo, and that it’s okay to not achieve that. This gives a clear thing to work towards, but also reduces the need to hedge.

Preconditions

You have to have psychological safety for this to work. And, importantly, you have to be able to introspect and have a culture where people are noticing when we can do better.

If you don’t have an environment that supports this type of safe, honest appraisal of the situation, then you may have to fall back on a commit based approach. Stick to sandbagging, carefully deciding what you can agree to, and making sure you look good.

There are times when predictability is more important than velocity. In those cases, definitely go for a commit based approach. But I think most teams should be focusing on operating in a high trust environment, setting clear goals, and introspecting and continuous improvement to deliver the most they can each week.

This is a general principal

This is bigger than just sprint commitments. This is a general aspect of human behavior. The more you increase the cost of something, the less you'll get of it.

Thank you

Image by un-perfekt from Pixabay

When you’re trying to optimize the flow of work, be on the lookout for piles.

Examples of piles

• Piles of bugs.
• Backlogs full of stuff.
• Unmerged engineering work.
• Piles of work waiting for QA.
• Work that needs to be reviewed.

Piles, piles, piles.

Why focus on piles?

Product development, and many forms of work, become more efficient as they become more continuous. Piles often point to where the inefficiencies in your system are. So every time you notice a pile, note it carefully as a potential bottleneck of the team.

Theory of constraints

The theory of constraints says that any manageable system is limited in achieving its goals by a very small number of constraints.

It is essential to optimize the part of the system that is a constraint. Otherwise, you risk making the problem worse.

Think of it like traffic. If you have a tunnel that is constraining traffic, adding another 3 lanes of traffic leading up to the tunnel isn’t going to make your traffic better. In fact, it will slow things down, because the tunnel bottleneck will become more acute.

Focus on the tunnel. Focus on your bottlenecks. Focus on your piles.

Agent based coding and piles

I'm seeing this in many places that are adopting agent based coding. For some reason, it's not speeding up the delivery of the team that much? Why?

Although speeding up coding seems desirable, it may not be your true constraint. Or it may shift the constraint to code review or some other place. Focus on your constraints.

Podcast on this topic Thank you

The book that helped me be sensitized to piles was the fabulous but very challenging book, The Principles of Product Development Flow.

Image by Gidon Pico from Pixabay

Work using the campsite rule: "Leave it better than you found it".

What is the Campsite Rule?

The basic principle is: "Leave it better than you found it". The idea comes from camping, where you should clean up your campsite after you're done, so that the place is in better shape than when you arrived.

The Campsite Rule is necessary because otherwise things deteriorate, and nobody can enjoy the camping spot.

Why follow this rule?

Like camping in nature, most things in companies get more complex and worse over time. Unless you follow the Campsite Rule the world gets worse. It will get harder to get things done.

Another reason for the Campsite Rule is that people often go off the rails when they try to fix problems. I often see engineers spend days on relatively unimportant work, because they get frustrated by the space they're in. The Campsite Rule gives you a standard so you know when to stop.

Balancing your time

The campsite rule means you shouldn't underinvest in making the space you're working better.

But it also means don't overinvest in making it better.

What does this look like in practice?

So for example, if you come across a part of the codebase that doesn't have tests, what do you do?

You have several options:

• You could rewrite the whole code to be more testable.
• You could write tests for the whole thing.
• You could add a test.
• You could add tests but timebox it to half an hour of testing.

In general, the guiding rule should be: leave it better than you found it. You also shouldn't leave it worse off. So any of these options could be reasonable, but likely the first one or two would be a stretch. The third option would be totally reasonable. And the last might be best of all.

Beyond code

This principle extends beyond just code. It can apply to anything you are doing:

• Documentation - Is the documentation better than when I started with it? When you're working in a wiki, is there a small improvement you can make before moving on?
• Process - Is this week's meeting a little better than last week's meeting?
• Team dynamics - Are we communicating better this week than last week? Do we understand our customers better?
• Tools and systems - Have we automated things a little more? Are things a little better than they were before? Is it easier to understand than it was when you started?

Improvements compound

When everyone on a team is applying the Campsite Rule, things get better and better over time. The codebase gets easier to work in. The meetings get better and better. You ship faster, and have more fun. It's exhilirating.

It's not about making everything perfect. It's about making everything a little bit better, consistently.

A few tips for the Campsite Rule

1. Red, green, refactor - Do the main work (make it green), then do a little cleanup until it's better (refactor). Then move on.
2. Timebox - Don't let the cleanup phase drag on forever. Timebox it, and be willing to throw away the work if you run out of time. Sometimes, you'll fail at the Campsite Rule, and that's okay.
3. Big things should be projects - If something is big enough that it violates the Campsite Rule, either get it in the priority list, or don't worry about it.

The Campsite Rule isn't about heroic efforts to fix everything. It's about sustainable, incremental improvement that makes your work environment better for everyone.

Thank you

I learned this too from Jim Shore.

Image by Kanenori from Pixabay

What do you do when you don’t have the authority to accomplish your goals? To answer that, I’d like to talk today about a concept I called Reflected Power.

What is Reflected Power?

The main idea with Reflected Power is that you use the power of the people above you. You reflect off of their authority. This allows you to proxy the power of the people further up in the organization, to achieve your organizational goals. You are basically using the power you don’t have as if you did have it.

I’m sure a lot of people use this approach, but I never had anybody really articulate it to me. So I had to figure it out myself. So today I hope to go into some detail about how to use Reflected Power.

Direct actions and Influencing Actions

When can use you Reflected Power? First of all, you have to identify that it is a situation that requires it.

Diana Larsen taught me a wonderful way to break this down. I’m definitely paraphrasing, but here is how I remember it:

Can you directly control the situation?

• Then take direct action.

Can you influence the outcome?

• Then take an influencing action.

Is it outside of your influence and control?

• Then don’t worry about it.

Influencing actions are a different class of actions than direct actions. Usually people can influence more than they realize.

Some direct actions might be:

• Create a plan and share it.
• Prioritize the work.
• Ask someone that reports to you to work on something.

Some influencing actions:

• Suggest to someone that they prioritize something.
• Share an observation with someone.
• Ask someone for help.

Influencing actions are actions for things not within your direct control. But they are within someone else’s direct control. Reflected Power is an example of using an influencing action.

So when you see that it is outside your direct authority, but someone in a position of authority could be influenced to lend you their authority, that’s Reflected Power.

How to use Reflected Power

The basic move is just to ask them.

Here are some examples:

• You can ask your manager to kick off the project publicly, so everyone has the organizational context, and your project is “blessed” as a priority.
• You can ask them to give you a temporary role or title.
• You can ask them to show up at a meeting to show it’s important.
• You can ghost write an email for them, and ask them to edit it and send it off.

If you lack the positional authority for something that is important to the organization and your manager, think through some options, and ask them.

Cost of using Reflected Power

The leader you’re reflecting power off of needs to be aligned with you and view this as important.

Ideally, it isn’t a lot of effort, and they will feel good about doing something to help make the situation better.

It does need to be clear that it is a good use of their time. And the greater the amount of work you’re asking of them, the more important it needs to be.

Do be careful not to overuse this. It’s a valuable tool for your toolbox, and something that can make you way more effective. But there is a cost to it.

Hierarchical thinking can limit us

Sometimes we get stuck in hierarchical thinking, and we think of delegation as something that always flows downhill. It can be a useful perspective to imagine the world from your boss’ perspective, or from the CEO’s perspective. Often you’ll notice actions that others can take that will drive the organization to benefits you aren’t able to do directly.

A lot of people either don’t ask their manager for any help. Or, they give their managers problems to solve for them.

In between these two extremes is a lot of room. Reflecting Power is often a lower level of effort from your manager, but allows you to solve a lot of the rest of the problem yourself. This can often be a much better approach.

Can be done with peers

Let's say you are a head of product and your engineering counterpart is just really stuck in the weeds. They’re dealing with a lot of fires, and haven’t put together the plan for the architectural and technical changes that need to happen over the next year.

I wouldn’t do this unilaterally or in a sneaky way. But let’s say you’ve seen that engineering concerns tend to fall to the wayside and become surprises throughout the year. And you’d like to see a little work done on this.

You can and should talk with the head of engineering about that. One thing you could suggest is that you two do a joint presentation on the high level themes of the year, in a month. And you can work together on that presentation.

This is actually delegating laterally – you’re giving them a lot of work to do. But you’re partnering on it, and you’ve hopefully given a lot of context as to why it’s important and worthwhile.

Other forms of this might be that you’d like some help reinforcing a message to another part of the organization. You can ask that other leader to reflect that to their org.

Can be done with other leaders

Reflecting Power isn’t restricted to your own manager. You can reflect power off of the CEO, or anyone with authority or who has the respect within the organization.

One of the more brilliant examples of this I saw was when Jim Shore came in as a consultant to New Relic. He surveyed the organization and asked the product development org members who they trusted to represent them in a large organizational change. He then brought that group in as the steering group for those organizational changes. He was using reflected power – all of our reputations, in order to make the organizational change more likely to succeed.

Can be done in a networked way

If you’re trying to influence the organization in a particular direction, having a couple of supporters for a direction means it will have a large chance of happening. Let’s say you’re going into a meeting on Friday with a bunch of points of view of the direction forward. If you can talk with a few people ahead of time, and ask them to speak in favor of a particular direction at that meeting, you’ve tilted the outcome significantly.

So you can approach this in a networked way. Perhaps asking your boss to give some valuable context, and asking the Chief Architect to talk about the need for a simple solution. Doing this work ahead of time can result in your driving outcomes even though they are outside your direct control.

Creating space for others to use your power

When you’re in a position of authority, you can also make space for people to use your power. You can even encourage people to delegate work to you. (I’ve used this for equity work in the past)

Explain the types of problems you’d like to see delegated to you, and the ways they can do so. Giving you problems, versus asking to use your authority. Give concrete examples!

Conclusion

So that's the idea of reflecting power. Let me know what you think!

Podcast on this topic

A podcast version of this post is available on Decoding Leadership:

Thank you

Craig Fecteau suggested this topic, a long while back. Thank you to Diana Larsen for the concept of “influencing action”. Thank you to the unknown person who shared the cover photo on Pixabay

I wanted to share some resources for product managers. Especially to highlight communities that product managers can participate in. This list is less complete than other lists you'll find (such as this example), and something I'll update periodically.

Online communities

• Lenny's Slack Community (paid).
• Rand's Leadership Slack #product-management channel. They also have a #product-management-donut to randomly pair you with other PMs to talk shop and pick people's brains.
• Women in Product
• You can often find others to form a local group of people to meet up with. You might go to a community like MindTheProduct to try and find local people to network with.

Newsletters

• Lenny's Newsletter
• John Cutler, also on LinkedIn. More about the nuance and complexity of product management.

Podcasts

• Lenny's Podcast

Articles

• Product management start here by Marty Cagan

People

• Marty Cagan author of several books on product management.
• Teresa Torres well known for product discovery topics. Author of Continuous Discovery Habits.
• Shreyas Doshi

Books

• Good strategy, bad strategy
• Competing against luck

Feedback

Let me know if you think I've overlooked anything major.

Thank you

Image by hartono subagio from Pixabay

Most leaders do not seem to be very good at seeing the long-term impact of things. There was a time I was very much in that camp, but I've learned a few techniques for seeing things more clearly. I'd like to share one of those techniques today. This technique can help you see the long-term impact of something.

Inputs and outputs

In leadership, you mostly hear about the "output" of something. You hear about problems. But you don't hear what caused that problem to occur. And specifically, you don't hear the dimension along which the problem is based.

So you will hear about problems like these:

• Support tickets are crushing the engineering team.
• We're spending too much time maintaining our software.
• The security team isn't growing fast enough to keep up with demand.
• Our reliability is getting worse and worse.

Let me describe an approach for identifying the "input" and doing somethign about it.

How to analyze long term trends

The step to take when looking at a problem like this are:

(1) Identify the largest driver to the problem. This is the "input". There can be multiple inputs, but try to look for the largest ones. And choose one that is varying right now. So for example, you might be hearing that the engineering team is spending more and more of their time with support tickets. The question is, what is varying that is making the support tickets increase? Is it that we're having a lot of new customers recently? Is it that we're releasing a lot of new functionality? Or is it that we are doing something else differently?

The varying part is important, because you're looking for an "input" that is driving the output. Let's say in this case, that it seems to be due to a lot of new customers. Customers are growing 10% a month, and the team's support load seems to also be growing proportionally. Maybe not 10%, but it seems to be varying depending on the customer growth.

If you can identify that, you've identified your "input" -- the largest driver to the problem.

(2) Look at the long term implications of that trend. The next step is to think about if that trend is going to continue. Are there counter forces that will make things better? Or does it continue like this?

I can't tell you how often I find that the answer is, it is going to continue like this. You should be at that moment thinking, "oh shit". Because you've just identified a "time bomb".

This part of the analysis is pretty important. You basically are trying to determine: is this going to get better, or get worse, over time?

(3) Look at how much time there is left on the time bomb. The danger with time bombs is that they're often not urgent. And they're often not visible. This means they often are easy to defer or ignore.

Figuring out the time horizon for the problem helps you get clear on how responsible you're being to defer it. Either way, you should have a plan or a way to track the time bomb, or you're neglecting your job as a leader.

One thing about deferring time bombs is that if you're doing that in one place, you're probably doing it in other places too. So a culture of deferment can result in all of your time bombs hitting at the same time.

(4) Look for two kinds of solutions. Even if you decide to defer it, it can be useful to have a high level thought on what type of solutions you might employ. I usually look for two types of solutions: (A) a solution that fixes it forever, and (B) a solution that merely gets you to a situation where things are always getting a little better. If it's always getting a little better, you pretty much don't have to worry about it. But if the gap between these two types of solutions isn't major, you might want to bite the bullet and solve it for good.

Example with software tests

Let's look at a few examples. The first example is you hear from a new team member that the software test suite is painfully slow, taking 20 minutes to run each time.

(For context for those with a less technical background: in software, developers write tests for their code. As they add more and more features, the library of tests increases. As you increase the number of tests, it takes longer and longer to run, making this a very common scenario).

Let's go through our steps:

(1) The driver in this case is the number of tests. There could be other factors, like tests that take too long. But the thing that varies the most is the number of tests, leading to the time it takes to run the tests to increase over time.

(2) Long term implications if the trend continues, the test suite will take a greater and greater amount of time. You can look for some evidence of how quickly it is growing. (A good activity might be to add this as a key performance indicator (KPI) you are tracking, so you know if it's getting better or worse over time, and can keep an eye on it).

Experienced engineers will tell you that there is a significant difference between a 10 second test suite and a 1 minute test suite. And a significant difference between that an a 10 minute test suite. It changes the nature of the work so much that it can transform the way you work.

(3) Time left This is very much an important but not urgent problem. It is not a time bomb that explodes at one point, but something that adds increasing friction over time. The effect on the team will be easy to underestimate. And it is a problem easy to ignore. But clearly ignoring it is irresponsible -- the trend is that you'll make engineering slower and slower, and that it will combine with other problems to make your whole organization worse off.

So it's not something that has to be done immediately, but it shouldn't be deferred very long either.

(4) Two kinds of solutions present themselves:

For a solution that makes sure things are incrementally better, you could put in place a rule that the build will fail if it takes over a certain amount of time. And you can slowly ratchet that down over time. Disruptive, yes, and maybe not the best solution. But it's an example of a "slowly get better" solution.

A "fix it forever" solution might be to run your tests in parallel. If you want to run more and more tests over time, you can divide it up into more jobs, and run more jobs in parallel. It might get more expensive as you run more jobs, and there might be some limits to how much you can divide them in parallel, but you should be able to run them at a fairly constant rate.

Looking at the difference in the amount of time it will take to do each of these things, the incremental solution is probably something you could do in half a day. But it will result in a lot of build failures, so it will have a higher long term cost. The fix it forever solution might take a week, and will result in some tests needing to be rewritten, but the savings will be pretty constant in the future.

This is a simplified view of the tradeoffs, but you can see how you might make a tradeoff on this, package up a project around this work, and make sure it's scheduled.

Example with security team not growing fast enough

Let's say you're working at a place with high security needs. You're talking with the manager of the application security team. Her team's job is to review the code from engineering and make sure security problems aren't being introduced.

She complains to you that they're swamped, and that they aren't hiring application security engineers as quickly as the rest of the organization.

You can mentally do these steps in your head:

(1) The driver is the number of engineers writing code, because as that increases, her teams' work increases.

(2) The long-term implication is that her team will get behind, and be ineffective at identifying security vulnerabilities. Ultimately, they won't be able to fulfill their function. If the company has high security needs, then this is a pretty bad situation.

(3) The amount of Time left depends on how quickly we're hiring, and how much her team is falling behind with the growth in engineers. This may not be a linear thing -- it may drop off dramatically at some point. You can ask her some questions about how it's evolved to get a sense of this.

(4) Two kinds of solutions, fix it forever, and incremental.

The root problem is that her team isn't able to keep up. This can be solved through headcount, or a better strategy for her team.

Ideally, an application security team would be increasingly effective over time, so they wouldn't need to grow at the same rate as the rest of engineering. The fact that they are probably speaks to a lack of automation. So getting better tooling, or changing their relationship with the rest of engineering might be the best approach. For example, they might employ tooling, and good partnership with engineering, and training, so that the rest of engineering takes a lot of responsibility for security, and gets better visibility into security issues. This is a high leverage approach, so I'd call it the fix it forever approach.

An incremental approach that would make things work would be to hire at a ratio of the rest of engineering. I think this is less than ideal, but it might be the stopgap solution until the more permanent solution is in place. The key issue here is to troubleshoot hiring, and make sure she has the right amount of headcount.

My suggestion to her would be, make the time bomb visible, show how you're addressing it. Make a plan that shows you're going to slow the growth of your team versus the rest of engineering. Ask for additional headcount to fill the gap until a solution is in place. Make it a business decision: "do we want to accept the security risk for this until a better, more scalable version of my team is in place, or do we want to increase the headcount by X amount?" This becomes a tradeoff decision.

Some tips on using this technique

The general thing I'm going for with this approach is to assess a situation, look at the long term implications, and think about whether it will be a problem or not.

Questions I ask myself are, "is this getting better or worse over time". If something makes engineering slower over time, that's a really bad thing. You want all of your long-term trends to be heading in a positive direction, so that your flywheel of value delivery is getting faster over time.

One thing I've found useful is to often add a comparison. So for example, if AWS costs are rising quickly, the input might be the amount of customer data coming in. The important question to answer is, "is the rate rising sustainably, or unsustainably". You can often look at whether the growth is above a line or below a line. In this case, are costs rising faster than customers are paying us? Or below that line?

This is the basic sense you're trying to build as you learn this skill. It takes some practice, but after a while, you'll have this new sixth sense, and notice problems that nobody else is thinking about yet.

This can also change the way you relate to other departments. You can talk with finance about the rate of growth, and the relative priority of profit margins versus capturing more market share. This makes you into more of an executive.

Talking about time bombs

The only thing I find unfortunate about building this skill is that it can be thankless. Yes, you'll feel very smart when you see problems other people aren't seeing. But a lot of time this means you're fixing issues nobody is aware of.

One thing I've found useful when talking about these type of problems is to come up with a really succinct phrase that gets at why it's important. Sometimes time bombs are existential, that's perfect. For example,

"Our AWS costs were rising at such a quick rate that if we don't address it, we'll be losing money on every customer within two years".

Something like that can really help people see why it's important. Another example might be:

"Support tickets are rising at an unsustainable rate. At the current trend, engineering will be completely focused on support within 18 months, and not delivering any new value to the business". Holy moly, that got my attention.

Other approaches

Molly Graham has a great rule of thumb called the "10x the problem" rule. The idea is to ask how bad the situation would be if you had 10x the scale (number of customers, number of tickets, amount of work, etc). Rules of thumb like this are an excellent way to help you explore long-term consequences.

Feedback

I hope you found that useful. Let me know your experiences as you try this skill out!

Thank you

George Sudarkoff and Linda Nguyen both found typos in the post and let me know about it. Really appreciate that!

Image from Pixabay.

Sometimes organizational boundaries make it hard to coordinate. There are too many leaders involved, or you have various groups that can’t align their work or activities. In this case, you often want a Task Force.

When you talk with older leaders, they’ll often talk about “putting the people in the same room” to work together. Task Forces are the equivalent of that, optimizing for high quality communication. You form a temporary team that lives for the duration of a project.

When to use Task Forces

Task Forces are always for cross-functional work. They are for urgent or important work.

Use a Task Force when you know it’s more important than the alternative, and you’re okay with the waste and chaos that it might cause. In particular, you should think carefully through the tradeoffs. You’re probably going to underestimate the disruption you’re causing. These are especially acute for full time Task Forces.

So think through these concerns:

• Task Forces disrupt roadmaps. They usually take some of the most experienced and capable people from projects, and move their focus to something else. This means all of the other roadmaps will be disrupted.
• Task Forces are often high pressure. While many people like to work on something that is perceived as high value, they may not appreciate the pressure. It also creates pressure on the team they came from.
• You can have issues with boundaries. Teams that have high ownership boundaries will feel like their boundaries are being violated if they need to support the work after the Task Force leaves. If you’re encouraging “high ownership” teams, this is directly opposed to high ownership.
• You’re adding work in progress. Although you’re locally creating more focus, you are adding to the total amount of work going on. Instead of focusing on less, you’re doing more.
• Participation in Task Forces can be political. If someone is selected for a Task Force, that often looks like they are “high performing”. Depending on your work culture, this can create a lot of morale issues.

If you see your organization always reaching for a Task Force, you might want to look deeper at why the Task Forces are necessary. There can be other issues, such as a poor organizational design, poor coordination and communication, or other issues. (Let me know if you want my help figuring it out!)

Different types of Task Forces have different levels of disruption. Setting up a weekly meeting often isn’t disruptive. But sometimes the Task Forces are a good signal of where other communication or coordination is lacking.

Examples of using a Task Force

To give you an idea of how Task Forces can be helpful, I’ll tell you two examples of using a Task Force.

A full-time Task Force to deliver something quickly

The first story is from earlier in my career when I was a VP of Engineering at New Relic. New Relic had waited too long to introduce distributed tracing. You don’t need to know what that is, but the end result was that the product was starting to feel stale compared to competitors.

Six weeks before our user conference, we decided to address this deficiency: we would demo a working example of distributed tracing. **This meant we had six weeks to deliver a working version of distributed tracing. **The problem with this was that historically this type of work had taken much longer than six weeks. We didn’t have a detailed plan for how to implement distributed tracing. The work had to be done across several teams. We needed to add “agent” support, APIs for the data, and data ingest. Each of the teams had highly different skillsets and quality concerns. For example, the “agent” team had an extremely high quality bar, because they could break the software of our customers. Each team also had their own roadmap and leadership. Coordinating work across these teams was complex.

With work like this, you often see a lot of integration pain. Technologists often like to take a problem like this and decompose it into separate problems that each team solves independently. They see the challenge as figuring out the boundaries between each team, and having each team work on their part of the project. Put them all together, and voila!

The problem with this approach is that it usually doesn’t go as you would expect. Invariably, you find that you’ve missed something. And your understanding of the problem evolves as you build the solution. I’ve almost never found the “decompose the problem” and integrate approach to work. Frequency of communication overwhelms speed of execution, every time.

I considered that approach. One variation could be to have a strong program manager leading and synchronizing between all the subprojects. With a lot of attention to detail, that could work. But it felt risky.

Another solution I thought about was to take one of the teams where most of the work was going to be, and steal a couple of people from other teams to work with them. This was pretty close to doing a Task Force, but the problem was the team was already a pretty large effort, and there was a way that this didn’t feel urgent enough. It kind of felt like business as usual, except they were working on a new project, with a few guests.

I was talking over this problem with Belinda Runkle, and she suggested creating a Task Force. Instead of creating this out of existing teams, cherry pick a team to focus exclusively on this project. Clearly, that was the solution.

We started out by figuring out who should be on the task force. And then I proceeded to craft what was the best kickoff meeting I’ve ever run. I had a vacation immediately after the kickoff that I couldn’t reschedule. My problem was: I have to kick this off so well that it will work without me for a couple of weeks. I put together a solid agenda and practiced the heck out of the content. And it worked, even better than I expected.

We spent the day going over what the market was like, what our customers were looking for, what we needed for the conference, and how to think about tradeoffs over the coming weeks. We talked about where we were going to sit, how we were going to work together. And we walked out of the meeting with everyone 100% on the same page about what we were doing.

The result was impressive. Our CEO demoed the feature at our conference. It went flawlessly, and our customers were excited to see the results.

Obviously the team did the work, and delivered the results. I can’t claim credit for all that work. But setting it up correctly was part of the success of this work. I credit Belinda for thinking of the right way to structure it, myself for the kickoff, and the team for the great work that delivered so well in such a short time. I think of it as some of my best management work.

A part-time task force to solve an overwhelmingly complex problem

I had another occasion recently when I saw a Task Force be employed, to very good results. Kaari Casey introduced a Task Force for a topic that a lot of leaders had been talking around. As soon as I saw it introduced, I kicked myself for not doing it myself.

I was in an Interim VP of Engineering and Product role at a company. They had recently gone through a merger. This meant that there were duplicative processes and technology across every department. From a technical perspective, we knew that we wanted to migrate everyone to one of the technical platforms. But we also knew this wasn’t going to be easy: there were a lot of processes and workflows that our platform didn’t support.

The technical work was inseparable from the process work. Each department needed to merge its processes, and some of that depended on technical work to support them. Yet the technical work would go a lot slower if I as product leader was directly leading the process changes. It needed a more coordinated approach.

Kaari came to this conclusion before I did. She said, “Clearly this is a big discussion, and we’re going back and forth about this. Let’s set up a weekly meeting to hammer out the plan and coordinate our work”.

She set up a great agenda, and in the first meeting we all walked out with action items and next steps. In retrospect, this seems completely obvious, but nobody else was doing it. I didn’t.

So that’s the beauty of a Task Force. You assemble a temporary team to solve a particular problem. It’s a valuable tool to have in your management toolbox.

When using Task Forces

• Decide whether the Task Force should be full-time or part-time. Usually part-time Task Forces are some sort of regular meeting. The downside to part-time Task Forces is that they can take a long time to produce results. Consider whether scrunching it to a day long meeting or offsite is warranted. If it’s urgent, that can produce results faster.
• Think through who will own the work. Because the team you create is temporary, it may not be clear who owns the work after the Task Force is done. Who will support the work afterwards? Can you make that clear? It can be good to think about future risks as well. For example, if the work is software, who will help out if there is an outage?
• Kick off the Task Force like your life depends on it. You’re creating a disruptive Task Force that is taking very talented people away from the valuable stuff they were already doing. Make it worthwhile! Kick off the Task Force well. Use an offsite if you can. Make sure everyone walks out of that kickoff meeting on the same page.
• Borrow legitimacy to bless the task force. You can sometimes get the Task Force blessed by someone higher up in the hierarchy. You might have them start the kick off meeting. Or you might have them bless the effort. Task Forces can be a sneaky way to use the resources of other departments. But other departments might feel bad about it if you don’t get the effort stamped as important.
• Select the right participants. Make the Task Force as small as you can, but make sure all fo the right people are there. I spend time thinking about who should attend, and test my thinking with other people.

Task Force versus other coordination models

These are some closely related models to the Task Force that you might consider:

• Merged Group is when you merge two groups together that were previously separated (like merging an operations and engineering team, or a frontend and backend team). This is a full time change, and makes sense when there were previously handoffs or dependencies. Often a Merged Group is then divided again along different lines, if it is too big. Merged Groups are more permanent in nature than Task Forces, but you might consider a Merged Group if your Task Forces are actually pointing in the direction of “vertically sliced” teams.

• Away teams are when you use your team in another team’s territory. They’re useful when the other team is too busy to help you out. Away teams are more for handling dependencies than for better coordination.

• Tiger teams are permanent Task Forces. They are similar in that they usually have a defined purpose. But since they are (fairly) permanent, they have an explicit “go anywhere” mandate.

Coordination models

Task Forces are one of many Coordination Models. I’ve written these up as a pattern language for coordinating humans in large groups.

See also

I had a Decoding Leadership podcast on Task Forces. A Youtube version of that is here.

Feedback

I think that's all I have to say today about Task Forces. Let me know if you have any questions!

Thank you

Belinda Runkle suggested the first Task Force. Kaari Casey spun up the second Task Force. Juan Ignacio Sánchez Lara helped me flesh out the section on the tradeoffs of Task Forces. He suggested a number of factors I hadn’t thought of.

Image by Dimitris Christou from Pixabay

I had a great conversation recently with Katie Wilde about the reliability space. (You can see or hear the conversation here). She made a wonderful observation about reliability work: all of the incentives in the reliability space are wrong. Doing the right thing in reliability is almost always the thing that isn't rewarded.

In the discussion, I found some lessons and techniques that I think are worth sharing, not just for people doing reliability work, but for any leader dealing with misaligned incentives. (That's a lot of leadership!)

People don't generally value reliability

When I was new to management, my first hire was a person to do IT work for the company. After he had been around for a while, making a lot of improvements, the CFO came to me and said: "Why do we need to have an IT person? We don't have any problems with our computer systems. We may have had a lot of issues a while ago, but they seem pretty good right now!"

I said, "Exactly". The IT person had made the problems go away, so the perception of his work was that it wasn't necessary. But that was actually a sign he was doing excellent work.

Reliability work is like that. You're making a class of problems disappear. And they only seem important if they don't disappear.

A lot of people in the reliability space complain about how nobody values reliability. Nobody values quality. They're kind of right. It's a natural bias. Like Katie said in our podcast, "people don't generally value reliability that much unless the site is down, or things are really bad."

I noticed this when I was leading product and engineering at Gremlin, a company that is all about being proactive with your reliability work, through the use of chaos engineering. The space was a lot harder to compete in than I expected, because honestly most companies don't want to focus on reliability. They view it as a waste of time.

So this means that if you want to create reliable software, and solid experiences, you're often fighting with the perception that it's not important. It's actually not important until it becomes the top priority when things go sideways.

Make it interesting, and intrinsic

Both Katie and I have approached this in similar ways: focusing on appealing to people's intrinsic motivations: their desire to learn, and their pride in their work. But Katie had a new approach I hadn't used, that I think is brilliant: make it fun and interesting!

Her approach is to make the incident review meeting and post-mortem writeup so interesting and spicy, that everyone wants to see it:

"It'll get very real and kind of juicy, I would say. So then people want to come because they want to see the things that are going to go down at incident review. And so they do come and then I do a write up, which I try to make approachable and amusing. Did you know about this login functionality that we have that broke? Well, we didn't know about it either! I try to make it really fun."

As a result, she's seen a lot more engagement in the reliability work: "I've seen people actually want to join because they're going to be where the action is. We're going to see things at scale and see things break at scale. And it's just going to be really interesting and fun, for some sort of perverted sense of fun."

I'm certainly copying that approach. The other thing that I've found work is to talk about what the work requires of us. As we work at greater scale, we have to grow and be a different sort of engineer -- to grow our skills and do things in a way different than what we're used to. I've found that if you paint a believable picture of where we're going, the whole team can get on board with that vision, and start to own it.

Heroics are also misaligned incentives

Another area in the reliability space where incentives are misaligned is with heroic behavior. You can build a dependency on your heroes. And your heroes will feel great, because they will step in and save the day every time.

This doesn't create a reliable system -- it creates a dysfunctional situation that is fragile. Yet all the incentives are to continue: we all praise the hero for saving the day. And when there is an emergency, of course you want to call the person most able to fix it, right?

The obvious thing to do is to fix the system so it doesn't rely on the hero. Cross-train people. Put the hero on the second line of paging, so other people have to learn it too. Do Gamedays so people have to practice.

But one of the things I've struggled with is how to show appreciation for heroism, without encouraging it. I've handled this in the past by contextualizing my praise for them. But Katie suggested phrasing it as lucky. "We were lucky that we have Alice, thank you Alice! What if Alice had been in the hospital?"

We were lucky to be bailed out by Alice, but we acknowledge that it's a dangerous dependency.

Use authority to emphasize the value of reliability

A lot of times you will find senior leaders who do care about reliability. You can use their authority by asking them to attend incident reviews. People notice these things. And for you as a leader, you should make sure you attend anything reliability related in your organization, to show you care about it.

Don't let a good incident go to waste

One of the things you can often do is to introduce changes that make people's incentives align with the needs of the company. One of the better examples of this was the use of the Don't Repeat Incidents rule. This rule is that when there is an incident, you have to do at least a little work to reduce the impact or likelihood of that happening again.

You may be able to get people to agree to a rule like that (particularly after an incident). When you do, you are adding leverage during future incidents, because every incident is now a little more expensive. Every incident affects the timeline for projects in that part of engineering. Which, in turn, makes it easier for your product managers. They don't want their timelines affected by incidents, so they may be more likely to support important reliability work.

What you've done is to rope all of your product team into caring about reliability a little bit more, because every time there's an incident, it affects their roadmap.

As a sidenote, this is an area you have to be careful with heroics. If an incident takes two days of reliability improvements by the team, that adds two days to the roadmap. If you heroically fix that, you're preventing the organization from caring about reliability work.

Thank you

A lot of the inspiration and content for this came from this Decoding Leadership interview with Katie Wilde. Check it out!

You might also enjoy this discussion with Beth Adele Long. It covers leaders role in incidents, and coordination during incidents, among other topics.

Image by Th G from Pixabay

You're working on a complicated engineering problem. An urgent Slack question catches your attention. By the time you get back to the problem, you’ve lost the chain of thought, and it usually takes quite a while to get back into that productive “zone”. This is common on engineering teams (and with other types of thoughtful work).

Engineering teams often face challenges because the work requires uninterrupted time if you want to be productive. But there are a lot of reasons to interrupt a team: support questions, small bugs, and communication needs (“is this expected behavior?”, “what’s our latest thinking on the estimate for this project?”). Coordination and responsiveness compete with focus.

You can destroy your team’s productivity if you requiring constant communication and allow interruptions. The other failure mode is walling your team from the rest of the company. This leads the rest of the company to be less effective.

My favorite way to solve this problem is a Hero Rotation. A Hero Rotation balances the needs of focused and responsive work:

• Most of the team is in Focus Mode: they are working on projects, and getting stuff done.
• The Hero is in Responsive Mode: they work on small, easily interruptible work. They try to be responsive to emergencies and communication needs. They aim to protect the team from interruptions.

A Hero Rotation gets you the best of both worlds: Focused AND Responsive.

Why have a Hero role?

• Enhance team focus: protect other team members from interruptions, allowing them to be more productive.
• Encourage knowledge sharing: you’ll be asked questions that only one person might know the answer to. Over time, being a Hero helps everyone share knowledge throughout the team.
• Build empathy across teams: everyone on the team gets exposure to what is happening in support and other parts of the company that interface with your team.

What does a Hero do?

• Handle on-call and respond to alerts from Pagerduty (or whatever).
• Act as the primary contact with the support team, and other teams that communicate with your team. Aim to respond within an hour or less.
• Minimize interruptions for the team by fielding communication throughout the week.
• There is an understanding that the Hero will be less productive in their project work during their rotation.

How to set up automations for Hero rotations

I like to set up some sort of automation that tells people when it’s their turn to be Hero, and handles overrides. I find a paging system to be a good source of truth. Why? They're built for exactly this use-case.

After setting it up in Pagerduty, I next will find a way to connect it to Slack. One easy but imperfect way of doing this is with Slack Workflows. Ask at the beginning of the week who is on the rotation, and post the result to all relevant channels.

Here are a few things I like to set up:

• Announce when someone goes on rotation, both for that person’s benefit, and awareness in the channels that communicate with the Heroes.
• Share a "rotation calendar", so it’s easy to look up who the Hero is.
• Set up an alias for the Hero in Slack, so that people can @ mention the Hero without knowing who it is. Train people to do that instead of pinging a certain person.

Tips for being a Hero

• You don’t have to know everything to be the Hero. It is expected that you won't know the answer to many topics. Spend a few minutes trying to figure it out and handle it if you can. Otherwise, escalate to the person who can help on the team. Do try and rotate the people you escalate to, but don’t feel bad about escalating – this is all a part of spreading the knowledge around. When you escalate, be sure to learn all you can about the topic. That way, the next time this comes up, you may not need to interrupt your teammate.
• If you’re paged, you don’t have to be able to fix everything. If there is a runbook, you go through the steps on the runbook, and then escalate. If there isn’t a runbook, escalate freely.

When you start your Hero rotation

• Check in with last week’s Hero - is anything carrying over from Friday? Is anything in progress that you may need context on before diving in? Are there any high priority issues that need your attention or continued focus?
• Monitor the support channel actively. Try to respond quickly to anything posted.
• If you see something needing attention, do one of these things:
  • Create tickets for new issues. Let the person know we are tracking it, and offer workarounds whenever possible.
  • You do not need to solve the issues you log unless they are urgent!
  • Escalate urgent issues that you can’t fix yourself. You can ask for help!
• Give the support team updates, and let them know what you are focusing on.

Advice on running Hero Rotations

• Consider two Heroes: If the load gets to be larger than an individual, you can sometimes have two Heroes, each doing a two week shift. This is actually kind of nice, because that gives you a person who is rotating out, and a person who is rotating in. That preserves more context between weeks. It also helps people feel less isolated and more like they have a teammate during their shift. This makes sense on larger teams that have a heavy support or operational burden. It also makes sense when the team has decided to shift a lot of responsibilities to the Hero.
• Monitor the responsibilities of the Hero: you will often find that if you have a Hero, it will be natural to add more and more responsibilities to the Hero. For example, you might have them do the bug fixes for the week. Or you might have them do the deployments, or be the person most responsive for code reviews. These things will balloon over time. Be sure to keep an eye on whether the level of investment makes sense for your team, and what the tradeoffs are. Also don't allow the Hero role to substitute for automation.
• Pay attention to handoffs: handoffs can be a challenge, especially if the Hero ends up doing a lot of bug fixing or operational improvements during the week. Anything that doesn’t end at the end of the rotation becomes problematic. I usually give the guidance that you should only do work that can be done by the end of your rotation. And escalate anything else that needs to get done to the manager, so that can be worked out.
• Set clear expectations: different people will invest different amounts in their rotations. So you need pretty clear guidance on how much to invest. Otherwise, you might find some people spend 100% of their time on Hero Rotation duties, and others hardly handle those responsibilities at all. It can be confusing to know how to spend your time when your turn comes, so spell it out really clearly and talk through it with the team.

About the Hero terminology

Some people find the Hero role to be challenging. They may get anxious about doing it well. And they may feel isolated. Others may find the role to be fun, because they get to fix a lot of little issues, and be helpful to a lot of people.

For the people that find the role less desirable, the Hero name can help, because they are helping their teammates be more focused. The name emphasizes that, so you'll feel like you're doing something for your team.

But one downside to the term is that it does imply a Hero who rushes in to save the day. And this can lead to hero syndrome, where people don’t proactively fix issues that should be solved systematically. It can lead to MORE emergencies, instead of less. Katie Wilde and I talk about that in this episode of Decoding Leadership:

So all of this is to say, feel free to use your own terminology. Here are a couple of alternative terms I've heard:

• Goalie. Try to stop the puck (interruption) from getting into the net (the rest of the team).
• Interrupt handler.
• Sprint protector.

Thank you

The Goalie and Interrupt handler terms are from Rich Lafferty. I learned and used this pattern from the "Support Hero" role at New Relic. I don’t remember the history behind that, so if I should be crediting someone (or crediting myself!) sorry about that. The Sprint Protector term was shared to me by Jenny Wanger.

Image by Nikki Luijpers from Pixabay

I wanted to share some notes on how I run remote retrospectives. These are not incident retrospectives, which I tend to run differently. But these are for project retrospectives, or for regular check-ins with a group of people to ask ourselves what we can do better.

Why use this format?

This retro style is more effective than other formats I’ve used. It has these advantages:

• You get almost universal participation.
• It’s extremely time efficient.
• It covers a very wide set of topics quickly.
• It allows for discussion of the most important topics.
• It often results in actionable insights.

How does it work?

Create a Google Doc ahead of time. You can also use Notion, Slite, or any other editing platform that allows for simultaneous editing of the same file.

Add two sections:

What went well, who do you want to appreciate? What didn’t go well, or what was interesting that you observed?

I have at times done other categories, but I think these two are the most important. You can have a separate Action Items section, for example.

Meeting format

At the retrospective, you start off by giving some context for retrospectives if you haven’t done them before. And then you say we’ll spend the first 10 minutes filling in the retro doc. And you share your screen and give an example. The example might look like this:

What went well, who do you want to appreciate?

• Jade: Shannon did an excellent job of running the incident response. Everything felt really well organized, and well communicated.

What didn’t go well, or what was interesting that you observed?

• Jade: Some of our dashboards aren’t working correctly, so during the incident, I spent a lot of time on a tangent that ended up being a dead end

Then you also show everyone that you can reply to other people with indented messages, or express agreement, using these patterns:

What went well, who do you want to appreciate?

• Jade: Shannon did an excellent job of running the incident response. Everything felt really well organized, and well communicated. +Jen +Max
  • Xavier: +1. Also really liked how she checked in periodically on all of the lines of inquiry.

What didn’t go well, or what was interesting that you observed?

• Jade: Some of our dashboards aren’t working correctly, so during the incident, I spent a lot of time on a tangent that ended up being a dead end
  • Jen: I think some of our dashboards broke when we broke out the user service. We should probably update them for the new service.
• Jen: My work for the week ended up being a lot easier than I expected, and I wasn’t sure where I could best help out after that
• Maria: Our standups aren’t feeling very useful to me. I’m finding it’s breaking up my focus for the morning. +Jen +Xavier
  • Jen: Is the goal to make sure we communicate? Or to share information? Or what? I think our team needs more context sharing, and wonder if doing more show and tell would be helpful.

Because everyone is editing at once, after ten minutes, you’ll have a full set of thoughts on the week. And you’ll be surprised, there might be entire conversations that have happened already.

Dot voting to go deep

You then spend the rest of the meeting talking about the topics everyone cares about the most. You first select the topics by using dot voting:

• Give everyone 3 or 5 votes.
• Tell them to put their initials (or an emoji they choose) next to the topics they’d like to discuss.
• Give people a minute to vote.
• Then go through the topics in order of those that got the most votes.

It might look like this:

What went well, who do you want to appreciate?

• Jade: Shannon did an excellent job of running the incident response. Everything felt really well organized, and well communicated. +Jen +Max MF
  • Xavier: +1. Also really liked how she checked in periodically on all of the lines of inquiry.

What didn’t go well, or what was interesting that you observed?

• Jade: Some of our dashboards aren’t working correctly, so during the incident, I spent a lot of time on a tangent that ended up being a dead end JT MF XF JR JR JR
  • Jen: I think some of our dashboards broke when we broke out the user service. We should probably update them for the new service.
• Jen: My work for the week ended up being a lot easier than I expected, and I wasn’t sure where I could best help out after that
• Maria: Our standups aren’t feeling very useful to me. I’m finding it’s breaking up my focus for the morning. +Jen +Xavier JT JT JT JT MF MF MF JR JR
  • Jen: Is the goal to make sure we communicate? Or to share information? Or what? I think our team needs more context sharing, and wonder if doing more show and tell would be helpful.

So in this case, you start with the topic of the standups, and then go to the dashboards topics. You have a retro sorted by the topics people most want to talk about.

How to facilitate the topic

The way I like to facilitate the topic is to say, “looks like our first topic is about the standups. Maria, can you kick off the topic for us?”

After she does, then ask other people to respond or give their thoughts. If people are quiet, ask them why they voted to talk about it.

After people have voiced some thoughts on it, action items might naturally emerge. If they don’t, then ask, “is there something we could do in the next week or two that would make this better?” Talk through the options, and come up with an action item, and make sure someone is assigned that action item. As manager, you might be the person taking the action item! Be sure to complete it quickly, and report back to the team when you do.

That’s basically it. It’s fairly simple. But be sure to treat the action items seriously. I usually paste action items into Slack or an email afterwards, and will often review the action items quickly at the beginning of the next retro.

Why this works

• You get high participation. Everyone feels like they are expected to contribute. People who are shy can easily participate.

• It is time efficient. Why? Because everyone is basically talking (and listening) at the same time.

• It covers a lot of topics quickly. Why? You get a lot of breadth, because people will spend time thinking and listing the topics from their week.

• It covers the most important topics deeply. The dot voting allows you to select and prioritize the topics in priority order. Then you spend whatever time you've allocated on the most important topics.

• You get a lot of action items. The format also encourages you coming up with a few action items each retro. This feeds a virtuous cycle, where people feel like the retro results in improvements, which makes the meeting more useful, which leads to more improvements.

Tips

One nice quality of these retrospectives is that you’re starting in a quiet, contemplative activity. I’ve found you can actually start the meeting whenever a few people have joined. As others join, they know the drill, and will just get started.

You can do an equivalent form of retrospective in person, using sticky notes, and calling things out as you put them up on the wall. It works, but it’s not quite the same. This is truly one of the few online forms of a meeting I find hard to replicate in person!

As a side note, I think this form of simultaneous editing of docs to be an interesting pattern. It seems underexploited -- I bet there are a lot of similar things you could do in other meetings.

Thank you

I learned about Dot Voting from Jim Shore. This took some inspiration from that. I think this format might have been something we came up with at Gremlin. I don’t remember who invented it – it might have been Alexa Stefanko, Jason Poole, or Shaun Yelle. Or me! Thank you to Cailin Laughlin for pointing out a typo.

Image by Tom from Pixabay

Let’s talk about the magic of small things: delivering mini-features, fixing bugs. These things are often the difference between a poor product and something you’re proud of. But they are often undervalued. I’ll share an experiment that helped us ship lots of small things: Tiny Thursdays.

The origin of Tiny Thursdays

I’m working as an interim product and engineering leader at a company called Campus. The engineering team is quite small – there are more stakeholders than engineers. Every team needs so much from engineering, that they all feel like they are always waiting on engineering.

The team has been delivering more incrementally (using milestones), and that has helped. But the demand is high for the team’s time, and little things don’t get a lot of attention.

The CEO of the company, Tade, had a small navigation improvement he suggested to an engineer at the company. The engineer said, “that’s tiny” and just did it. Everyone loved the change – it was one of those things that was bugging everyone, and the improvement got as much attention as some of our feature releases.

Tade came to me afterwards and pitched the idea of Tiny Thursdays:

• Tell everyone we’re setting time aside for doing small improvements.
• Share a form people can use to submit suggestions.
• The engineering team spends all Thursday working on small improvements.
• On Friday, we tell everyone what we shipped.

I was intrigued. I was already thinking about two related problems:

• There were a lot of little improvements the product needed, and we weren’t prioritizing them.
• We weren’t triaging bugs, or fixing anything that wasn’t urgent. The customer experience was getting worse.

I also saw Tiny Thursdays as a useful way to keep a steady stream of value coming from engineering. Even if we’re working on something larger, that will take a while, it ensures a continuous drip of value.

I decided to try it out and see how it felt. The tradeoff, as I saw it, was we would potentially slow down our projects by 20%. Would it be worth it?.

How did Tiny Thursdays go?

It was so worth it.

On the first Tiny Thursday, the team delivered fixed some critical bugs, and shipped a few minor improvements. The response was enthusiastic!

At a recent senior leadership meeting, I mentioned Tiny Thursdays, and the whole leadership team perked up and spoke lavishly about how happy they were with the change. It's been really appreciated!

How to do Tiny Thursdays

The way we implemented Tiny Thursdays was:

• We created a feature request form. It fed into our Notion database of feature requests.
• We agreed as a product team to go through those requests frequently. As we do, we add rough estimates. And if anything seems especially important, we prioritize it from P1 to P4 in priority.
• Anything that is estimated as a day or less of work automatically gets put in the Tiny Thursday bucket.
• Every Thursday, we have a repeating Slack message that reminds the team of how Tiny Thursday works, and has links to Bugs and Tiny Features. Team members are encouraged to start with a bug, and go back and forth between bugs and small features. They’re also encouraged to review other people’s PRs rapidly, so everything is done by the end of the day. They also have the latitude to fix things that are bugging them or that they've noticed recently.

A few tips

The most valuable part of the definition of Tiny is this: “you can deliver it by the end of the day”. One danger of Tiny Thursdays is that work can bleed into the next day or take several days longer. “Tiny == Today”!

Another thing we learned is that it’s important to communicate the changes. The way we did it was in both Jira and Notion, we use a status field. After the work is done, it doesn't end in "Done" or "In production". The final status is called “Communicated”. We make sure we communicate all the improvements as a natural part of our work.

Why were Tiny Thursdays necessary?

One of the challenges product teams often have is that it is difficult to prioritize small things.

Most prioritization schemes tend to be some variation of effort divided by time (my favorite might be cost of delay). So you would expect them to naturally surface low effort work to the top of the list. But in practice, I haven’t seen that be the case.

This surprises me a little. Perhaps part of it is that prioritization has overhead. A lot of the process you build around prioritization is too heavy-weight for small things. My experience with Tiny Thursdays was that it bleeded off smaller work outside of the more heavy-weight pile. It creates a separate group of work that was small, important, but could mostly happen at any time.

If your team is already delivering things in very small increments, Tiny Thursdays may not be necessary or a big improvement.

Challenges with Tiny Thursday

I don’t have any regrets about Tiny Thursday. It’s been overall a big improvement. But a couple of things you might find worth noting:

• People on a hard deadline will opt out. I think this is fine, but I ask people to be public about when they plan to not participate using “I intend to” language.
• Most people have no idea what is tiny and what isn’t. I recently had someone suggest a Tiny Thursday item that would have been a year long project! I’ve generally found it best to not communicate timeline expectations around any items, unless they’re truly urgent. A lot of the requests will end up being good roadmap items, but won’t be real candidates for Tiny Thursday.

Let me know your experience

Tiny Thursdays has been a great change. Let me know if you experiment with it, or do something similar!

Decoding leadership podcast

I covered much the same content in this podcast:

Thank you

Tade Oyerinde came up with the idea of Tiny Thursdays, I believe. Iris Hou, Sonia Sidhu, and Mateo Marcano all helped implement and improve Tiny Thursdays.

Image by Doris Rohmann from Pixabay

I'd like to share a few things I've learned about conducting skip level 1-1s.

What is a skip level 1-1?

Skip level 1-1s are a way to sense what is happening in your organization. The basic idea is:

1. You schedule meetings with your "skip levels". A skip level is the person two levels down. Your direct reports' direct report. So if you manage Alex, and they manage Beth, you would be setting up the skip level with Beth.
2. You rotate through the people in your organization in some way, eventually meeting everyone before starting over again.

Why do skip levels?

The further up you go in an organization, the less you know what's going on. A lot of your effectiveness depends on learning to sense your organization.

Skip levels allow you to create relationships with people throughout your organization. And perhaps more importantly, they give you information about what is happening within your organization. You can learn important signals such as:

• How well your managers are managing.
• How people are feeling about recent changes.
• What struggles people are having in their jobs. What is geting in their way.
• What people are worried about.

Skip levels are like sampling -- they are information rich, but not very well targeted. So use them to get deep context.

How do you organize them?

First, you have to decide your capacity for skip levels. That depends on you -- I've generally done one or two a week.

I've managed the meetings a couple of ways:

• One way was to keep track in a spreadsheet, and add meetings with people every week.
• I've also used a tool like Donut. Donut allows you to automatically invite people to 1-1s based on mutual availability on both people's calendars. The list of people to attend is based on who is in a Slack channel. It basically handles everything except getting the people into the Slack channel. The nice thing about this method is that you decide your capacity, and make sure people are in the room, and Donut handles everything else.

Who do you meet with?

I've generally worked within small to medium sized organizations. In theory you can use skip levels at any sized organization. One thing you'll probably need to decide in a larger organization is whether you want to meet with 2 levels down, or all the way down. I suspect it's usually valuable to go all the way down, because that keeps you grounded to the work being built. But the tradeoff is that you get less opportunity to build relationships with managers in your organization.

Since I generally keep the number of meetings limited to what I'm comfortable with every week (usually one or two), this means as your organization size changes, the frequency you meet with people changes. It could take a year to get through the whole organization!

Some tips for skip levels Start off by telling people what to expect

Meeting with a leader with a fancy title can be scary. I make sure my invite says that it is a Skip Level 1-1, and that it is a chance to talk about anything on their mind, or give feedback on problems they're seeing. They can give feedback on their manager, or talk about things getting in their way. Etc. Give them a list of things they might talk about, so they can feel prepared and know what to expect.

I also start off with human stuff: introducing myself, asking about them and getting to know them. If you're aware of anything they've done well, mention it. It can mean a lot to them.

You also need to let the whole organization know you're doing skip levels. Your managers should be aware, for one thing. I usually just mention it in Slack, and tell my managers in a staff meeting.

Use skip levels to assess recent changes

Leaders often rely on quantitative methods like metrics to assess the effectiveness of a change. But qualitative methods can miss a lot. Asking a lot of open ended questions can often cue you in to problems you weren't even aware of.

One thing I love about skip levels is that I can ask about recent changes, and ask people for critique and feedback. Do they have any suggestions or recommendations on ways to improve on that change? Are they noticing ways it might be causing problems? Ask specific questions that require an answer, instead of vague queries. So for example, ask "what problems are you seeing with this reorg", rather than, "do you have any feedback on this reorg".

Use skip levels for deep-dives

If you're doing skip levels, people won't be surprised to see a meeting invite from you. This gives you an opportunity to investigate anything you're concerned about. For example, if a manager seems to be having problems, you can reshuffle your skip levels and talk to two of their direct reports that week.

You can even do deep dives. If something makes you nervous, pack your schedule with skip levels. This is a way of really quickly assessing the situation. You'll know more about the situation than anyone within a day or two.

Use skip levels to test your plans

Skip levels allow you to query people about topics that will affect them. If you're planning a big change, for example, you can ask people about ways it might affect them. This gives you a chance to understand the implications of your work. Yet you do it as a natural part of your organizational work.

I sometimes use skip levels to think through topics that are nascent and early.

One danger of doing this is that you may signal to people that something needs attention. So this has to be done in a delicate way. I try to make it clear I'm speculating.

Use skip levels to verify communication or alignment

One thing you can do with skip levels is ask questions about things you believe you've communicated. You'll be surprised how often people haven't really internalized it. It's a good sanity check for communication.

You can also ask questions about the company strategy, or other alignment issues. It can be a good test of whether people are really aligned.

Be sure to create a safe space for this. The goal is to test and learn, not to go after any individual. If you make anyone feel threatened, the entire skip level approach is sabotaged, and then you're just wasting everyone's time.

Use skip levels to see patterns

One thing you'll notice with skip levels is that you'll start hearing the same things from different people. This can often clue you in to problems you weren't paying attention to. But it can also give you false signals. Sometimes people that are really good at amplifying the problems they're concerned about will get a lot of other people to ALSO worry about those things. Even so, it's useful to keep a pulse on your organization.

Be careful what you take on from skip levels

Sometimes you can be eager to help, and to show results. So you may be inclined to intervene in situations you shouldn't. Just be sensitive to the fact that this person reports to your manager. Sometimes the best course of action is to redirect them, but let them know you will help if they're not able to address the situation.

Thank you

Emily Nakashima sparked this post in a conversation, making me want to record some thoughts on it. Thank you, Emily! Keizan Shaffer shared some elements of his skip level practice that helped me flesh out this post. He reminded me that you can use skip levels for communication and alignment testing. And also reminded me to emphasize that you will get through everyone, but the amount of time will vary depending on the size of your organization.

Image by DWilliam from Pixabay

Today, I’ll share the best organizational approach I’ve found to improving reliability. It is effective at organizations that focus on shipping, and struggle to prioritize reliability work.

With this approach, you can enjoy the benefit of rare, artisanally curated incidents, instead of repeating the same boring incidents you’ve experienced before.

This approach is called “Don’t Repeat Incidents” work. Or, DRI work.

The origin of Don't Repeat Incidents

We developed this approach at New Relic. New Relic’s engineering culture was quite focused on delivering features. All of the pressure on teams was around delivery. Yet, there were periods when our systems became fragile, and we started to have serious incidents. Customers would complain, and sometimes leave.

As a result, the leadership would make hard pivots towards reliability work, stopping all current projects. This made leadership feel better, and customers felt like their concerns were taken seriously. But for many of the engineers, it felt wrong. We would switch to focusing on reliability, and then go back to focusing on projects. Until it got bad enough that we focused on reliability again.

And the thing was, we were underinvesting in reliability. Even though there was a lot of business value in focusing on reliability work, it was hard to reason about when to make it a priority versus some other project. Usually, we didn’t prioritize as highly as we should have.

Reliability wasn’t baked into the way the teams were operating. Even though we were doing fairly modern practices of having teams own their own operations, and being on call, all of the pressure was to deliver features.

During this time, New Relic gradually developed a world-class reliability program. Our internal tools for this were the model for some of the incident tooling modern teams use to manage incidents. We improved our competency in many ways. For example, we got pretty good at running blameless retrospectives. We learned from our incidents. And during these retrospectives, we would capture lists of follow-up work. Incident write-ups often were quite well written, and thoughtful.

Why we didn’t do follow-up work

One of our bigger problems was that we would uncover a lot of follow-up work, and it would never get worked on. It would get put in the backlog somewhere, and seldom get prioritized.

It wasn't that the people were lazy or that they didn't care. It would go something like this:

1. We would put the incident follow-up items in Jira.
2. We would prioritize it. When we created it, we would think, “yes, this is really important. We need to do that. We’ll schedule it after this urgent thing we are working on.”
3. When the urgent thing we were working on was finished, the urgency from the incident would have faded.
4. The follow-up item would then be prioritized against a few other things. Since it felt less urgent, it seemed like something that could happen later. After all, this other important project needs to get out!

This would happen over and over. It was being scheduled in a way that seems rational (after all, someone is prioritizing it). But it wasn’t improving reliability.

Often, very similar incidents would occur a few weeks later. These were especially frustrating. Often, the plan was to start work on it the next week. The incidents would be so disruptive, because they would take the whole team’s time for a day or two. And it would be extra disappointing because something similar had happened before.

During this time, someone observed something interesting about our incident retrospectives. They usually came up with a pretty long list of followup work. All of them were good ideas, but the larger items were big projects. And the smaller things would often fall by the wayside. In this was the seed of “Don’t Repeat Incidents”.

How Don't Repeat Incidents works

There were basically three parts to “Don’t Repeat Incidents”:

Step 1: fast retro or post-mortem

After an incident, you always schedule a retrospective. It should be done within a day or two.

This is not a common practice. Most of the time, teams won’t schedule a post-mortem or retrospective right away. They might spend a week writing up what happened. Or they might wait until the next sprint retrospective, and talk about it as a part of that retrospective.

Doing retrospectives within 24 or 48 hours is a dramatically better approach. Why?

• Everything is fresh in your mind. You learn more from the incident by examining it within a day or two.
• You could have a similar incident within that time period, if you don’t act more quickly.
• The longer you wait, the less likely you’ll have the retrospective. This will mean you’ve incurred the cost of the incident, without receiving the benefit of the incident, which is the learning and improvements you make.

We did this for all incidents above a certain severity level.

Step 2: identify “DRI items”

At the retrospective, you identify what happened, learn what you can about your systems, and how the team works. And identify actions you can take to make your systems more resilient.

You identify these, and tag them as “DRI work”.

This is the way we defined DRI work:

1. Takes a maximum of X weeks, for the full team to complete.
2. Will substantially reduce the likelihood of the issue occurring again (MTBF), or substantially reduce the severity (eg, scope of impact, MTTD, or MTTR) if the issue does occur again.
3. Ideally, mitigates the general class of issue that occurred, rather than a specific issue. For example, if an issue was caused by disk space on server X filling up, the DRI solution should be addressed to the general case of preventing disk space from filling up on any of the team’s servers rather than specifically about server X.
4. Ideally, addresses the underlying causes of the issue rather than its symptoms (for at least the owning team).

For DRI work to be effective, the items you select and tag as “DRI work” need to be small. We capped it at two weeks of work, but you can set this according to your company’s needs. I've made it shorter at other companies. DRI Work is often a hack, or small temporary fix. Ideally, it is a day or two of work. The smaller the unit of time, the easier it is from a process perspective.

Step 3: follow the “DRI rule”

Then a new rule kicks in: we do the “Don’t Repeat Incidents” work before we get back to anything else. This work is automatically more important than deadlines, projects, or anything else.

This work can be technical work, improved monitoring, or a new checklist in our process.

That was the whole change.

Don't repeat incidents helped, a lot

This small rule ended up being one of the best changes we made. It had a significant impact on reliability.

It’s something I recommend, if your team has a lot of pressure to focus on projects, and isn’t doing enough reliability work.

Why DRI?

When I step back and look at this change, I think it worked for a couple of reasons.

First of all, we had a constant pressure to ship. The DRI rule gave teams the leverage to actually do reliability work – it became a counterpressure to the pressure to ship. When there was an incident, no matter what, you had the space, and even the obligation, to take a few days to prevent it from happening again, or to make it not as bad next time.

It was not only effective for the fixes to software and alerting. It had an important influence on the culture of operational excellence. The policy admitted that incidents were a normal cost of writing software. The DRI expectation meant every incident was a built-in reminder that reliability demanded continuous improvement, and was a first-class concern, not an afterthought.

It also kept the responsibility for identification of problems and solutions directly in the experts on the teams closest to the ground truth in production.

Limitations of “Don’t repeat incidents”

One of the problems with “Don’t Repeat Incidents” is that it allows for only small pieces of work. This is incredibly valuable, because you get bandaids that help your reliability. But there can be some type of reliability issues where the bandaids are not sufficient.

For example, when I was supporting a team at New Relic, we had some challenges that required large changes to improve the reliability of the product. These weren’t something that could be done in a couple of days. Yet the amount of work required was substantial.

In some cases, teams working on reliability projects would get interrupted frequently by DRI work. So even if they were working on a solution to a class of problems, the demand for a short term solution would interrupt the larger project. Don’t repeat incidents can focus your attention too narrowly on the technical problems. The need for larger systemic changes can be hidden by the small changes. And the emotional relief from the quick fix can contribute to teams delaying larger projects that require more substantial change. DRI helped, but wasn’t sufficient to drive larger reliability work. We developed other approaches to deal with that.

Another way you can handle this is “error budgets”. This is well covered in the Google Site Reliability Engineering book. Error budgets are more complicated. You can implement don’t repeat incidents just by getting people to agree to it – error budgets require more tooling and measurement.

Like all organizational changes, it does require people to agree to it. You need to have your leadership team agree that it’s the way to do things. And it’s important to agree on the parameters and definition of DRI work. One thing that can be helpful is to provide an explicit prioritization across all types of work (features / security / customer escalation / incident follow-up work).

Podcast on this topic

Decoding Leadership is my podcast on leadership. I cover this topic in one of the episodes:

You can also view it on Youtube.

Thank you

Marcos Wright-Kuhns helped me locate the video of a talk that Beth Adele Long and Matthew Flaming did on the history of the reliability improvements at New Relic. It’s a wonderful, insightful look at all the changes we made over a couple of years. Eric Dobbs pointed out some limitations of DRI, and shared a post I linked to on missing the forest for the trees. He also had some very helpful suggestions about similar incidents versus repeat incidents. He contributed language to the Why DRI section on the impact on operational culture and practices, and also the respect for local expertise. He also added language around larger reliability projects and how they related to DRI work. Beth Adele Long added nuance to “what is a repeat incident”. Andrew Bloomgarden pointed out that doing retrospectives within a few days wasn’t a common practice, and needed more emphasis. Elisa Binette pointed out some of the preconditions for DRI work to be effective, and also emphasized the importance of similar incidents versus repeat incidents. She also added some nuance to the fast retrospective section, and improved some of the language around monitoring versus alerting. She shared some specific definitions of DRI work, that I included.

Image by Andrea from Pixabay

If you're doing an important job, you will end up being a bottleneck at some point. There is an art to disentangle yourself from doing things directly. This skill is hard to learn. So in this post, I go into detail on a lot of ways you can remove yourself from being a bottleneck.

Many leaders struggle to balance being involved, with the fact that you can’t do it all yourself. As your organization becomes larger and larger, you’ll be crushed by the amount of decisions and work that come your way. Your organization will suffer if you don’t learn these skills. These problems are quite common and natural. But it is an essential skill to learn as a leader.

The framing for this is that you want your organization to produce the outcomes you would produce yourself.

So what are these techniques for removing yourself as a bottleneck?

Use indicators to set up an alerting system

The first technique is to use what I call “indicators”. Indicators are metrics, but they’re metrics you’re using to guide your attention, rather than to assess or measure. This is one of my favorite ways to use metrics, because it doesn’t automatically produce the negative surprise results you get from a lot of metrics. You can use imperfect measurements as indicators, which gives you a much wider set of things you can monitor than with metrics.

You use indicators to set up an “alert system” for yourself. This allows you to preserve your valuable focus time for things that are most important. But it gives you awareness of when something might require your attention.

For example, you can use the number of Jira tickets, broken down by team, as an indicator. If the number is ever-increasing, that is a sign that the team is operating in a particular way. You may or may not think that is a problem, but if you’re wanting your team to keep their bug inventory low, it could be a way to trigger some conversations with your team.

Set up an information system for yourself

A lot of director and above leadership is really about setting up an information system for yourself. You’ll often find that leaders have less idea of what is going on than anyone, because they’re furthest from the work. A good information system can help you stay abreast of what you value in your organization.

Indicators are one part of that information system, but you should look for anything that gives you a signal on how things are going. I list a number of ways to set up your information system in my “advice for new directors” post, including demos, skip level 1-1s, and indicators. You can also set up reporting so that people are feeding you the information you need.

How do you set up this information system?

The way I usually go about setting up an information system is to list the things I care the most about. And then think about how I will get a signal on how well things are going. I might attend demos to see how the teams are doing, and how the work is progressing. I might try out the software weekly, to get a sense of how quality is doing. I might have my managers report on a few indicators I care about. There are a lot of ways to get at this information, but the important thing is that you’ll need it. Otherwise, you’ll find that you’re always harassing your direct reports to get information from them. And you’ll drive them nuts.

How does this remove you as a bottleneck? An information system allows you to step back from directly overseeing the work, but still be confident it is going well. It’s a tool for trusting and verifying at the same time.

Use the Ladder of Leadership to grow and evaluate your team

One of the best tools I’ve found for removing yourself as a bottleneck is the use of the Ladder of Leadership. This term was coined by David Marquet, the author of Turn the Ship Around (although this concept doesn’t appear in that book).

Here’s a slightly modified Ladder of Leadership. You start at the bottom, and try to work towards the top:

• I've been doing...
• I've done...
• I intend to...
• I request permission to...
• I recommend ...
• I see or I think...
• Tell me what to do

At the bottom of the ladder, you have the least amount of autonomy and mastery. For any given task, a person at the bottom of the ladder will ask for direction. If they have slightly more autonomy, they might share their observations, or recommendations. And the more they own their area, the more they are able to act independently to solve problems.

This Ladder is domain-specific. You can have people that are at the bottom of the ladder with one thing and at the top of the ladder, with something else.

The magic of the Ladder of Leadership is that it specifies how to move people further up the Ladder. You always prompt people for the next level above where they are today. For example, if your subordinate says, “I’m not sure what to work on,” that is at the bottom of the ladder. They are saying, “tell me what to do.” What you want is for them to share their observations or thoughts. Your goal is to get them to start saying “I see” or “I think”. So, you might ask them, “what are you seeing nowadays that seems important?” Then you can hear what they say, and help them reason about what they should be working on. You still may end up telling them what to do, but you’re training them to get further up that Ladder.

Moving them up this Ladder helps them to be more effective leaders. It also helps you, because it gives them increasing levels of autonomy under you. This frees up more of your time.

The Ladder can also help you move people back from dangerous situations. You can move them down the ladder. The key is to assess how well it’s working. If people aren’t doing well at the “I’ve done” stage, for example, you should move them down to “I intend to”, and make sure you’re aligned with their approach.

The Ladder of Leadership can also be used to evaluate your organization. Where are the parts of your organization that are high on this ladder? What are the parts that are lower?

You also have to keep in mind your own part of this. You may be less comfortable letting someone be fully autonomous when you don’t understand their area. And you might be naturally inclined to be more involved in areas you understand better. Be wary of your part in this.

The Delegation Ladder of Leadership

I’ve developed another variant of the Ladder of Leadership, specifically focused on leaders who are delegating. This Ladder is less tested than the above, so let me know your experience using it. It looks like this:

• Someone has been doing it...
• Someone has done it...
• Someone is telling me their intention before doing it...
• Someone is requesting my permission before doing it...
• Someone recommends approaches before doing it...
• Someone is sharing how they think it should be done….
• You’re doing the work yourself

Again, you start from the lowest part of the ladder, and you want to empower your organization to go higher and higher on the ladder. So this is how you gradually increase your level of delegation. You start with something you’re doing directly, and your next step is to get them to think about how it should be done. You might say, “I would like your help with this. I'm trying to solve this problem. Can you walk me through how you think you might go about solving it?” You can decide based on how they describe it whether you think they are ready to do the work themself.

If you do start giving them parts of the work, you give them what you think they have the skills to take on.

As you give away work, you oversee it depending on how far up the ladder you go. If it’s towards the bottom, you’ll probably want to have them check in with you frequently. You might have them report on progress every few days, or talk through how it’s going with you. The more you’ve developed trust in the person, and can delegate to them, the more you can step back and give them meatier problems.

The reason I think this is important is that many leaders struggle with delegation because they don’t feel like they can give it away because they have to give up control for how well the work is done. But when you see it as a continuum, you can do it in smaller ways, and gradually develop trust in your team’s ability to take on the work you delegate. I think it helps both the leader and the subordinate to work better together.

Delegation removes you as a bottleneck in the leadership chain. And if you gradually build up trust with your team and delegate to a greater and greater extent, they will be fully owning their part of the work and not be blocked by you from getting things done.

Use Completed Staff Work to empower your team to make decisions

Another concept I’ve found useful is Completed Staff Work. This came out of the US military. Completed Staff Work is a standard you can apply to help you understand if any given management work is sufficient or not. The standard is: “can you present the work to your manager, and have them just give a thumbs up to approve it?” If you have answered any questions they might have, and made it clear it’s a good decision, then you’ve done enough management work, and you’ve done Completed Staff Work.

Completed Staff Work helped me become a better subordinate. I learned from it how to manage up better. When I read about Completed Staff Work, I realized I wasn’t giving my manager the information he needed to make decisions. I wasn’t completely owning my choices, and making it easy for my manager to make decisions.

Completed Staff Work gives your manager a chance to inspect your work and reasoning. And if you’ve done the work, they can just approve it.

There is a lot of nuance to Completed Staff Work, and if you’re not careful you can waste a lot of time with it. I've written an entire post on Completed Staff Work, and I recommend reading about it.

But it is a concept you can share with your team to help them do better management work. Doing so frees up your time, because your subordinates are doing better, more complete work before they present it to you. It reduces back and forth with them. And importantly, it trains them to better own their areas, so you can step back and stop being a bottleneck.

Use process to remove bottlenecks

Sometimes you can use process to replace work you’re doing directly. For example, let’s say you are the one that always approves certain changes to the product or to the website. You do this because you understand it best, and want to control the quality of it. And you know the results will be good if you oversee it.

In such a case, you could use a process to help guide people. You could make checklist that people could go down to make sure they’ve considered all of the things you think about. And if there is a set of things you think is dangerous, you can have them escalate to you for those things only. This allows the default case to not come to you, but still gives you involvement for things that are dangerous enough that you haven’t fully figured out how to hand it off yet.

The time you spend on these type of process improvements is almost always worth it, because your organization becomes unblocked, and there is less that has to come to you. That's what you want.

Go from doing the work to inspecting the work

Another pattern I've seen is to move from doing the work directly to inspecting the results of the work. So another way of saying this is that you review people's decisions rather than deciding yourself.

Let’s use an example of hiring. In a lot of startups, the founders are fully focused on hiring the right employees. They realize how crucial this is, and feel like if they’re not doing it, the right employees won’t get hired. But… the fact that you don’t trust your organization to hire is pointing out your actual problem.

One way you could handle this is to gradually move your position in the hiring process towards the end of the interview loop. As you gradually move further back in the interviews, ask yourself, “are we still making good decisions? Am I happy with the assessments? If not, you work on that until you are satisfied. Then you can gradually move even further back.

It’s not going to work for you to hire every person in the company. So gradually move from doing the work, to inspecting it.  This helps train your organization to produce the outcomes you’re looking for.

Focus on outcomes and constraints

One of the biggest mistakes bottleneck leaders make is to focus on activities.

This is natural. When you're doing the work yourself, your evaluation of the situation naturally includes how it is done.

But when leaders are trying to remove themselves as bottlenecks, they'll often dictate the activities, and delegate them to their subordinates.

This is harmful for a number of reasons:

1. The person doesn't feel ownership of the work. Why? Because they aren't really grappling with the problem space. So they're just operating as a remote controlled version of you.
2. You're not setting up the work for real delegation. Why? You're operating at too granular a level. This drags you into the weeds, and keeps you in a position of being a bottleneck.

What you want to work towards (assuming your subordinate has the capacity to take on the work) is giving people a few things when you delegate:

1. The outcome you're wanting to see.
2. Some constraints to guide their decision-making.
3. The next steps for synchronizing your thinking or inspecting the work, or making sure you're on the same page.

When you are a bottleneck, you should aim to give your subordinates Problems, not Solutions.

Use Tenets to give high level guidance

Another way to disentangle yourself from being a bottleneck is through the use of Tenets. Tenets are a way to give people mental shortcuts that help them reason about common problems within the company. If you see people making decisions repeatedly that follow a pattern that you’re not happy with, tenets can be a good solution.

I’ve written a whole post about Tenets. There is a lot of nuance to using them effectively. But the basic idea of a Tenet is that it is a rule of thumb. Ideally, a Tenet should be something where you could reverse it and it would still be a valid Tenet.

An example Tenet might be: “we are frugal with the company's money, and take extra time to reduce costs”. An opposite Tenet might be, “We are willing to spend money to make us go faster”. With each of these, there is nuance, and you often provide more detail. But the high level Tenet becomes something people remember, and it helps them make decisions in their daily job.

If you don’t use Tenets, people will decide based on what they think is best, and what they think is expected of them. Tenets provide explicit guidance on what to do in these type of situations.

Tenets are useful to leaders because you shape the decision-making for a large group of people, and don’t have to be involved in correcting for decisions that are made outside the direction you desire.  

Use 'I Intend To' and have a default towards action

In Turn the Ship Around, David Marquet emphasizes using “I intend to” as a standard practice for all employees. He recommends spreading a culture where everyone in the company says the course of action they plan to take as a regular part of their work. He pioneered this on a nuclear sub. There, a sailor would say, “I'm going to do this”, and the person they said it to would say, “Got it. You're going to do that.”

This is quite useful when your work coordinates with others. When people say I intend to do X, it does a number of things:

1. It helps people know what's going on. So you can coordinate with that person.
2. It gives people a chance to intervene, if there would be bad consequences.
3. There's a default to action. Instead of requiring permission, when you say “I intend to” you’re saying something will happen unless anyone does something.

So you can train your team to broadcast “I intend to” language, and that can often require less intervention and decision-making from you. For example, one of your managers might say, “hey everyone, I wrote up a plan for how we're going to roll out this new feature. I'd love your input and improvements. Absent any changes or feedback that affects the plan, I intend to move forward on Friday.”

Notice how if nobody weighs in, the plan is approved by default. But they’re inviting input, critique, feedback, and possible intervention.

“I intend to” is a wonderful model of operating. So I recommend sharing it with your team, and modeling it yourself. It removes you from being a bottleneck, but also improves the way your team coordinates with each other.

Create real teams so communication doesn't always go through you

The final thing I want to mention is that I see a lot of leaders become hubs, where they work directly with their subordinates. If you are an information hub where everything's coming through you, you are necessarily going to be a bottleneck.

The alternative to that is to get your team to work together on problems together. Instead of working through you, they work with each other. I talked about how to do this in my recent post on organizational work. You want a leadership team, not a group of individuals.

Did I miss anything?

I made this list of approached when I was writing down a list of advice for bottleneck leaders. What did I miss? Let me know!

Podcast on this topic

Decoding Leadership is my podcast on leadership. I cover this topic in these episodes:

Thank you

Image courtesy of Pixabay.

Organizational work is work that you do ON an organization, to make it more effective.

Organizational work is a critical concept for leaders, a sort of "second job". I’ll describe how I think about organizational work, and some surprising ways it helps build an effective organization.

People at small companies mostly focus on getting things done. Before you really start scaling a company, it makes sense to focus on whatever you need to do to get things out the door.

As a company grows, focusing only on getting stuff done can perversely slow things down. Why? The organization becomes more and more complex. Unless you do something to fight that complexity, things will get harder as time goes on. So, you have to continually fight to streamline the way work is done. The best way I’ve found to do that is “organizational work”.

Defining organizational work

What do I mean by organizational work?

Let’s use the example of how hiring is done. In many companies, you’ll find that the hiring manager is responsible for defining the hiring process. When it’s your turn to hire someone, you mostly reinvent the wheel. There might be some lightweight process you follow. Or you may copy what you did last time. But it’s never really been designed very well, or improved upon. And after you hire your person, you go back to whatever else you were doing.

Organizational work would be to not only focus on what you need to hire your employee, but to also create a method someone else can follow to make it easier for the next person. You don’t just define your own hiring process, but you make one for everyone.

Thus, organizational work is doing work on the organization itself, to make the organizational machinery more efficient.

Do you see your leaders streamlining things? Is the way you go about things improving continually over time? Are you building a _system _that is continually improving? Are there people in your company that are continually focusing on what’s broken? Those are all organizational work.

You have two jobs

I was introduced to organizational work by a mentor of mine named Bjorn Freeman-Benson.

He communicated the job of a leader as having two jobs:

1. The first job is the job you were hired for. You have the results you are expected to deliver. Projects, and people management, and the stuff on your job description.
2. You also have a second job. The second job is to make things better around you, to make things easier for the next person.

He said you should always be thinking of these two parts of the job. I loved this description of the job of a leader. He baked it into the culture of the organizations he built.

Organizational work makes a great workplace

Organizational work is about creating a flywheel effect, where it gets easier and easier over time to create the same impact.

When the leaders in an organization are focused on organizational work, there is a compounding benefit. Over time, the compounding “interest” of this work can transform the organization. You benefit from a flywheel of improvements, instead of a continual slog of things getting harder and harder. You see a lot of things become automatically easy. Companies that continually invest in organizational work can become pretty amazing places to work.

When I look back to the best place I’ve ever worked, I think this was the most foundational element behind why everyone loved to work there. It’s a bit hidden, because most of the people at the company probably had no idea it was going on. But it was what was behind all of the continual improvements that gradually led to an exceptional work culture.

Organizational work and complexity

If you don’t invest in organizational work, what happens? The default drift in an organization is toward entropy and increasing complexity. The organization grows, and becomes more and more complex. Work gets harder and harder to do. Things become more political, more challenging. People add process to try to manage the pain, often making it even worse. It can result in a death spiral. Or often, the company just plateaus, at the level of complexity it can manage, unable to rise any further.

There are all sorts of ways of reducing complexity, but organizational work is an approach to tackling these things. It’s a concept and an element you can build into your culture to make it more resilient. It can even make the work culture antifragile, because new problems and stresses can lead to improvements in the organization and make it stronger.

Building strong management teams

One thing I love about organizational work is that they are one of the best ways to form a strong management team.

Typically you will see management teams formed of individuals that don't really have any shared problems to work on. They are responsible for their own areas, which doesn’t give them much room to work together. What happens then is that their manager becomes a hub of communication, and they rarely work together on anything.

In fact, one of the most common patterns I see for CEOs is that they operate in a hub and spoke model: they are the hub, trying to direct everything. And the CEO has many leaders under them who work directly with the CEO. The disadvantage of this is that the leaders don’t communicate very much with each other, and they don’t become a real leadership team. Instead, they work in their own areas, on their own problems. This can be problematic because the leadership team doesn’t build a habit of collaborating and working together on company problems. They’re not able to learn from each other, and help each other be better.

I see this all the time, from the very top of organizations, all the way down.

The way to build strong teams is to have them work on shared problems together. And often, organizational work is a way of doing that. I think all managers and leaders should be aware of the concept of organizational work, and should be using it to improve their organization. If you manage managers, it’s a valuable lever for building a strong management team.

When you work on organizational work together, you have a natural common cause: making the organization work better. You have natural partners in that cause. Other leaders are people that can help you improve your plans, critique them, and partner with you on them. This also makes leadership less lonely, and helps accelerate leadership growth. You’re able to see how other leaders operate, and copy what they’re good at.

If you create a leadership team that has a high degree of trust, where everyone is helping each other out, where people are motivated to success but don’t have to put each other down to succeed, where they are sincerely interested in doing good work, and learning from each other, you can create a remarkable environment for learning.

I think of organizational work as a foundation to creating strong leadership teams.

Implementing organizational work

When you are aware of the concept of organizational work, you have a way to point people at problems.

As a leader of leaders, what I will often do is spend some time coming up with my best idea of the most important problems to solve in the organization. I might list a few potential solutions to those problems as well. Then I’ll work with my team and say, “hey, do we agree that these are the main things that need to improve? Anybody have anything else that belongs on this list?” We discuss and agree on the top priorities, and then we sign up for them, or I assign them. We all come up with proposals for our projects, and get critique from each other on those proposals. We report on progress to each other. And we help each other with suggestions, ideas, and active collaboration.

It then becomes natural to work together. A leader might say, “I’m struggling with this part of my plan. Does anybody have any ideas of how I should approach this?”

I’ve even used this to recruit people into a company. At one place I worked, they faced substantial challenges, and had very little leadership in place. I recruited an unusually seasoned team to join the company and work on the organization itself. They joined partially because they knew they would be part of the team fixing the whole organization, and they were excited by that problem. They knew I wasn’t just hiring engineering managers, I was hiring co-conspirators.

Leaders are often motivated by a desire to solve big, meaty problems, that have a lot of impact. Organizational work often feels substantial in that way. It can stretch you and make you a better leader.

Developing leaders through organizational work

Organizational work doesn’t just build strong teams. It also is effective at growing your leaders at a fast pace.You can use organizational work projects as a way to deliberately train people in areas they need to grow and don’t have experience.

For example, I’ve done reorgs a million times. But I once worked with a new leader who didn’t have a lot of experience with them. So I had them develop a first cut of the plan, and then I filled in the gaps to the plan. Then I had them talk through how they would go about implementing the plan, and I filled in a few things they missed. It’s an effective way to give someone a new skill set.

You may find that your managers don’t have the bandwidth to take on additional work. Management roles can be incredibly time intensive, and it may not be realistic to add to their plate. It might just feel like additional work to them. This is actually a really good signal. Your manager is working at capacity. Don’t give them organizational work at that point. If all of your managers don't have the capacity or the time to make the organization around you better, that’s an important thing to know. You can be certain the situation is not going to improve without some sort of intervention.

Organizational work and your experience in the role

New managers are especially often not able to take on organizational work. I explicitly will set the expectation that this will be part of their role in the future, but it’s not realistic today. “Organizational work is something you’ll want to make a regular part of your management practice later on. But right now, just focus on the skills you’re learning. It will feel more natural to tackle it later.”

As your position in the company rises further and further up, the percent of your time spent on organizational work will naturally increase. It also becomes a more and more required part of your work! For a front-line manager, it’s a sign of experience that they are making the environment around them better. For a director, they should be continually improving their organization. For a VP, it becomes a significant part of the job, because your altitude is mostly about creating the machinery of your organization.

Balancing your two jobs

I have seen leaders get into trouble with organizational work. Your first job is your first job, and organizational work, while important, is something you do if you're doing well at your first job.

If you focus exclusively on organizational work, you might neglect things that others view as your main job. I’ve seen leaders fall into this trap with things that seem like “cultural improvements”. Yes, they make the organization more effective. But if you do that work without holding down your core responsibilities, you'll brand yourself as a leader that is ineffective. And that will compromise your organizational work. (This is not to say cultural improvements aren't valuable, of course.)

All organizational work is not equally valuable

All organizational work is not equal. And some organizational work can be essential to the company's success. For example, streamlining your hiring process may be part of your ability to hit your engineering goals over the next year.

Sometimes when people describe the second job, they mean work that "isn't important" or core to the business. I don't usually use it this way. To me, the second job is about improving the organization around you to be more effective. Cultural work can fall into that category, but I don't use "second job" to refer to work that other people don't value.

You also fail by focusing on work that isn’t organizational work, because you’re not really building a better environment around you. You can also fail if your organizational work isn't focused in the most valuable direction.

Documentation

Organizational work is easier to iterate on when process is like "source code": the source of truth for process, and something that is fluid and easy to change.

When you have things written down, it is easier to improve upon. You can practice the steps yourself when you use the process. And improving the process can often be as simple as updating the documentation.

I've written about how to make process be valuable and easy to change.

How to identify organizational work, and examples

Usually the way I come up with organizational work is I spend some focused time on a writing assignment for myself. I’ll list the problems I see in my organization, and try to identify the most important ones to address. I think about what I might do about those most important problems, and those become potential organizational work projects.

Here is an example of how I might go about it, writing down problems and potential solutions.

• Problem: The development workflow is terrible. Changes have to be rolled back when there are conflicts. No use of feature flags. Things aren’t getting reviews. Lots of work sitting in PRs, never merged.
  • Possible org work: introduce feature flags, train team on them.
  • Possible org work: have an experienced engineer or two design a better workflow and put it in place. Make it so engineers can merge their own work.
  • Possible org work: explicitly encourage pairing to reduce back and forth.
  • Possible org work: get some metrics in place and review with team, along with goals.
  • Etc.
• Problem: Our hiring process is poor, doesn’t give a good impression of our company. No hiring process. We often don’t get back to candidates very quickly. Sometimes not at all.
  • Possible org work: write down a very simple hiring process we can iterate on.
  • Possible org work: communicate a “turn-around SLA” for getting back to candidates.
  • Etc.
• Problem: Communication in Slack is poor. We don’t write much down, so everyone is always asking the same questions.
  • Possible org work: communicate it’s not official until it’s in the wiki.
  • Possible org work: encourage better communication practices and some standards for large rooms in Slack.
• Problem: we have a bottleneck team that is really suffering. Heavy oncall, high attrition, nobody is happy with their work.
  • Possible org work: hmm, not sure of best course of action. So course of action would be identifying a plan.

Once I have that list of issues and potential organizational work, I'll align people on the top problems, and select a few organizational work projects that leaders can focus on. I'll work on some of them myself, and get my management team to take others.

Steps for organizational work

There are usually some clear steps to take when doing organizational work.

1. Write up a plan for what you’re going to do. It can range from a paragraph, to a full document with a lot of detail.
2. Get critique on that plan. This can be quite quick, or more intensive, depending on how large the project is.
3. Then get to work on it. Treat it like a normal project you would report on. Estimate when you’ll do what, report on it. Etc.

Final thoughts

Is organizational work happening in your organization? Should it be? Would your leaders benefit from having the notion of “organizational work”, or “two jobs”? I’d love to hear your experiences as you introduce this into your workplace!

Podcast on this topic

Decoding Leadership is my podcast on leadership. I cover this topic in one of the episodes:

Thank you

Bjorn Freeman-Benson introduced me to the two jobs approach. He probably got it from somewhere else, because I do see other people using the same terminology. But he is the one I attribute it to because he baked it into my brain. Ralph Bodenner gave me some good ideas for improving this post, especially around how your second job is affected by your level of experience in the role. Ben Darfler referred me to Charity Major's post on making sure you do your first job well. Jeff Gray initiated a discussion on LinkedIn on the foundational importance of the first job.

Cover image by kikkuru0606 from Pixabay. Illustrations by Dalle.

Let’s talk about leadership within a macho or bro environment. We’ll explore the elements of these environments, discuss how to navigate your own behavior within these cultures, and talk about what you can do to influence them.

Recognizing macho culture

First of all, let’s talk about how to know you’re actually in an environment that has a macho culture. Often, this is something you’ll just know. But if you’re not sure, some indicators can be:

• Lots of war metaphors.
• Use of aggressive, tough language.
• Jockeying for position or attention.
• Criticizing others to look good.
• Blaming without context.

You also may notice that the way people communicate is either:

• Everyone agrees, because they don’t want to stick out. Or,
• People are fighting and sniping over everything, because they look good if they are “tough”.

A sign that you’re dealing with a toxic version of macho culture is if you see signs of sexist behavior:

• Lots of double standards, based on what type of person you are.
• People being evaluated not by the outcome they produce, but the (aggressive) way they do it.

The downsides of macho and bro culture

What is wrong with having a “tough” environment, or a macho or bro culture?

It’s not always a completely bad thing. There are some aspects of these cultures that are useful in a business environment. Risk-taking behavior, for example, is often valuable. Some people that come across as "bro-ish" are great coworkers!

The two main challenges I see are when:

1. Macho and bro environments can reinforce conformity.
2. The culture leaves certain types of people out.

Macho and bro culture often encourage a sort of monoculture. This may surprise the bros, because they “encourage arguments” and “encourage conflict”. But the price of engaging in conflict varied greatly depending on who you are, and your power within the organization. The founders and the other bros will engage in their macho behavior and be rewarded for it. But in my experience, this is rarely done uniformly, because most macho cultures don’t focus on making true dissent and conflict safe. They come from a place where the individual is supposed to be tough, not where the environment is supposed to make it safe for people to be contrarian. So the rules are selectively applied, and only the bros end up “winning”.

This in turn ends up leaving out a lot of people. When you are in an environment where people are afraid to think or act differently, you automatically make some people’s contributions less valued. A lot of people will see that they don’t fit in, so they’ll be discouraged from participating fully. It’s like having a football team where half the players aren’t trying to win – obviously, you won’t win the game.

If you want a team to produce better outcomes, you need a team that can have high quality conversations. Improving the way they perform is often done by improving the level of discussion they have. Macho and bro environments limit the number of people who can participate in those high quality conversations. I have seen some examples of smaller “bro” groups being effective. But they fail when they try to broaden it out beyond that group, because they aren’t able to incorporate other types of people.

They can cause talented people to leave your organization. So it's really not best for the business, and it can be harmful for the individuals involved.

Why macho behavior thrives

You will see macho behavior in a lot of places. I believe it develops for a couple of reasons.

One reason is that many people have stereotyped and narrow views of leadership. They only interpret aggressive behavior as leadership behavior. This is a filter on their perception, so anything besides that doesn’t look like leadership to them. This is understandable – I would argue it’s a bias most of us have about leadership. But it does result in a limited view of what a leader can look like and what a leader does.

A second reason this thrives is that people seem to naturally emulate their leaders. So if the top few people of an organization act in a particular way, the others will emulate them. I’ve seen this firsthand many times. I noticed myself emulating mannerisms from people I’ve worked for. I first noticed this when watching a documentary on Enron. Many of the leaders there started copying the CEO’s manner of dress, and showy behavior.

We talk like our leaders. We copy their behavior. It’s a natural thing for humans to do. So macho and bro behavior tends to be copied by the rest of the leadership and become entrenched in the work culture.

These behaviors can then be further reinforced by hiring decisions. Because everyone is looking for certain aggressive forms of leadership, all the leaders hired will exhibit these traits. It can become a deep part of a work culture.

Strategies for leaders in macho environments

What do you do when you find yourself working for a manager with these traits? Or when you find yourself working at a bro environment.?

Often, there is not a lot you can do. You can try to influence the situation, but you usually have little control of it.

What I generally do is try to assess the level of influence I have in the situation. If the leader I’m working with is self-aware, and cares about creating a high performing work culture, they may be open to feedback once you have a working relationship. You might be able to say something like, “I’ve noticed that all of us managers are all trying really hard to look ‘tough’ to you. For example, at the last meeting, I saw XYZ. I think part of what’s going on is people are trying to impress you, and they don’t feel safe bringing up bad news.” Or whatever your particular observation is.

But often, I have found that you have to play a sort of dumb game within bro environments. Let’s talk about that.

Using language to navigate bro culture

What I’ve generally found is that you almost have to take all of your work and translate it into macho language.

One funny thing you can use LLMs for is to do this translation or you. One prompt I used once is: “I have a status update. I'd like you to make it sound super macho. Like really, really aggressive. Extra machismo.”

I gave it this update: “The Slack integration project is going to be delayed. We chose to push it back two weeks because an urgent request came in from sales and we decided supporting our biggest deal of the quarter was most important. As a result, the Slack integration will be available at the end of September.”

This is kind of dumb, but it is the sort of translation you may need to use so you are heard. It’s silly, but real.

Advocating for others in bro culture

Another thing you can do is make sure that you translate other people's behavior into language the bros can understand. They will naturally miss the contributions of many people, so you may have to be the interpreter. You might see someone be criticized or punished because they’re not acting in a way the bro filter can make sense of.

So sometimes you have to translate for others. One thing I’ve found helpful is to create a motto that helps bros understand the value of other people. Or that helps them understand how I’m thinking about it.

For example, you might say, “I don't care how people do it. I care about the results.”

It sounds good, right? It’s speaking the action language that bros can hear, but it also makes space for people to produce results in different ways.

When to leave a toxic environment

Some types of macho environments can be toxic, and in that case you may (or may not) have the option of leaving.

I had a company I worked at a few years ago that I decided I couldn’t be a part of. I actually liked all the people involved, but the leadership had a macho culture and they couldn’t see the huge contributions the women leaders were making. I just saw it over and over again.

There was always criticism and scrutiny for the women in leadership. But there was lots of room for learning and making mistakes for the men. These double standards are common, but in this place it was egregious enough that I couldn’t be a part of it. If you have the ability to leave, that’s often a good choice.

How to shape bro culture

When you’re in a bro or macho culture, are there ways you can shape it as a leader? I have seen cultures change. But ultimately a lot of this does seem to come from the top of the organization, so the best thing you can do is to leverage any credibility or relationship you have to influence that person. If they are open to it, the team can work on the way they use language, and they can try to bring more diverse leadership styles into the team. They can praise people in public who bring up hard information, or counter the narrative.

This is hard, but typical leadership work. It can be rewarding if it changes, but quite frustrating if there isn’t an openness to change.

Podcast on this topic

I cover macho leadership and bro culture in this episode of Decoding Leadership. It covers almost exactly the same content:

Thank you

Image by Davie Bicker from Pixabay

One of the harder situations you might find yourself in is managing a bottleneck team. What is a bottleneck team? When other teams can’t get their work done unless you do something for them, you’re a bottleneck team.

Examples of bottleneck teams

• You might own an API or capability that requires extensions for all your clients. Any time they need changes, you need to do work for them.
• You might have to do work to spin up infrastructure for your clients.
• You might own legacy code in a different language than the rest of the organization, and many other parts of the business rely on integrating with that legacy code, but only you have the skills to work in that area.
• You might provide a service or have expertise the rest of the company uses. For example, you might have database expertise.

Why bottleneck teams are hard

There are many reasons bottleneck teams tend to be challenging. Mostly, it’s that the math is against you:

• Bottleneck teams are typically not funded at the same rate of growth as the company. This means that if the demand for your team’s work doesn’t decrease over time, your team’s time will be more scarce than the need for it.
• If there is a certain probability of a project needing work from your team, and you multiply that by all the projects at the company, then over time, your team will have an ever-growing backlog of work. And that will grow faster than your team.

If you’re a bottleneck team, you’re always overloaded. There will be many important initiatives that ultimately depend on your team to be successful. And there will be talented, influential leaders with big titles that view their success in terms of getting their priorities to happen.

How common are bottleneck teams?

There are usually a couple of bottleneck teams in any sufficiently large organization. You could say that that's just true by definition. The theory of constraints basically says there's always a single constraint in your system. By dealing with that constraint, you make your whole system more efficient.

I usually find a couple of places where all of the key projects funnel through. These teams are usually stressful, challenging places to be.

The people are often holding things down and keeping the business running. But they are often criticized and underappreciated. Why? Everything looks like failure because you can never get enough work out of them. And often they're dealing with so much pressure that their effectiveness is reduced.

Problems can stem from architecture

I have a couple of things that I usually suggest for bottleneck teams. One is that often the underlying issues are architectural in nature. It's often a good idea to solicit the help of people that can think deeply about your team’s situation, and how it could be restructured to be better.

Get protection from stakeholder bullying

One of the most important steps to take when you’re on a bottleneck team is to align carefully with your management chain. They need to understand the situation well enough that they don’t make the situation worse. And you may need to get help shielding your team from stakeholder bullying.

What is stakeholder bullying? What can often happen is that people with big titles will see that their success is tied to getting your team to do work for them. They are persuasive people, and used to making things happen, even when it doesn’t look reasonable. So they may not accept a no, and they’ll push hard. It can be difficult to say no to them.

How should you align your management team? The main thing you want is helping other people to see how their interests are aligned with yours. So you have to think very carefully about what it is that you want. Do you want more funding? Do you want a period of time where nobody interferes? Spend some time thinking through what you want, and then talk with your management chain about it. In an ideal world, the VPs will be advocating for the same thing as you, because they see it as a way to get what they need for their organization. But this is getting into the world of politics, and it’s likely that your management chain will be better at this game that you are (at least for now!)

Moving to self-service can free your team

I’ve seen teams extricate themselves from being a bottleneck team through transforming their work products to a self-service model. They were teams that were doing work for other teams. They would code something. They would create or update APIs.

To whatever degree you can, self-service is the best way to decouple teams. I have written a whole blog post on moving to self-service.

Open source approach to contributions

I’ve seen some teams take on an “open source” model approach to contributions. In this approach, the team isn’t doing the work for other people. But they do have a role in vetting the work and integrating it into their codebase. This can be less work. It doesn’t solve all bottleneck problems, because sometimes your capacity is so constrained you’re not able to help out even with reviewing other teams’ work. And sometimes the other teams’ contributions are hard to accept, because they don’t have the same level of context your team does.

But it is an approach that you can often employ.

Using “Heroes” or “Cones of Silence” to protect focus

When you’re a bottleneck team, you are often highly thrashed and it can be hard to dig yourself out of the situation. Often I’ve found it useful to preserve focus. I typically use one of two approaches:

Hero rotation

A hero role is a rotation. The person (or persons) are responsible for taking all the interruptions for the team. They are the first line of defense, so they field questions, bug reports, support inquiries, and any other communication tasks. I have a post on Hero Rotations.

Cone of silence role

A cone of silence is similar to a hero role, except it is the opposite. A cone of silence is when you protect a couple of people from interruptions. The team tries to give them space, as much as possible, to focus on their work. Often the cone of silence can be employed for critical work that will unblock the team from being a bottleneck. For example, the work that helps make your team’s work product more self-service. Or the architectural work that helps distribute your team’s ownership more broadly.

Both of these approaches can be done as rotations. And can help you while you dig yourself out of the bottleneck situation.

Rename the team

Sometimes teams can be named in ways that encourage bottleneck behavior. For example, a team called the Platform team will end up owning all the services and horizontally designed work in the company. You can sometimes rename your team to something more narrowly defined, to help clarify that your team shouldn’t own as much as people thing it should.

Shift team ownership and responsibilities

Another thing I've seen work is to shift responsibilities. It should be done in a way that makes sense for the whole company.

For example, I remember a team that was responsible for all of the scheduled jobs for the entire engineering organization. They owned hundreds of scheduled jobs. And this was one of their many responsibilities. In this case, it didn’t make a lot of sense for them to own these things, because the work these jobs did were more aligned with other teams’ responsibilities. So what the team successfully advocated for is to own the underlying scheduled jobs infrastructure, and the patterns for how scheduled jobs were run. They shifted the jobs themselves to the teams that were more closely aligned with those jobs. That sort of shifting of responsibilities can be a good move, because it reduces the surface area of your team. But it has to make sense for the wider organization, as these type of moves can be political.

Podcast on bottleneck teams

I cover bottleneck teams in this episode of Decoding Leadership:

Thank you

Image by Matthias Böckel from Pixabay

Imagine an engineer who decides to make a breaking change to an API. They think it saves a ton of time. What they may not realize is that this decision will ripple out, causing countless hours of rework for multiple teams across the organization. This is the hidden danger of fan out work.

Fan out work is a topic I haven’t seen written about before. It is something I've observed mostly in organizations that are medium-sized or larger.

Fan out work happens when a team can generate work that “fans out” to a very wide set of teams.

Examples of fan out work Platform teams

Platform teams can sometimes be a source of fan out work. This is not to vilify them. They are reasonable people doing things that they believe are really good for the company. You’ll see projects like:

• Large technical migrations. For example, a mandate to move off MongoDB to Postgres.
• API deprecation. Whenever there isn’t an easy migration path, a team creates work for all of its customers.
• Large scale capabilities. Features like RBAC can sometimes originate in a platform team out of customer needs, but the work involved can spread out to override other priorities.

Architects

Architects are another source of generated work. They create plans that represent things they think need to happen, but these plans require work and can come across as unfunded mandates, introducing slowness across every project.

For example, if an architect mandates a new pattern in a codebase, that means either of these must happen:

1. Every time engineers work in a part of the codebase, they will need to understand how to make that change, and make it.
2. Or, someone will need to go through and change all the current code to match the new standard.

What’s wrong with Fan Out Work? The cost of fan out work is often hidden

Often the easiest decision for a team member is to do the easiest thing for their local team. For example, an engineer may decide to make a breaking change to save their team a month of work. If ten other teams each spend two weeks adapting to that change, the organization loses five months of engineering time. It is an inexpensive decision for the team member, but an expensive decision for the whole organization, due to externalized costs.

Fan out work causes poor incentives

So fan out work can result in hidden inefficiencies in your organization.

If we return to the example of the architect mandating a new way of doing things: if that person calculated the amount of work mandating the new pattern in the codebase would be, it could be months of engineering work. Is the change worth a single individual spending several months making that improvement? Often, the answer is no.

Addressing fan out work

Here are a couple of approaches for dealing with fan out work:

1. Teams generating the work should do the work. For instance, if a platform team mandates a migration, they should do most of that work themselves, in all the dependent team’s codebases. This approach forces costs to not be externalized, so they fully grasp the impact of their decisions. It provides a feedback mechanism. You will need to fund teams sufficiently so they can do the work. Like on-call, this puts the feedback in the right place. This can result in better outcomes, and is my favorite approach.

2. Limit the number of major projects. You can also put limits in place for the number of major projects a team can produce. For example, you might tell architects they get a limited number of changes. Or you might tell a platform org they get one big migration project a year.

3. Calculate and consider costs. You can require people to do calculations before any fan out work is undertaken. This does require some sort of awareness of fan out work, or a way to detect it. But having people think about the expense can help. If you’re using a prioritization approach that considers effort or costs, this can plug into the value / time equation and help you plan more rationally.

4. Decouple work to local teams’ timelines. To reduce the impact of fan out work, you can allow teams to integrate the fan out work into their own priorities on their own timeline. This reduces disruption and allows better local decision-making.

I'd love to hear your experiences with fan out work. Have you encountered similar challenges in your organization? How have you addressed them, and what strategies worked best for you?

Podcast on fan out work

I cover the topic of fan out work in this episode of Decoding Leadership:

Thank you

Image by Werner Weisser from Pixabay

Kenneth Rose interviewed me for the Level Up podcast. We covered the topic of strategy work.

Here is an outline of the topics we covered:

• Where to focus when creating a product strategy.
• Common pitfalls in developing a product strategy.
• Getting leadership to problem-solve strategy.
• Implementing a product council.
• Practical advice for developing product strategy.

You can find the episode here.

Thank you

Kenneth Rose was a great interviewer. Really appreciate being on his show!

Image by Jake Parkinson from Pixabay

I’m excited to share some innovative but experimental practices today. You can use them to improve the reliability of your engineering organization. They are based on ideas from chaos engineering.

What is chaos engineering?

The idea behind chaos engineering is to test your systems by deliberately injecting failure into those systems. By doing so, you inoculate your software to those classes of failure. For example, you might fill up the disk on a server, and see if anything breaks. If it does, you learn how it fails, and you fix it. If it doesn’t break, then you’ve learned your software is currently resilient to that class of failure.

The way I remember it, the techniques of chaos engineering largely emerged with cloud computing. Early cloud computing systems were not reliable. And they encouraged an architecture which was more distributed and microservice based. Companies like Netflix and Amazon were essentially trying to create reliable systems on a substrate of unreliable systems. An early example of this was Chaos Monkey, a tool at Netflix which randomly took down hosts.

Typically, chaos engineering uses tools that inject failure, along with an approach where you use gamedays to learn about your systems, and deliberately make them more reliable.

Gamedays are a wonderful practice. I encourage you to use them! But I’m going to talk about some chaos engineering approaches that are more systemic. They are also more experimental, and less common. If executed well, they should guarantee good results. And, you might use gamedays as a regular practice while you roll them out.

😈 Evil Staging

The first technique is “evil staging”. What is this diabolical practice?

In software engineering, the idea of a deployment pipeline is to take software and gradually release it into more and more realistic and rigorous stages, until you release it to customers. Thus, a common practice is to have engineering teams…

1. Work locally (on dev).
2. Then push their work into a shared environment (staging).
3. And then finally to deploy it to production (prod).

Typically, each stage is more and more “life-like” and rigorous. For example, typically your staging environment will have more realistic data in it than dev. It will have more people using it. It will be closer to what is happening in production.

The key idea behind Evil Staging is that you make your staging environment more evil 😈. For example, it might have no free disk space. Or the processes might be restarted every hour. Or it might frequently run out of memory. All of these are realistic scenarios, that will happen to your application in production. But Evil Staging makes them happen all the time.

There are a million ways you could do Evil Staging:

• You can do it as a staging environment, or do it as a separate production environment.
• You could organize the failures with some sort of regular schedule, or do them randomly.

But the basic idea is to make these failures happen more often.

Will this improve reliability? It might not. If people respond to failures in staging by investigating them and fixing them, then this might work for you! If alerts on staging don’t cause any response, then it’s probably not a good fit.

If it’s not a good fit, read on for Gauntlet programs.

Gauntlet Programs

The second technique is what I call a Gauntlet Program.

A Gauntlet Program is where you gradually introduce faults into your staging and then production environments. But, you do it on a gradual schedule. It might take a year or two to fully ramp up the gauntlet.

The idea is that you are gradually inoculating your software from entire classes of failure. So a gauntlet program might look like this:

• Month 1: Introduce the Gauntlet Program to the engineering team. Train engineers on using failure injection tools (it could be Gremlin, or any of the alternative tools available). Communicate it in a fun way, give teams time to prepare their systems, and then they can get back to whatever other work they were doing. Then, you start the challenge. The initial challenge is to make systems resilient to unexpected host restarts within 30 days. First this will happen on staging, then it will happen in production (yes, truly!). You give teams a playbook that includes observability, gamedays, and a lot of support, as they prepare their systems for the first stage of the Gauntlet. Then the time comes, and there are some minor issues, but they are resolved, and fixed. Congratulations, you’ve just eliminated an entire category of failure from your system. And since it’s ongoing, it won’t reoccur.
• Month 3: Focus shifts to preparing systems to withstand hosts running out of disk space. The failures start in staging, and two weeks later, move to production. (Yes, truly!)
• Month 5: The challenge involves making systems resilient against running out of memory. The failures start in staging, and two weeks later, move to production.
• Month 7: Teams work to ensure their systems can handle maximum disk I/O consumption. With the same sequence of staging and production failures.
• Month 9: The focus shifts to time being out of sync. Various time offsets are introduced, following the standard pattern.
• Month 11: The challenge for this period is that latency is added to some or all network traffic.
• Month 13: The teams now face the fact that a percentage of all network traffic fails.
• Month 15: Next, the teams face the challenge of dealing with making their systems resilient to network partitions.
• Month 17: The challenge is to run out of file handles. Sneaky one that!
• Month 19: The final challenge involves making systems resilient to DNS failures. They should ensure that things degrade in a reasonable way, until DNS resumes.

Each challenge is designed to innoculate your software from a new category of common failure. As you progress through the Gauntlet, your systems should become gradually more reliable. They should also learn to be more proactive about reliability issues, and think about their software in an adversarial and proactive way. This builds a culture of proactive and systematic reliability.

You can make this as aggressive or gradual as you like. Adding a challenge per quarter might be a more reasonable approach in some environments. Or, if reliability is truly a concern, you might make it happen even faster!

Note that you can add additional types of challenges to the gauntlet. You might add some security related challenges, for example. You could even add some usability challenges, to guarantee that basic workflows in your application are never broken (this would pair nicely with training on Synthetic monitoring). And, there are a lot of limits I didn't include in the list above.

Make the gauntlet into a Reliability Race

You could design a gauntlet as a Reliability Race, where you have a maturity model, and each team turns up the difficulty level until they reach MAX LEVEL.

It could look like this:

• Newbie: have an on-call set up. Get a basic level of monitoring in place for the team.
• Level 1: have run a game day.
• Level 2: added the “host restart” failure to their software (in staging and production), and it’s running fine.
• Level 3: added the “disk is full” failure to their software, and it’s running fine.
• Level 4: added the “memory full” failure to their software, and it’s running fine.
• Level 5: added “maxed out disk I/O” failure to their software, and it’s running fine.
• Level 6: added the “time drift” failure to their software, and it’s running fine.
• Level 7: added the “network latency” failure to their software, and it’s running fine.
• Level 8: added the “packet loss” failure to their software, and it’s running fine.
• Level 9: added the “network down” failure to their software, and it’s running fine.
• Level 10: added the “file handles full” failure to their software, and it’s running fine.
• Max level, you win! Added the “DNS is down” failure to their software, and it’s running fine.

By the way, running fine can mean whatever makes sense for your application. It may mean graceful failure, it might mean you’ve decided the failure is acceptable, or it might mean doing extraordinary things to make the experience okay. That’s for you to decide. Leadership should probably provide some guidance on how to reason about these things, so every team doesn’t interpret it differently.

Making it a Reliability Race could make it more enjoyable for participants. It could also make it easier for some teams to reason about their local priorities versus the Reliability Race. For example, a team could already be doing some important reliability work, and decide to prioritize that before dealing with “host restart” failures. Every team will also have a different amount of work they have to do to meet the Gauntlet, so this makes that more flexible.

The Newbie Gauntlet

A variation of the Gauntlet Program is a Newbie Gauntlet. This is easier to introduce because you only do it for new things.

Whenever you spin up a new service, you start with all the failures in your staging and production environment. This makes it so you have to build things to be resilient at every point. It forces good reliability and engineering practices in everything you do.

The advantage of this is that it makes all your new services reliable by default. But you would have to be careful you don’t make it so onerous that people want to avoid creating new services!

Prereqs

For this to be successful:

• Your leadership team needs to value reliability.
• You probably need to have an on call rotation.
• You probably also need observability tooling (something like Honeycomb, New Relic, or Datadog).
• The rollout should ideally be communicated well, and by leadership that has empathy and can do good change management.
• I will note that a lot of SRE organizations do not have this level of support. It is similar in some ways to error budgets or SLAs in that it needs a high degree of support to be successful.

Why I like these approaches

I like these approaches because they can guarantee good results. They are systemic in nature, so they make it necessary to build your software in a different way.

You’ll notice that they are analogous to test suites. They are always present, always defending you against making the mistakes you are certain to make. They make the expectations of reliability explicit in your software, and give you safeguards to ensure you don’t violate them.

The objections you’ll hear

Objection: you shouldn’t do this because it will cause failures in production. Those failures will happen anyway – they will just come at an inconvenient time. This method gives teams time to prepare. It also communicates that the expectation is that you build things in a reliable way. Often, teams do not get this type of explicit guidance, and they can ironically be punished for building things in a reliable way.

Objection: production is already evil. You're already getting lots of signal from your production environment, because things are failing a lot! So it can seem like a stretch to add more failures to your approach. But it is valuable to have the failures happen more predictably and consistently. If you only run out of disk space once a year, you're unlikely to know about that until you run out of disk space! But if you have that failure right after you commit a change that will cause a problem, it's much better to be able to know about that right away.

I hope more companies experiment with variations like this! Let me know your experiences, and what you come up with!

Thank yous

Evil staging was inspired by some of the practices I heard from Netflix. I remember it being talked about at New Relic when I was there. I don’t remember who coined the term. I believe Gauntlet programs are my own invention, but of course I’ve been in several environments where a lot of these ideas were floating around. And I’ve worked with people who have been at Netflix, Amazon, and other places where reliability practices were pioneered. I developed Gauntlet Programs as a potential product strategy for Gremlin, when I was VP of Engineering and Product there.

Thank you to Tim Tischler for sharing with me the list of limits. And Katie Wilde for feedback on the post. Ryan Kennedy suggested moving the prereqs section earlier. Martin Wellard and Daniel Kruger asked some questions which caused me to flesh out the section on chaos engineering.

Images were generated by Dalle. I'm sure they'll look very dated in a year. Very 2024 AI generated. It will seem vintage, I'm sure.

Ash Patel interviewed me for the SRE Path podcast. He said the conversation got "spicier and spicier. At one point, I was hunting for my favorite sparkling passionfruit drink to cool thinks down."

This was the spicy part:

"One of the challenges you’ll sometimes find in organizations is that platform engineering teams can be the source of a lot of work for the rest of the organization. Every time you deprecate an API or create a standard approach that requires migration, you are creating work for all your consumers. The purpose of a platform team is to drive efficiencies in the rest of engineering, not cause additional work!"

We also talked about:

• Focusing platform organizations on their customers.
• How I've gone about diagnosing team issues.

You can find the episode here.

Thank you

Ash Patel wrote up a great summary of the conversation. Really appreciated his invitation to be on the podcast!

Image by Florante Valdez from Pixabay

What do the most effective people do to increase the velocity of their teams?

There are several roles people take on teams, that make their teams move faster. I describe one of these roles today, and how you might learn from them.

Role #4: The Nowist

The Nowist moves everything towards the present.

For a Nowist, every action either makes things move faster, or makes them move slower, and they can’t tolerate anything going slower. So they’re continually looking for ways to eliminate waiting.

Some typical questions that the Nowist might ask include the following:

• Can we just go talk with them now?
• Do we need a meeting? Could we decide it right now?
• Could we do it in three weeks instead of six weeks?

A Nowist has a different relationship to timelines than most people do. To a Nowist, a deadline isn’t something you work against—it’s the final point of failure. The Nowist wants to have things started immediately and finished tomorrow. If you have three days worth of work to complete this week, the Nowist is the person who starts it on Monday morning and has it done Wednesday afternoon.

How do Nowists work?

To a Nowist, the worst words they can imagine are “I’m waiting for so and so to do X.” For them, there is always a way to do things that are non-blocking. For example, instead of waiting for a decision on something, they’ll move ahead and inform people what they plan to do. Generally, a Nowist seeks guidance instead of approval, and broadcasts intention rather than hesitating.

So a Nowist might say, “Here are the various options and tradeoffs. I’m going to proceed with X, but if you have any concerns or feedback, please let me know.” It’s usually better to course correct and throw away a little work than to sit around and wait for consensus.

One thing a Nowist will often insist on is a default course of action. If people are deciding between M and N on a project, they might say, “you two go ahead and hash out which is the better option. We’ll assume it’s M unless you tell us otherwise by the end of the week”. This reduces the need for communication if M is indeed the option they choose. It also means that if M is a bad option, the two of them will be motivated to make a strong decision in favor of N. Defaults move things to the present.

Some Nowists think in terms of momentum. They need for there to be a milestone or deliverable every couple of weeks, or they think the team isn’t being propelled forward. A deliverable that is three months away is too far for a Nowist. They need a more immediate goal.

What are the dangers of being a Nowist?

• If you're in a position of power, and there isn't a lot of trust, Nowists can distort their organizations. People will want to go along with the Nowist, and end up creating unrealistic goals and expectations.

• Nowists can be intolerant of obstacles. So they have to take extra care when people are slowing things down. They have to really listen and understand the concerns.

Want to practice being a Nowist?

• Try this Nowist exercise. When you're in a meeting you can observe, try to observe every part of the meeting with this lens: is this making things faster, or slower? You'll find that a lot of patterns in communication slow things down, without necessarily providing a benefit. Start with the assumption that there is always a "speed things up" version of every pattern of communication. What would that be? You can then take these insights, and try to analyze your own behavior. And you might be able to influence others in their behavior.

Other velocity roles

• Subtractionist
• Mole
• Dodger
• Nowist
• (and I have some more planned for the future)

Implementing velocity roles: team approach

Having someone on your team with this role can speed things up. But there are some things you can do to practice these roles. Here’s an exercise you can do on your team.

• Have each member of a team choose one of the velocity roles. Have them choose so that each person has a role, but there are no duplicates.
• Have them keep it secret which one they’ve chosen.
• See if after a month everyone else on the team can guess which role the other team members chose.
• Rotate the roles each month, so everyone gets to try each role.
• Talk about what you learn.

Your team should be more focused, efficient, and productive. And the team members should have an appreciation for different work styles.

Implementing velocity roles: individual approach

If you are practicing velocity roles as an individual, it’s easiest if you have someone to observe. So start out by looking to see if anyone you know is good at that role. Then watch what they do carefully, and emulate anything you think is effective.

If you don’t have anyone in your work life that is good at that role, then you can create reminders to yourself to practice that behavior. I’ve also found it useful to set aside some time to think about how I could act if I were better at that Role. Thinking through it should give you some concrete actions you can take if you were better at it. Through practice, you may find you exhibiting that role in a lot of your daily life.

A speech, resurrected

I’ve presented this content in a couple of forms: several talks, internal to a company, and external. I’ve also written it up on a corporate blog (it’s since been deleted).

For some reason, this content has resonated more than perhaps anything else I’ve done. I’ve had people come up to me years after one of those presentations and talk about it! After I gave the presentation at an internal company event, someone made laminated cards for all of the roles and handed them out to every team in engineering!

I’m reworking the roles a little as I write them up, so many years later. And let me know what you think – I have never had a reliable way to understand what resonates!

Have you known a Nowist?

Are you a Nowist? Have you known other people that you now recognize as Nowists? Let me know if I missed any observations or could describe this role better!

Thank you

The first version of this was created when Kirby Frugia, Darin Swanson, and I huddled in a room and brainstormed behaviors. I believe Keizan Shaffer and I also brainstormed a version of this for managers.

Image by Annette from Pixabay

What do the most effective people do to increase the velocity of their teams?

There are several roles people take on teams, that make their teams move faster. I describe one of these roles today, and how you might learn from them.

Role #3: The Dodger

The Dodger’s job is to dodge obstacles.

The Dodger is the person on the team who finds that perfect workaround for things that might set back the team. This role makes the difference between a team that is blocked, and a team that comes up with the solution that keeps things moving forward. If you’ve ever worked with a Dodger, you know it’s an amazing role to have on the team.

How do Dodgers work?

The Dodger is great at coming up with ways to prevent obstacles from being real. The Dodger sometimes does this by refusing to accept that an obstacle is actually a true obstacle. They give themselves a little time to question and challenge obstacles before acknowledging their existence.

Some Dodgers do this through deep technical knowledge. They can go deep on a problem and come out with a solution that saves the day. Or they might just think more deeply about the constraints and situation. Other Dodgers accomplish their goals through other people—they bring others together to solve any blockages impeding the team. These Dodgers are great faciliators. They help a group come up with a solution.

Dodgers are effective because they have an unconventional approach to problem-solving. They question assumptions and constraints. They often are willing to entertain “crazy” or “bad” ideas, and use them in ways that end up being effective.

The defining characteristic of a Dodger, though, is that they make obstacles disappear.

What are the dangers of being a Dodger?

• Dodging obstacles is important, but sometimes Dodgers will focus too much on avoiding obstacles, and not enough on solving the upstream problem that causes the obstacles in the first place.
• Dodgers can sometimes avoid complete work. Hacking around problems can be its own sort of problem. Make sure your solutions aren't temporary. And make sure other people aren't always finishing your work for you.

Want to practice being a Dodger?

• Practice questioning constraints. When faced with a new obstacle, explicitly list out the constraints you believe you’re operating under. Consciously go through each of them and question whether they are necessary or true.
• Do deep dives and develop expertise. If you know an obstacle will be discussed, aim to be the person that knows more about the topic than anyone else before it is discussed. Clear your schedule and immerse yourself in the problem space.
• Practice developing multiple solutions to every problem. Try to come up with three solutions to every problem. Explicitly list out the tradeoffs of each approach.

Other velocity roles

• Subtractionist
• Mole
• Dodger
• Nowist
• (and I have some more planned for the future)

Implementing velocity roles: team approach

Having someone on your team with this role can speed things up. But there are some things you can do to practice these roles. Here’s an exercise you can do on your team.

• Have each member of a team choose one of the velocity roles. Have them choose so that each person has a role, but there are no duplicates.
• Have them keep it secret which one they’ve chosen.
• See if after a month everyone else on the team can guess which role the other team members chose.
• Rotate the roles each month, so everyone gets to try each role.
• Talk about what you learn.

Your team should be more focused, efficient, and productive. And the team members should have an appreciation for different work styles.

Implementing velocity roles: individual approach

If you are practicing velocity roles as an individual, it’s easiest if you have someone to observe. So start out by looking to see if anyone you know is good at that role. Then watch what they do carefully, and emulate anything you think is effective.

If you don’t have anyone in your work life that is good at that role, then you can create reminders to yourself to practice that behavior. I’ve also found it useful to set aside some time to think about how I could act if I were better at that Role. Thinking through it should give you some concrete actions you can take if you were better at it. Through practice, you may find you exhibiting that role in a lot of your daily life.

A speech, resurrected

I’ve presented this content in a couple of forms: several talks, internal to a company, and external. I’ve also written it up on a corporate blog (it’s since been deleted).

For some reason, this content has resonated more than perhaps anything else I’ve done. I’ve had people come up to me years after one of those presentations and talk about it! After I gave the presentation at an internal company event, someone made laminated cards for all of the roles and handed them out to every team in engineering!

I’m reworking the roles a little as I write them up, so many years later. And let me know what you think – I have never had a reliable way to understand what resonates!

Have you known a Dodger?

Are you a Dodger? Have you known other people that you now recognize as Dodgers? Let me know if I missed any observations or could describe this role better!

Thank you

The first version of this was created when Kirby Frugia, Darin Swanson, and I huddled in a room and brainstormed behaviors. I believe Keizan Shaffer and I also brainstormed a version of this for managers.

Image by Jae Rue from Pixabay

What do the most effective people do to increase the velocity of their teams?

There are several roles people take on teams, that make their teams move faster. I describe one of these roles today, and how you might learn from them.

Role #2: The Mole

Moles spy on the rest of the organization to give you intel.

They help an organization coordinate better. They might have insights like:

• Our dependency on that team’s project is a risk, because Angela just got transferred to another project.
• The support organization is about to go through a reorg so they are more aligned with product engineering. They’re doing that because the product requires a lot of specialization.
• The way we’re thinking about building this is nonstandard and could cause issues for the tools teams. We should let them know what we’re doing.

Moles are obsessed with communication. They tend to move across the organization a lot, and consume a lot of information. They read more Slack channels than anyone. They talk with more people. They also act as double agents, passing information on your team to the rest of the company.

(Also, for the non-native English speaker, a Mole is a spy)

How do Moles work?

Moles speed things up because they improve the flow of communication. They reduce surprises. They ask questions like:

• Who needs to know about this?
• How is this going to be communicated?
• Did we put this update in the wiki?

I am a Mole. I always have a mental model of who is good at what. I generally have a better understanding of what is going on than the people around me. I have found that I am able to help the people around me coordinate more effectively, because I monitor a lot of information, and talk with a lot of people.

A tip for fellow Moles: if you are a Mole, you can accelerate your learning by playing a game with yourself. You have an internal catalog of the skills of the people around you. You can pretend that they are characters for you to call upon. “What would Nadya do in this situation – she’s really good at getting people on the same page”. “What would Alex do in this situation – he’s really good at giving big areas of responsibility to people.”

If you’re working with a Mole, consider asking them questions like:

• Who would be the best person to talk with about X?
• Who are the people we might want to keep in the loop as stakeholders?

What are the dangers of being a Mole?

• At their worst, Moles can be gossips. Just passing on rumors.
• Moles also have to be careful to center themselves on the work. Some moles can enjoy the connections over the results.
• I use the idea of "spying" on other teams for fun, but if a Mole is truly passing on secret information, that would undermine trust. Moles should be transparent about the fact that they're communicating with other parts of the organization.

Want to practice being a Mole?

• Deliberately expand your information network. Look at adjacent teams, and consider how you would monitor what is going on there. Evaluate external meetings by the possibility you would acquire useful information, divided by the amount of time it will take. Look at the people around you and consider whether more communication with them would be beneficial. Then experiment and see what you learn.
• Take a skill inventory of the people around you. A fun way to do this is to create “character sheets”, which have various attributes: project management, ability to align people, technical depth, ability to get people to work together. Then score the people around you based on your subjective guess. These are people you can learn from, or use when you’re needing help in these areas. (They might also be people you can help if they have low scores in particular areas).
• Think of yourself as a liaision. Read about liaisions, which is a formal way of using Moles. Try to represent your team elsewhere. And try to represent other teams at your home team.

Other velocity roles

• Subtractionist
• Mole
• Dodger
• Nowist
• (and I have some more planned for the future)

Implementing velocity roles: team approach

Having someone on your team with this role can speed things up. But there are some things you can do to practice these roles. Here’s an exercise you can do on your team.

• Have each member of a team choose one of the velocity roles. Have them choose so that each person has a role, but there are no duplicates.
• Have them keep it secret which one they’ve chosen.
• See if after a month everyone else on the team can guess which role the other team members chose.
• Rotate the roles each month, so everyone gets to try each role.
• Talk about what you learn.

Your team should be more focused, efficient, and productive. And the team members should have an appreciation for different work styles.

Implementing velocity roles: individual approach

If you are practicing velocity roles as an individual, it’s easiest if you have someone to observe. So start out by looking to see if anyone you know is good at that role. Then watch what they do carefully, and emulate anything you think is effective.

If you don’t have anyone in your work life that is good at that role, then you can create reminders to yourself to practice that behavior. I’ve also found it useful to set aside some time to think about how I could act if I were better at that Role. Thinking through it should give you some concrete actions you can take if you were better at it. Through practice, you may find you exhibiting that role in a lot of your daily life.

A speech, resurrected

I’ve presented this content in a couple of forms: several talks, internal to a company, and external. I’ve also written it up on a corporate blog (it’s since been deleted).

For some reason, this content has resonated more than perhaps anything else I’ve done. I’ve had people come up to me years after one of those presentations and talk about it! After I gave the presentation at an internal company event, someone made laminated cards for all of the roles and handed them out to every team in engineering!

I’m reworking the roles a little as I write them up, so many years later. And let me know what you think – I have never had a reliable way to understand what resonates!

Have you known a Mole?

Are you a Mole? Have you known other people that you now recognize as Moles? Let me know if I missed any observations or could describe this role better!

Thank you

The first version of this was created when Kirby Frugia, Darin Swanson, and I huddled in a room and brainstormed behaviors. I believe Keizan Shaffer and I also brainstormed a version of this for managers. Thank you to Paulette Johnson for the point about Moles undermining trust.

Image by Roberto Lee Cortes from Pixabay

What do the most effective people do to increase the velocity of their teams?

There are several roles people take on teams, that make their teams move faster. I describe one of these roles today, and how you might learn from them.

Role #1: The Subtractionist

A Subtractionist simplifies everything.

Subtractionists see the world in terms of simplification. They take any complex situation, and distill the problem into a sentence or two. Or they ask the critical question that helps everyone focus on what’s important.

They do this by removing the non-essential. They are unusually clear thinkers, because they actually have a less complex version of the world to reason about.

On projects, a Subtractionist prevents feature creep. They cut through bullshit and focus on the core. They speed things up by removing work from consideration. A Subtractionist will say “the best project is one you don’t have to do”. Or, “the easiest way to double your speed is to do half as much.”

How do Subtractionists work?

Part of it is they ask questions differently:

• Do we really need to do this?
• How is this important to our customers?
• Could we cut ___ out and do it later?
• What if we did ___ instead? Wouldn’t that give us almost the same value, but be a lot simpler?

You’ll notice that Subtractionists use questions that require simplification. For example:

• If we only did one thing, what would it be?
• If we only had a month, is there a part of this project we could deliver that would still be valuable?
• If the quarter had 6 weeks in it, what goals would we choose?
• What is the list of things we should remove from our todo list?

What are the dangers of being a Subtractionist?

• Mostly, you have to be careful you’re distilling and not cutting out anything essential.
• Subtractionists can get in trouble when they believe they understand a situation, but they’ve oversimplified the situation or left out something critical.
• Subtractionists tend to ignore the things they have subtracted. So they tend to have bigger blind spots.
• Subtractionists can also sometimes be impatient with complexity. They can think that people who engage with complexity are overcomplicating things. Sometimes systems thinkers can conflict with Subtractionists.

Want to practice being a Subtractionist?

• Practice interrogating yourself with simplifying questions. The questions I listed above can be used to subtract. Schedule some time each week to review your projects and plans and see what happens when you simplify them. Look at your todo list and ask yourself what the one thing you have to get done today is.
• Practice using Subtractionist questions in meetings. There will be times that are ripe for Subtractionist thinking. Try and look for those opportunities, and see how these queries affect the meeting.

Other velocity roles

• Subtractionist
• Mole
• Dodger
• Nowist
• (and I have some more planned for the future)

Implementing velocity roles: team approach

Having someone on your team with this role can speed things up. But there are some things you can do to practice these roles. Here’s an exercise you can do on your team.

• Have each member of a team choose one of the velocity roles. Have them choose so that each person has a role, but there are no duplicates.
• Have them keep it secret which one they’ve chosen.
• See if after a month everyone else on the team can guess which role the other team members chose.
• Rotate the roles each month, so everyone gets to try each role.
• Talk about what you learn.

Your team should be more focused, efficient, and productive. And the team members should have an appreciation for different work styles.

Implementing velocity roles: individual approach

If you are practicing velocity roles as an individual, it’s easiest if you have someone to observe. So start out by looking to see if anyone you know is good at that role. Then watch what they do carefully, and emulate anything you think is effective.

If you don’t have anyone in your work life that is good at that role, then you can create reminders to yourself to practice that behavior. I’ve also found it useful to set aside some time to think about how I could act if I were better at that Role. Thinking through it should give you some concrete actions you can take if you were better at it. Through practice, you may find you exhibiting that role in a lot of your daily life.

A speech, resurrected

I’ve presented this content in a couple of forms: several talks, internal to a company, and external. I’ve also written it up on a corporate blog (it’s since been deleted).

For some reason, this content has resonated more than perhaps anything else I’ve done. I’ve had people come up to me years after one of those presentations and talk about it! After I gave the presentation at an internal company event, someone made laminated cards for all of the roles and handed them out to every team in engineering!

I’m reworking the roles a little as I write them up, so many years later. And let me know what you think – I have never had a reliable way to understand what resonates!

Have you known a Subtractionist?

Are you a Subtractionist? Have you known other people that you now recognize as Subtractionists? Let me know if I missed any observations or could describe this role better!

Thank you

The first version of this was created when Kirby Frugia, Darin Swanson, and I huddled in a room and brainstormed behaviors. I believe Keizan Shaffer and I also brainstormed a version of this for managers.

Image by Jae Rue from Pixabay

I often have people approach me about coaching or advisory work. But they find individual coaching to be too expensive.

So, I run group leadership coaching sessions. These are small groups of leaders who meet regularly. Each person has some time for their own topics. They are well facilitated, and everybody learns from each other. You can often get better coaching from a well run group than from an individual.

I run several groups. Participants are technical or organizational leaders, and people aspiring to these roles.

If you are interested in group leadership coaching, fill out this Google Form. I'll send you a proposal that includes the proposed time, cost, and detailed information on how it works. You can then decide if you want to proceed. Perhaps I'll see you in a group leadership session soon!

Thank you

Image by ar130405 from Pixabay

I've noticed two danger zones that organizations run into. Today I'll describe these two danger zones, and give some advice for navigating them.

I'll talk about this for engineering organizations. But I suspect it's applicable to any group of humans working at these scales.

Why are these danger zones important?

I've seen several companies waste years getting trapped in these danger zones. That time is precious for a startup, and can result in the business failing.

I've seen people that are smarter than me get into these traps over and over. I believe the reason for that is that these are structural problems. Solving them requires some deep refactoring of your organization, and most people haven't done this type of work before. So, they fail, and their company and employees suffer.

The growth traps in these danger zones interact with other leadership and organizational problems in harmful ways. For example, if you have a leadership team that tends to micromanage, these growth traps will make it worse. Or if you have a lack of leadership alignment, that will be more prominent. So the traps in these danger zones have a disproportionate impact.

Why listen to me on this?

I have both breadth and depth of experience in these danger zones. At New Relic, we had an experienced leader who helped us navigate the first danger zone. But I saw areas where we struggled, and was involved throughout the process. I spent years grappling with the second danger zone at New Relic. It was the most critical deficiency in our engineering organization for years. It was something we continuously grappled with. We worked with Jim Shore, author of The Art of Agile Development, to successfully address it. I was part of the team that fixed it, and it shaped my thinking about the patterns behind how groups of humans can operate at increasingly larger scales. It became a major theme in my leadership work ever since.

I've since worked at almost twenty-five startups, and any of them that have grown through the danger zones have run into these same growth traps. I've focused a lot of my consulting practice on helping organizations get through these traps and avoid much of the suffering you would expect.

Danger zone #1: The team trap

The first of these two danger zones is what I call the “team trap”. It generally happens sometime between ten and twenty people.

You start with a co-founder or leader that is in charge of engineering. The engineers all report to this person. There are lots of projects going on, and they get busier and busier as the team grows. Often, you'll have each individual focusing on their own project!

Because the team is small enough, everyone has a pretty decent sense of what is going on. You know what projects are being worked on, and how things are progressing. Priorities are fairly clear, and communication is uncomplicated. Usually it all happens in one place, with everyone reading everything or in the same room. Communication is many to many. The future is bright.

What happens with the team trap

As the team grows, however, things start to break down. The leader works longer and longer hours. Yet, it seems harder and harder to get work done, and execution and quality seem to be slipping. Communication seems muddled, and you hear more people wondering about priorities or what the strategy is.

This is the Team Trap. You're heading towards failure, and unless you reshape things, it will get worse and worse!

Why it's hard to fix the team trap

You will face a few obstacles to make the right changes. First of all, the founders and early people are often really smart, incredibly dedicated people. They will work harder and harder, and try to brute force themselves through the Team Trap. That won't work – it will only delay the solution.

Second, some of the solutions to the Team Trap will feel like bureaucracy to the founders and early people in the company. They'll resist the changes because they want to preserve the way the company felt early on – everything could happen quickly and effortlessly. They will often have an aversion to the changes they might need to make.

What to do about the team trap

The changes you typically need to make at this phase are structural changes:

• You need to set up cross-functional teams with ownership. This is a hard thing to do well, and if done incorrectly, can actually make everything worse! I advise you to read Team Topologies, and to get help with this. You can also try an alternative approach: FAST agile collectives.
• You'll need to start thinking about your organization's design. You may need managers. But you may need a different type of manager than you think. You'll need to think about how to design your meetings. You may need some lightweight role definition. Ultimately, someone has to start thinking about the way your organization is structured, and how all the pieces will fit together.
• Part of this organizational design is to also think through your communication design. You probably want to start segmenting communication, so that people know what they need to, but aren't flooded with a lot of information they don't need.

There is a balance to this. You don't want to overtilt towards structure, but you also don't want to avoid necessary structure.

All of this is pretty hard, and I've built a business helping engineering organizations with this (so definitely reach out if you need help with it, or find someone else to help you).

Why does the team trap happen?

Incidentally, I believe the reason this seems to happen at between ten and twenty engineers is because that's when one person can no longer reasonably manage everyone in engineering. You have to start to split the world. And once you do that, it forces a lot of other changes to happen at the same time.

It's a little like when you have a web server that is delivering content over the internet. As soon as you want a second server for redundancy or scaling reasons, all the sudden, you need a lot more in place to make things work. You may need a load balancer. You may need to think about state and caching, since it is done independently for each server. All of these concerns happen at the same time.

If you're successful in your design, you'll have a structure that will take you pretty far. Your teams will be autonomously creating value for the company. And things should go pretty well, until you hit the second danger zone.

Danger zone #2: The cross-team project trap

The next trap is with how teams work with each other. You reach a level of complexity where the primary challenge for your organization is how to ensure that anything that crosses team boundaries can be successful.

As the number of teams grows, each of them delivers value. But they aren't perfect encapsulations of delivery. Teams need things from each other. And as your product grows, you'll need things from multiple teams.

The themes for this stage are coordination and dependencies. How do you get teams to coordinate, to deliver something bigger than themselves? And how do you deal with the fact that dependencies often aren't reasonable? How do you sort through those dependencies, and minimize them?

The cross-team project danger zone occurs somewhere after about forty people. I often see it happen between forty and sixty people in an organization. At New Relic, we tried valiantly to fix cross-team projects, but we didn't really succeed until we worked with Jim Shore, and at that time there were probably 200 engineers in the organization. It was long overdue.

As an aside: It's plausible that our failure to address this earlier was instrumental in Datadog's ascendance. Why? We were much less effective in engineering at the time, and this slowed our ability to succeed in the enterprise market. Most of the bigger projects were enterprise features. Our focus on growing into the enterprise distracted us from Datadog's rise, and prevented us from addressing shifts in the way developers were working with microservices. It's possible that handling this earlier could have resulted in a completely different outcome, though there were a lot of factors involved.

What happens with the cross-team project trap

To understand the cross-team project trap, consider a couple of examples.

First, you may want to do some work that affects many teams. For example, let's say your customers are asking for role-based access controls. This is work that many teams will need to focus on. Yet the enabling work might be done by one team. This can require coordination.

Another need you'll see increasingly is that multiple teams need similar functionality. They might both want a similar table user interface. Or they might both require a similar API. Or they might depend on similar data. This type of work tends to require teams to depend on other teams to do work for them. This is a growth in dependencies.

At some point, coordination and dependencies grow to become your most serious obstacle to delivery.

You'll know you're in this second danger zone if you see some of these symptoms:

• There is a lack of confidence from the rest of the company that engineering can deliver large, important initiatives. The general track record is that engineering ships late, if at all.
• You see lots of heroic effort to deliver anything that crosses team boundaries.
• You may have a few people who are unusually good program managers, but even they have failures. You might see opposing instincts to add more structure, or operate more “like a startup”. A few really experienced engineers who can get things done that are held up as saviors. But the general default is that things don't ship well.
• You have areas of the organization that are such hot spots that they go through waves of failure. Often because they are the hot spot for dependencies.

Why it's hard to fix the cross-team project trap

When I was at New Relic, I was leading up the engineering side of a new analytics product. It was an ambitious product. We had widespread agreement that it was a top priority. However, we needed things from many parts of engineering in order to deliver on the complete vision for the product.

Those dependencies didn't seem optional. So the way I attempted to handle this was by acting as a program manager. I tried to organize my dependencies as projects. Each of them would update on progress and risks. Fairly standard stuff for program management.

But what I found is that the structure of the organization didn't make execution on this type of project possible. It was mathematically impossible. Every team had their own priorities. Even if they thought we were the top priority, that was subject to change. If they had a reliability problem, they had to do work to address that problem. Sometimes something new would come up, and bump our priority back. I was essentially making plans that weren't based on anything structurally sound.

It was difficult to fix at that point. I couldn't tell all of engineering how to operate! At the time, I didn't know what would fix the situation. We had tried for years at New Relic to crack the “cross-team project” problem. We rewarded people who were good at project and program management. We hired people that were good at it. We even made delivery of cross-team projects part of our promotion criteria! But ultimately, we didn't make the structural changes we needed.

The challenges you face to fixing these at this stage are mostly that there is a lot of organizational inertia. Changes can feel threatening. People can feel demotivated to make changes when they are under so much stress. Working within the existing system can take all of your bandwidth, so people will be reluctant to work extra to extricate themselves from the mess. And structural fixes can intrude on leadership turf, so you need a high level of support from the very top of the organization to make your changes.

This is not something you can just ignore. It will continue to get worse and worse, until any project that crosses team boundaries ends up being impossible to complete.

What to do about the cross-team project trap

These are the types of changes I recommend if you run into the cross-team project trap:

• Centralize cross-team priorities (possibly with a product council) and teach teams how to work with those central priorities.
• Define organizational and team coordination models. For example, move platform teams from service model teams to self-service. Make your product teams act as independent executors, assuming no dependencies in their projects. Carefully design which teams act in an embedded model.
• Use program managers for cross-team initiatives.
• Limit the number of cross-team initiatives.
• Reorganize your teams mostly along cross-functional lines. Reduce dependencies between teams. Or, you might experiment with FAST agile teams.
• Reduce the size of projects (using milestones or increments).
• Ideally, get help!

Why does the cross-team project trap happen?

I have a hunch these growth traps are the result of complexity jumps that occur when you add a layer of management. The first danger zone corresponds to when you add managers, the second danger zone is often when you add directors.

When you have this jump in complexity, you have to shift the structure of the organization. Otherwise, you have a mismatch between what is necessary for that structure to be successful, and the way it really is.

This is not to say that adding managers or directors is what causes the problem. You can run into these growth traps even if you have managers or directors. It's just that you need them both at the same time.

Incidentally, this is why I am bullish on FAST agile. I think it may allow you to have a simpler organizational structure for a longer period of time. Combined with some of these other structures, I think the potential benefits outweigh the fact that it is a new, less well-developed practice.

Thank you

I try to credit people who have influenced my thinking or directly affected my approach.

Much of the first danger zone is through my own observation. I'm not sure I've seen anyone else articulate it. But I would guess others have seen the same thing – when I talk with other leaders or venture capitalists, they have a look of “yeah, that sounds familiar”.

For the second danger zone, there isn't a place I can point to (that I remember) that highlights this as a scaling barrier for organizations. I'm pretty sure something must exist! For how to address it, my biggest source of credit goes to Jim Shore. His work at New Relic was quite effective, and it was a career highlight to work with him and the Upscale team to design and implement the solutions. While the coordination models have been my own pattern language for organizational and cross-team work, you'll notice he is credited on many of them.

Image by atul prajapati from Pixabay

Hiring junior engineers is challenging. I've seen so many junior engineers get hired but then be supported poorly.

There is a lot to write about this topic, but today I'd like to highlight a wonderful writeup by John Hyland on what is the best example of new engineer hiring I've encountered.

John set up and ran the "Ignite" program for several years, and shares how it ran. It's kind of perfect!

How the Ignite program works

1. The junior engineers are hired (either in cohorts or continuously).
2. They go through a project-based onboarding for two weeks. The project is designed to introduce the basics of working with internal systems: Git, JIRA, standups, code reviews, retros, and so on.
3. They're assigned mentors.
4. They go through several rotations with teams. These last several weeks. They work on real work for those teams, contributing in a couple of defined ways.
5. They have various social and networking meetings, such as an alumni event, so new engineers can see what it looks like to be successful and also build social connections within the company.
6. After a few rotations, they're placed on a permanent team.

There are a lot of interesting hacks they used to make this program work, so I highly recommend it if you're interested in the topic!

What I like best about this program is it just seems so effective:

Advantage for companies

• Save money. These programs can pay for themselves. It's less expensive to hire newer engineers. If successful, these programs can make it possible for engineering organizations to tilt hiring towards less experienced engineers, potentially saving signficant amounts of money. Companies that are able to more successfully harness less experienced engineers will have a competitive advantage to other companies, due to improved cost efficiency. New engineers can have a higher retention rate than with inferior onboarding.

• Better value delivery. Better onboarded engineers will deliver value sooner and better, producing more value. The program increases internal connections within the company and across teams, which can lead to innovation and better collaboration. During onboarding, can be a way to address high importance, but low criticality work, like tech debt. This can improve long-term company performance, because these things tend to drag down long-term delivery if left unattended to. Companies can benefit from greater internal diversity (which has been shown to improve product quality and decision-making)

• Less disruption. New engineers are better oriented and less disruptive to their teams. They're able to contribute almost immediately.

Advantage for new engineers

• They are more likely to succeed.
• They feel more valued because they're contributing more to the company.
• They understand engineering processes so feel less disoriented as they join teams.
• They are a better fit for their teams, because they've had the chance to see a couple of options.
• They have a cohort of fellow junior engineers they can network with and get support from.

Advantage for the industry

• Creates a better pathway for engineers to succeed and grow. The trades have well established apprenticeship programs -- these could fill a similar niche in software.
• Improves diversity and inclusion in our field.

Applicability

This program is most appropriate for medium sized companies or larger. But it mostly depends on how quickly you're hiring. At New Relic, we did it when we were well over 60 teams, but it could be done at an earlier stage.

John: "if you hire at least 20-30 engineers annually, it's likely that 10-15 of them could be early career, which is easily enough to justify putting some resources into doing a good job with it... That's a higher ratio of early career engineers than many companies go with, and depending on the specific kind of engineering the company does, that may be reasonable. So, you'll want to adjust those ratios to whatever you're comfortable with, and consider what resulting hiring rate is needed to warrant an early career hiring program based on your particular circumstances."

That said, a lot of the lessons from this can be applied to earlier stage companies. I think the design of this program is a really good thing to emulate.

Where is this fabulous post?

The post is a three part series. You can start here: Running New Relic’s Ignite Program, Part 1 - Hiring.

Interview with John Hyland

I interviewed John about their advice for people to make hiring and onboarding engineers successful.

Thank you

John Hyland is a former colleague of mine, and the author of the post I feature here. Contact them if you're interested in further details after reading this post. I suspect they'd jump at a chance to do similar programs at other companies, and even to help others who are doing so.

Image by kim_hester from Pixabay

Shomik Ghosh, Partner at Boldstart Ventures, interviewed me for the Software Snack Bites podcast. One founder described said, "this was fire" 🔥.

I was really happy with the interview. The main theme was what are the things that startups struggle with as they grow. It's worth listening to if you're interested in these topics:

• Common anti-patterns in engineering teams
• How teams can better communicate
• What hiring practices work well
• How to think about structuring career paths
• How to adjust to the "do more with less" environment today.
• What common advice do I give after seeing 23+ startups?

As Shomik put it: "For anyone interested in learning how to run effective eng orgs, this is the episode for you."

You can find the episode here. And it's on Apple and Spotify.

Thank you

Shomik Ghosh was a fabulous interviewer. Really appreciated the care he put into every part of the interview. The prep was meticulous, the questions were thoughtful, and the editing and promotion afterwards was quite well done.

Image by Positive_Images from Pixabay

FaST is the most interesting organizational practice I've learned about in a while. Today I share what FaST is, and explore whether it's worth experimenting with. TL;DR? Quite compelling, but still experimental.

What is Fluid Scaling Technology (FaST)?

FaST is an agile software variant. FaST focuses on self-organization within very large teams. These large teams are called Collectives. They can be a few people, to as many as 150 people[^1].

Twice a week, the Collective meets. At the meeting:

• Team members show and tell what progress they’ve made.
• The product leader reiterates the vision and sets direction. They give context on the top projects. They introduce new priorities as work completes, or priorities change. And typically they show the priorities in a visual way, on a wall or virtual equivalent.
• People within the Collective self-organize and self-select into smaller teams to deliver the top priority work.

The most interesting part of FaST is these self-organizing teams, so let’s describe them in a little more detail.

Every meeting, people volunteer to lead teams, and people volunteer to participate in those teams. When someone volunteers to steward a team, they step forward and say, “I’m going to work on [one of the top projects]”. People then join the teams that have been created and work in that area. The minimum team size is two, so you need people to “vote with their feet” to form the team.

The teams formed do not have to be around the priorities that product has laid out. For example, a team can form around improving the deploy pipeline, or improving monitoring for some flaky services.

Outside of the hour a week in meetings, people spend most of their time in their self-selected, self-organized teams. Theoretically, you could change teams twice a week. But in practice, people end up figuring out who they like to work with, and teams tend to settle down after the first month or so. They remain fluid enough that people can move as business needs shift, or people with expertise are needed in different areas. Moving teams is an expected, low-effort change. So what this means is that during the Collective meetings, there is a pattern of people recreating their teams, but the composition tends to not change as much as you might imagine.

There are a lot of other details like how bugs and on-call are handled. I’ll cover some of these topics later. But the basic core of FaST is these self-selecting teams, working within a larger Collective, on projects. The Collective meetings are structured, and focused on giving context on the needs of the business, and communicating progress against those priorities.

Summary of differences

FaST is much different than the way most agile software organizations run today. Here are the differences as I see them:

Common practice (SCRUM, Kanban, or “Lightweight Agile”) FaST

Meeting density Medium to High Varies from company to company, but commonly daily standups, demos, grooming, estimation, retros, etc). Low Two, half-hour meetings a week. Others as desired.

Ability to scale Unit of scaling is the team. Growing and splitting and creating teams (and their areas of responsibility) is a major focus of management. As business priorities change, requires reorgs to reflect the change in priorities. Unit of scaling is the Collective. Practically speaking, the number of tracks of work will scale similar to common practice. However, adding people and shifting flows of work is more fluid and self-organizing. Less challenges with “too small” or “too large” teams. Less concerns about viability of teams, and matching teams to areas of responsibility. Untested at very large companies, but theoretically seems sound.

Code ownership and expertise alignment Strict code ownership, which helps divide the world into what individuals can master. Good alignment of incentives because people work in their area. Challenges are to make sure each team can maintain and operate its codebase. Moving responsibilities tends to be painful. Changing business priorities often results in reorgs. Practices can be decentralized and fragmented, although this can cause challenges. For example, you can let teams choose their own programming languages, choose different ways of working, and different coding styles. Shared code ownership. Requires good standards, skilled team or good review practices. Changing priorities is less difficult. Reorgs are almost non-existent, unless you’re changing Collective boundaries. (Think of how much management time is spent in reorgs!) Practices need to be more standardized, or you will have problems. Standardizing practices is a good thing, so incentives are in the right place, but failure mode is probably worse.

Architectural patterns The ideal architecture is microservices. Large monoliths need to be well-factored. Generally messy outside microservice environments. I would expect FaST to be more agnostic about architecture. Should work with monolithic codebases or microservice environments. And with a blend. Should make moving to microservice environments more gradual.

Rollout patterns Strict code ownership tends to require organization-wide design of which team owns what. Implementation of these changes is always big-bang. Requires disruptive, large reorgs. Can be done incrementally. You can start with a team or two, and gradually add additional teams to it if it works. Reorgs should be infrequent. Will still happen at a Collective level, but that should be less frequent.

Alignment mechanisms Typically use OKRs, all hands meetings, and written communication to align people. The Collective meetings are a built-in All Hands meeting twice a week, where you reinforce goals and set context for the team. If leadership treats that meeting seriously, seems very high leverage.

Employee retention Practices can range from feature-factory grind and top-down, high-control environments, to high performance teams that understand their business context and customers, and continually deliver value for their business. Thus, retention will vary significantly by company. I expect practices can vary, similar to common agile practices. However, FaST lends itself to employee retention because it is aligned with intrinsic motivation. Employees can choose who they work with, what they work on, and get reinforcement that their work matters. I would expect this to result in better overall satisfaction. One challenge might be politics. Self-governed groups sometimes deal with conflict poorly, so that could backfire.

As you can see, FaST is a very different approach than what most companies are doing today.

So what are the tradeoffs?

As I see it, these are the tradeoffs of FaST agile:

• FaST may offer better organizational scalability
• FaST can be experimented with incrementally
• FaST should result in fewer reorgs
• FaST makes it easier to respond to changing priorities
• FaST could be better for intrinsic motivation and retention

But…

• Many environments are incompatible with FaST
• You might have to manage self-management
• FaST requires a lot of work
• FaST can run counter to a desire to allocate people to areas
• The FaST materials are full of jargon

We’ll discuss each of these in turn.

FaST may offer better organizational scalability

When you scale organizations, you typically do so by scaling them “horizontally”. You grow the organization a team at a time. Each of these teams own a slice of the product, and you often have a platform organization that makes the product teams more effective.

The complexity of the organization explodes as the number of people increases. If you have ten people in engineering, they can all communicate with each other. A hundred people in engineering cannot all talk with each other. And the codebase won’t fit in any individual’s head. So you segment this complexity into teams, and each team has an area they can focus on.

FaST is interesting because it is an attempt to scale organizations “vertically”. Instead of adding more teams, FaST attempts to make bigger teams, called Collectives. The Collectives still own code, but the self-organized teams within that Collective do not. And you do still have to segment communication. But the teams are more dynamic and continually evolved. Instead of reshaping teams through reorgs, it is a continual, expected part of the way you operate.

Team size is a familiar topic for many heads of engineering. How large should teams be? Let’s delve into this topic a bit, as it helps us understand how FaST claims to scale better.

How large should teams be?

When you spend time around VP of Engineering folks, occasionally the topic of engineering team size will come up. You’ll hear arguments for small teams, and arguments for large teams. Both sides have merits.

Advantages of small teams

• Less coordination and communication overhead within the team.
• Can move quickly as a result.
• Less streams of work, so easier to focus and produce results.
• Easier to manage.

Advantages of large teams

• Flatter organization. An engineering organization with 150 engineers in it will have 3-4 directors with ten-person teams. It will have 6-8 directors and 1-2 VPs with five-person teams. Where does the confusion in an organization come from? Probably the leadership, so large teams can reduce confusion.
• More resiliency. Someone can take vacation and the team can continue just fine.

So small teams are great within their team, but less ideal because there are so many of them that it increases organizational complexity. Larger teams are better for overall organizational complexity, but less ideal for performance within each team.

The approach many leaders take is to make large teams that focus on fewer streams of work. This reduces internal team complexity, and also reduces organizational complexity.

You can look at FaST as an approach which tries to achieve even greater organizational simplicity through ridiculously large teams. And it tried to get the benefits of having small self-assembling teams as well.

Scaling organizations is challenging work

Much of my consulting business is helping organizations deal with scale. There are several phases of scaling challenges organizations run into.

The first barrier is typically at the 15-25 engineer mark. This is the time that your amoeba has gotten too big, and needs to split. You see these types of problems:

• Communication channels become ineffective.
• Engineering starts having delivery and/or quality problems.
• Managers or leaders feel overwhelmed. Adding more people will make it worse, not better.

And a second barrier is at around sixty engineers. At this point, you start seeing a lot more coordination problems:

• Cross-team projects rarely are successful.
• Product work swamps platform teams, or vice versa.
• Inability to complete large initiatives.

I’ve seen brilliant people fail again and again to navigate these step-order changes in organizational complexity. My hunch is that these step order increases in complexity correspond to when you add an additional layer of management.

Growing engineering organizations is a real skill. It requires a deep understanding of how organizations and people work. You learn about things like:

• Coordination models to get groups of people to work together.
• Stream-aligned teams versus functionally designed teams.
• How to do reorgs, where you shift people and code around as business priorities change.
• How to divide products into areas of technical responsibilities overlapping with product investment.
• Setting up communication channels around teams.
• Setting up on call and reliability programs to support your product effectively.
• Setting up platform organizations and developer experience initiatives to combat complexity.
• Etc.

Even if you have people with this expertise, it requires a lot of effort from your management team to deal with scaling. So if FaST can scale better, you’ll unlock a lot of the potential of your management team to focus on something else. And, because your step-order changes occur so much less frequently, you could potentially be making your organization much easier to manage.

A thought experiment around FaST agile

So, imagine you could scale team size. Instead of teams being five or ten people, what if you could have effective twenty-person teams? Or sixty-person teams?

On the face of it, this is a ridiculous proposition. Teams that large are not effective. Even twelve-person teams are a stretch. I sometimes see the resumes of people who have had twenty or thirty people report to them. My immediate reaction is to think they worked at a terrible company. Having that many direct reports is an “organizational smell”.

The central premise of FaST agile is that if you add self-organization, and self-selecting teams within this larger Collective, you can scale your teams more horizontally. Each team (FaST calls them Collectives) can be much larger. And then the teams people work with on a day to day basis are self-assembled within those Collectives.

If the larger Collectives were effective, they would reduce organizational complexity significantly. The typical software startup could go years longer before having to divide up into areas of responsibility. The codebase wouldn’t need to be rigorously segmented out into areas that each team owns. This would free up significant management time that managers could use to focus on the product and teams.

So this is the central thing we need to test: does self-assembling, and self-organization offer a way to operate that is as effective as smaller, designed teams?

If it does, then it will offer a way to scale organizations that is potentially an order of magnitude better at scaling organizations. Why that much? Let’s compare an organization growing with teams of size six people versus an organization scaling with “teams/Collectives” of size sixty people.

Org size How many six person teams (conventional teams) How many Collectives (FaST)

18 people 3 1

60 people 10 1

90 people 15 2

120 people 20 2

…

300 people 50 5

There are a lot of assumptions built into this argument, and you might be able to poke some holes in it. But even if you do, FaST is essentially allowing for a larger organizational unit to own an area of the product and codebase.

FaST is new, so there isn’t a ton of evidence right now about whether FaST teams operate as effectively as conventional agile teams. But I suspect the answer to that question is yes. I have a couple of reasons to believe that. First of all, Jim Shore’s preliminary experience with it was quite positive, and he’s someone I trust.

A second reason I am bullish on FaST is that I’ve had some experience with large groups of people doing self-organization. And I was surprised at how well it worked. And I have some experience with FaST, and found it to be an effective approach, though I didn't do it in a large Collective.

In small companies, of course, you see a lot of self-organization. But as companies grow, you standardize things and remove self-organization except within teams. However, one of the most interesting experiments I’ve been a part of was a large-scale self-organization event at New Relic. We redesigned all of our teams, and asked everyone in fifty software teams to select which team they wanted to be a part of. We had constraints on the skills needed for each team, and had defined what everyone needed to own. It ended up being way more successful than anybody anticipated, even us organizers.

So although the jury is out on this, my hunch is that this practice has a lot of potential in the right contexts.

FaST can be experimented with incrementally

This isn’t mentioned in the FaST materials, but one thing that jumped out at me as being interesting about FaST is that you can experiment with it incrementally.

You can implement FaST Agile on one team as an experiment, and then use a blob approach where you swallow additional teams over time. If it goes well, continue to swallow additional teams. If it doesn’t, abort the experiment.

FaST should result in fewer reorgs

Change happens in organizations. Priorities shift, people leave, areas of the product have unexpected growth. Previous technical decisions bite you and you have to do additional work. New opportunities arise that require investment.

All of these things have an impact on the structure of your organization. You have a set of teams, each of which have their own areas of ownership. As this change happens, you refactor your organization over time to perform the best you can. These refactorings are “reorgs”, and in growing organizations, they can happen frequently!

Reorgs should be a lot less frequent with FaST. The unit of reorganization is so large that you shouldn’t have to do it as often.

A potential downside of this is that you don’t have teams with defined areas of responsibility. So you could have a diffusion of responsibility and have a poorly maintained code commons. (We’ll talk about this a bit later)

FaST makes it easier to respond to changing priorities

When priorities change in conventionally structured teams, you have teams set up to do the work. If that team structure doesn’t support the new priorities, you have to refactor your organization.

For FaST, changes in priority are more fluid and continuous. As long as the responsibilities lie within the Collective, the teams just organize themselves around the work, and it’s like another day.

The Collective meeting structure also makes changing priorities less dramatic. You essentially have a built-in All Hands twice a week. At the Collective meetings, you can continually communicate context and educate your team on your customers and market. This should help keep your team better aligned.

When I contrast that approach to something like Quarterly OKRs, it seems more fluid and like it would do a better job of aligning people. It does require a committed effort from the product leader to continually share context and priorities. But some of the best teams I’ve been a part of had a product leader acting in this fashion, so I suspect this is time well spent.

FaST could be better for intrinsic motivation and retention

Many teams can feel like their process is designed as tools to control people. And it can feel like working on an assembly line.

FaST embraces self-organization as a fundamental component of its design. It’s so fundamental, it won’t work without self-organization. It requires a completely different toolkit for management, and it’s mostly (though not completely) incompatible with top down, hierarchical approaches.

Daniel Pink famously outlined three aspects of intrinsic motivation:

• Autonomy – A desire to be self directed, which increases engagement over compliance.
• Mastery – The urge to get better skilled.
• Purpose – The desire to do something that has meaning and is important. Businesses that only focus on profits without valuing purpose will end up with poor customer service and unhappy employees.

With FaST, you’re able to self-direct your work, helping you feel autonomy. You’re able to choose to focus on work that improves your skills, achieving a sense of mastery. And you’re constantly reminded of how your work is connected to what’s important for the business, helping you feel a sense of purpose. So although it’s not guaranteed, I see a potential for organizations implementing FaST to see high retention rates.

Self-organization also provides some handy signals that can keep a community of people working better together. The “jerk manager” type of person gets a signal that their behavior isn’t welcome. How? They step forward to lead a team, and say the work they want to do. If nobody joins their team, the work doesn’t happen, and the team doesn’t form (teams have a minimum of two or three people on them, or they don’t happen).

Self-organization also allows people to deal with painful aspects of work that may not get attention in other companies. For example, if the test suite is terrible, someone can step forward to lead a team fixing that issue. It’s perfectly reasonable to do work that isn’t explicitly a business priority. But it’s in the open, and others have to join in for it to happen. This can help avoid feeling like you’re in a feature factory.

Finally, you’re able to choose the people you work with every day. When I’ve seen self-organizing teams in the past, this was an unexpected benefit: people chose to work with people they wanted to work with. And those teams were much stronger than I expected.

The thing this all would be balanced against is any new pains people would experience with FaST. One challenge might be with informal power dynamics, or potential politics.

Many environments are incompatible with FaST

Because FaST is so new, it is riskier to implement. FaST won’t make sense in every situation. And there are more unknowns. It’s best to think of it as an experimental approach.

Which environments are most conducive to trying FaST out? And where should you avoid FaST?

The more qualities from this list your company has, the better the fit for FaST:

• An interest in management innovation and experimentation.
• High trust, low-fear environments. Or at least a desire to avoid command and control in leadership.
• A good guide or facilitator of the process. Someone who can run an effective Collective meeting, set up the process, evaluate and actively manage it. Someone who understands organizational dynamics and how humans in groups work together. (You could hire me for this).
• A medium to high rate of change in priorities. FaST is less useful if priorities aren’t shifting.
• A team that is collaborative, self-directed, high-performing, and continuously learning.
• A leadership team that can figure out its priorities.
• You have a product leader who can talk about the priorities and business context at each Collective meeting. And they can communicate effectively.

The less these traits are true, the more headwinds you’ll face with FaST. I don’t think you need to be in a perfect environment for it. In some ways, I think wanting to be these things is more important than actually being these things. So perhaps the most important success factor is that leadership is on board with making space for it.

You might have to manage self-management

Self-organizing systems can work effectively. But they aren't free. They are more like managing a community.

You'll need to create constraints and means to incenvitize the right behavior. And you'll need to carefully watch and manage to make sure things don't go off-track.

Any organization can have bad actors, or people that aren’t aligned with the interests of that community. I remember once talking with an engineer who told me (I kid you not) that they didn’t think they had an obligation to produce value for their employer. Some engineers might want to focus on things that develop their skills or make them a more valuable employee, rather than things that are good for the company that employs them.

With FaST, you are signing up for managing a community of people.

In conventional situations, the hierarchy makes clear who is responsible for this. In FaST, I suspect it would be easy to try and avoid conflict and “let the members figure it out”. This could result in informal power networks dominating in a sort of tyranny of structurelessness. If you do use FaST, this is something I would guard against.

The other extreme would be to not let the group figure problems out, and to have management handle it completely. This could also be harmful.

So FaST will require good facilitation, careful observation, and occasional intervention.

FaST requires a lot of work

The biggest reason you might not want to experiment with FaST is that there is a lot of hidden work with FaST. It requires a lot of change in your organization, to an unfamiliar way of operating.

You can compare FaST to working in a new computer programming language. The new language seems promising because it has so many new ergonomic features that it seems way better than anything you’ve seen before. But, you don’t have frameworks and libraries written in this new language. So you’ll have to build a lot of it yourself.

So the biggest disadvantage of FaST is that it’s not a well established practice. You’ll need to make up a lot of things to deal with incompatibilities between how FaST operates, and conventional practices. Here are some examples:

Someone may undo all of your work

FaST is incremental, so the cost of trying it out is low. But honestly a lot of leaders have their own approaches, and may see FaST as weird and non-standard. So a change in leadership can result in completely eliminating all the work you've done to create this innovative system. That's just a practical downside.

How does performance management work?

On conventional teams, you have a manager who oversees the work, and coaches individuals. You have performance reviews, and if someone isn’t a good fit, the manager can intervene. Although there are problems with performance reviews, most people understand how they work, and they serve a purpose in the organization.

On FaST teams, there isn’t really an obvious way to do performance reviews. Does the person’s manager even know what they’re doing or how they’re working? You can’t even evaluate “teams”, because they are transitory.

So the whole notion of performance management needs to be reinvented. While that may be a good idea anyway, it’s something that will require thought, and invention.

What is the role of engineering managers in FaST?

There are a lot of ways to structure an engineering manager role. I generally recommend having the engineering manager responsible for project management because it gets them to work side by side with their team, without putting them in an area where the power differential causes problems. There are a lot of valid ways of defining the engineering manager role, but that’s one I tend to gravitate to.

In FaST, the engineering manager role is less clear. You don’t have long-lived teams, so it’s harder for the engineering manager to work side by side with their direct reports.

You could have engineering managers lead the self-assembled teams. Or you could have them be pure people managers. Or you could have them be player-coaches.

All of these have tradeoffs, and some pretty big downsides. FaST seems to diminish the need for as much engineering management. You still need people to hire, do performance management, and oversee process. But maybe not as much as in a conventional organization? I’ve seen many organizations suffer because they don’t value management as a discipline. But I would guess that FaST organizations need less management.

I’m really curious to see if this plays out, and want to hear from organizations that experiment with FaST. What do you do with management?

How will you ensure code quality without areas of ownership?

Code ownership provides natural incentives for keeping code quality high. It’s in your self-interest, because your future self has to work in your present code.

In a shared code ownership situation, you have to go to greater lengths to ensure code quality. My recommendation would be pairing or mobbing as a required practice.

You’ll also need stronger standards for code patterns. You’ll need code linting. And you’ll need some sort of architectural decision-making. You don’t want to have twenty patterns for how your frontend code handles state. These are problems you’ll have in most software organizations, but they’ll be much more pressing in an organization using FaST.

One thing a lot of organizations under-invest in is training and communication around effective software design patterns. And conversations to make sure everyone is aligned on which approaches to use, and when. These are probably even more important in a FaST organization.

Shared code ownership is a practice that has some precedent. It would be worth spending the time to dig into how to be successful with it if you proceed with FaST.

What happens with multiple Collectives and crossing Collective boundaries?

Work that crosses Collective boundaries is essentially undefined in FaST. And large initiatives requiring high degrees of coordination are also under-specified by FaST. So you’d have to invent solutions to these situations.

Any organization large enough to have multiple Collectives using FaST will run into cross-Collective projects. The patterns for addressing these things will be similar or analogous to how you resolve it on a smaller scale. But this is largely uncharted territory, as far as I know. So you’ll need to be using some coordination models to solve these problems when they arise. It will be an act of invention.

How does FaST deal with on-call?

On-call isn’t specifically mentioned in the FaST guide. So you’ll need to invent a scheme to deal with it.

In conventional teams, the standard advice for on-call is to have each team responsible for its own operations. And to have everyone on the team on call. This implies you should have teams of at least four people so the burden isn’t too bad. The thinking is that having teams on call for their work product incentivizes them to bake reliability in. This helps them ensure they have a decent on-call schedule.

With FaST, you might create an on-call rotation that has tiers (follow this runbook and escalate if that doesn’t fix it). And you’ll need to explicitly keep a map of who is able to support what. This won’t map to your teams.

This isn’t super hard to do, but it is less common than conventional on-call practices, and will require management attention.

What about fixing bugs and support escalations?

FaST has an optional role called a “Feature Steward”. They are kind of an expert for a particular feature, and are the point of contact around that feature for stakeholders. They aren’t required to work continuously on that feature, but are required to have a continuous understanding of it.

Like on-call, you’ll need a mapping of features to people with expertise on those features. This will be important for both bugs and support escalations. And you might as well combine it with on-call as well. You’ll need some way to triage issues to the right people.

You’ll also need to make sure you have a mechanism for filling in knowledge gaps when expertise is isolated to one or two people.

One thing you’ll have to decide is your policy for fixing bugs. Do you want them to always be fixed, slowing down other feature work? Are the bugs fixed by the person who introduced them, or do you want some other scheme? And how will you determine who is responsible for triaging bugs? Probably a rotation of some sort?

Support escalations will also require some effort. The point of contact is the Feature Steward for an area. But what if the Feature Steward is on vacation? You probably want more than one person knowledgeable about each area. And what if the support question ends up requiring work? Do you just do the work, or feed it into the central priorities?

None of these are hard problems to solve, but they are management work. And everything you do will have tradeoffs of focus versus responsiveness.

What about reliability incidents and incident retrospectives?

FaST does not define how incidents are handled. Your on-call patterns will largely determine how incidents are handled, and will likely involve whoever knows how to fix the situation being pulled in to fix it.

FaST doesn’t describe how to handle incident retrospectives. I would recommend doing something like the following: schedule an incident retrospective to happen before the next Collective meeting. One of the aims of the retrospective would be to identify a few pieces of work that could be done that would either reduce the likelihood or severity of whatever caused the incident. Another would be to learn all you can from the incident.

At the Collective meeting immediately following the incident, I would share what we learned, and also make an automatic priority for the next cycle be to do some of the work identified in the retrospective.

At New Relic, we didn’t use FaST, but we had a similar policy. We called it “Don't Repeat Incidents” (DRI). It was among the best things we ever did for reliability. The rule was that DRI work was automatically more important than other priorities. Thus, an incident always resulted in either less scope, or a deadline getting pushed back. I have a whole post on Don't Repeat Incidents.

How do designers work in FaST?

A challenge many designers have is focusing on working with the engineering team, or doing work ahead of the engineering team. Or have them operate both ways. You’ll hear strong arguments for both ways of working.

FaST doesn’t specify how this should work, so you’ll need to decide for yourself. Do designers sign up for the teams to work on? Or are they a service organization, which work ahead on things, and act as consultants when people need something from them? You’ll need to decide. I would probably have the designers sign up and be part of the teams, but again this is an example of the type of decisions you’ll need to make when adopting FaST.

How do product managers work in FaST?

The product role in FaST is mostly to prioritize and communicate priorities. There is a lot that isn’t really described in the FaST manual about the work outside of that.

The assumption in the FaST manual seems to be that the product managers inspire and communicate, and then the engineers work on the features.

To the degree that you can, I would get engineers talking with customers, and have them gain expertise in the product area they are working in. Ideally, they would be doing the work, demoing it to customers, and getting feedback on it from them. Having the product managers facilitate that, and leveling up the engineers’ ability to do that effectively, could be a valuable part of their work.

There is a lot of flexibility in this model. You could do the typical product-to-engineering handoff, or you could have them discover and problem-solve the space together, with customers.

So as you can see, there are many gaps in FaST. You’ll need to problem-solve the rest.

FaST can run counter to a desire to allocate people to areas

When I ran a FaST Collective, one unexpected source of friction with FaST I found was that you can have some situations where parts of the organization are wanting to see a dedicated effort to be allocated against their needs. You can do this with FaST, but there can be a pull towards allocating a team or a set of people against a stream of work.

The FaST materials are full of jargon

You may find the FaST website and manual confusing. You’ll have to navigate a lot of jargon and annoying terminology to get at the gold of FaST. As an example, here’s the awful definition of FaST Agile that attempts to define the practice in version 2.12 of the FaST Guide:

“What is Fluid Scaling Technology? Fluid Scaling Technology combines Open Space Technology and Open Allocation to create a lightweight, simple to understand, and simple to master method for organising people around work - that scales. FAST is the acronym for Fluid Scaling Technology. Fluid Scaling Technology for Agile is FAST Agile.”

So. Bad. And this is very representative. You’ll see lots of references to Collectives, Value Cycles, Open Technology, Teal transformation, Theory Y, etc. This is all presented with no explanation, and really even if it were explained it wouldn’t be helpful. They’re speaking to a niche audience of agile theorists, and focusing on attribution rather than usefulness.

They also will change terminology frequently. They used to be FAST Agile, then they were FaST. I find it excrutiating.

Conclusion

So where does that leave us? Is FaST worth experimenting with?

I believe the answer to that is yes, but with caution, and in the right contexts. You can play around with it within individual teams. And if you have leadership support, experiment with it gradually within larger organizations.

Please share with me if you do experiment with it. And if you’d like help rolling it out in your organization, contact me. I’ll either help you directly, or connect you with others who can help.

Further discussion

I talked with Jim Shore about his experience with FaST on Decoding Leadership. One topic we talked about that I haven't seen anywhere else is the role of management on FaST teams.

Thank you

Early versions of this post were… really bad. I’d like to thank Seth Falcon and Davy Stevenson for their critique. They both pointed out some major problems with the post that made me back up and rethink my approach. And they made some excellent points that I incorporated into their own sections of the post. Eventually, I rewrote most of the post, and I was much happier with the outcome. Thank you! Seth has a newsletter on engineering leadership worth checking out, and Davy (like me) does startup advising and coaching.

Jim Shore introduced me to FaST agile. He has some excellent talks on his experiences with it. And we’ve met a few times and discussed the implications and implementation of FaST. Jim is the author of The Art of Agile Development.

Image by Kanenori from Pixabay

Footnotes

[^1]: I believe 150 people is probably unwieldy for a Collective. I suspect sixty people is a better maximum size.

I’ve been doing advisory, fractional, and interim engineering leadership consulting for more than five years now. People in my network contact me often because they are considering it for themselves. I wrote this up to share what I’ve learned over the years about being a consultant.

Why consult?

You can optimize for money or for schedule flexibility. Optimizing for money requires working a lot harder than at a normal job. If you’re willing to focus on building a business (instead of whatever your specialty skills are), you can have a higher financial upside.

For me the appeal was schedule flexibility rather than money. I enjoy the ability to shape my schedule. And I like that there are downtimes between large contracts. I make less than I would as a full-time engineering executive, but I have a lot more flexibility in my schedule.

A second advantage for me is that I have accelerated certain types of learning. I’ve now worked at more than thirty startups. That gives me a skillset and a perspective I never had previously. I have much more certainty about which of my skills to apply in what situations. In a world where many engineering leaders learn by copying from their environment, this gives me a rare amount of breadth.

There are many disadvantages and tradeoffs that you should be aware of before jumping into consulting.

You are running a business, and businesses have overhead

The first thing to be aware of is that there is a lot of overhead in being a consultant. At times, the overhead can be half the work you’re doing in a week.

What overhead?

• Constantly meet with potential clients.
• Write up proposals.
• Lots of correspondence to remind people to make decisions. (You’d be surprised at how many people take the time to meet with you and want to see a proposal, but ghost you after you send the proposal).
• Market yourself to find potential clients.
• Register and renew your business.
• Maintain your books.
• Invoice customers. Track payments, remind people who are late.
• Administer your health insurance.
• Set up your own 401k plan.
• Set up liability insurance.
• Deal with contracts. Negotiate contracts. Hire lawyers to create standard contracts.
• Pay estimated taxes. Pay additional local taxes if you’re a business. Potentially declare yourself as an S-Corporation, so you can save tens of thousands of US dollars a year on taxes. But deal with all the overhead of running a corporation.
• Network, and be helpful to people. Talk with other contractors to figure all this stuff out.

The amount of time this takes varies depending on the type of contracting you do. And it can vary seasonally or between contracts. But you should expect to spend a significant amount of time on these things.

Note that a lot of these things are flexible. They may depend on your appetite for risk, and your willingness to be less profitable. And you can ease into them. You may decide to not incorporate as a business, or forgo S-corp tax advantages, just to keep things simple. However, even if you keep it simple, you should expect a lot of overhead.

Finding clients is the major challenge

One of the most important factors for your success is the rate of incoming leads (potential clients). If you have a high rate of leads, you can probably build a sustainable business.

You’ll need a strategy for finding those leads. Here are approaches I see people take:

1. Your network. If people know you do exceptional work, they are more likely to consider hiring you to do work for them. And they’ll be more likely to refer you to other people. So you need the people that know your work are aware that you’re available to be hired for consulting work. They need to easily pattern-match your skills against problems they hear about. The stronger your network is, the more this is an option. So if you have a large following on Bluesky, or a lot of former coworkers that are now at companies all around, your network will help a lot. My own assessment of my network wasn’t accurate: I thought I didn’t have an especially strong network until I went into consulting. Then I realized it was actually quite amazing!

2. Partners or benefactors. It can often be in someone else’s best interest to refer you. If you can come up with people that have good connections with potential clients, and will make introductions for you, then you are getting referrals that are much more likely to use your help. For this, you would need connections who can see obvious ways you can help them out. Usually this means it’s their business to do so. Venture capitalists, or consultants who need specialized skills and would refer you.

3. Content marketing. If you can offer content that is useful to people, it can also drive awareness of the work you do. This requires some skill in writing and a perspective that is unusual or interesting in some way.

4. Be a subcontractor. If you know someone that could use your help, you can often get work from them.

I’ve used the first three approaches, and get referrals from all three sources.

When you don’t have enough leads coming in, you’ll end up spending all your time finding clients. This can feel like self-promotion, and you’ll be doing it a lot.

If focusing on self-promotion sounds icky, my advice is to focus on being helpful and on finding clients the ways that you enjoy the most. I write my blog because I enjoy it and do it in my free time. Writing has a side benefit of sometimes resulting in people wanting to employ me.

Finding initial clients

I had good results by letting my whole network know I was doing consulting work. I let people know what I was doing, and that I was available to help companies with certain types of problems. I was surprised how many people set up connections with companies that could use my help.

The first time I did consulting, I converted my full-time role into a consulting position as I left the company. If I had wanted to, I could have used this as a bridge into full-time consulting. I’ve seen others do that. This approach won’t always work. But if you have a specialist skill, this can be a route into consulting.

You should either have some savings, or some form of “bridge” work you can do until you have enough work to have a sustainable business. For myself, I gave myself a deadline, and told myself I would try it out and see if it worked. If it didn’t, I would go back to searching for full-time work. To my surprise, I had a sustainable business within a couple of months. I’ve been told that this is unusual, so you probably should expect it to take a lot longer than that.

You need to have skills that are valuable to people

One piece of advice I received early on was to narrow my focus so that it’s easy for others to understand what I do (thanks Mike Dauber!). And be careful what early clients you choose, as they’ll guide how people perceive your work.

People will pigeonhole you. So it’s often better to specialize than to be a generalist. Why? People will only pattern match you against one or two things. You want people to think about you as an option when a problem comes up.

I struggled with specializing, because I’ve been doing this work for a long time, and I’ve been fortunate enough to work in situations where I acquired deep skills in many areas. But nobody is going to pattern-match me into all of those areas. So I chose a couple and focused all of my communication around those areas. But I probably don't do this as much as I should.

You should have a level of experience in your area that is unassailable. If you’re doing leadership consulting, for example, you probably should have gotten a VP title or two. If you’re focused on founder consulting for startups, you should have started a few startups.

This is not to say there isn’t space for less experienced people in consulting. But I think it’s a tougher road to travel. You’d need a deeper network, or better connections. Something else to balance out the lack of depth in your area.

Variability can be a large stress factor

Consulting work is often feast or famine. You’ll have more work than you know what to do with. And then a big contract will end, or a few clients will leave at the same time, and you’ll have less work than you like.

This can be stressful. When you’re busy, you’re not spending time developing new business. And you may take on more work than you should (be careful and avoid this if possible).

When you’re not busy, it can be difficult for many people to enjoy the extra time. After all, you don’t have the income, and you may be stressed about where the next client is going to come from.

I’ve had people tell me this stress is what caused them to leave consulting.

The key is to have a lot of savings, so you can ride out months without income if necessary. And you need to find a way to ensure that enough clients are contacting you about work all the time, that you have some confidence you’ll always find work.

One thing that helped me was to track and model my business. I kept track of the leads coming in and the percent that became customers. And I track the mean and average length of time a customer will use my services.

This has given me a model I have some trust in. I know how many leads I need to see each month for my business to be sustainable. I know that there have been times when I have a lot more or a lot less leads, so I’m used to the variability. Even as the economy worsened in 2022 and 2023, this has given me some ability to feel less anxiety and enjoy my “time off” between larger contracts.

In general, many people that do consulting say they become much more aware of the money they make. When you’re working full-time, you have paychecks come in, and you don’t have to think about it much. With consulting, I notice every invoice that is paid, and am very aware of the work that is bringing in next month’s money.

Disadvantage: lack of long-term work relationships

Humans are a tribal, group-oriented species. When you’re a consultant, you’re usually an outsider to the companies you serve. You’re often a temporary person. So people don’t invest in their relationship with you.

As a result, many consultants feel like they lack a team. This can feed a sense of disconnection with people.

I personally find this varies quite a bit, depending on the work. But if you crave a social circle at work, or like to build long-term relationships, consulting can feel unfulfilling.

This is not to say you don’t have work relationships as a consultant. They’re just different, and they tend to shift more. You might have a broader network of people you know, but interact less deeply with most of them.

Advantage: breadth first learning

When I left my first VP of Engineering role, I had been at that company for almost nine years. I had no idea how applicable the things I had learned were to other companies.

Several years into consulting, I’ve now worked at thirty startups. I have a very clear perspective on what works in what contexts.

I like to describe full-time work as depth-focused learning. You’re learning deeply within a certain company context. You’re going deep on a particular company and situation.

In consulting, your learning tends to be more shallow from the places you’re at, but you have more of them to draw from. So it is breadth-focused learning.

Both seem desirable to me, and if you’re focusing on learning, it is probably beneficial to move back and forth between the two.

Disadvantage: not building anything durable

One of the joys of being the person in charge of an organization is that you can make it better and better over time. It can be a thrill to make an organization thrive. And you get to see the compounding interest of all your work.

Consulting is usually less direct. There are moments where you see the impact, like when a client recently took stock of things and said, “six months ago, we were struggling to deliver. And, now we’re really delivering at a new pace.” That can feel amazing. But you don’t truly own things long term.

Advantage: don’t have to care as much about politics

Most consultants have multiple clients at a time. This and the nature of the work insulate you from work politics.

This can be liberating. You are being paid to be objective and do the right thing for the company. As a consultant, it can be easier for you to bring up hard topics, because it’s often something they expect of a consultant. There is a weird dynamic where companies often pay consultants to make changes that an internal person wouldn’t be able to make. I’ve even seen leaders at companies use this approach deliberately: they hire a consultant to roll out changes they know are important. When the outside person is pushing for it, it can be easier to make it happen.

Part of the dynamic is that they’re paying for this advice. A colleague once told me that if you are a consultant, and you want people to make a big change, you have to get them to pay a lot of money for it. This investment helps them to literally “buy in to” the change. While I have no idea if that is good advice or not, it’s always stuck with me as an interesting observation.

Disadvantage: clients don’t always listen

Depending on the contract, you may be advising people, or you may be directly doing the work. When I’m in an advising situation, I find that my effectiveness varies. I’ve had some clients that mostly want to talk about fairly unimportant topics. I try to challenge them to think bigger or redirect, but there can be a limit to how far that will extend.

I also have had clients that don’t act on the discussions we have had. This can be frustrating, because they aren’t truly benefitting from your expertise if they’re not putting it into action. In these cases, I share the observation that things aren’t changing, and attempt to address the fact that they’re not acting upon their environment.

Your effectiveness can be hampered by the focus or topics your client wants to discuss. But it’s your job to provide outsized, highly leveraged value.

About interim roles

I’ve had six interim roles to date, so let me share some observations of what these roles are like.

In my experience, they average about six months in length. The shortest was three months, and the longest was nine months long.

It can be difficult to always be doing interim work. This is for a couple of reasons. The first is that it can be challenging to line up the next interim role when you’re currently doing one. You may not know how long the current role will last, so setting up a second role to start can be fraught.

It’s hard to do multiple interim roles at a time, because they tend to want to be full-time roles. So it compliments other types of work, and can be a good way to start in consulting.

The money for interim roles is fine. In my testing of the market, I’ve found it to be more price sensitive than I would expect. Pay is higher than being full-time, but only a little. Due to gaps between roles, it ends up not being lucrative.

Interim roles also tend to be intense. Some of the most challenging work I’ve ever done has been in interim roles. Why? People usually just want to hire a permanent person into a role. An interim role is often required when things are on fire. They can also be useful when there isn’t a competent person to do the hiring for the full time role. But generally I’ve found these roles to involve a lot of organizational firefighting. As a result, I’ve flirted with burnout a couple of times during interim roles.

The flip side of things is that if you enjoy fixing urgent organizational challenges and making situations better for people (as I do), it can be rewarding work. And I’ve learned a great deal from my interim engagements. It’s a way of doing shorter-term engagements as a company but still have the ownership for the organization on your shoulders.

One oddity of interim roles is that you’re not free to make change in the same way you are as a permanent person. You only want to make changes that the next person wouldn’t want to revert. I avoid some of the more innovative practices in my toolkit, because I don't want the next leader to roll them back. In a full-time role, I would have more time to institutionalize these things, so that provides more flexibility.

One thing to watch out for is the hatchet job. You’ll sometimes be brought in to do the dirty work, like laying off a bunch of people. My biggest learning doing interim roles is that it is important to understand the situation you’re getting into, and make sure you’re aligned with the people hiring you on the course of action to take.

Another important factor is to assess how many people you can do work through. I’ve had a couple of times I’ve done interim roles only to discover there weren’t any managers to do the work. I ended up doing mostly hiring. That is fine, but it ended up delaying my impact on the company by months.

About fractional roles

Fractional roles are roles where they’re hiring you for a fraction of your time. These roles tend to come in a couple of varieties:

Project work

This is work where the person hiring you points you at a problem, and your job is to make it better. For example, a leader might hire you to uplevel the whole management team. Or to hire up some new teams, and improve their hiring process. Or they might hire you to roll out an incident management process. To get them onto continuous integration. Or to set up engineering levels. Or to set up a data team for engineering.

These roles make sense when they have confidence in your ability to do this work. And when they don’t have someone internally they can point at that problem. Or alternatively, when you have expertise they don’t have in-house.

Evaluations

Evaluations are when the leaders need information from an objective expert. They want a diagnosis from a trusted party. Why is engineering not delivering, for example? Or can you evaluate our project management practices and give us your recommendations?

Fraction of a leader roles

These are roles similar to a normal full-time role, except you’re doing it on a fractional basis. You might work two days a week, for example.

The challenge with these roles is assessing how much work is actually required of the role, and crafting the role to be effective. Often an experienced executive can accomplish a lot in less time, especially if your job is narrowly focused.

About advisory and coaching work

Advisory work is where you have working sessions together. They’re similar to coaching. I’ve written a whole post on advisory work.

This type of work is enjoyable, but the main challenge is finding enough clients to make the work sustainable. Most people seem to merge this work with other types of work, because they can't get enough advisory clients.

Tip: how much should I charge?

When you first start out, you may think you’d like to charge an hourly rate. Generally the rule of thumb is two (or three) times what you would get paid hourly for a full time job. Why that much? Your rates need to include unpaid vacation and sick time, time between gigs, additional risk, insurance, 401k, etc. And consider all of the overhead I mentioned above.

It’s fine if you want to start out hourly. But you’ll probably undercharge. And the psychology of hourly rates is that you’re setting yourself up to be a commodity. Why? When you’re being compared against others, they’ll mostly be thinking about their evaluation, and then they’ll directly compare your hourly rate. This can lead to a race to the bottom and make it harder to make a living.

Instead, you want them to be thinking about the business impact you’ll have. Pricing based on projects or a retainer seem to be the best models to strive for. I typically charge based on a monthly retainer. I’ve had this book recommended to me (let me know if you find it useful or not).

So what is the answer to how much you should charge? You should charge as much as people will pay you for. That’s why it is so important that you have a lot of incoming leads. The busier you are, the more you can test higher rates. The less busy you are, the more you should lower your rates. This is how to set your pricing.

About the economy

The startup and technical markets have been brutal. Many of my clients have had layoffs, and many have cut back on advisors or contractors. I’ve lost some clients due to the economic climate. But I also seem to be doing better every year, even through these times.

My overall read of the situation is that it’s becoming more competitive. More people are entering fractional, advisory, coaching, and interim work. I once joined a Slack group with 5000+ fractional leaders. I’m sure many of those people are curious, but that’s a lot of leaders!

As a result, people are needing to differentiate themselves, and the most experienced people are still making a good living at it. And the rest of the market is becoming less profitable and less desirable. This will probably result in non-differentiated work being paid less well, and overall being less desirable.

It’s possible that every person entering consulting is also making space for additional consultants to fill those roles. And if this trend continues, we might see “contract executives” become a more accepted practice in the future.

Should I try consulting?

It’s up to you whether to try consulting. It’s not a better or worse thing to do. Hopefully, this post illuminates the tradeoffs.

For me, the main benefits have been that I can sometimes work a part-time schedule, and get a decent compensation for that time. And I've been able to support people across a many companies. And I've learned a great deal from working at so many companies. But there have been a lot of disadvantages as well: cyclical work, less permanent relationships, and a lot of overhead.

If you’re thinking about consulting, my advice is to only do it when you have a nest egg and can live without the income for six months. Give yourself a set amount of time. See if you get the traction you need. And if you don’t, you can always go back to full time work!

The easiest way to become a consultant is to make your current employer into your first client. Is that a possibility?

Thank you

Seth Falcon reviewed a draft version of this and made some helpful suggestions.

Darin Swanson is the person that got me into consulting. He suggested some improvements to my conclusion in an earlier draft of this post.

John Hartley reviewed early versions of this post and provided excellent feedback and suggestions. Much of the structure was influenced by his suggestions!

Mike Dauber gave me the advice to focus my offering as a consultant to be narrow enough that people could easily understand what I was doing.

Image by SusuMa from Pixabay

I'd like to share a complete guide to how to hire and interview engineering managers. I'll cover every step of the process, and even include a sample interview plan you can use. I think you'll find a lot of surprises, and some genuinely useful templates and questions to use.

This approach can be used for other roles as well.

Figure out what the role is you’re interviewing for.

First of all, you need to be really clear on what you’re looking for.

Hiring an “engineering manager” isn’t actually possible. There are many different ways that role is defined. So you need to figure out the role you’re actually hiring for.

I have a post on the many forms of Engineering Manager. Read it, and figure out the areas of responsibility for your engineering managers.

Determine which assertions you’re making

After you’ve figured out what archetype of manager you’re hiring, you next need to write down the specific qualities and skills you’re looking for. I call this “assertion-based interviewing”, because you’re making a list of things you would like to assert are true about the person you’ll hire.

A few example assertions are:

• Has product engineering experience in a startup environment.
• Understands our product area well.
• Shows an ability to break down projects into smaller pieces that can be incrementally delivered.

Each of these are areas you’d like to assess with the candidate. But you can’t do everything, so you’ll need a list of prioritized assertions you care about.

You can read about how to create a list of assertions, and this general approach, in my post on Coordinating your interviews with assertion-based interview plans.

Create the interview format

Your next step is to design the interview flow. If you’re working with an internal or external recruiter, they’ll probably be the first person the candidates talk with. What steps happen after that?

You should have as few steps as possible. So I like to define interviews with these steps:

Others will often insist on adding steps. For example, a founder or VP might want to interview finalists. Ideally you will have three steps, but four can be okay. If you’re getting above that, your process is too onerous.

Note that sometimes you’ll schedule the final interviews over multiple days. To save time for your interviewers, you can add a “circuit breaker”. Do a quick evaluation of the candidate after the first day. If it’s trending really poorly, to stop the interview from proceeding to the second day.

Create an interview plan

Based on the type of Engineering Manager, and list of assertions you’ve come up with, you can then define an interview plan.

You can start with my template for an Engineering Manager interview. This is based on an Engineering Manager interview for someone that is responsible for People, Projects, and Process.

You will want to modify this based on the role, and your individual needs.

Start from my template. This is for Engineering Managers that are responsible for People, Projects, and Process. You can modify it as you see fit.

Determine who will do the interviews

After you’ve put together a draft of the interview plan, you’ll need to put people against each of the interviews.

I recommend pairing people on interviews. Although it’s expensive to do so, I find that people often get into a rut with how they interview. Then their practices won’t improve, and you’ll get weird results from your interviews. Pairing also increases the amount of people that are trained to interview. And you can ensure that good notes are being taken.

You’ll also learn a lot from interviews where the two people present give different feedback on the candidate. I had an interview recently where two people interviewing a candidate at the same time ended up giving completely opposite feedback: a strong yes, and a strong no! You can learn a lot from that.

Become best friends with your recruiter

Set up some time with the recruiter you’re working with. And talk through the interview process. Make sure you’re really clear on who will be doing what. I’ve found a wide variety in the expectations of recruiters. Some do a lot more than others. The best recruiters I’ve worked with can own parts of the process for you, and have a good mind for improving that process continually. Other great recruiters have been able to take on an increasing amount of the screening interview. But there are also a lot of different places a recruiter can focus, and you should be clear on how much they’re doing sourcing, candidate screening, scheduling for candidates, and so on.

I usually set up weekly half hour sessions to talk through how things are going and make improvements.

One thing I like to do is to request a few things from the recruiters that can speed up the whole process. This ultimately can make them more successful, because it can reduce the amount of time it takes for them to fill a role. And that’s often something they’re evaluated on. But it can also demand a lot of their time, which can reduce their ability to focus on the role. But I have a list of things I’ve seen improve hiring speed that I like to float by them whenever I can.

Set up communication channels per role

At a recent role, a recruiter suggested that we set up a Slack channel for each of the roles we were interviewing for. We were having a lot of communication that was happening in side channels. This streamlined our communication significantly, and is now something I recommend.

It’s important to set up expectations for what can be communicated in the channel. For example, you don’t want people biasing other candidates. But typically that’s not a major challenge.

Then do a kickoff, where you go through what you’re looking for.

Whenever I introduce a new role we’re interviewing for, I like to do a kickoff meeting. It’s typically an hour-long meeting, where I talk through the position. I talk about what we’re looking for, and go through the interview plan.

I do sometimes schedule this after we’ve kicked off interviewing, because it can be difficult to schedule so many people. But it’s important to get this in place within a week or so of the role being opened.

Schedule time to meet with each of the interview sections, and go over the interview and customize it together.

After I kick off the interview, I aim to meet with all the interviewers and go through their part of the interview. And ask them to refine their part of the interview.

Honestly, I sometimes skip this step, or schedule it a few weeks later. But I think it’s important. Why? You want to make it clear to the interviewer that they own their portion of the interview, and that they can alter the format of the interview.

And this extra step will result in improvements to the interview that will pay off in the quality of the assessment.

After each screen, calibrate with the recruiter

I’m usually the person doing the hiring manager screen. Sometimes the screening interview doesn’t go well. At those times, I look for any signals that I can share with the recruiter. I like to share my feedback after every screen with the recruiter. So I’ll post my notes, then say whether or not the candidate will proceed to the next steps. And most importantly, if they didn’t proceed to the next step, I try to mention anything that would be helpful to the recruiter as they’re talking to future candidates.

For example, I might say:

• “Interview with Lisa went quite well. Proceeding to interview”
• “I’m passing on Ari. Nothing I can think of that we can calibrate on in future recruiter screens – they seemed like a reasonable candidate”.
• “I’m passing on Jeff. He didn’t seem like he had enough experience in unstructured environments. That might be worth screening for in the future. Let’s talk on Wednesday about that.”

After each final interview, do a debrief session

Most debrief sessions on candidates are dull affairs. Instead of just reiterating the feedback everyone wrote down, use this candidate debrief session format.

My goal is to get a good signal on whether a candidate should go to a larger, more expensive interview. So I view the hiring manager screen as a shallow version of the whole interview. So after each debrief session, I consider whether I need to tweak my screening interview. I’ll often learn about things I could be screening for. For example, if I see that I’m not doing a good enough job screening the technical skills of candidates, I might talk with the person doing that interview and figure out what I can ask during the hiring manager screen.

Thank you

The idea of a communication channel per role is something Mailani Burton came up with when we worked together.

Image by Eak K. from Pixabay

Most candidate debrief meetings are super-boring. Everyone just reads the feedback they already put into the application tracking system.

A couple of years ago I was introduced to a much better approach, and I'd like to share that today.

A better format for candidate review meetings

Instead of a meeting where people summarize their feedback, you as hiring manager do some work ahead of time. Go through all the written feedback.

1. Make a list of observations that seem to be common between people.
2. Then, more importantly, make a list of observations that contradict each other. Did some people have feedback that doesn’t match up? Did people bring up concerns that didn’t get responded to?

During the meeting:

1. Quickly summarize the feedback on the candidate
2. Then move to the important part: go through the points people had disagreement about, and dig deeper on them, in a safe and curious way. For example: “Lisa, I noticed you were concerned about how the candidate communicated about their employer. But Josh said glowing things about how they talked about their team. Can you both share a little about that portion of the interview, what questions you asked and what you uncovered?”

Your objective is to get to a set of tradeoffs with the candidate. You want the hiring manager to be able to say, Candidate A is stronger in these respects, and weaker in these respects. Candidate B has these other unique strengths, but is less well developed in these other areas. This gives the hiring manager the information they need to make an informed decision.

Why this format works

I’ve found that a lot of observations and assumptions break down when people have these type of discussions. An observation like, “they’re not very good at receiving feedback”, will become way more nuanced when you have a discussion about how you asked the question and what they said. And when people have opposing feedback, it forces your interviews to improve, because that conflict becomes apparent.

Thank you

Shaun Yelle introduced me to this meeting format.

Image by Stefan Parnet from Pixabay

When you’re navigating new territory, it is essential to have a guide. And if you want to grow your skills as rapidly as possible, it helps to learn from someone who has done it before.

Why use an executive advisor?

An advisor is someone who helps you produce better results with your work. They…

• Suggest new ways to solve problems.
• Help you navigate problems you haven’t dealt with before.
• Point out mistakes you might be making.
• Accelerate your career growth.
• Provide new mental models for thinking about your work. Giving you more tools for your leadership toolkit.

How do executive advisors work?

Every advisor is different, so you should find out what approach a prospective advisor uses. I think my approach is fairly common, so let me share how it works with my practice:

With most of my clients, we conduct working sessions. They are one hour sessions, and are weekly, biweekly, or monthly. In these sessions, my client brings their current problems, and we problem-solve them together.

Each of us brings an essential part to the working session. I bring a lot of experience (both in terms of years doing this, and across many companies). I’ve seen it all before. My clients have their own experience and expertise. But most importantly, they have much better context on the problem and environment.

We talk through a few ways to solve each problem. And we review the tradeoffs of those approaches. I usually suggest solutions my clients wouldn’t consider. And I share tradeoffs they would miss. They in turn point out ways the environment may bias the solution in a particular direction.

As we work through these examples, we use them to clarify principles and mental models. That way, we are both solving an immediate problem, and building a muscle for addressing future problems.

Some of my clients use me to test their plans. They’ll share what they’re thinking, and have me walk through the tradeoffs, or suggest some alternatives they might not have thought of.

We walk out of these sessions with solid plans and new ways of addressing problems. It’s like pair programming, but for leadership!

Other advisors may have different styles or ways of operating, so be sure to inquire into their approach.

When to use an advisor

I see several profiles of leaders who reach out for advisory help:

Leaders who are in new territory. Typically leaders approach me when they’re getting in a little over their heads, or their current approaches seem to break down. (This often happens at ~20 engineers or ~50 engineers in an organization). Symptoms are too many direct reports, too much relying on them, engineering velocity slowing down, or quality problems emerging.

Leaders who are promising but need support. Often very talented leaders who have a little less years of experience seek out advisors to make sure they’re well supported. Sometimes their boss may even insist on it. This can help bright and capable leaders step into situations that may otherwise feel risky for the company. (Hint: you can use that to your advantage to get companies to take a risk on you by pairing the responsibility with an advisor)

Leaders who are growth minded and want to be as effective as possible. These leaders are aware that their growth is magnified if they get good feedback on their work. So they seek to accelerate their growth by choosing an advisor who will give them new perspectives.

Leaders who are part of a work culture that encourages growing their leadership. Some companies invest in their leaders, because they view it as a high leverage place to invest. Making a leader 10% better has a greater than 10% impact on their area of the company. In these companies, most of the senior leaders have executive advisors or coaches.

How to choose an advisor

I recommend you interview a couple of advisors to determine which person is best for you. Ask people in your network for recommendations. Select a few, and contact them.

With my potential clients, I like to do a practice session. This gives the potential client a feeling of what it’s like to work together. Ideally they will do the same practice session with a couple of advisors, and see what kind of insight they get from each person. And they should also evaluate how comfortable they feel being vulnerable or showing their weak areas with the advisor.

Do keep in mind that sessions will get better over time, as you develop a working relationship together. But the sample session will give you an idea of what it’s like to work with the person. So I recommend asking for a sample session.

What to look for

What should you look for in an advisor?

• Someone who has been through the stages you’re going through.
• Someone you communicate well with.
• Someone you can be vulnerable with.
• Someone who listens as much as they speak.
• Someone who doesn’t have the same solution to every situation.

The last one is something I think it’s important to emphasize. How flexible is the advisor’s thinking? Do they respond to every situation with the same solution? That’s often a red flag.

The problem with determining the effectiveness of an advisor is that many people will confidently give you bullshit suggestions that can actually make your life worse. As humans, we often conflate confidence with competence. What I would look for is the level of insight they give you. And compare it with what others give you when given the same information. Another suggestion is to ask them about their thought process and why they are making that recommendation.

Rates for advisors

Rates vary greatly depending on the advisor. I’ve seen rates that vary by an order of magnitude! If you’re on a budget, you can probably find someone fairly inexpensive.

But you have to be careful with your choices. You’re hiring someone for their expertise, not just for them to coach you (more on the difference between coaches and advisors later).

Ideally, your company should be paying for the advisor. What I tell companies is to use this heuristic to determine whether the rates are reasonable for an advisor:

If you took the yearly cost of the executive advisor, and compared it to how much it would cost to hire an additional engineer, how does it compare? Compare both the anticipated impact of the advisor, and the cost. When you consider the impact on the company, it’s generally a very smart investment, even if it looks quite expensive. A good advisor can help you avoid expensive mistakes, and improve the trajectory for an entire department. That’s often worth it even if it’s quite expensive.

Advisors can be expensive, and even feel out of reach for individuals. Why? Consider the economics of the situation. A highly experienced engineering leader can command a salary of $300-500k/year, plus 0.5-2% of the equity of a company, and bonuses. Total comp can be $350-$1MM a year.

Advisors also have a lot of overhead to just be an advisor. To give you an idea from my own consulting: I have 4 or 5 hours of meetings with prospective clients for every client that signs up (and I think that’s a good ratio). I need to have about four hours of meetings a month just to be sustainable as a business, and it fluctuates greatly. I have to manage taxes, health insurance, and deal with unpaid vacation. When you add all these things up, the rates can end up being higher than you might expect.

I personally provide a 50% discount for individuals, because otherwise many of them can’t afford it. And many of them find even the discounted rate out of their reach. (This is why I started offering group leadership coaching.)

If possible, go through your company. An easy way to ask about this is to ask if they offer executive coaches or executive advisors. If they don’t, ask if they’d be open to it. Many leaders don’t want to seem “weak” by asking for this. You would be surprised how common executive advisors are. Many companies do offer this kind of investment in their employees, or are willing to. You may be surprised that the person you’re asking already uses an advisor!

Most advisors charge either hourly, or by the number of meetings. My own rates are monthly charges, based on the frequency we plan to meet. That seems to be pretty common.

What about using an executive coach?

Here’s what ChatGPT says is the difference between using a coach and an advisor. I have to admit it’s a good summary of the difference.

"Both advisors and coaches can provide guidance and support, but there are some key differences between the two roles.

An advisor typically has specialized knowledge or expertise in a particular area and offers advice based on that expertise. They may provide recommendations, opinions, or solutions based on their experience and knowledge to help a person make informed decisions.

On the other hand, a coach is more focused on helping a person achieve their goals and develop their skills or abilities. Coaches typically ask questions and use various techniques to help their clients identify their own strengths and weaknesses and work towards self-improvement. Coaches may also help their clients create action plans and hold them accountable for their progress.

Another key difference is that advisors tend to have a more directive approach, while coaches are more non-directive. Advisors often offer specific recommendations and may even make decisions for their clients, while coaches help their clients find their own solutions and strategies.

Overall, both advisors and coaches can be valuable resources for people seeking guidance and support, but they have different areas of focus and approaches."

You’ll find that there is a continuum between the two. Most advisors also do some coaching. Some coaches also do some advising. I do both.

My advice for engineering leaders is to find someone who leans towards advising. The reason for that is that having deep experience in engineering leadership is something that is worth a lot. There is a wide body of knowledge and expertise that you will be able to tap into from advisors that understand that domain well.

But this does depend on the goals you have in mind. If you’re looking to be a more effective communicator, a coaching approach may be more appropriate. And if you’re wanting to have more perspective as a leader, a coach may be more appropriate.

Both are reasonable options. I would look at this as a spectrum that you can use to evaluate people you’re considering.

Any questions?

Let me know if you’re looking for advisory help. I both do this myself, and also maintain an online community of people doing similar work.

Also, if this article didn’t address something you think it should, I’d love to hear your feedback!

Image by Simon from Pixabay

Steel Threads are a powerful but obscure software design approach. Learning about Steel Threads will make you a better engineer. You can use them to avoid common problems like integration pain. And you can use them to cut through the complexity of system design.

So obscure it was deleted on Wikipedia in 2013

How unknown are Steel Threads? The concept was deleted from Wikipedia in 2013 because “the idea is not notable within Software Engineering, and hasn’t received significant coverage from notable sources.” Let’s add to the coverage, and also talk through why it is such a useful approach.

What are Steel Threads?

A steel thread is a very thin slice of functionality that threads through a software system. They are called a “thread” because they weave through the various parts of the software system and implement an important use case. They are called “steel” because the thread becomes a solid foundation for later improvements.

With a Steel Thread approach, you build the thinnest possible version that crosses the boundaries of the system and covers an important use case.

Example of conventional, problematic approach

Let’s say you’re building a new service to replace a part of your monolithic codebase. The most common way to do this would be

1. Look at the old code, and figure out the needs of the new system.
2. Design and build out the APIs that provide the capabilities you need.
3. Go into the old code, and update references to use the new APIs. Do it behind a feature flag.
4. Cut over using the feature flag.
5. Fix any issues that come up until it’s working, turning off the feature flag if necessary to go back to the old code path.
6. When it’s stable, remove the old code paths.

Sounds reasonable, right? Well, this is the most common way software engineers operate, but this approach has a lot of landmines.

What problems would I expect in this project?

1. It may be appealing to build the new service in a way disconnected from the old system. After all, the design might feel more pure. But you’re also introducing significantly more structural change and you’re making these changes without any integration to the old system. This increases integration pain significantly. My expectation would be that all the estimates for the project are unrealistic. And I’d expect the project to be considered a failure after it is completed, even if the resulting service has a generally good design.
2. I would expect the switchover to the new system to be problematic. There will be a series of problems uncovered as you switch over, that will require switching back to the old code paths or working intensely to fix problems in the final stages of the project.

Both of these things are avoidable, by not having a huge cutover. Note that even cutting over one percent of traffic to the new service with a feature flag is a cutover approach. Why? You’re cutting over all that one percent of traffic to all the changes at the same time. I still would not expect it to go well. You are taking steps that are too large.

Example using a Steel Thread

Contrast that approach with the Steel Thread way of doing it.

1. Think about the new system you’re building. Come up with some narrow use cases that represent Steel Threads of the system – they cover useful functionality into the system, but don’t handle all use cases, or are constrained in some ways.
2. Choose a starting use case that is as narrow as possible, that provides some value. For example, you might choose one API that you think would be part of the new service.
3. Build out the new API in a new service.
4. Make it work for just that narrow use case. For any other use case, use the old code path. Get it out to production, into full use. (Tip: you could even do both the new AND old code path, and compare!)
5. Then you gradually add the additional use cases, until you’ve moved all of the functionality you need to, to the new service. Each use case is in production.
6. Once you’re done, you rip out the old code and feature flags. This isn’t risky at all, since you’re already running on the new system.

Isn't this also the "strangler" pattern? Yes, but this can be used for new projects too. Read on for a greenfield example.

Steel threads avoid integration pain, and give you higher confidence

Integration pain is one of the bigger causes of last minute problems in projects. When you cut over to a new system, you always find problems you don’t expect. You should be suspicious of anything that involves a cut-over. Do things in small increments. Steel Threads integrate from the beginning, so you never have a lot of integration pain to wade through. Instead, you have small integration pain, all along the way.

Also, your service never needs to be tested before it goes live, because you’ve tested it incrementally, along the way. You know it can handle production loads. You’ve already added network latency, so you know the implications of that. All the surprises are moved forward, and handled incrementally, as just part of the way you gradually roll out the service.

The important thing is that you have a working, integrated system, and as you work on it, you keep it working. And you flesh it out over time.

Steel threads can help cut through complexity

When you’re designing a system, you have a LOT of complexity. Building a set of requirements for the new system can be a challenging endeavor.

When using a Steel Thread approach, you choose some of the core requirements and phrase them in a way that cuts through the layers of the system, and exercises your design. It provides a sort of skeletal structure for the whole system. The implementation of that Steel Thread then becomes the bones upon which further requirements can be built.

Thus, Steel Threads are a subset of the requirements of a system.

Steel threads can be used on greenfield work as well

Let’s say you’re implementing a clone of Slack. Your initial Steel Thread might be something like:

“Any unauthenticated person can post a message in a hardcoded #general room in a hardcoded account. Messages persist through page refreshes.”

Note how limited this initial Steel Thread is. It doesn’t handle authentication, users, or accounts. It does handle writing messages, and persisting them.

Your second Steel Thread can move the system towards being more useful. You could, for example, have a Steel Thread that allows the message poster to choose the name they post under.

This second Steel Thread hasn’t actually done much. You still don’t have authentication, accounts, or even a concept of a user. But you have made a chat room that works enough that you can start using it.

Also note that you haven't pulled the Steel Thread through every portion of the system. But you have stubbed out the concepts of users and accounts.

Steel Threads provide early feedback

Note that in this Slack clone example, you can get early feedback on the system you’re building, even though you haven’t built that much yet. This is another powerful reason for using Steel Threads.

After just those two Steel Threads, your team could start using the chat room full time. Think about how much your team will learn from using your system. It’s a working system.

Compare that to what you would have learned building out the User and Account systems, and hooking everything up, and finally building out a chat room.

Start with Steel Threads

Steel Threads are often a good place to start when designing your projects. They create a skeleton for the rest of the work to come. They nail down the core parts of the system so that there are natural places to flesh out.

I encourage you to try a Steel Threaded approach. I think you’ll find it can transform your projects. Let me know your experiences with it!

Steel Threads relation to other patterns

Steel threads are also referred to as Tracer Bullets, especially when doing it for new work. They have also been referred to as a Walking Skeleton approach. When used with migration projects, the pattern is referred to as a Strangler Fig Pattern.

Steel threads are closely related to vertical slices. I describe the concept in my post on Milestones.

Steel Threads are a software design technique that result in delivering your software in vertical slices. The term tends to be used to describe the initial vertical slices of a system. They’re closely related concepts, but not completely the same.

Thank you

Image by Steen Jepsen from Pixabay

Today, I'll share an interesting approach we used at New Relic to solve executive alignment on product direction, and cross-team execution.

What is a Product Council?

A Product Council is a way to create a leadership group that coordinates the most critical product work for the company. It provides a product strategy, and a way to prioritize and protect critical projects. It can help solve the challenges of cross-team projects. And it can also be a way to get executives aligned on product strategy.

The Product Council is led by the head of product. They also involve other key people in the council: perhaps the CEO, some Product leaders, and the head of engineering.

Key responsibilities for the Council are to write and communicate a product strategy, and to prioritize and communicate the key initiatives in the company.

That seems straightforward, so why is this useful?

How does a Product Council help with cross-team projects?

The year prior to us forming a Product Council at New Relic, I was responsible for the engineering effort behind a new product. It was an ambitious project, requiring work from many teams in the organization.

Everyone knew that the project was important, because we talked about its importance continually. I was able to secure the commitments of five or six teams to do the work we needed to deliver this new analytics product.

But as any experienced project manager will tell you, we were building on sand. There were other important projects too. And emergencies that came up. And one by one, the teams who have given me iron-clad assurances started to waver. New priorities came in, and piece by piece, the plan fell apart.

I came to the conclusion that a project of that size was simply infeasible given the way we were organized.

After suffering for a couple years of failed cross-team projects, we finally hired a consultant, Jim Shore, who helped us dig our way out of the situation. An expert in high performing agile teams at scale, Jim created a set of solutions to help address our cross-team project problem.

I’ll describe how it worked in a second, but what it did for cross-team projects was transformative. Simply put, it gave us the ability to execute on large projects and to have confidence they would move forward. It also gave us assurance that smaller critical projects could be guaranteed protection. It was a huge improvement.

How does a Product Council help get executives aligned on product strategy?

Another reason to consider using a Product Council is to align your leaders on a product strategy. Nadya Duke Boone, Chief Product Officer at Huntress, told me she is setting up a Product Council there: “We’re setting up a Product Council and have just eight teams [New Relic had fifty when we established the Product Council]. The first problem we want to solve with the Product Council is making sure the right people are involved in decisions about what teams are working on at the ‘big rock’ level. We have three executives who have unique insights that I want to make sure are heard.” Nadya also invited the functional heads / directors of engineering, product, and design to participate.

Executive misalignment is incredibly expensive. A Product Council can help.

How does a Product Council Work?

“The biggest benefits of a Product Council are the cross-functional membership of the council and that it creates a single official system for the approval and prioritization of work. Without a Product Council or General Manager-ish, the only place where there is cross-functional conversation about priorities is on the CEOs staff. That might work fine for a tiny company, but very soon the distance from an individual worker up to the nearest resolution point becomes vast. The Product Council addresses this explicitly by saying ‘everyone needed for a decision is here’” – Nic Benders

The Product Council was led by the head of product. They also chose the other members. At New Relic, it was composed initially of three VPs of Product, the head of engineering, and the chief architect. But I could easily imagine a CEO or cofounder participating.

The output of the Product Council was a written product strategy, and an ordered list of Product Priorities. At New Relic, the product strategy was revised at least annually, and the Product Priorities were updated about quarterly.

The Product Priorities was a merged list of these two types of projects:

1. Programs: Important initiatives that required coordination across a lot of teams -- programs rather than projects. For example, an RBAC program that needed almost every team to do work for it.
2. Critical projects: Projects that required protection from the rest of the organization, even if they didn’t require a lot of coordination. For example, the team working on an absolutely business critical feature, that didn’t need to participate in the RBAC program or anything else until it was done.

Participating in the Product Council is almost a full time role – it’s not just attending a meeting. Being on the Council was basically a full time role. Writing a product strategy, communicating that strategy, and synthesizing and coordinating the most important projects of a large organization is a lot of work. Be sure to set appropriate expectations!

The Product Council also served a governance function. We focused on delivering milestones instead of projects, and made our milestones small so they shouldn’t need to be interrupted. We further ruled that to interrupt a milestone in progress required the consent of the Product Council. Nadya Duke Boone: “The need to go to the Product Council to interrupt work in progress was an A+ feature of the Product Council – it really put meat into protecting a milestone and letting teams deliver before pivoting.”

Why a Product Council?

Companies with software teams tend to have centralized priorities. Or they have team-by-team priorities. Both can work fine, but will fail to scale as you get large enough. Eventually, you will need both. Why?

• Centralized prioritization eventually fails because the amount of things to prioritize increases with the number of teams. The central team has to maintain an understanding of an ever-increasing amount of things. And they lack the knowledge of the local teams, so will prioritize many things poorly.
• Decentralized prioritization will eventually fail because dependencies between teams can increase in complexity at an exponential rate. Without some form of coordination, all requests that cross teams have equal weight, and important projects will just be unable to move forward because there won’t be a shared understanding of what is important.

How the Product Council worked with local teams

A Product Council merges the advantages of centralized and decentralized prioritization. You get local teams using their local expertise to develop plans that make sense for their part of the organization. And you have a centralized coordinating function that provides overall business context and direction, and coordinates the success of the most critical projects.

The Product Council existed to ensure the big boulders could be protected. But the emphasis was on building strong local teams, with expertise in their area. We had product managers on each local team (and for teams that didn’t, we added a transitional role until we could hire a product manager for each team). These local teams thus also had their own roadmaps.

Product managers would take high-level direction from the Product Council’s product strategy. They’d develop their own local plans and have conversations with adjacent groups and the product council leadership to rationalize their plans. They could also pitch ideas to the Product Council and shape the overall direction of the company.

So the product strategy was influential because it was essentially a distillation of the market needs and product thinking of the leadership. This set the overall direction. And the local teams would craft their own strategies in response. But what was interesting was the interplay between the two.

The Product Priorities were also influential and helpful to local teams.

For product teams, they made our plans realistic. Prior to the Product Council, I could proceed with the illusion that my new product might possibly happen. After the Product Council existed, that illusion was dispelled, because everything had to be ranked. Product teams were forced to make realistic plans without excessive dependencies, unless they were near the top of the priority list. This is where the independent executor model comes from: make plans, but any dependency on another team needed to either be a Product Council priority, or it needed to be optional.

For platform teams, they found this prioritization useful as well. A platform team gets competing requests from all over the organization. If you are helping expand your API for several different teams, it’s hard to know which of them to prioritize. Prior to the Product Council, you’d guess or make your own assessment. Or you would be influenced by the most influential leader. But after the Product Council, you could reply to the #2 priority that the other team had to come first, because they were first on the list. This allowed them to give a clear “not able to do that for at least 3-6 months” answer, which allowed product teams to reconsider their plans with better information on what was actually feasible.

Product Council tips

The Product Council should be very restrained in what it prioritizes. In general, leaders will want to tackle everything. But doing this takes up all the spare capacity of engineering, and leaves the local teams with little autonomy or ability to use their local expertise in service of the company. Because the opportunity cost isn’t very visible to the Product Council leaders, they need to remind themselves of this all the time.

We found that there was a limited number of programs that could operate successfully at the same time. Programs tend to each spread their tendrils throughout the organization, because they are large and tend to involve a lot of work across teams. The top program always went well. As we got further down the list, our confidence it would succeed dropped. At some point there was a clear line – that represented our program capacity. We capped the number of program managers to that natural number, and tried not to exceed that number of programs we prioritized.

One danger I’ve seen of centralized prioritization is that the project that gets the “top priority” billing can use that designation to request unlimited resources from the rest of the product development group. It’s important to restrain things to the most minimal size possible, because you’re locking up the engineering organization. There should be some sort of scrutiny of the top priorities so they stay lean.

You may have a need to set some ground rules for reliability and security work. At New Relic, we had a rule that “do not repeat incidents” work trumped project priorities, even if deadlines would not be met. Just because something is the top project priority, doesn’t mean it’s more important than reliability or security matters.

Each program has a web of teams it hits within the organization. The top three priorities can often hit the same platform or infra team. We found that these hot spots were often fully dedicated to Product Council priorities. This meant they had no ability to do any other work. This can result in unanticipated tradeoffs. Be on the lookout for where the bottlenecks are, and always be fixing your bottlenecks. At the very least, talk with those teams about the impact it is having on them.

Also, you do need your leaders to play by the rules. Brent Miller pointed out: “One of the failure modes the Product Council had was there was an attitude from [some] leadership that they wanted all the things, so the reality on the ground was that there wasn’t one top priority, there would be five. For bottleneck teams that was just untenable; they would get in trouble for blocking somebody’s pet project no matter what they did.” This is a cultural challenge – you have to constantly reinforce good behavior and discourage people from pressuring teams for lower priority work.

Another important factor in a Product Council is that the conversations have to go both ways. Encourage people to critique the priorities or pitch other things to be the top priority. I saw some pretty dramatic improvements in the product strategy at New Relic when people successfully pitched new directions to the Product Council. That was one of its best features, but it was also one of the hardest to achieve:

“Making this work was really hard and it never worked particularly well from my view. Many people complained that the Product Council were the ‘secret masters’ who decided everything in secret. So even though the entire strategy would be made up of work proposed by various people in the org, we were always accused of making the whole thing up ourselves. Part of that is life, some is the communication strategy, but some is around a need for real transparency and openness in how you run it.” – Nic Benders.

Finally, over time a natural complement to the Product Council developed, which was an Execution Meeting for key projects. This was a large, tightly run meeting where we reviewed the key projects each week. We went over all the projects, talked through the red/yellow/green status for each, and called out risks and concerns. A typical meeting resulted in a couple of realizations of ways the projects were at odds with each other, or needed attention or assistance of various kinds.

When to use a Product Council

The main value of the Product Council is in creating a cross-functional, unified product direction. And the second value is in setting priorities that help resolve cross-team deadlock.

So I would consider using a Product Council when you have a group of executives or founders that are poorly aligned on the direction the product should go. Or when the company direction is poorly distributed throughout the organization. Another indication that it might be a good time to consider a Product Council is if you’re seeing cross-team project problems.

A Product Council can work in engineering organizations as small as eight teams, or as large as fifty. Nic Benders said this felt a bit like an upper bound of the size that the Product Council could be effective.

Scaling Product Councils

“Already with fifty-ish teams the Product Council felt overwhelming. We couldn't manage fifty workstreams and milestones per month. So some way to scale it, either in a fractal nature or by changing altitude, is needed. We ended up moving to a GM [General Manager] model as we outgrew the single Product Council” – Nic Benders.

For changing altitude, what Nic is referring to is the granularity of the work the Product Council looks at. The size of the organization influences the altitude of the work the Product Council will be capable of evaluating. Originally, the plan was for the Product Council to focus on milestones instead of projects. But this quickly became infeasible.

If your organization gets larger, you either need to subdivide into multiple Product Councils, reduce the amount of things the Product Council considers, or change the scope of what the Product Council considers. Or you might want to consider other approaches.

New Relic was about 50 software engineering teams when we added the Product Council. It would have been helpful with many fewer teams. But I wouldn’t probably do it before cross-team coordination was a large problem.

Spend more time implementing strategy, less on writing strategy

Alex Kroman was on the Product Council for many years. When I asked him his thoughts on it, he shared these thoughts:

The Product Council focused too much on writing strategy and not enough on implementing strategy.

It's relatively easy and risk free to write a good strategy doc. But it is difficult, and involves a lot of risk, to implement. Implementing strategy involves making difficult tradeoffs, and can be really unpopular and use a lot of political capital.

Decisions to support strategy almost always cross organizational lines or functional boundaries.

What I would do if I were doing it over again is identify a small number of cross-functional initiatives that will have the most impact on the business. Then communicate about them.

Then I would set up the Product Council to work with the leaders of those cross-functional initiatives, and help the organization make the right tradeoff decisions. These things won't happen on their own because they cross boundaries. So you need to align leaders across boundaries on these priority tradeoffs.

I would judge the effectiveness of the Product Council based on the number of hard tradeoff decisions the leaders make and implement themselves, in support of the highest priority initiatives.

Too often, I see organizations give "directly responsible individual" type accountability for projects. You tell this leader they have our permission to work across teams and do this work and do all these things. What I typically see is they don’t give that person that authority. What needs to happen is the leaders need to actually make hard decisions based on that person's recommendations.

Product council versus other coordination models

One alternative to a Product Council is a General manager approach (which is a type of Single Threaded Owner) or a Division structure. This subdivides the problem in smaller pieces, which can scale.

Jim Shore said: “Since the work at New Relic, I've been looking at ways to vertically scale (fewer bigger teams) rather than horizontally scale (lots of little teams). If I were to do it again today, I'd use FAST to create a handful of large teams, each responsible for a well-defined area, similar to how the General Manager model had people responsible for particular areas. FAST solves a lot of the problems I've seen with horizontal scaling: it's more amenable to specialties like UX, more flexible to changing business goals, and—particularly valuable—allows for incremental change rather than requiring a big-bang transformation.”

You can also opt to make teams so independent that what teams offer is essentially self-service.

You could attempt to resolve this through some sort of currency, real or virtual.

You could also do this work directly through the Product Management hierarchy.

Coordination models

The Product Council is one of many coordination models. Coordination models give you a menu of choices to choose from when solving your leadership coordination issues.

Feedback

See anything I missed? Disagree with this? Please let me know your thoughts!

Thank you

Jim Shore introduced me (and all of New Relic) to the Product Council. It was part of a larger set of changes he introduced – the most successful organizational transformation I’ve seen. Jim reviewed this post and shared his thoughts on FAST as an alternative approach.

Alex Kroman left voice memos about the Product Council and product strategy that were far more coherent than I would have been able to make while driving. I edited them into the section above on implementing the product strategy of the Product Council.

Nic Benders was one of the key executives behind the agile transformation at New Relic. He worked with Jim Shore and myself to implement these changes and was the executive responsible for the work. He was on the Product Council for many years, so he brings a unique perspective to bear. Nic reviewed a draft of this article and I included a quote of his on the cross-functional and prioritization value the Product Council provides. He emphasized Brent’s point about the time commitment of being on the Product Council. Nic also contributed a quote about the power dynamics and perception issues of the Product Council, and the need for transparency.

Nadya Duke Boone shared her experience planning a Product Council at another company. She also pointed out some things that I had forgotten about, such as the Product Council being a point of escalation for interrupting in-flight milestones. Nadya pointed out the importance of “do not repeat incidents” and security priorities, and how having those ground rules in place help. She also reminded me of the project execution meetings, which emerged after the Product Council as a way to track execution on key projects.

Brent Miller pointed out some of the challenges local teams would have from leaders who didn’t necessarily pay attention to the Product Council priorities. He also mentioned that the time commitment for participation in the Product Council was something the leaders tended to underestimate. He also reminded me of some of the challenges around the size of work to focus on.

Thank you to Bessi on Pixabay for the cover image.

This recounts the organizational history of Mini-Ms. Histories like this are important for a couple of reasons:

• They can provide valuable context.
• You can use that context to give you a better ability to adapt practices to your local situation.
• And you can learn from the history about the environment from which the practice emerged, helping you to create a similarly fertile place for other practices to grow.

This post is last in a series:

1. We first looked at how Mini-Ms work and what they are.
2. Then, we covered how to implement Mini-Ms.
3. We next discussed variations you may want to consider.
4. And finally we cover the history of Mini-Ms.

Before we had enough managers to have Mini-Ms, we had one management group for engineering. We called this the M-team meeting (and the name Mini-M came about when we split it up).

In the early days, this meeting combined operational matters and topics designed to make us better managers. Our VP of Engineering would present on topics about how to be an effective manager. For example, how to run effective meetings, or on project management. But a lot of our topics were conversational in nature: like a discussion group on management.

The management meeting format varied depending on the problems we were solving. For example, during review periods, we would read each other’s reviews and critique them.

I participated in these early review review sessions, and they taught me almost everything I know about writing reviews. “Overall, the performance reviews at New Relic were better than most places I've worked before or since, and I attribute much of it to this practice… There were multiple sets of eyes on reviews which helped identify bias and also ensured nobody 'phoned it in'.” – Jason Poole, former Mini-M Organizer.

In the early days at New Relic, we had weekly status updates that were due every Friday. Managers would gather and write them together. We called these meetings “Status Jams”. This was importantly different than the official M-team meetings, because Bjorn Freeman-Benson, our VP of Engineering, wasn't there. It is very hard to create peer connections if a hierarchical leader is present, so those Friday meetings gave us a time to check-in with each other, decompress, and occasionally kvetch about things we didn’t agree with or understand.

Nic Benders was in charge of a team that was developing a new product. At one point, Nic left that team to go create Site Engineering, and another manager, Etan, was selected for that team. Nic and Etan started having a weekly 1-1, where they talked about the team, management, and other topics.

The seed for Mini-Ms happened when Jon, another manager, joined the management team to build yet another product. Bjorn said, “Hey, Nic, can you add Jon to your 1-1 with Etan and help him get up to speed?” This group became the first Mini-M.

This went so well that Bjorn decided to make it into something that should be scaled out to everyone. He designed it as an organizational practice. He assigned managers to Mini-Ms and insisted everyone attend one. And he tweaked the practice over time.

Bjorn described the thinking behind it this way: “The purpose of the mini-Ms (originally) was to help the managers improve their craft just as pair programming or mob programming or code reviews are used in the development team to improve that craft. It had additional benefits such as creating a more consistent management practice across the whole org and being emotional support for managers (because it is a tough job), but the original goal was about people stronger in one skill helping people weaker in that skill get better. And because nobody was better in all the skills, having a group help each other improved everyone more rapidly than 1-1 coaching, etc.”

This part was related by Nic Benders, then reworked by me to make it more intelligible to people who weren’t there. If you have more to contribute to the history, let me know!

Thank yous

Not a lot has been written about Mini-Ms, but there is one post currently on the New Relic blog, and another that has mysteriously vanished.

Bjorn Freeman-Benson was the founder of the Mini-M practice. He shared a lot of his thinking about the principles behind why Mini-Ms were successful and what they were aiming for. He advised me to break up this post into sections and make it easier to get to the implementation section. And shared overall feedback.

Nic Benders ran the first Mini-M, and established and evolved the practice. He also ran one of the more influential Mini-Ms for years. He reviewed this post, offered feedback, and contributed to the history section and described some of the design goals for Mini-Ms.

Darin Swanson authored some content that were inspirations for large parts of this article. He and I have worked together on helping other companies implement Mini-Ms, so contact me if you’re interested in help. He also provided feedback and suggestions on drafts of this article. He encouraged me to explain the first team concept more fully, and to describe why pre-product market fit companies may not be a good match. And he suggested I split the content into separate articles.

Elaine May provided feedback, some of which was so good I just ended up quoting her. She was gracious enough to offer some talk to talk about her experiences setting up or participating in similar programs at other companies. And she talked about her experiences with me in New Relic’s Mini-Ms. Elaine introduced me to the Chatham House Rules, which I incorporated into the post.

Merlyn Albery-Speyer helped me improve the section on “when to use Mini-Ms” by pointing out some preconditions for success. He pointed out that the structure became less important after the Mini-M is established. Merlyn pointed out that we tried to keep people from being in the same Mini-M as their managers, or other people reporting to the same manager. He also shared the theory about VPs not being willing to be vulnerable as a possible explanation for why the Mini-Ms never took off in manager of manager groups to the same extent they did for frontline managers.

Molly Graham contributed the section on “Manager coaching circles” in the variations post. She has an insightful newsletter.

Jason Poole shared his experience as an Organizer of Mini-Ms. He pointed out a now mostly disappeared second post on Mini-Ms. He also suggested ways Facilitators could counter unproductive ranting, and pointed out how effective the Mini-Ms were for improving our performance reviews.

Marty Matheny shared feedback based on his experience as an Organizer. He helped improve the advice for Organizers. And he pointed out that engineering adjacent departments participated.

Chris Hansen pointed out that confidentiality was an issue with whiteboards. He noticed an error that would have been embarrassing. He pointed out the value of M-teams in distributed organizations to counter isolation among managers. He also added some advice for participants. Chris also helped with a point about the value of the first few meetings.

Teresa Martyny reviewed a draft of this post and pointed out some redundant assertions I was making. And she recommended I break this into multiple posts or edit for brevity.

Natasha Litt was another early Mini-M leader, and she reviewed a draft of the post and contributed to it.

Marco Rogers influenced some of my thinking on building communities in a recent conversation, so some of his ideas were reflected in that section.

Image by Tide He from Pixabay

Low trust High trust

Focus on accountability Focus on learning

Doesn’t require alignment or context Requires alignment and lots of context

Command and control Autonomy and alignment

Scary to reveal problems, risk. Eager to explore problems, risks

Leaders get poor information. Leaders get good information.

Focus on commitments. Changes require lots of scrutiny and explanation. Focus on estimates, update as you get new information. Changes are inexpensive.

Fatalism Optimism

Low retention High retention

Individual performance Team performance

Leaders focus on activities Leaders focus on outcomes

Functions blame each other for problems Functions (product, eng, design) work together, share problems

People that raise problems are criticized People that raise problems are praised

People with a high tolerance for conflict speak up. People who can contribute speak up

Thank yous

Image by Lisa Caroselli from Pixabay

I'm often asked about my advice for new directors. Here I lay out the ten things I wish I had known when I started as a manager of managers.

Being a director is a very different job

First of all, the move to being a director was a bigger change for me than I expected. I thought managing managers would be similar to managing engineers. That was naive.

The shift wasn’t quite as big as moving into management. But it was close.

Being a half director isn’t the preparation you may think it is

As a senior manager, I started to transition into a manager of managers role. But I did so in a hybrid way: managing a team directly, and managing another manager at the same time.

I thought this meant that I understood the job of managing another manager. I did learn a lot, but it wasn’t the preparation I thought it would be.

Why? The skills to manage that manager were new, so I did the job in a limited way. Only when my role switched to fully managing other managers did I have enough ability to focus on the job. It was then that I realized how little I actually knew about that work, and started to grow at a faster pace.

You’ll do most of your work through others

One of the larger shifts in the nature of your work is that Directors do most of their work through other people. You may think you’re doing work through other people when you’re managing engineers as a front-line manager. But Directors have to operate at a much higher level of indirection.

A consequence of this is that your weekly meeting with your managers, and your 1-1s with them, become your most important meetings of your week. Consider lengthening your 1-1s with them to an hour, and make them weekly. And spend serious time planning your management meetings and 1-1s. They are where you should be doing most of your work.

Operating with so much indirection can be an adjustment. Most successful managers have probably been successful largely through their direct efforts. They’ve managed projects well, or hired well, or made improvements on their team.

When you’re a Director, you’re usually working through someone else who is doing the work. Suddenly your success is based on how well your managers are running their projects. Your success is based on how well they hire. Your success is based on the improvements they make on their team.

This leads to a few common pitfalls. One is the overinvolved Director, who doesn’t make space for their managers to do the work themselves. One sign this is happening is if you’re in all the same meetings as your manager.

The second is the underinvolved Director, who views their role as hiring the right people and supporting them. This is similar to the shit shield school of management.

Instead, I encourage you to think about your level of involvement as something you flex depending on the circumstances. Your goal is to be less involved, but it should depend on the level of expertise of the manager in this particular area, and the complexity and challenge of the situation they’re facing.

When a situation is challenging for a manager, you might be more proscriptive, giving them a pattern to follow. You might review their plans and offer more feedback. You should interact more frequently, and talk through the actions they plan to take.

A lot of your job is training managers

This flexing of your level of attention is an important part of your role as Director. And it leads to the next thing to be aware of as a Director: your role is to train your management team.

Ideally, any of your managers should be able to step into your shoes, and do any part of your role. And even if they all can’t do so, your job is to prepare at least one of them to do so.

To understand this topic, I first recommend your read my post on Completed Staff Work. Pay special attention to the end, where I review Marquet’s Ladder of Leadership.

The way I like to look at my management team is that they all have varying levels of skills in different areas. Sometimes their skills will exceed my own in certain areas! But my role is to help them develop their skills as rapidly as possible.

Part of this is that the more skilled they are, the more autonomously they can operate. This is at the heart of a scalable organization. If all your managers rely on you for everything, you have an ineffective organization.

So your job is to create an increasingly autonomous and skilled organization. One that is able to produce good results independently. To do this, you need to be expanding their skill set, and creating the right environment for them to thrive. Coach and develop them to build their skills.

Biggest skill to learn: sensing your organization

The biggest surprise for me when I moved to a pure manager of managers role was how little I knew what was going on. It was like someone had turned off all the lights. I couldn’t see anything that was happening any more.

You may find the information vacuum unsettling. You’re simultaneously put in a position where your job is to make things better, but you have much worse information about where the problems are.

I see some Directors become destructive to their organizations at this point. They rely on their gut and pride themselves on making decisions without full information. This can work sometimes, but it can also result in problems. It’s like a doctor that doesn’t diagnose the disease, and instead starts filling you with random drugs, and starts surgery in random parts of your body. Correctly diagnosing and understanding the cause of things is essential.

You need to build a way to understand what is happening in your organization. You need to set up observability of your organization, so you’ll know if things are going off the rails. Pay a lot of attention to this. It gives you the ability to support your managers, and it provides opportunities for intervention.

A few things you might try:

• Look for meetings that give you signals that things are going well, or not going well. I found demos to be a particularly rich source of information, for example.
• Do skip level 1-1s, to get a random sense of how things are going, and to establish connections throughout your organization.
• Collect metrics from your managers, so you can have conversations about trends or things that seem to be going off track.
• Look at the information tools can give you. Stats on reliability, how often people are paged, product usage metrics, and analytics tools can help fill in your picture of how things are going.

One trap to be wary of is that your need for information may entice you to ask your managers for information. You’ll probably need information frequently enough that you can be a source of annoyance to them. Consider adding some structure around your information needs. Think about what you really need to know each week, and ask your managers to push it to you, instead of pulling it from them all the time.

You’ll need a new perspective

One question to ask yourself is where you can be helpful to the organization. What are things you can do that nobody else can do?

One thing you do more as a Director is to plan further into the future. You should have a higher-level perspective on how your organization’s work fits into the broader offering of the company. This perspective is something you can use to shape the direction your organization heads, and is something your managers cannot usually do.

The skill to learn here is outlined in my post: Leaders make their own problems. I recommend looking at that post carefully.

You can also look at this post on upstream thinking, as I think it outlines some of the mindset required as a Director.

You should focus on systems

A weird thing about being a Director is that you’re operating more at a meta level. What I mean by that is that instead of directly tinkering with a team, you’re working with a system of teams. Your focus should be shifting to be more about the patterns of things, than the details themselves.

This may come naturally to the rare people who tend to be systems thinkers. For everyone else, this is a skill to build. My suggestion is to always be operating at two levels: solving both the immediate problem, and looking at the level of abstraction above that.

For example, if there is a project that is going off the rails, you should be thinking both about how to help with that. But you should also think about what your playbook is for off-the-rails projects. Or how to notice these projects earlier. Or how to systematically reduce the prevalence of this type of project.

You also should think about ways you can influence the whole system. Your toolkit is different, because you’re operating at a systems level – at an organizational level.

A few suggestions:

• You are in a unique position to offer clarity. You can simplify things for people. You can allow them to focus on less things. This is almost always a helpful thing for you to be doing, so pay attention to how you can both simplify things in your own mind, but also how to communicate them.

• You’re also in a unique position to offer context. You will have a lot more context than you used to, because you’ll be interacting with higher levels of leadership than you did in the past. Think of that context sharing as a service you offer your organization.

• Constraints are a tool that you may wield more as a Director. For example, you can make simple rules for teams that help nudge them in the right direction. An example? Teams can only have a project or two they work on at a time. Don’t use over-use constraints as a tool, but sometimes it can be helpful to ask yourself: if I could only do one thing right now, what would have the biggest impact?

You should also familiarize yourself with the levers of coordination models. These are patterns in the way that humans work together in groups to be effective. You’ll need to learn how to organize groups of people, do reorganizations, and so on.

Some of your instincts may be untrustworthy. For example, when deciding whether to organize a team based on skillset or around a product area, you may have suspicions of the best way to do it. But most likely, you have no idea the kinds of tradeoffs you’re dealing with. Which leads to my next point.

Get support and mentorship

Many managers gradually reduce the amount of support and mentorship they receive as they go up the hierarchy. I think they do this because they’re expected to be experts.

This is foolish. As you change roles, and move through different parts of the organization, your skills will need to grow. So seek out people who can mentor you. Seek out peers that can give you feedback. Start your own Mini-M support group for Directors. Contact me or another experienced leader to advise or coach you. Read books and subscribe to newsletters that stretch your thinking (such as my newsletter for engineering leaders)

Beware the distortions of power

Another thing you need to be aware of is that the higher you go in an organization, the more there is an invisible distortion field around you. It affects how people interact with you, and the information you see from your environment.

This can have a harmful impact on your ability to understand the true situation in your organization. It requires a specific set of skills to counter. Without doing so, you’ll be operating in lala land, unaware of the problems you’re creating.

I wrote a whole post on this topic: Everyone Lies to Leaders. As a Director, you’re going to start getting the first tastes of this, so be on the lookout and start building your habits early.

You’re judged by the difference you make on your organization.

To close, I’d like to leave you with my mental model for how you can assess yourself as a Director.

You are judged by the output of your organization. And specifically, you’re judged by the difference you make on the organization. What do you make better? How do you improve things? What is the diff you apply to that organization?

Podcast on these topics

Decoding Leadership is my podcast on leadership. I continue this topic in my episode on removing yourself from being a bottleneck.

Katie Wilde and I also talked about this topic at length in the episode below. We had very similar challenges when we were promoted into director roles, and developed similar strategies to cope:

Thank yous

The mental model for how to assess yourself as a Director comes pretty directly from the book High Output Management. It is one of the best books on engineering management.

Image by stokpic from Pixabay

This contains my experience choosing tooling for an engineering handbook. It also has some points that might be relevant if you're evaluating a wiki.

The ideal tool would have these qualities:

A short edit loop. A challenge with maintaining process documentation is that the cost of making changes is often too high. The more expensive it is to suggest changes, the less participation you’ll have.

This is something I think most leaders underestimate the importance of. I suggest you look at two scenarios when evaluating tooling:

1. How much effort does it take to correct a typo?
2. How much effort does it take to add a new page?

You know this is an issue when people will propose a workflow like this:

1. We’ll use Google Docs to collaborate on things before they’re official.
2. Then we’ll copy them into Confluence when we’re done.

This is acknowledging that Confluence is terrible at the job of collaboration. So why would you choose to store your content there long-term?

Github Google Docs Confluence Slite/Notion-like wikis

Correct a typo Days Seconds Tens of seconds Seconds

Add a page Days Tens of seconds Tens of seconds Tens of seconds

Based on this criteria alone, I tend to order these solutions like this:

Modern wikis (Slite/Notion) or Google Docs > Confluence > Github

Easy for many types of people to contribute. If you’re an engineering leader, you may care mostly about engineering. So you may think you should focus on engineering team needs for the handbook. But you’ll soon find that Product and Design and Support will all want to be a part of your process docs. It’s useful to be able to work on a shared process with Support.

Some tooling is harder for other specialties to use. Github is primarily an engineering tool, for example, and is challenging for many types of people to engage with.

Based on this criteria, I tend to order these solutions like this:

Google Docs > Modern wikis (Slite/Notion) > Confluence > Github

Have flexible permissions and an approval process. Ideally, you’d have some sort of an approval process, and you can have a discussion about a set of changes.

For larger changes, Github is pretty ideal. You can write up a set of changes and have an approval process that the change goes through. But the approval process for Github takes so long that I find it disincentivizes minor edits, and your process documents will degrade over time. In fact, I find it so bad that I’d rather have no approval process and just open up process docs to allow anyone to edit everything than to have a rigorous approval process. But your organization may have different preferences.

Google Docs is pretty hard to beat, from both a permissions perspective, and the ability to suggest changes. You can set your permissions based on the context, which is a major advantage. Sensitive content can be set up so anyone can suggest changes. But less sensitive content can be world editable. And the “suggest changes” functionality is truly useful. However, Google Docs does have some awkwardness around inheriting permissions from parent folders.

Based on this criteria, I tend to order these solutions like this:

Google Docs > Github > Confluence > Modern wikis (Slite/Notion)

Easy to organize your content. Finally, you want to be able to navigate your content and organize it.

Github Google Docs Confluence Slite/Notion-like wikis

Ability to organize content Slow, requires technical skills. Slow. Have to manually insert links for everything. Fairly easy Fairly easy

For ease of content organization:

Confluence and modern wikis (Slite/Notion) > Google Docs > Github

Ease of marketing. Finally, you need to get an organization of people to use these process docs. Some tools are easier to market than others.

Github Google Docs Confluence Slite/Notion-like wikis

Marketability Engineers will love to use it. Signals you care about their participation. Others will feel excluded. Everyone will complain that it seems wrong to use Google Docs for a handbook. Everyone will think it’s a natural place to put things. But won’t use it without prodding. If you’re already using the tools, people will think it’s natural.

Based on this criteria, I tend to order these solutions like this:

Github or Slite/Notion-like wikis > Confluence > Google Docs

Conclusion

I recommend using a modern wiki for process docs (Slite or Notion are the ones I would start with). I would also consider using Google Docs.

(Note I get a referral fee if you click the link above for Slite, but recommended it long before I included that link and will remove the link if I find another wiki I prefer).

If I wanted to kickstart the process with engineering, I would consider using Github for a while. You could then migrate to another tool when other functions like product and design and support want to participate.

Image by Pexels from Pixabay

Process: a series of actions or events performed to make something or achieve a particular result – Cambridge English Dictionary

Today, we delve into the great debate about Process. Is Process the solution to the chaos in organizations? A way of applying good discipline and operations to make stuff happen? Or is Process itself the problem, adding overhead and getting in the way of people doing their best work?

I’ll argue there is a way to manage Process that preserves its benefits, while eliminating most of its downsides. It’s probably something you’ve never seen before. And I’ll describe how to roll it out, what the rules are for operating in in this way, and some of the surprising benefits.

Process is how we work together

The first thing to realize about Process is that it exists already, whether you acknowledge it or not. Process is the way we work together.

Process can be explicit and written down. Or it can be implicit, something people understand just because they’re used to working in a particular way as a team.

Process can be something a group of people are aligned on. They all agree how things work. Or it can be something that people don’t really agree about. They may have different ideas of how things get done.

Because Process is a part of humans working in groups, it’s nonsensical to be opposed to Process. That doesn’t stop people from trying! But the real question is how you shape and define the way we work together.

Process can be problematic

Most of the opposition to Process comes from people’s bad experiences with it. This is natural. Process is a bit like code – without attention, Process can degrade and cause problems.

The challenge with Process is that it has a life of its own.

For example, let’s say a software engineering team is only focusing on new work, and not doing any work on bugs that are coming in from the support team.

Ignoring that you should probably look deeper into why this is happening, and let’s pretend we’re just going to make a process response to this problem. You could:

• Have the engineering manager and product manager decide to schedule some bugs every sprint.
• Decide that the team will have the on-call person do bugs for the week they’re on call.
• Add an SLA around bugs.

Whatever you decide, you’re seeing a problem, and you’re changing the team’s behavior to try and address that problem in the future. This is Process work.

So let’s say you decide that the engineering manager and product manager will meet with their support liaison once a week and prioritize the most important bugs each week. You set up the meeting, and it starts happening every week.

The problem is that two years later, the problem might evolve. And the people who created that Process may not even be there. So the people involved in that meeting two years later may not understand the problems the meeting was intended to solve.

This uncertainty about the origins of Process can make people reluctant to change things. This is particularly true if the Process is important or around something that seems dangerous. So Process can take on a life of its own, where it operates as a sort of zombie. And people follow it because they have to, even if it is bad or harmful.

Again, this is like legacy code. It works, but anytime you touch it, there is a danger that you’re going to cause other problems. So mostly people try to avoid touching it.

What if Process were fluid and dynamic?

What would ideal Process look like?

• It would feel fluid and dynamic. Easy to adapt to the needs of the organization.
• It would embed good context inside of it. This would make changes easier to make.
• It would be clear, concise, and up to date.
• It would be the source of truth, so you could rely on it.

Treat your Process like code

In short, good Process is a bit like good code:

• It can be updated easily.
• You have version control. You have good historical information to see how it has changed over time.
• The cost of change is minimized.
• You have good context that allows you to see why the process exists and what it’s trying to do.
• The process is what describes how things work.

Solution: open source process

So how do you get flexible Process, that can adjust to the changing needs of your organization?

The best solution I’ve found is to open source your process. Write down your process, and allow anyone to suggest changes. Tell everyone they can change the way the company runs.

While this can be a lot of work, it can also be immensely clarifying. And it can feel empowering to be able to say: the way we work together is written down. And anyone can make changes to it.

Advantages of a written culture

Writing down your process is a core practice for an organization with a written culture. And there are a number of advantages to writing things down:

• Writing is clearer thinking. It requires precision. It provides clarity.
• Writing is asynchronous and always available. Writing is more scalable, because it doesn’t require meeting to share information.
• Writing is thus better for distributed teams.

The primary disadvantage is that documentation requires effort to maintain.

How to open source your process: an engineering handbook

So how does one go about building a written culture, and create a flexible, maintainable process for an organization?

You create an engineering handbook. It is a repository of how things work in your organization. It represents your process documentation.

I’ve done this numerous times. Here’s the general steps I recommend.

1. Choose a format for your engineering handbook

The first challenge you’ll have is to figure out where to keep your process documentation. I’ve never found anything that works perfectly. I have a separate post which shares my notes on which tools to use for employee handbooks. The important step is to choose something. You can change your mind later.

2. Bootstrap the content

The next thing to do is to start filling in content. Anytime you encounter a process, write it up, even briefly. Describe how things currently work.

This works especially well as you join an organization. You can use these writeups to clarify how things work. You can even show your writeups to people and ask them if it is correct.

I might do this for a couple of months before proceeding to roll it out further. As I make organizational changes, I will document them, and have a before and after section. After I make the change, I’ll put the before section into a history part of the doc, so people can see how our process is evolving over time.

Often you’ll find that there are previous versions of process docs that are sitting around somewhere. They might be incomplete, or unmaintained. But they can shortcut a lot of this process. Organize them and start maintaining them.

Having enough content to interact with is important. It shows that there is momentum behind the practice, and that it will be taken seriously. This allows people to invest their own time in it.

3. Choose a maintainer: it’s probably you

You will need someone who makes sure these process docs are taken seriously. That they’re updated and organized. You can enlist help, but ultimately it needs someone to be invested in them. This might not be something you can easily delegate.

Ideally, it’s something that all management takes seriously, and you can model it but have a larger group of people helping out.

4. Communicate how it all works, and roll out the changes

The next step is to roll it out. If you’ve been bootstrapping the content, it may not be a surprise to the whole organization that you’re rolling it out. Some people might have already seen it before.

What I’ve typically done is to talk about the reasons why we’re emphasizing a written culture, and then share the rules we’re following to ensure we keep our process docs up to date (more on that in a second). And then I try to share how anyone can improve the way the organization works, and describe how that works. This can be an exciting, empowering thing for the team.

Rules for process

So what are the rules for process documentation? Here are the things I like to share.

1. Reply with documentation

The first thing I like to share is the concept of replying with documentation. I’ll usually share it with the whole engineering team, in an All Engineering meeting, or in a newsletter to the organization. And I’ll try to model it myself. What is “replying with documentation”?

When someone asks you a question, ideally you want to reply with a URL that answers that question. You do this by going to the wiki, making sure the answer isn’t there, and adding it if it isn’t. Then you send them the URL.

This does a couple of things:

1. It ensures that the question can be answered in the future without you having to reply to it.
2. It trains the person asking where they can find answers to future questions.

Replying with documentation is putting a caching layer in front of all the questions people might ask you. It makes your expertise more widely available.

For the organization, the impact of Replying with Documentation is that you are future proofing your answers. Instead of answering the question a million times, you’re ensuring it only needs to be answered once. You’re systematically solving problems rather than just doing a one-off.

2. It’s not official until it’s in the handbook

The next rule is something I repeat over and over to the management team. It’s not official until you put it in the handbook. Whenever a manager is making a change, or rolling out a plan, I say the plan needs to include a URL.

• Is the team moving to two week sprints? That should be in the handbook.
• Is someone moving to a different team? There should be a list of people on that team, in the handbook.

This rule is important because otherwise, the handbook is not the source of truth. People can’t rely on it to be accurate. This degrades trust in the handbook, and makes it a less useful source of information. But adding this constraint makes what is in the handbook the truth. It’s a completely necessary rule if you care about keeping your process docs up to date.

3. Editing a process document isn’t sufficient to make a change

A third rule I share is that you can’t make people do something just by editing a document. That’s not how humans work. But you need this rule as otherwise, many people will attempt it.

You will need a page that describes how changes work. I usually call it something like “How to change the way we work”. It describes:

• Who the maintainer of the handbook is.
• That anyone can make changes.
• That you can’t dictate how things work just by editing things in the wiki. You have to communicate and get buy-in from the appropriate people. But that improvements are encouraged, and process should be fluid.
• That the hierarchy of power is still in existence. Anyone can make improvements, but ultimately the VP of Engineering is delegating these decisions to the whole organization.

You will also need to describe the rules for approval and change. I’ve done things through a full approval process, and I’ve done things in a wiki, where anyone can make changes. When you do things in the wiki, it can be good to monitor the changes periodically, to make sure inappropriate changes aren’t introduced. I also would generate release notes periodically, so people knew what had changed in the way we operated over time.

4. Process should include context and history

You’ll probably want some sort of a template for your process pages, which explicitly includes a section for context and history. This can feel like extra overhead. But it’s helpful. You can explicitly call out the situation that warranted the process you’re implementing. And you can sometimes even mention when it won’t be useful any more, or when you might want to rethink this way of doing things.

Adding history can help people understand the changes that have been made over time. And the context can help people understand how to shape future changes.

How does AI affect handbooks?

With the rapid rise of AI from 2025 on, your handbook's format may want to be different than it was previously. One thing to consider is that you should think about LLMs as one of the most important output formats for your handbook. You probably want people to be able to ask about something, and have the LLM reply.

Will Larsen wrote a piece about how he improved on existing documentation in Notion that's worth reading as you think about the implications, and also about the topic of cleaning up existing messes as you move towards better documentation.

Take it to the next level: make it public

If you’ve rolled out your Engineering Handbook, and it’s been successful, you might consider taking it to the next level: and opening it up to the whole world to see.

This is a radical step, and probably not a fit at a lot of companies. And it may be difficult to get people on board with the idea. But it does have some advantages:

• You’re marketing your organization to the rest of the world. It can attract candidates, who see how well your organization is run, and the principles by which you’re organized.
• Having a public audience will automatically result in a higher quality of process. Wouldn’t you do a better writeup of your team’s process if the whole world could see it?
• Having your process in the open will also help others learn from you.

The company most notable for doing this is Gitlab. The Gitlab handbook is the operating manual for the company, and it’s quite well done.

Thank you

Bjorn Freeman-Benson came up with the idea of and practice of putting the New Relic process in Github.

Rebecca Campbell was my partner in crime for a few years with the New Relic process docs. I learned lots from our time maintaining the process docs there.

Alex Kroman asked me to write the New Relic engineering handbook, which was essentially our book on how we did product development. It included our practices, roles, and standards.

Michael Davis introduced me to the idea of Replying with Documentation.

Image by Pexels from Pixabay

Today we cover a variety of ways that managers uplevel each other by meeting in groups. Some of these variations are variations in the Mini-M format, but we also include some practices that are completely different.

This post is third of a series:

1. We first described what Mini-M manager support groups are and why they’re so effective.
2. Then, we learned how to implement Mini-M management support groups.
3. Now, we discuss variations of Mini-Ms and other ways to encourage manager growth.
4. Next, we'll cover the history of Mini-Ms, including where the name came from.

There is no right way to run Mini-Ms, as long as it is working for participants. So let’s dive into the variations!

Rotating members in a Mini-M

One topic we discussed periodically was whether it was best for groups to be long-lived, or for there to be a lot of variation in members. We settled into a combination of the two. Groups were long-lived, but we would rotate a couple of members from the group to another group every six months. This led to more cross-organization relationships, but allowed the groups to feel relatively stable and preserve their identity.

Mini-Ms can be like a conspiracy

Another perennial topic was whether the Mini-Ms should be a source of work and a force for change. Or whether the groups should mostly be focused on supporting each other.

At New Relic, we actively encouraged the idea that managers were responsible for improving the organization around them. So it seemed natural that Mini-Ms would be a place where change would originate. And the original Mini-M was a sort of cabal, or cell, within the larger organization:

“I deliberately modeled the group as a conspiracy, where anything could be discussed privately, then I would take the meaningful results (without names) up to my org peers and boss. Every system has official and unofficial ways of getting things done. One of my goals with the Mini-M was to blur the lines and officially build that structure for doing things the unofficial way.” – Nic Benders, one of the cofounders of Mini-Ms.

It was fun to plot improvements with co-conspirators, and make them happen. You can encourage this, or let it develop naturally.

Mini-M demo management work

If you have Mini-Ms doing work, you can also try this variation.

Management is often seen as this mysterious group doing work behind closed doors. We use demos in Engineering organizations to show the work we have completed. Mini-Ms can demo as well: presenting iterations on process or a different bit of insight on performance reviews. A secondary benefit is these demos can pique the interest of those who are management curious by showing the tangible work that managers do.

Mini-Ms with managers from adjacent departments

When we started Mini-Ms, we included managers from just Engineering. But we tried to make the participation seem like you were meeting with people from across the organization.

“As the program grew in popularity, we started to have folks join from outside Engineering (e.g. support, design, security), as they'd heard the program was useful. This worked pretty well for "engineering adjacent" orgs” – Marty Matheny, former Organizer

Hearing the challenges of design managers helped me to have more empathy and understanding for their challenges. And that helped me work more effectively with design. The same thing happened with support managers and managers of product managers.

This probably works most effectively with roles just adjacent to engineering. If the work doesn’t overlap, it will probably be less effective.

Organizationally aligned Mini-Ms

In the original formulation of Mini-Ms, people from the same organization were not allowed to be in the same Mini-M. The intention was to improve connections across the organization. We even allowed people in from outside engineering, and encouraged cross-timezone groups. We also wanted to maximize psychological safety, and thought it would be easier if you weren’t working closely with the people you were talking about.

This had a lot of advantages, and I think a lot of the people that participated viewed this as a core benefit of Mini-Ms. So it might be heretical to say you could design Mini-Ms to focus on the opposite extreme: organizationally aligned Mini-Ms.

I haven’t seen this in action, but I can see a lot of benefits. You would combine managers from closely organizationally related groups until you had large enough Mini-M groups.

This would encourage stronger working relationships within the sets of people working most closely together. They would probably need to share some of the results with their management hierarchy, but they would need to meet without leadership involved.

Probably the best reason to choose this variation is if you’re unable to do an organization-wide change, and your area of influence is within a smaller part of the organization.

Manager coaching circles

This next way to level up managers comes from Molly Graham:

“Every start up I know struggles with "manager training" at some point, meaning they feel that the layer under their leadership is under-served and possibly under-powered. My main point to start-ups is that you can (and should) invest in things like manager training but one of the most powerful and lightweight things you can do from an early stage is create manager coaching circles that meet either monthly or bi-weekly. I called them "manager breakfasts" but Mini-M is better :) It is a scalable way to create a culture of managers helping and learning from each other. If you ensure that there is a range of experience within each group and do the work to ensure the groups have a light structure that is easy to maintain, it is very scalable and less expensive than training all managers every N months.”

Random 1-1s with other managers

Apparently at Slack there is a practice of having the Donut app schedule managers for random 1-1 time with other randomly paired managers.

The advantage of this is it is quite easy to set up. And you get to build some shallow relationships with a lot of the other managers. This type of acquaintance type relationship can be really valuable in a large organization.

The disadvantage is that you don’t build deep enough connections with these people to really improve your practice much.

Manager communities

Managers can build community outside of the company, in other forums.

Many managers participate in the Rands Leadership Slack. It’s probably the largest community for engineering managers in the world. It can be a good place to find certain types of support. However, it is also so large that it is basically a public group, so any topics that are sensitive are harder to bring up there.

Smaller, private groups can be an alternative. It is easy to spin up a free Discord or Slack server, and invite some people you know to participate.

A few suggestions for building communities of this type. First of all, the initial group will largely set the tone for people that come later. So try and optimize for some initial participants that will be active, but also show vulnerability and empathy with each other early on.

Be deliberate about adding new people. You essentially want the rate of growth to be slow enough that the people are indoctrinated by the current participants into what behavior is to be expected. Some groups are invite-only for that reason.

However, groups like this also can have a minimum viable threshold before they begin to take off. I was pretty aggressive about adding people to one of the communities I started until I began to see people participate.

You will need to model the behavior you desire of your participants. And you might conspire with a couple of other people to make sure there is enough content and dialogue that other people decide to invest in participating themselves.

A growth-focused management meeting

Bjorn Freeman-Benson founded Mini-Ms. Early on at New Relic, our management meetings were a blend of tactical topics and topics intended to grow our management team. He treated it like a professor would, in an upper-level seminar class, with discussion amongst participants.

Here is an example agenda:

• Review Nic’s new engineering hire first day/week/month checklist.
• Big release positivity: what is working to instill excitement and urgency on your teams? Bring specific examples, and your continued action plan. (For me, I shared context with the team on why we were doing this release and how it was important)
• Improving/expanding experiment. We are four weeks into the experiment. What has changed and what is working for your team and process?
• Management Growth. How are you doing? What can the team be doing for you?
• Vacations
• Manager Hiring Retrospective. We had two manager interviews last week, and have two more next week. What worked? What can we improve on?

As you can see, there was a lot of room for discussion, but we had a full agenda and didn’t spend the whole meeting in discussion. The meetings were tightly run and had action items that came out of them that resulted in further improvements. But he did involve us in thinking about the ways we were improving the organization. And he had us share examples of how we were solving management problems with each other, to help each of us learn from the other.

What else?

If you experiment with Mini-Ms, please let me know your experiences and variations. I’d also love to hear of any other ways groups of managers have learned from each other. Please share your experiences, and perhaps I’ll incorporate them into this post!

Thank you

Not a lot has been written about Mini-Ms, but there is one post currently on the New Relic blog, and another that has mysteriously vanished.

Bjorn Freeman-Benson was the founder of the Mini-M practice. He shared a lot of his thinking about the principles behind why Mini-Ms were successful and what they were aiming for. He advised me to break up this post into sections and make it easier to get to the implementation section. And shared overall feedback.

Nic Benders ran the first Mini-M, and established and evolved the practice. He also ran one of the more influential Mini-Ms for years. He reviewed this post, offered feedback, and contributed to the history section and described some of the design goals for Mini-Ms.

Darin Swanson authored some content that were inspirations for large parts of this article. He and I have worked together on helping other companies implement Mini-Ms, so contact me if you’re interested in help. He also provided feedback and suggestions on drafts of this article. He encouraged me to explain the first team concept more fully, and to describe why pre-product market fit companies may not be a good match. And he suggested I split the content into separate articles.

Elaine May provided feedback, some of which was so good I just ended up quoting her. She was gracious enough to offer some talk to talk about her experiences setting up or participating in similar programs at other companies. And she talked about her experiences with me in New Relic’s Mini-Ms. Elaine introduced me to the Chatham House Rules, which I incorporated into the post.

Merlyn Albery-Speyer helped me improve the section on “when to use Mini-Ms” by pointing out some preconditions for success. He pointed out that the structure became less important after the Mini-M is established. Merlyn pointed out that we tried to keep people from being in the same Mini-M as their managers, or other people reporting to the same manager. He also shared the theory about VPs not being willing to be vulnerable as a possible explanation for why the Mini-Ms never took off in manager of manager groups to the same extent they did for frontline managers.

Molly Graham contributed the section on “Manager coaching circles” in the variations post. She has an insightful newsletter.

Jason Poole shared his experience as an Organizer of Mini-Ms. He pointed out a now mostly disappeared second post on Mini-Ms. He also suggested ways Facilitators could counter unproductive ranting, and pointed out how effective the Mini-Ms were for improving our performance reviews.

Marty Matheny shared feedback based on his experience as an Organizer. He helped improve the advice for Organizers. And he pointed out that engineering adjacent departments participated.

Chris Hansen pointed out that confidentiality was an issue with whiteboards. He noticed an error that would have been embarrassing. He pointed out the value of M-teams in distributed organizations to counter isolation among managers. He also added some advice for participants. Chris also helped with a point about the value of the first few meetings.

Teresa Martyny reviewed a draft of this post and pointed out some redundant assertions I was making. And she recommended I break this into multiple posts or edit for brevity.

Natasha Litt was another early Mini-M leader, and she reviewed a draft of the post and contributed to it.

Marco Rogers influenced some of my thinking on building communities in a recent conversation, so some of his ideas were reflected in that section.

Image by Klaus Hausmann from Pixabay

In the last post, we looked at how Mini-Ms work and what they are. In this post, we cover how to implement Mini-M support groups.

This post is second in a series:

1. Last post, we looked at how Mini-Ms work and what they are.
2. In this post, we cover how to implement Mini-Ms.
3. Next, we discuss variations you may want to consider.
4. Finally, we share a history of Mini-Ms, including where the name came from.

Mini-M implementation principles

To roll out Mini-Ms, you need to know what to optimize for. These principles can help you as you introduce Mini-Ms to make sure you’re focusing your efforts.

1. The single most important thing is that participants feel safe and able to be vulnerable. All of your decisions around how to set things up should optimize for psychological safety.
   “[Mini-Ms should be] a safe place to vent, both about frustrations with the team that you manage or frustrations with your own manager or senior leaders. This is much healthier than bonding inappropriately with your tech leads or your organizational aunts and uncles. Encouraging this, without becoming overly negative, is always a balancing act. But I think a good Mini-M should feel slightly conspiratorial.” – Nic Benders

2. Mini-Ms are diverse, deliberately. You want participants to feel like they are from a random selection of teams. This helps avoid groupthink. Ideally, you’re talking with managers from product engineering, infrastructure engineering, and platform teams. They all have different ways of working and contribute different perspectives.

3. “Everyone should be about the same level, hierarchically and energy wise. A bunch of keen new managers will learn way more from each other than they will learn from someone who has seen it all.” – Nic Benders. I’ll add that the facilitator can be a little more experienced as long as they are careful not to make it a teaching session.

Steps to roll out Mini-Ms

1. Identify the Organizer.
2. Create an implementation plan, modeled on this and modified to your purposes.
3. Identify Facilitators.
4. Train Facilitators.
5. Communicate program to managers.
6. Organizer creates Mini-M assignments.
7. Run first sessions.
8. Retrospect and adapt.

Roles in a Mini-M

You have three main roles:

Organizer

• Usually a senior leader in the Engineering organization. Needs to be trusted.
• Responsible for the overall program.
• Decides who goes in what group.
• Decides who the facilitators are.
• Monitors how well the groups are doing.
• Shuffles members when groups aren’t working.
• Encourages Facilitators to learn from each other. This often meant lunches or meetings to check in with Facilitators.
• Assigns new people to groups.
• Responsible for the overall effectiveness of the program, to senior leadership.

Facilitator

• Leads a Mini-M group.
• Facilitates each Mini-M meeting. Ensure quiet people speak, and that people have equivalent time speaking.
• Schedules meetings.
• Makes the group function as a team.
• Make sure participants are fully engaged, with cameras on. Follow up with people who aren’t participating, to understand why and to encourage participation.
• Make space for venting, but ensure that the group problem-solves. Don’t allow the group to become a complaint session.
• Consult with the Organizer when there are challenges or the group isn’t gelling.

Participant

• Attends Mini-M meetings.
• Brings problems to share with their group.
• Provides a sounding board and shares solutions. One of the most powerful things a Mini-M can do is to reassure a participant that they are not alone in their problem, provide a different perspective that might ameliorate the problem, or help workshop a solution.
• Maintains confidentiality of conversations held within the group. This of course is as long as it isn’t something egregious that would require reporting to PeopleOps.

Advice for Mini-M Facilitators

The most critical factor that determines the success of a group is that the first few meetings must establish a sense of vulnerability. People need to build a sense of trust with each other, and must feel comfortable bringing up their challenges. In organizations with low trust or high competitiveness, this can be a challenge.

One of the larger things Facilitators need to watch out for is participants that are showing off, criticizing others, or trying to look good. This is a meeting for showing your underbelly. It is a meeting where showing your weakness is desirable.

Facilitators can explicitly discuss this in the first meeting. But they also need to model this behavior. Because the first few meetings are so critical, they should also intervene to stop unwanted behavior. Waiting until after the meeting is over may not be sufficiently timely. Intervention can be a delicate matter, requiring tact, attentiveness, and sometimes humor.

Why are the first few meetings so important? Social psychology suggests that groups settle into patterns of behavior within the first couple of meetings. After that, it takes much more effort to change the expectations of a group. It's also a crucial window to show the value of the group. If participants see the value in the first few meetings, they make it a priority to attend consistently. If they don't, you will struggle with flagging attendance.

A good working agreement might be to abide by "Chatham House Rules”.

The group should feel like they own the meeting. You as Facilitator can modify the format of the meeting to suit the needs of your group. You should feel some freedom to experiment. One Facilitator told me they made a point to throw out the structure, and that it worked as well or even better after doing so. The initial structure was helpful as people were getting used to it, but after that it was most important that the participants were getting what they needed out of it.

Groups can sometimes devolve into complaining. “Having someone ask ‘What can this group do about that right now?’ and leaving with action items was a sign of a useful meeting. This often turned into small projects with the collaborators being the Engineering Managers of that group.” – Jason Poole, former Organizer.

Facilitators can help build empathy across the organization by asking skillful questions. For example, if someone is complaining about a decision a leader in the organization made, the Facilitator might ask, "Why do you think the leader made that decision?" The Facilitator is not the mouthpiece for the organization's leadership, but they can encourage an appreciation for the tough calls that leadership has to make.

One issue for in-person groups can be putting things on a whiteboard. You need to make sure the room is private, so topics aren’t visible to passersby.

Advice for Mini-M Organizers

One of the most important parts of your role is to ensure the groups are going well. This means you should be periodically polling participants, and talking with Facilitators about problems.

When you find a group that isn’t working well, don’t let it persist. It’s often better to explode the group and shuffle some group members to form new Mini-Ms than to let people have a bad experience with Mini-Ms.

You can vary the size of the Mini-Ms depending on the rate of participation. When participation is lower, raise the size of the groups. You need at least three people to attend meetings for them to be useful.

It can be difficult to know if the Facilitator is an effective Facilitator or not. Some groups don’t gel for reasons unrelated to the Facilitator’s effectiveness. If you see a pattern of groups not performing well under a Facilitator, change the Facilitator. The effectiveness of the group is the most important thing. You might choose to coach them, but don’t have them compromise the effectiveness of the group.

When organizing participants into groups, we took care to ensure that people were never with their managers. And we generally aimed to have people in the group not report to the same manager. This made for more of a safe space, and also bred cross-organizational ties. Bjorn: “Many people, when I first describe mini-Ms, don't grasp that the boss cannot be part of the Mini-M. It's a peer group and having any senior executive or boss as a part of group, whether it was me or someone from human resources or anyone else would destroy that peer group.”

At New Relic, “manager of manager” groups were largely less successful than frontline manager groups. Participation was lower, and perhaps the perceived value was lower. To counter this, we made the groups larger. A core group would attend, and the rest would either not attend at all, or attend more sporadically. One theory for this was that VPs and Directors had more difficulty being vulnerable with each other, making their groups less successful.

Want help?

Contact me if you’re interested in getting some help rolling this out. What we will provide is:

1. A detailed implementation plan, customized to the needs of your organization.
2. A timeline to roll it out.
3. A description of the artifacts each role will need to track.
4. A checklist of steps each person will need to do.
5. A training for Facilitators and the Organizer.
6. Implementation assistance while you’re kicking off the program.
7. (Optionally) We’ll run a few Mini-M sessions, to help set the pattern for how they should operate.

Let me know your experiences

I'd love to hear from you if your organization is experimenting with Mini-Ms or something similar.

Thank you

Not a lot has been written about Mini-Ms, but there is one post currently on the New Relic blog, and another that has mysteriously vanished.

Bjorn Freeman-Benson was the founder of the Mini-M practice. He shared a lot of his thinking about the principles behind why Mini-Ms were successful and what they were aiming for. He advised me to break up this post into sections and make it easier to get to the implementation section. And shared overall feedback.

Nic Benders ran the first Mini-M, and established and evolved the practice. He also ran one of the more influential Mini-Ms for years. He reviewed this post, offered feedback, and contributed to the history section and described some of the design goals for Mini-Ms.

Darin Swanson authored some content that were inspirations for large parts of this article. He and I have worked together on helping other companies implement Mini-Ms, so contact me if you’re interested in help. He also provided feedback and suggestions on drafts of this article. He encouraged me to explain the first team concept more fully, and to describe why pre-product market fit companies may not be a good match. And he suggested I split the content into separate articles.

Elaine May provided feedback, some of which was so good I just ended up quoting her. She was gracious enough to offer some talk to talk about her experiences setting up or participating in similar programs at other companies. And she talked about her experiences with me in New Relic’s Mini-Ms. Elaine introduced me to the Chatham House Rules, which I incorporated into the post.

Merlyn Albery-Speyer helped me improve the section on “when to use Mini-Ms” by pointing out some preconditions for success. He pointed out that the structure became less important after the Mini-M is established. Merlyn pointed out that we tried to keep people from being in the same Mini-M as their managers, or other people reporting to the same manager. He also shared the theory about VPs not being willing to be vulnerable as a possible explanation for why the Mini-Ms never took off in manager of manager groups to the same extent they did for frontline managers.

Molly Graham contributed the section on “Manager coaching circles” in the variations post. She has an insightful newsletter.

Jason Poole shared his experience as an Organizer of Mini-Ms. He pointed out a now mostly disappeared second post on Mini-Ms. He also suggested ways Facilitators could counter unproductive ranting, and pointed out how effective the Mini-Ms were for improving our performance reviews.

Marty Matheny shared feedback based on his experience as an Organizer. He helped improve the advice for Organizers. And he pointed out that engineering adjacent departments participated.

Chris Hansen pointed out that confidentiality was an issue with whiteboards. He noticed an error that would have been embarrassing. He pointed out the value of M-teams in distributed organizations to counter isolation among managers. He also added some advice for participants. Chris also helped with a point about the value of the first few meetings.

Teresa Martyny reviewed a draft of this post and pointed out some redundant assertions I was making. And she recommended I break this into multiple posts or edit for brevity.

Natasha Litt was another early Mini-M leader, and she reviewed a draft of the post and contributed to it.

Marco Rogers influenced some of my thinking on building communities in a recent conversation, so some of his ideas were reflected in that section.

Image by Ekaterina Ershova from Pixabay

Today, I would like to share a management practice we developed at New Relic. It was one of the best things we did as an engineering organization. The practice is called “Mini-Ms”. I believe it’s as important a practice for managers as code reviews are for engineers.

This post is first of a series:

1. In this post, we look at how Mini-Ms work and what they are.
2. Next, we learn how to implement Mini-Ms.
3. Then, we discuss variations you may want to consider.
4. Finally, we share a history of Mini-Ms, including where the name came from.

What is a Mini-M?

A Mini-M is a group of managers that meet weekly or biweekly. The meeting is a combination support group and working session. In each session, managers share challenges they face. The other participants offer support and help problem-solve.

Why are Mini-Ms effective?

What is the value in meeting with other managers and talking about your problems? It may not seem like it is worth the time. Yet, many managers claimed Mini-Ms helped them grow more than anything else.

Mini-Ms help you develop skills

There are a few reasons for that. Elaine May describes three reasons:

1. "Learning to manage is very different from other types of learning (like engineering, for instance). In management, the concepts are simple and the execution is not. I've seen this trip up many engineering managers who read a book or attend a class and find the concepts trivial and therefore easy or not very important. Learning management is more of an apprenticeship versus an intellectual pursuit."

2. "In a Mini-M, you have many other people to learn from. If you are only learning from a coach, manager, or mentor, their approach may not fit very well for you, but in a Mini-M, you are getting multiple perspectives."

3. Teaching clarifies your thinking. When you share your experiences and ideas with other managers, you’re forced to articulate your own practice, and consider it in a critical way.

I have a few more explanations:

1. Skills are unevenly distributed. You might be good at running meetings. I might be good at 1-1s. Another participant might be an effective project manager. During a Mini-M, we all share our experience with each other.
   
   Imagine having to fill in those skills some other way. You would have to study each of those topics. Or learn each of them the hard way. That would be much slower! The diversity of the group allows the group to grow faster.

2. Mini-Ms meet you where your skills are. Most alternative methods of learning involve assumptions of where your current skills are. For example, a management course tries to teach you a certain set of concepts or practices. But Mini-Ms allow you to expand your skills from wherever they may be. If I am struggling with how to run a particular meeting, I can immediately add a little to my skills during a single Mini-M session.

This makes the M-team into a combination of a support group and training session for your management team.

Mini-Ms are a source of valuable support

Management is a difficult profession. As hierarchical creatures, being “higher” in the hierarchy separates you from your team members. And other managers are often busy and unavailable.

Because Mini-Ms were largely modeled on support groups, your group can become part of your support network. Even outside of the meetings, you might turn to your fellow managers for help.

Having a Mini-M helped me to build up a mental map of the skills of the managers around me. This was helpful, because I started to understand the skills that I needed to build. When I ran into problems running projects, I felt like I could go to the person I knew was good at it and ask for help. And because we had a relationship from our Mini-M, they were more likely to want to help.

In addition, many companies are distributed nowadays. This can result in less accidental interaction with peers. And it can contribute to isolation. Mini-Ms can be a source of connection with other managers.

Mini-Ms help retain and attract managers

Because managers feel better supported by their peers, they are more likely to stay at the company. Or stay in management. I anticipated that benefit.

What surprised me was that people outside the company seemed to keep hearing about these Mini-M groups. And they would bring it up during interviews and mention it was why they applied in the first place!

Why were they so attracted by Mini-Ms?

• They demonstrated that we took management seriously as a role.
• They showed that we valued developing our people, which was a signal that their job as a manager would be better here.
• They showed we experimented with our organizational practices. And would welcome further innovation.

Mini-Ms help build a management culture

Along with a support group, the other inspiration for Mini-Ms was a conspiracy. The idea was to bring a few managers together, gripe about the problems in the organization, and come up with solutions. Mini-Ms naturally became the graph along which many management practices spread through the organization.

In the Five Dysfunctions of a Team, Lencioni talks about the importance of managers seeing their “first team” as the management team. Mini-Ms helped develop a community of managers that all were focused on improving their skills and their teams. And many of us felt empowered to address the problems we saw in the organization around us.

How does a Mini-M work?

There have been a lot of variations to the way Mini-Ms have run over the years. What follows is a good way to start Mini-Ms.

The Organizer of the Mini-Ms assigns each manager to a Mini-M. Each Mini-M has five to eight managers. The Organizer sets the expectation that managers attend.

Each group has a Facilitator. They schedule and facilitate meetings. They usually start with a standard meeting format. Here’s an example format:

• Each person brings a topic: something that was challenging from last week, or something they are facing now.
• In an in-person meeting, the Facilitator would go around and ask everyone for their topics and put them on a whiteboard. In a distributed setting, all the participants would add their topics at the same time to a shared doc.
• Everyone gets three votes. They put them on the topics they would like to discuss.
• The group goes through the topics and discusses them in turn, ordered by the topics that have the most votes.
• The Facilitator checks in every five minutes to see if the group is finished with the current topic, or wants to continue the discussion.

This is based on the Lean Coffee meeting format.

The Facilitator sets the expectation that meetings are safe places. Participants discuss sensitive topics and there is an expectation of confidentiality. For example, a manager might talk about the challenges they were having with their manager.

When to use Mini-Ms

You should design your Mini-Ms based on the particulars of your company. The size and company culture are particularly important.

One precondition to this being successful is that the environment be a supportive one. If the leaders value learning and growth, they are more likely to support Mini-Ms properly. A competitive culture may not be a good fit.

I recommend Mini-Ms to companies with product-market fit. Early in the growth of a company, product-market fit is the most important factor in a company’s success. After you’ve found a good fit with the market, organizational effectiveness moves up in importance to become one of the most critical factors for your success. At that point, M-Teams begin to contribute more directly to the success of the company. While you can lay the groundwork for M-Teams earlier, I wouldn’t recommend them in most early stage companies.

You can use Mini-Ms globally, across an organization, or within a smaller group. You can even implement a Mini-M informally, with a smaller group of managers. If you start informally, be sure to read the History section to see how they evolved at New Relic.

How do I implement Mini-Ms?

We get to that in our next post.

Thank you

Not a lot has been written about Mini-Ms, but there is one post currently on the New Relic blog, and another that has mysteriously vanished.

Bjorn Freeman-Benson was the founder of the Mini-M practice. He shared a lot of his thinking about the principles behind why Mini-Ms were successful and what they were aiming for. He advised me to break up this post into sections and make it easier to get to the implementation section. And shared overall feedback.

Nic Benders ran the first Mini-M, and established and evolved the practice. He also ran one of the more influential Mini-Ms for years. He reviewed this post, offered feedback, and contributed to the history section and described some of the design goals for Mini-Ms.

Darin Swanson authored some content that were inspirations for large parts of this article. He and I have worked together on helping other companies implement Mini-Ms, so contact me if you’re interested in help. He also provided feedback and suggestions on drafts of this article. He encouraged me to explain the first team concept more fully, and to describe why pre-product market fit companies may not be a good match. And he suggested I split the content into separate articles.

Elaine May provided feedback, some of which was so good I just ended up quoting her. She was gracious enough to offer some talk to talk about her experiences setting up or participating in similar programs at other companies. And she talked about her experiences with me in New Relic’s Mini-Ms. Elaine introduced me to the Chatham House Rules, which I incorporated into the post.

Merlyn Albery-Speyer helped me improve the section on “when to use Mini-Ms” by pointing out some preconditions for success. He pointed out that the structure became less important after the Mini-M is established. Merlyn pointed out that we tried to keep people from being in the same Mini-M as their managers, or other people reporting to the same manager. He also shared the theory about VPs not being willing to be vulnerable as a possible explanation for why the Mini-Ms never took off in manager of manager groups to the same extent they did for frontline managers.

Molly Graham contributed the section on “Manager coaching circles” in the variations post. She has an insightful newsletter.

Jason Poole shared his experience as an Organizer of Mini-Ms. He pointed out a now mostly disappeared second post on Mini-Ms. He also suggested ways Facilitators could counter unproductive ranting, and pointed out how effective the Mini-Ms were for improving our performance reviews.

Marty Matheny shared feedback based on his experience as an Organizer. He helped improve the advice for Organizers. And he pointed out that engineering adjacent departments participated.

Chris Hansen pointed out that confidentiality was an issue with whiteboards. He noticed an error that would have been embarrassing. He pointed out the value of M-teams in distributed organizations to counter isolation among managers. He also added some advice for participants. Chris also helped with a point about the value of the first few meetings.

Teresa Martyny reviewed a draft of this post and pointed out some redundant assertions I was making. And she recommended I break this into multiple posts or edit for brevity.

Natasha Litt was another early Mini-M leader, and she reviewed a draft of the post and contributed to it.

Marco Rogers influenced some of my thinking on building communities in a recent conversation, so some of his ideas were reflected in that section.

Image by J Garget from Pixabay

Introduction

Management is a deep topic, and there isn't a formal program for learning. Books are an essential part of deepening your skills as a manager. Here are my recommendations for books.

Overall management

• High output management. One of the better all-around engineering management books.

• An Elegant Puzzle, Will Larson. Covers a lot of topics of engineering leadership.

• The Louis Allen books on management are out of print and there are a lot of versions, some of which aren't as good as others. These are perhaps the best overall books on management I've read, but they're out of date. I'm working with someone else to get these revised and republished. Let me know if you would buy a copy.

• The Five Dysfunctions of a Team. Has some useful frameworks of how to think about management.

• Drive by Daniel Pink, introduces the basics of motivation. It is important to understand the importance of autonomy, mastery, and purpose. You may not need to read the book, but these concepts are essential to good management practice.

Decentralized leadership

• Turn the ship around. How to make an organization where everyone leads.

• Warfighting Some of the best info on decentralized leadership comes from the US military.

Org design

• Team Topologies. The nuts and bolts of how to organize teams into organizations.

Strategy

• Understanding Michael Porter. This is an essential intro to business strategy.

• Good strategy, bad strategy. How to make strategy that is actually strategy and not wishful thinking.

Lean thinking

• Principles of product development flow. This is a hard book, reminding me of text books from graduate school. However, it covers the principles and laws behind making product development work effectively. It's extraordinary, but a lot of effort. I have had at least one person I recommended this to say they didn't like the tone of the book and couldn't get into it. But I think an essential part of engineering leadership is understanding Lean approaches, and this book, though difficult, covers it better than anything else I've seen.

• Black Swan Farming How to find the most highly valuable work to focus on.

Meetings

• Your Best meeting Ever by Rebecca Hinds, PhD, is a great overview of how to run effective meetings. It's full of a lot of interesting ideas on how to improve your meetings and meeting culture. I interviewed Rebecca for Decoding Leadership. Will share the link when it's available.

Thank you

Image by Mystic Art Design from Pixabay

Introduction

As a consultant, I work at a lot of startups. Many of them find a need to implement engineering levels. Because I've done this many times before, it has become second nature. And, one of the companies I worked at gave me permission to share the levels I developed while there.

So today I'm sharing example engineering levels, and an outline of the process I go through to roll out engineering levels.

Why use these levels instead of any other levels?

These have been used at multiple companies and refined over time.

In my experience, most people copy engineering levels and then let them languish. These have received updates every year. And because I've done this at many companies, I have a lot of experience I've brought to bear on building them.

My recommendation is to review the design choices below and decide if you think it matches your needs.

There are a lot of examples of engineering levels you can steal from, such as the Rent the Runway engineering levels. Carta does a nice job of explaining the philosophy of their engineering levels.

Whichever you choose to base your engineering levels on, you can still use the process below to roll them out. Most companies customize their levels to match their individual needs.

What design choices did we make with these levels?

• I prefer systems that have job titles that map to what people think about rather than arbitrary numbers. So these levels use titles like "Software Engineer", rather than "L2".

• I like having about $US10K bumps in salary for every level. This leads to more frequent promotions, which makes the promotions less high-stake. I’ve seen some evidence from an analysis at a previous company that suggests that high-stakes promotions go more poorly for underrepresented minorities (but don't have the reference handy, unfortunately). Having more frequent promotions can help eliminate that. I’ve also heard many horror stories from big tech companies with high-stakes promotions. These have shaped my thinking. I now believe you should optimize for more promotions, with smaller bumps. But you want the raise to feel meaningful, so there is a minimum amount.

• I have a lot of experience with and prefer fair pay systems -- where your pay is more dependent on your impact than your ability to negotiate.

• These leveling systems can be used with both geographically adjusted compensation and without.

When should you introduce engineering levels

The conventional wisdom (which means a source I can't remember) is that you should wait for product-market fit before introducing levels.

I think that's generally pretty good advice. After product market fit, you're hopefully doing more hiring and growing the organization. Having engineering levels helps you hire and retain your people better.

Introducing engineering levels can have downsides. It's time-intensive, and it can shift the focus of every 1-1 towards getting promoted for a while. In the early days of a startup, you haven't proven your company can survive, so it can seem premature to introduce levels before you've demonstrated product market fit.

How to roll out engineering levels

1. Choose the engineering levels you want to start from. I offer my preferred version below.
2. Make any customizations you think necessary to match your company. When doing so, consider what you want to incentivize.
3. Do a sanity-checking exercise. Go through a representative sample of your team, and rate where each person is with the criteria. I like to take the criteria, make a copy for each person, and highlight the criteria that apply to each person. Make a spreadsheet that compares where people are leveled today versus the new system, and think about how you feel about that.
4. Do a very rough estimate of salaries for each level. Do this by just looking at current salaries for people and where you're leveling them. You want to get an idea of how large the bumps will be, and how many steps there are.
5. Work with HR to develop salary bands. You can read more about this in my post on implementing pay equity, even if you decide not to implement pay equity.
6. Make alterations until you feel like you've got a good system in place.
7. Then, you can start using it privately. Look for people that might be underleveled. You can also use the new levels with hiring. Keep improving the levels until you're happy with them.
8. Think about your promotion rate. What type of budget will you have for promotions? With your leveling system, how often will that mean people will be promoted, on average? This is something you can usually work out with finance, but the sophistication of finance will vary greatly by company.
9. Publish the levels. This involves some deliberate change management. You'll want to talk through the criteria and philosophy behind it. And take care with people who might be sensitive to anything you're introducing.
10. Now use them!

A few suggestions

• I wouldn't include titles you're not using unless you think there will soon be a need for them. Many places won't need Principal Engineer levels, for example.

• When customizing engineering levels, be sure to only choose things that vary depending on the level of experience. Sometimes people want to add items that are baseline expectations. For example, the values of the company. This is a mistake unless it is something that scales with the level of experience and impact the person has.

Note the thank yous

• The work below is the result of the work of a lot of people. See the thank you section below for the credits. The text portion especially owes a debt to the Carta blog post on their engineering levels.

Our titles

We have four engineering titles:

• Software Engineer
• Senior Engineer
• Staff Engineer
• Principal Engineer

There are no standard titles for levels used in our industry. Every company’s system is different. This makes comparisons between companies fraught. A senior level at one company is not the same as a senior level at another place. It’s worth understanding our levels and the way we assess them, as you think about your career.

It is difficult to prescribe the requirements for performing at a certain level. If the criteria are too broad, employees may not know what they need to do to advance. If the requirements are too specific, employees may become focused on checking boxes. We aim to incentivize a focus on areas that grow your skills.

The single most important thing for your leveling is your impact on the company. We can sum up the entire system by describing the impact we expect employees to have as they progress.

• Software Engineers have an impact on tasks and features. An example would be shipping features effectively.
• Senior Engineers have an impact on problems and teams. An example would be helping make a few teams work more effectively together, or making sure a set of features have a better impact on customers.
• Staff Engineers have a sustained impact on the whole engineering and product development organization. An example would be substantially impacting the roadmap for the company, or mapping out the technical plans that will guide a large part of the engineering effort for the next year.
• Principal engineers have a sustained company and industry level of impact.

When we talk about impact, we’re talking about making a difference in that area that we would reasonably mention to an outside observer. So having an impact at the company level means something you could mention at a board meeting. Having an impact on the engineering organization is something you would bring up in an executive meeting.

You’ll note that this is not a linear progression, especially at the top of the range; moving the trajectory of our field (on the broader industry) is exponentially harder than moving the trajectory of the product development organization. This is why our levels aren’t linear, and neither is compensation.

Fairness

Leveling decisions can sometimes seem unfair, and we aim to create a fair promotion process. One important thing to understand is that levels are conservative by default. Once we put someone in a level, there isn’t a practical way to “take it back”. It can cause an otherwise excellent employee to look like they are underperforming at their level, which in some cases can only be remedied by managing them out of the company. So when we promote someone, we try to be sure that it’s the right decision. It’s better for everyone to have you perform solidly at a Senior 3 level than to struggle at a Staff 1 level.

Because levels are conservative, you’ll often hear managers say that promotions are a lagging indicator of performance: You have to demonstrate that you’re performing at the next level, consistently, over a period of time. This can feel unfair. You might ask, "why am I delivering value at a Senior 1 level and only getting paid at a Software Engineer 3 level". But remember that promoting you is a risk for the company, and since the company doesn’t want to make a mistake and lose you, they’re risk-averse. You might be certain that you have all the skills of a senior engineer, but the more obvious that is, the more likely the promotion is to occur.

However, there is also a risk in not promoting people that are performing at a high level. So the way we account for this during our review period is that we evaluate your performance over the entire review period and weigh the amount of time we’ve seen your performance at that level. So if the review period is for six months, and we’ve seen exceptional performance over the last two months, that only gets about a third of the weighting it would get if you had performed that way over the whole period. Similarly, if you’ve had a rough period, we’ll do the same weighting in your favor.

One of the more common mistakes engineers make is they evaluate themselves based on their skills. This is a part of the way you’re assessed, and the code you deliver is part of the value you create. But you’re judged much more by the sum of your impact on the organization and the business.

There will be times when the situation is unfair. You will have done good work, but some “bad luck” affects your project and it isn’t seen as successful. Or your part of the work is excellent but something else causes it to fail. The further up you go on the ladder, the less fair it may seem. The reason for this is that the more senior you are, the more impact you’re having on the organization. And the more you’re expected to fix the issues of how your work integrates with other people. That also means you’re likely doing work that is riskier, or more challenging. More experienced people tend to take this as part of the job, and know that their efforts will tend to be rewarded over time. We can only reward the impact you have, while trying to account for the amount of risk you’re taking, and the difficulty of the situation. This is one of the areas our managers have to take particular care while they evaluate people, because it can also be the most prone to bias.

Another important thing to understand is that the way people are leveled in the organization won’t always feel “pairwise fair.” Sometimes you may think that you’re performing better than someone else in the organization who is at a higher level. And sometimes you may be right. But it’s crucial to realize that making these kinds of comparisons won’t necessarily help convince your manager to put you up for promotion. First, your assessment of your coworker is likely to be incomplete: you have full knowledge of your work, but only a tiny window into that of your peers. But even if you’re right, someone else’s performance problem isn’t grounds for your promotion—imagine what would happen over time if we let the worst-performing employee in a level set the standard. Keep your focus on your accomplishments and the impact you’re driving for the company; don’t waste time and energy worrying about your coworkers.

Compensation and pay equity

We have three levels within each title: 1, 2, and 3. These correspond to three standard salary levels within each engineering title. The three levels have these meanings:

1. Establishing. You’re demonstrating behavior at this level, but it’s not completely uniform or consistent.
2. Solidly executing. You routinely demonstrate this behavior.
3. Mastered. Your behavior at this level sets the standard for that level.

We use structured compensation in engineering, which means salary is dependent on your level. We do this to make compensation fair, so you don’t have to guess if you’re getting paid fairly. Everyone at the same level should get paid the same amount.

There are some caveats here. We haven’t always had this system, so it will take some time for it to evolve to that point. When we promote people, we’ll move them to the standard pay for their level.

We also provide equity at a standard amount per level. We use a system called the implied value method, which ensures that employees not only get a standard amount, but that we correct for prior mistakes in equity grants. This is to ensure our employees get treated fairly so that people who are better negotiators don’t get paid more than people that have more impact. Note that people who join a company earlier will get more equity in the company. They incur more risk by joining a less mature company, so the equity is valued less, and they get more of it. As we get additional rounds of funding, we offer less equity to new employees, because it’s worth more and it’s lower risk. But we also strive to ensure that equity becomes a more regular part of your compensation.

To set compensation, we use [XXX] to benchmark our salary rates. And we target the XX% rate for the market. [also include something on whether you use geo-based compensation or not]

Promotions

The natural question most people have when looking at levels is, how do I prioritize my efforts to get promoted?

This leveling system is a way of measuring your impact on the organization. The more impact you have, and the more you improve things for the company, the more you should be promoted.

The purpose of this document is to serve as a proxy for impact. We spell out the types of impact we value, and help you reason about what to focus on to have more impact. It will never be a perfect reflection of impact, so we expect it to evolve. It’s a bad sign if these criteria do not change every year.

You should work with your manager to review these levels, and assess both how you think you are doing against the aspects of the levels, and how they view how you are doing. Discuss the differences in how you both perceive things, and come to an agreement on where you should spend your time building skills, both technical and with how you work with the people around you. Ideally, your manager’s reviews should reflect what you both agree will make you more valuable to the company, and if you improve in those dimensions you should see career growth.

It is a common trap to get into that your manager will identify something to work on. You’ll work on that, make progress, and then they’ll point out something else to work on to get promoted. This can feel like moving the goalposts. There _will _often be a couple of things to improve to get promoted. Your manager might not always recognize everything at the beginning. They also might realize additional areas when they try to get you promoted. But they should make a good effort to use the leveling system to identify where to focus your efforts. You should work together to create a shared understanding of the progress you’re working towards, and how you’re developing towards those goals.

Manager versus individual contributor

“Engineering manager” and “individual contributor” are parallel tracks. Management is never a promotion from being an engineer, it’s a lateral move.

Since you can’t manage engineers if you don’t deeply understand the work they do, we expect most people who make the transition to management to reach at least Senior Engineer. Many people who decide to become engineering managers stick with it once they’ve made the transition, but some managers eventually decide they’d be happier and more productive as ICs. Both of these transitions are perfectly okay. And some people go between the two types of roles.

If you do decide to transition between tracks, your compensation won’t go down, but your title will change, and you might have a “lower” level. This isn’t an insult: it’s a reflection of the fact that management and software engineering are different jobs that require different skills. Being good at one doesn’t automatically make you good at the other. In fact, it may feel like you’re starting over. If this happens to you, your manager will keep a close eye on your progress toward your next promotion. If your pay is higher than the standard for your level, you should expect smaller compensation increases when you are promoted.

Definitions

• Technical Impact: What qualities do we expect from the solutions you deliver?
• Business Alignment: How do you use your understanding of the company’s strategy to deliver better outcomes? How do you improve the outcomes your team provides to your team’s customers?
• Interacting with Others: To what extent do you coordinate with others effectively to deliver your work? Who is acting on the feedback you provide regularly as a part of your work?
• Autonomy & Ambiguity: What kinds of direction do you need to deliver your work? How do you contribute to project and roadmap planning? What degree of uncertainty are you expected to navigate regularly and successfully?
• Problem-Solving: How complex are the problems you solve? When something unexpected happens, how do you handle the situation?
• Process Improvement: What kinds of organizational processes do you improve, and how do you improve them?

Engineering levels

Title Software Engineer Senior Engineer Staff Engineer Principal Engineer

Impact Tasks and features Problems and team Organization and Product development Company and industry

Level 1: Establishing 2: Solidly Executing 3: Mastered 1: Establishing 2: Solidly Executing 3: Mastered 1: Establishing 2: Solidly Executing 3: Mastered 1: Establishing 2: Solidly Executing 3: Mastered

Technical Impact Solutions solve immediate needs, but may need refinement to scale. Solutions are efficient, maintainable, and pragmatic. Solutions improve the speed of future work & preserve optionality without unnecessary complication. Solutions are exemplary in terms of scalability and cost-effectiveness. Makes trade-offs between opportunity vs. complexity.

Business Alignment Understands the objectives of the projects that their tasks support. Familiar with the value their team delivers to their internal or external customers. Understands the objectives of the project and uses that to clarify and improve project plans. Identifies ways the team could provide better value for their customers, especially where value overlaps with their own work. Suggests and delivers improvements based on these observations. Understands the corporate direction and uses it to improve the definition for teams’ projects. Identifies ways the team could provide better value for their customers across everything that their team owns. Suggests and delivers improvements based on these observations, even when this requires work on areas outside of their team’s area of ownership. Clarifies strategic outcomes and influences roadmaps and projects. Identifies improvements in the entire end-to-end experience their customers go through, even for parts of the experience that other teams own. Suggests and drives improvements based on those observations (either directly or through others).

Interacting with Others Work does not require them to coordinate with others (and their tasks) to deliver. Work does not require them to influence others. Advises peers on their team. Work requires coordination with people on their team, including product and design. Influences improvements for their team and the projects they work on. Advises peers on their team, their direct manager. Work requires coordination with people outside their team, including product and design leadership. Influences the roadmaps of other teams, especially to get work prioritized that’s required for their own team. Advises people across their VP-level org. Work requires coordination across the entire company. Influences the entire org to make changes to support their work. Advises teams across the company, outside of their VP’s reporting chain.

Autonomy & Ambiguity Implements non-ambiguous tasks with limited direction. Can break down their portion of a project. May need oversight to validate the approach. Requires no direction on tasks. With limited direction, defines and breaks down ambiguous projects or projects with non-obvious solutions. Breakdown is incremental and cross-functional. Clarifies requirements. Contributes to task estimation. Capable of negotiating and working out solutions to ambiguity that are effective for the company. Simplifies. Requires no direction on project plans. Requires limited direction on designing to support a long-term roadmap. Effective at considering how work breakdown will be effective across the team. Considers and plans around cross-team dependencies. Defines & breaks down challenging projects (e.g. projects that are hard to derive the benefits until the end of the project, projects that are very large, or projects that have a lot of uncertainty or require novel solutions). Requires no direction on designing to support a long-term roadmap. Translates customer & business needs and strategic direction into projects. Consistently simplifies high complexity situations.

Problem-Solving Solves straight-forward problems. Learning planning approaches. Plans their own work. Contributes to team decomposition and estimation. Escalates when projects hit roadblocks. Troubleshoots issues and addresses immediate causes. Solves difficult problems. Removes bottlenecks. Can sometimes find a way forward without escalating, but still provides visibility into risks. Escalates in difficult situations. Is able to identify trade-offs and make recommendations. Identifies and addresses root causes. Makes trade-offs between short-term and long-term solutions. Evaluates (and is accountable for) trade-offs others are making. Independently finds a way forward in difficult situations. Effective at recognizing/mitigating risk and removing roadblocks. Anticipates future risks and removes them before they become a problem. Decomposes strategic direction into projects. Plans, communicates, and executes on the most challenging problems in the company. Anticipates most risks. Drives simplification to mitigate risks ahead of time. Helps the rest of the organization reason about risk.

Process Improvement Works within defined team processes. Raises concerns when it breaks down or fails. Improves team documentation. Identifies issues with their team’s processes, communicates the cause, and proposes improvements. Automates manual tasks to improve speed of delivery. Influences & implements best practices. Owns projects to streamline team processes. Work often speeds or simplifies the work for people on other teams and is accepted as broader standards. Proposes new organization-level processes to improve key areas such as team throughput, employee happiness, or product engagement. Drives best practices across the organization.

Want help?

I've done this many times before, and I'm available to help guide your organization through it.

Future improvements

• Jade likes the simplicity, visualization, and mental model proposed by Amy Chanta here.
• Honeycomb has some interesting elements of their engineering ladder Jade might want to steal from.

Terms of use

• You can modify this for your company’s purposes as you like.
• If you make big improvements to this document for your own purposes, please share them with Jade. He’d like to improve the original over time. This is a request, not a requirement.

Thank you

• The original version of this was developed by Shaun Yelle, Alexa Stefanko, and Jade Rubick at Gremlin. It has since been modified and improved as it’s been used at a couple of companies.
• I'd like to thank Gremlin, and in particular Matthew Fornaciari, for granting permission to share these levels outside of Gremlin.
• The inspiration for this format was developed at New Relic, after a couple of iterations. Jade Rubick did a lot of the work on those versions.
• Carta's excellent post on their engineering levels had some really well done language. I adopted some of that language and have modified it for use here.
• Image by stokpic from Pixabay

At some point, you begin to lead. This is very different than managing. The difference can be summed up with the phrase, "leaders make their own problems". I'll explain that in a bit, but first, let me tell you a story.

When I first realized I was missing something

When I first became a director, I had a conversation with another Director named Jim.

Jim: "So, you've been in this role for a month now. Have you figured out your vision for your organization?"

Me: [What is he talking about?]

I realized I was used to taking direction from others, and I had become a sort of "execution" focused manager. I didn't have any idea what my organization needed. I had no idea what the future should look like. I was managing my organization, not leading it.

Mental models: try and catch them all!

At around this time, my manager challenged me to start using more mental models in how I assessed my organization. He suggested looking at the situation from the point of view of:

• A systems thinker. (this was my default)
• An architect
• A product manager
• A designer
• Etc.

Most leaders use their own experience as their main mental model. Being able to add new mental models to your toolkit makes you more flexible. Each mental model can alter what you think the future should look like. And what the present should look like. It can also inform you of the actions you should take to make things happen.

As I started developing a habit of looking at things from other mental models, I began to see blind spots in my previous ways of looking at the world.

The limitations of role-based thinking

It's useful to have a clear definition of your role. But your role definition can be overly constraining. If you define your success by if you're doing your role well, you're limiting the type of problem-solving you will engage in.

For example, you might think of the role of Engineering as execution. This will limit you from thinking about the Product perspective. Yet, some of the magic happens when you're both thining about the intersection between these functions.

Try using "we" thinking instead of "me" thinking. What can we do as a wider leadership group? How can my organization support what the company needs to be successful?

The limitations of incremental thinking

Another challenge most leaders get into is thinking of situations incrementally, instead of thinking back from what MUST HAPPEN.

An example of this was when I was working in a place that was a Rails shop. We were building some technology that would have more ideally been built in Java, and this was a long term effort, well worth doing it in the most appropriate language. Because I was thinking incrementally, I was thinking about how to build the software with our existing team. Not about how to create the team to build the software we needed to build.

Interrogate yourself

I brought up being execution focused with the executive coach I was working with at the time. He encouraged me to set time aside every week to think deeply about my area of the product.

At first, it was hard to preserve the time I set aside. In my busy schedule, it seemed luxurious to save a couple hours just for thinking. But it quickly became some of the most valuable time I spent each week.

I would spend time each week thinking abount my organization, about the product, about how things were going. I researched other products that competed with mine. I spent time using the product. I thought critically about our technical and product failings.

I ended up developing a better sense of what possible futures I could create. I started to see possibilities that were invisible to me. I started to learn how to lead.

Interrogate yourself with what exactly?

Each week, I sat down and looked at an evolving set of questions. I would go through the list and write answers to the questions on the list. Here is my list of questions:

Review high level objectives

• What are the results that are needed? What needs to be true in a year? X needs to be Y by Z. What would need to change for that to happen? What are the implications of that need?
• How are the teams performing? On time, hitting the mark, right characteristics? How are they performing AS a team?
• Where are things on the path to fail?
• What's the worst thing that could happen?
• Is there anything I think my boss is making a mistake on?
• Can all of my reports take over my job in 1-2 years?
• What seems like it's not working right now?
• What's going well that could be turned into a repeatable process?
• What should I worry about?
• What do my managers need?
• What does my org need?
• What do I need?

Review progress against objectives

• How on track is everything?
• How long should the rope be for all of my objectives? Agree to them with other people. For ex: when should we expect reliability to improve?
• Review team health
• Review progress against goals.
• How are team members doing against their goals?

Identify next actions

• What actions should be taken?

I found writing down the answers to these questions to be tremendously helpful.

If you're an external processor, you might find it more useful to talk with others about these questions. For me, I did a bit of both. I found a few other people that I would talk about these things with. But I found the private thinking time valuable -- it gave me more material to test with others in conversation.

Leaders make their own problems

A leader is a person that asserts a future for a group of people.

If you want to lead, you have to learn to make your own problems. You want the problems you're working on to be partially something you've generated. Because that is really the role of a leader: to scout out the upcoming terrain and prepare a group of people to navigate it. While you don't need to do all the scouting yourself, you need to have a way to make sure the future is explored and thought out.

Hopefully, you'll find this post useful to develop and approach to navigating that future. If you've found other methods to do so, please share them with me!

Thank you

Jim Ruppert helped me see that I didn't have a vision for my organization, and his gentle nudge helped me expand my horizons. Robert Goldmann helped me learn the perspective I needed to lead better. He helped me develop some of the questions on this list. Alex Kroman challenged me to consider things through multiple mental models. Thank you to George Sudarkoff for feedback on the post and some minor edits. Nate Saul pointed out typos (oops!).

Image by Arek Socha from Pixabay

Today, I'll share a few thoughts on what makes a good project plan. And I'll provide a sample project plan.

Why have project plans

Many agile teams focus on sprints or chunks of work. But they don't really plan -- instead, they do what they can each sprint, plot out their velocity, and determine what they can accomplish over the next sprints.

This is fine, but project plans are a tool for getting you to think about the contours of your projects. They have the following advantages:

• You can play with different ways of structuring the project, so you can sequence the value of delivery easier. This makes it easier to play with scope, incremental delivery, and to adjust to changes.

• You think through risks better with a project plan. You have to explicitly list things like when people are on vacation or on call. This can result in better planning.

• Many teams use sprint cycles or kanban cycles that are longer than a week. Weekly project plans give you more frequent signals if you're on track or not.

• Sprint plans track all the work, which is useful. But that also makes the plan harder for stakeholders to understand.

Keep your project plans simple

Typical failure modes for project plans are no project plans at all, or overly complex project plans.

Complex project plans...

• can have many project artifacts.
• require time to keep up to date.
• can be brittle, requiring fixes or maintenance. They are highly dependent on the person who made the plan.
• sometimes create the illusion of more certainty. For example, the project "will be done in 23.53 to 27.55 days".

I know a project plan is in dangerous territory when I see people allocated as fractional "resources". Or when the plan is something that only one person can update.

Why keep it simple

One of the dangers of plans is that they can cement things into place. You want a project plan that allows you to make changes in seconds, not minutes or hours. Most complex project plans disincentivize change.

You also want a project plan that clearly communicates. An outside observer should be able to look at your project plan and figure out what will happen when. This requires the right altitude. Keep it simple.

Qualities of a simple project plan

Here are some things I recommend in a simple project plan:

• Make it week-by-week. List out what should happen each week, with a couple of bullet points. Doesn’t need to be more complex than that. What should the bullet points be? Things you will demo at the end of each week.

• Estimate milestones, not projects. You should plan milestones, not projects. Yes, a high-level project plan can be important, but you also shouldn't overinvest in it by planning out the whole project in advance. That prevents you from making changes to sequencing or adapting based on things you've learned. You should make a high-level technical plan, and make a list of sequenced milestones. And to estimate the overall project, you might do some high-level estimates. But I recommend only having a week-by-week plan for the current milestone. [Does that mean these are really milestone plans? Yes, but I call them project plans anyway, because that's what people are used to.]

• Milestones should be less than a month in length. See the milestones post for more details. The key insight here is that small projects typically go way better than large projects, so make all the projects small projects.

• Combine plans with weekly communication. It’s helpful to combine project plans with project reporting. Use a style of communication that is concise, represents the state of things dispassionately, surfaces risks, but maintains a “I’m taking care of things” tone. Combining plans with project reporting ensures the plan will be updated at least once a week. I’ll write about project reporting soon.

• Make text-based plans. I’ve found it more useful to have project plans be text-based, rather than tied to tooling like Jira. Tooling-based approaches are fine. But text-based approaches force people to think through everything differently. You can use links to link to sprint plans or individual stories, or whatever. But it keeps it easy to understand for someone not aware of the project. I sometimes don't insist on this, but it depends on the circumstances.

Weekly project plan template

This is for a milestone within a larger project

Week of Jan 4th

• Single chart shows up in Slack. Data is canned.
• Schedule risk: we're validating our list of chart types are all technically feasible. We'll demo outcome of that investigation.

Week of Jan 11th

• Chart data reflects live information, and is functional in Slack chart.
• Additional chart type shows in Slack room, with most basic visual design.
• We’ve shown to at least one alpha customer for feedback. We start sharing with them every week from here on out.
• Jessica is on-call and doing interrupt-driven work for week.

Week of Jan 18th

• Most important feedback from alpha customers incorporated. Other work is prioritized to future milestones.
• Last chart type added.

Week of Jan 25th

• Holiday Jan 26th.
• Charts look great and are thoroughly tested, instrumented. We’ll show usage dashboards.
• Release end of week.

Thank you

Thanks for putting up with the click-batey title. It's truly terrible.

Image by Gerd Altmann from Pixabay

As a consultant, I have a view across many companies. So I’ve seen a lot of product strategies. Most have been problematic. Occasionally, I see an example of a product strategy that stands out. Today, I review common problems with product strategies. Then, we’ll cover how to craft a product strategy that avoids these problems.

Problem: product strategies take too long to create

Putting together a product strategy is fucking hard. And while it is important, it’s easy to defer.

Sometimes, you don’t have all the information you need. You need to develop deep expertise in your area or market. That requires time – contacting people, having lots of conversations, and developing a deep sense of the dynamics of your space.

Writing down your thoughts requires sustained, focused time. The people who usually write product strategy (CEOs or heads of product) have hard jobs that lure them from that focused effort. This can result in it taking a long time to put together.

Creating the strategy also requires influencing and collaborating with many people. All of these interactions require time to get people on the same page, discuss disagreements, and incorporate improvements or changes.

Finally, your market can change quickly. New competitors can emerge, technologies change, and customer feedback can shift. These all can result in changes in perspective or emphasis, which can further slow down putting together a product strategy.

And finally, even after you’ve done all the hard work putting the strategy together, you have a lot of work to do communicating that strategy, and getting people to understand it. This also takes a lot of time.

The end result of all these steps is that a common failure mode is “the product strategy is coming”.

My recommendation is to always have a working product strategy. Because strategy work takes time, you shouldn’t make people wait for it. If you don’t have a real strategy, start with a temporary, short-term strategy, based on your best thinking at the moment. Saying the product strategy will be created in the next few months creates the opposite of clarity, and is a common product management failure mode.

When you’re starting out, it will be a bad product strategy. And you can be up front about the fact that it is temporary. But people often need some sort of guidance about direction, and having temporary guidance is way better than no guidance at all.

Problem: product strategies that are a roadmap

I’ve seen some leaders view the product strategy as a communication challenge. They know what needs to be done. They just need to tell their vision to a bunch of people.

So, they write down some themes to focus on. And they share a roadmap, and talk about why it is important. Then they clap their hands together and say, “done!”. They’re not done.

Usually, they then get feedback that people are still confused. So they give another presentation on the “strategy”, and it helps, a little. But all they really did is communicate what the company will be working on. People are still confused.

A strategy isn’t just a list of things to do. It’s not merely a roadmap. So calling a roadmap your product strategy doesn’t work, and that’s part of the reason it doesn’t clarify as much as you would like.

We’ll discuss the parts of a real strategy later in this post. But for now, my advice is to view a product strategy as a larger piece of work than a communication exercise, or a prioritization exercise.

Problem: product strategies that are goals

Leaders often view the product strategy as a tool to get people aligned. And goals are a way to get people aligned. So many leaders muddle the two together.

They say:

• “Our strategy is to land five big enterprise customers next quarter”
• “Our strategy is to grow our business 20% compared to last year.”

A goal is not a strategy.

Your goals should emerge naturally from your strategy.

This isn’t to say using goals is a bad idea. You should be familiar with all the tools at your disposal: Goals, tenets, standards, metrics, and centralized prioritization are all valid and effective coordination models. A product strategy is another. I recommend looking through each of them and understanding their tradeoffs. The product strategy is probably the most foundational of all of these options, though, so it’s the place I would recommend you start.

Problem: your product strategy doesn't need to cover everything

"One issue I see is that there is a natural inclination to create a strategy that encompasses 100% of the activities of the company. It's not bad to do it that way, but it's not necessary. Some things have inertia. You only really need to include things in the strategy when they need to change. Focus on those things and everything else will be fine." -- Alex Kroman

A few other common problems with strategies.

I've written about some other common failure modes of strategies: they don't hurt, they don't outline "the diff" against now, or they don't tell a story.

How to put together a good strategy

So how do you go about crafting a product strategy? Here is some advice I’ve given. And I asked product leaders I respect what they thought, and they added their advice as well. Here’s my synthesized set of recommendations.

First, stop what you are doing and read these two books

Clear your schedule, and read this book: Understanding Michael Porter. If you’re not sure if it’s worth reading or not, start by reading this summary of Understanding Michael Porter. You’ll learn more about strategy from reading that book than anything else I’ve encountered.

A second book on putting together your strategy is Good Strategy, Bad Strategy.

Second, study your business and market

You need to understand the market you’re in to make a good strategy. This seems obvious, but I’ve seen a lot of product leaders try to make a product strategy without fully understanding their customers.

You should be able to answer these questions:

1. What are the various segments of customers you have? (This should be as detailed as possible. It can include size of business, types of problems they face, how they purchase the product, what other products they use, and many other factors)
2. Which customers are your best customers?
3. Why do your customers choose your product? Why do customers choose competing products?
4. What are the trends in your area of the market?
5. Which trends are you well positioned to take advantage of? Which are you poorly positioned for?
6. What is it like to purchase your product? What kind of an experience do people have when getting support, or learning your product?
7. What is it like to sell your product?

Third, talk to a lot of people

How can you possibly learn the answers to all these questions about your market?

1. You need to be talking with as many customers every week as you can. In some companies this is a challenge. This has to be solved immediately – if you and your peers aren’t connected with customers, your company will eventually fail.
2. Go through the experience of signing up for your product. Try it out. Pretend you are various types of customer, and go through the experience you think they would have. Try the same thing on competitors’ and see what they are like.
3. Talk with people within your company that have good information for you. Sales people. Analytics people. Support people. Partnership people. Finance people. Come prepared with lots of questions.

Fourth, start sketching out a draft strategy

Your strategy work should be iterative. So start clarifying your thinking by putting together a rough draft of an early strategy.

I recommend an outline similar to the following (similar to Good Strategy, Bad Strategy’s format):

1. The situation. Outline the environment the business is in. This should include the business environment (runway, financial strengths and weaknesses, overall trends), what changes are happening in your space, what direction things are going (generally and with each competitor), if there are timing considerations where opportunities will close or open, and what the competitive environment is like. After reading this, everyone in your company should have an elevated understanding of the business situation and market conditions.
2. The current product. Describe the state of the product. What are its capabilities? What are your liabilities, both in terms of what the product can do, and your ability to evolve it? What tech debt issues are you facing? Are you meeting goals for usage, user experience, and adoption? How does it interact with the current situation the business is in?
3. The diagnosis. What is the most effective path forward? Describe what should be true in the future that isn’t true today, the path forward, and the implications of that choice. The path to being successful should be clear, and risks outlined. The purpose of this section is to be crystal clear -- everyone who reads this section should understand why the company is focusing where it is and how that will connect with the company being successful. This section should outline your target customers, and how you will differentiate or compete.
4. The argument and alternatives. You almost certainly have alternative approaches floating around. There are people within your organization that are committed to them. This is the section where you make the case for all the alternatives, and explain why they are less compelling choices than the one you’ve chosen.

As you’re writing the strategy, you’ll realize there are gaps in your understanding. That’s fine -- highlight that there is missing information, and start tracking it down. Strategy work takes time and is a lot of work.

Use offsites and regular strategy sessions to talk through the various ways your company can succeed. And make sure you can articulate what the rest of the actors in the market are aiming to do strategy-wise.

Your work on this should be iterative, and even after you produce the first version, you’ll want to revisit it periodically.

Some tips while developing a strategy Good strategy takes advantage of your strengths, and minimizes your weaknesses

I like to think of companies as being in an ecosystem in the same way that species are part of an ecosystem. If your strategy is to compete directly with another company, you better have a really strong reason to believe that you’ll outcompete that competitor. Head to head competition is usually not good for either party, so you have to have some sort of innate advantage that your competitor can’t compensate for. This is likely something like a capability they can’t easily add, or because your product is much less expensive than theirs due to some sort of structural reason.

When you key in on your competitive advantage, you have to be consistent with using that as a part of your strategy. For example, Google Docs was able to outperform Microsoft Word mostly because the value of collaborating on a document was so much greater in Google Docs, because it was web-based, and able to maintain centralized state. New Relic was able to establish itself in a whole new category of Application Performance Monitoring, because it was SaaS based and had a product led growth approach to purchase.

Strengths are usually structural like that, based on

1. Network effects. The more customers you have, the more valuable your offering.
2. Cost differentials. You can offer your product much less expensively than your competition. Or,
3. Capabilities that aren’t easily replicated. You provide capabilities that a competitor would struggle to replicate, or it would degrade their product to replicate.

These are worth thinking deeply about. How can you build a business where someone can’t just come along and replicate everything you’ve done and destroy your business?

One exercise I like to do if you’re a market leader is to imagine what product you would build if you could take the best people from your company and to go off and create a competing product that would aim to destroy the original company. If you do this exercise, and your competition is pretty similar to what you come up with, you might be in trouble.

How do you know if you've thought enough about this?

"An extremely simple test of whether you have thought deeply enough about the strategy is to ask if the strategy could be followed by another competitor. If the answer is yes then you haven’t gone deep enough to understand your strengths/weaknesses" -- Alex Kroman.

Good strategy includes where you don’t plan to win

Organizations also tend to want to do everything, so it’s important to make it explicit what you’re not planning to do.

Nadya Boone Duke: “So much of strategy is focusing on few things with enough vigor to make a difference. When I outlined a recent strategy I made an explicit section for ‘What we're not doing.’ There are plenty of good ideas and they will ooze into the workstream unless you build a way to keep them out, and then dilute your focus on the big rocks.”

Henry Shapiro: “[A particularly good strategy we saw at New Relic] outlined the things we were not doing as a result of the strategy. This was part of jettisoning [an important but off-strategy new product], deciding to focus on our core audience, and not get sidetracked with adjacent markets.”

Good strategy addresses how you’ll overcome obstacles

Many times you won’t be in an ideal situation. You might have some big hairy problem you need to solve with product-market fit. Or you may have a competitor that you haven’t figured out how to compete against. You can outline the bets you’ll make, even if you don’t fully understand them yet.

Your strategy is a plan to be successful in the environment you’re in. It’s incomplete if it doesn’t meet that goal. It’s fine for your strategy to be incomplete, but you should continue to work on it.

Get buy in from the right people

The best strategy in the world won’t go anywhere if you can’t convince people it’s the strategy to follow.

Henry Shapiro: “I think one of the central challenges of the product strategy stuff is that strategy often has to balance buy-in with inertia. That is, the larger your cohort of ‘writers / feedback-givers’, the more the strategy suffers from groupthink and long collective bargaining efforts. I think [the thing to do is] to get buy-in from the right-people, which is a skill unto itself. Part of that is working with a single partner on the eng side to assess capacity and skill gaps, and having a similar partner on the biz / sales side, and maybe an exec who is going to have to buy in as well.”

Use “invent the future” meetings to focus creativity on the future

I read once that Steve Jobs dedicated most of his Mondays to talking with his most trusted advisors about the future. They spent all Monday talking through various aspects of how the future could play out, and how they could shape it. I don’t know if he actually did this, but after reading it, I played with something I started to call “Invent the Future meetings”.

Sometimes groups can have a vision that is too focused on the next month. If you need more focus on where you’re going, try an “invent the future meeting”. Here’s how I ran it:

1. I’d select a topic, like, “what should customer sign-up look like over the next few years?”, or “what should our product look like in two years, so that it’s much more useful to site reliability engineers?”. I’d ask people to think about the question, and come prepared to present their thoughts. (No slides or excessive preparation).
2. At the meeting, we’d go around the table and each present what we thought the future should look like, and why.
3. People could ask clarifying questions, but after going through all participants, we would discuss the tradeoffs of the different approaches.

What I found is that we would often settle on some new possibilities that we weren’t even considering previously.

How to know when you’re done

Gregory Kim: “It might be worth mentioning adding rules for knowing when your strategy is good enough. I.e., would I stake my future livelihood on this? The more critical the need for success, the higher the bar should be for the strategy.”

The strategy needs to solve these three things:

1. Profitable
2. Not easy to replicate
3. Valuble to customers.

Remember the opposite test

The opposite of your strategy should be a viable strategy.

Nadya Duke Boone suggests checking to see if anyone would ever do the opposite approach. If no one ever would do the opposite, then your strategy is too generic. This helps you recognize goals masquerading as strategy.

Note that the opposite test is a good way to show a strategy is not a viable strategy. But it doesn't necessarily show that a strategy is a good one. For example, a strategy of "focusing on growth in the Enterprise market" passes the test, because another company might focus on SMB customers. However, focusing on the Enterprise market is just the beginning of a good strategy.

Update it regularly

Gregory Kim: “I think one of the arts of implementing a strategy is knowing when to be fluid and adapt to incoming information and when to stay the course. Supporting this is a couple of mindsets. Be confident in what you propose and understand what it is based upon. Be flexible to change if important underlying assumptions are proving to be inaccurate or if new unexpected opportunities become known.”

Make multiple versions

Henry Shapiro: “I think what made Patrick Lightbody’s product strategy so successful is that it was 12-13 pages long. Having something that is visual and instructive and brief is powerful. The other thing Patrick did was make a slide version of it, which was helpful as a way to communicate it to that 20% that didn't read it.”

What if you’re not responsible for Product Strategy but need it?

Ask the VP of Product or CEO to present the Product Strategy to your group, or ask for a written version of it for your team to use as guidance. It likely will be insufficient at first, but it can start the conversation. Or send them to this post.

Communicating the strategy

Like most things, strategy work benefits from repetition. Here’s what I’ve seen work:

1. Get leadership aligned on it.
2. Send it out in written form.
3. Communicate it at an all-hands.
4. Do follow-up Q&A sessions
5. Repeat yourself.

You may be surprised at the appetite for hearing about product strategy. When Patrick Lightbody and his team came up with New Relic’s first written product strategy, they sent out a link to it ahead of the All Hands. Patrick asked who had read the product strategy, and a stunning 80% of the audience raised their hands to say they had read it!

Executing on the strategy

Once the Product Strategy has been finalized, it’s often a good next step to have various parties write down their Execution Plans in response. You want them to think deeply about the implications of the plan, and reconcile it with what they were already planning on doing. And then you want to rationalize these plans against each other -- sharing what people are thinking, and dealing with the inevitable conflict between plans and reality. Sometimes this will require further revisions to your strategy, if you uncover anything major.

You should make sure you make space for people to surface the tradeoffs and implications of executing against this new strategy. Your product strategy is a tool for improving and communicating your strategy, not holy scripture. You may not realize some of the tradeoffs you are making, so ask people to be explicit about what implications they see. Team should volunteer the things they’re giving up or deprioritizing, so everyone has full visibility into it.

Alex Kroman shares some tips on how to execute on a strategy in my post on Product Councils.

I talk further about this topic in a Level Up podcast: Product strategy pitfalls.

See also

I really liked this post on Lenny's Newsletter about the mechanics of putting together a product strategy.

Thank you

Many people weighed in on early drafts of this post – any mistakes since then are probably my fault. Big thank you to Nadya Boone Duke, Henry Shapiro, and Gregory Kim for substantial critique and improvements. Gregory Kim suggested contrasting the value of product strategy to other coordination models. Nadya suggested adding in the current state of the product to the outline for a product strategy. Alex Kroman not only introduced me to Michael Porter, but taught me the nuts and bolts of strategy. And he sent me a great message that provided a couple of quotes on strategy. Patrick Lightbody showed me an example of what a good product strategy could look like. Todd Etchieson, Kevin McGuire, and Nic Benders taught me a lot about strategic thinking. Thank you to @gabor on Twitter for the great strategy quote.

The profitable, not easy to copy, and valuable to customers framing was something I heard on Lenny's Podcast. However, I wasn't able to find out which episode, so wasn't able to attribute it correctly.

Cover image by Steve Buissinne from Pixabay.

A common leadership mistake is process gates. What are process gates? Let’s see if you can spot the pattern:

• Engineering is shipping too many bugs, so you bring in a QA team that reviews everything before it goes out to production.
• There has been a history of making poor architectural decisions, so you put in place an architecture review.
• The team keeps shipping completely unusable UIs, so you add a designer review step before anything can ship to production.
• The team keeps having cost overruns in AWS, so you have a central team that controls all infrastructure code.
• People are doing a shitty job with code reviews, so you add a group of the most senior people who are the only people able to merge PRs, and they review everything before it goes out.

The name for what you’re doing in all of these examples is adding “process gates”.

What are the problems with process gates?

Like any management technique, process gates are a tool you can use. But they have consequences that tend to be much worse than most people imagine.

The issues with process gates are many:

1. Process gates tend to persist over time
2. Process gates increase cycle time, and that is a very, very bad thing.
3. Process gates tend to proliferate into a death spiral (things aren’t working, add more)

Process gates tend to persist over time

One of the dangers of process gates is that they’re difficult to remove. Because they’re usually done as a response to something bad happening, removing them feels scary. Removing a process gate is usually seen as reintroducing the original pain. So they persist, and tend to cause subtle damage over time.

Process gates increase cycle time, and that is very bad

Most engineering organizations focus on speed, but they should be focused on cycle time. Cycle time is the primary indicator of how successful your engineering organization will be. Or at least this is true for 99% of modern software-based product development.

This is an important concept, so if you don’t believe this already, please read my post on this topic: What can air combat can teach us about software project failure?

The reason process gates increase cycle time is that they’re adding steps to a process. Mathematically, there is no way they cannot increase cycle time. So what that means in practice is, they do something like the following:

• Increase the amount of time before a PR is merged.
• Increase the amount of time before a release goes out.
• Increase the amount of time before an infrastructure change can be made.
• Increase the amount of time before an architectural design can get approval.

The problem with these cycle times is that they’re also difficult to see. They tend to accumulate over time, and not be the obvious problem to why engineering is slowing down or delivering less. And they result in an overall less effective, less responsive organization.

Another way process gates increase cycle time is that they often increase the amount of passing back and forth between people. For example, if you add a QA step before work goes to production, you’re adding a whole set of work that needs to go back and forth before things can be released. While this may seem desirable, what it often accomplishes is to create a whole new category of work that bounces between teams. This work has its own latency, because it may only be reviewed once a day. This can add days and days of cycle time to releasing code.

Process gates tend to proliferate into a death spiral

When cycle time increases, the predictable result is that batch size increases. So this also means you have larger chunks of work moving through the system, instead of smaller chunks of work. This decreases the flow of your overall product development, and results in lower quality, less responsive work overall.

So the thing you’re trying to do (improve quality, for example), tends to actually get worse as a result of adding a process gate for quality.

There can also be moral hazard issues with process gates. If the development team is handing off their work for testing, then they are less on the hook for quality themselves. Again, this can result in things backfiring, and actually getting worse instead of better.

What should you do instead of process gates?

So instead of process gates, you should reach for alternatives:

1. Temporary or narrowly focused process gates
2. Automation and alerting
3. Non-gated checks
4. Do it in the same cycle
5. Learn more from failure

Make your process gates temporary or narrowly focused

It is possible to use process gates responsibly. The best way to do so, is to make them temporary or narrowly focused.

Temporary process gates are something you use to get people to pay attention to things. Or you can use them so you can be made aware of exceptions that might show your process changes are inadequate.

My favorite example of this is as a method for moving to services. If you have a long-term initiative to move your organization to services, it can be quite painful at first to use services. The most elegant way to handle a microservices initiative is to spin up a team completely focused on eliminating the obstacles to services. Their job is to always work on the thing that is most standing in the way of people using services in new projects. The process gate to add is to say that all teams that are planning work need to share any pain points that are preventing them from doing work as services rather than in the monolith.

Note that in this example, the process gate is very lightweight. You’re making people think about these things, and share some information with the team that is working on services. But you’re not truly making a process gate, because they’re not being blocked by this team. So in a lot of ways, it’s not even a real process gate.

Also note that it is temporary in nature. Over the course of time, this process gate should become less and less necessary. As services become easier, this process gate becomes less relevant.

And also note that this process gate is helping this team focus its efforts in reducing cycle time. So it’s counterbalancing a lot of the disadvantages of typical process gates. So this is a nice example of how to do it right.

You can also focus on making your process gate very narrowly scoped. If you have a high number of problematic PRs getting merged, you might want to have all PRs go through a set of experienced reviewers, for example. You can flip it around so it’s much more narrowly scoped: people go through a probationary period for a few months where other people review their PRs, and then they graduate to being able to merge them for peers.

As another example, you can have people tag their PR based on the risk level that it will cause problems. High risk PRs can be reviewed by the most experienced team members. One of the advantages of doing this narrow scoping is that you avoid some of the problems with process gates. If the most experienced team members have to review everything, they’ll be overwhelmed and everything will take much longer. If it’s narrowly scoped, they’ll have less overall burden, so the increase in cycle time will be more modest.

A common way to scope down a process gate for PRs, for example, is for changes that involve a database migration to receive more scrutiny, because they can be more devastating when things go wrong.

Use automation and alerting instead

One of the better ways to avoid process gates is to use alerting or automation instead of process gates.

So instead of reviewing PRs for their impact on infrastructure costs, alert when the infrastructure costs spike. Run automated QA tests to make sure functionality isn’t breaking, instead of having a human do it. It can be helpful to consider: “how could I alert on this instead of have an extra step for it”.

Use non-gated checks

Automation and alerting are two ways to make your checks non-gated. Non-gated checking is when the checks happen, but in a way that doesn’t block things from moving forward.

For example, a security team might do automated security checks in production, instead of adding a step before things go out to production. This is a non-gated check. If the security checks find a problem, you can quickly roll things back to a safe checkpoint. Or have a well defined way to quickly resolve the issue within a certain SLA.

This is basically pipelining, and it's why our computers are so fast.

The challenge with non-gated checks is that they work best when you have low cycle times in your engineering organization. A team that can notice a significant problem and push out a fix within a few hours can use non-gated checks much easier than a team that takes a few weeks to push out a change. So moving to this tends to go hand in hand with other efforts to reduce cycle time.

Do it in the same cycle

Another way to avoid process gates is to do it in at the same time as other work. So you can have a QA person that works side by side with engineering, instead of reviewing their work afterwards. You can have an SRE that is working like any other member of the team, but has the expertise on operational matters. Having them work side by side, they can often respond to requests quickly, or even be pairing on the same part of the code together. Embedding expertise so that the work is done side by side is often an effective way around process gates.

This type of working together at the same time is something that takes a while to get right. You’ll face resistance and it won’t feel natural at first. But it’s a competency worth building. Have your designers and QA and ops people working more closely with your engineering team. You may be surprised at the results!

You can also employ peer to peer options. So instead of having a centralized team reviewing PR requests, have their peers on the same team do so.

Learn more from failure

Finally, process gates are usually a response to failure. Something bad happens, so you add a process gate in response.

This can be the result of a culture where mistakes and quality problems are to avoided. Sometimes, I’ve found that it can be useful in these cases to try and embrace the failure, because the avoidance is part of the problem.

So when you encounter a problem, try to approach it with curiosity instead of blame. Why did this happen? Is it likely to happen again? How bad is it really?

Try to dig in and see if there are ways you can make this class of problem less prominent that don’t require process gates. Is it training for this individual? Is it better observability?

Sometimes the thing to be learned from a failure is that you have a problem that shouldn't be solved with process. For example, you might find that you don't have enough senior engineers on your teams.

Adding process gates is usually not the best solution to failure. But failure is often a good way to learn a lot about how your organization works, and with curiosity and imagination, you can often come up with ways it can improve your team.

Thank yous

Image by Ulrike Leone from Pixabay.

💩🛡️

Many new managers think their job is to “protect the team”. In fact, they see that as their primary function.

This is a mistake. I’ve interviewed hundreds of managers in my career. Usually, when people talk about protecting the team, it is a sign of inexperience. It is a common trap for new managers.

Why do new managers focus on being a “shit shield”?

If you’re a new manager, you may be stepping out of an individual contributor role. This makes you biased towards your old needs. And you are unaware of the new skills you may need to master.

You'll hear phrases like, “you can be a shit shield, or a shit funnel”. The response to that seems obvious: "be the shit shield". It sounds honorable. And you don't want to expose your team to dysfunction, right?

Most people find their first exposure to management politics to be painful. My first thought was, “oh my god, has this been going on all the time?” It’s easy to come to the conclusion that protecting your team from that mess is desirable. You may see that as a large part of your role.

Sometimes it is necessary

There is some value in shielding your team. In some organizations, it may even be necessary. Dysfunction is common! You’ll find it everywhere you go.

So why isn't positioning yourself as a shit shield a good idea?

Protection sets up an adversarial relationship

When you’re focused on protecting your team, you’re setting up the rest of the organization to be your enemy. You’re setting up an adversarial relationship when that may not be necessary.

This can prevent you from being effective. Sometimes, the solutions to your team’s problems will come from collaboration. Sometimes, the solution requires structural changes. People won’t see you as a person to engage with that type of work if they see you being defensive all the time.

This can leak into how your team behaves. The team can begin to see the world as "us versus them". I often see these teams look down on sales or support.

Adversarial relationships tend to propagate dysfunction. Look for better options.

You’re focusing on a local maximum, not a global maximum

An organization composed of self-protecting teams isn’t an effective organization. They are all focused on their own needs, but there isn’t an ability to fix problems that are larger than the team level.

For example, you may run into a problem where support is sending you too many tickets. Instead of focusing on your team’s needs, it’s preferable to look at the larger picture. Work together with the support person, and identify what is better for both of you. You might ask these questions:

• Why are there so many tickets?
• Are we creating features that aren’t supportable?
• Are the tickets created by support because they’re not trained on new features?
• Is our documentation poor?

This meeting will result in better outcomes. A “shielder” would take a different approach:

• Tell the support leader to send less tickets.
• Or tell the support leader you’re not going to work on many of those tickets.

This is inadequate, and results in more dysfunction.

You shouldn’t compromise on your team’s needs, but see it within a larger ecosystem. This raises your effectiveness as a leader, because you’re solving larger macro-problems, rather than just the issues within your team.

You need many management tools, not one

Protecting a team is one of many tools you need in your management toolkit.

For example, let’s say there is some conflict over the product strategy. You could try to shield the team from the messy conversations. Or you could take the approach of an interpreter: explain the situation to them so they understand what is going on.

Having the “interpreter” management tool makes you more flexible. This is an approach that improves empathy and context for your team. That context will help even if you’re not available, because the rest of the company will make more sense to the team.

You’ll find that using the “many tools to apply” as a manager is a way better mental model than the “shit shield” approach. You can be an interpreter, a system’s thinker, a collaborator, or a shield.

Managing “distributed humans”

Being a manager is working within a distributed system of humans. The more capable you become, the larger the portion of that system you’ll be able to affect. Acting as a shit shield is confining your vision to just one node of that system. That will limit your growth and potential as a manager.

So don’t be a shit shield, or a shit funnel. Instead, focus on interpreting. Focus on solving the problems around your team. Focus on delivery. And focus on making things better for your team, and the people around you.

Thank you

Image by Thomas Anderson from Pixabay

Today, I'd like to cover the weekly life of a project manager. When I'm managing a project, these are the things I do every week:

1. Identify the next milestone. Do you have a goal that is less than a month away? If not, make one up as soon as you can. Talk about the next milestone every meeting with the team.
2. Update your project plan. Schedule an hour or two every Friday to review and update your project plan.
3. Update your risk registry. During your project planning time, update your risk registry.
4. Send a weekly project update. After updating the project plan and risk registry, I send out an update that summarizes where things are at with all the projects I'm responsible for.

Putting these things together will often require meetings or conversations. But having a concrete idea of what you're delivering each week can make it more clear what to focus on.

Identify the next milestone

People work best when they're working towards a goal that is close and easy to wrap your mind around. engaged, more productive, and more effective when working on a milestone that is less than a month away.

This also cleaves complexity into a chunk you can manage. People can reason about about that much work. They work more effectively towards it.

Most weeks, you should already have a next milestone identified. Fantastic! However, you'll often be close to finishing that milestone, or things will shift in a project. Or the project won't be broken down enough to have a milestone that is soon enough.

Why a month, as opposed to something sooner or later? The short answer is that this is what I've seen both with my own experience, and with evidence from some limited data I've seen. You can read more about this in my post on milestones, not projects.

I recommend getting the product manager, designer, and people who have thought the most about the technical contours of the project in a room, and tell them you'd like to identify a milestones: "What would the best milestone be that is less than a month away? Let's come up with a few possibilities, and choose the one we like best."

Start with the constraint that you always need a milestone that is less than a month away. If you don't have one, or you're about to finish a milestone, make sure you have the next milestone identified. Refer to that milestones post to learn more about the art of breaking up projects. And also remember that this is a real skill that takes time and practice to develop.

Update your project plan

I book time every week to update my projects. I review the project plan, and make sure it reflects my current thinking. I ask myself if it still seems realistic. I think through the rest of the project, and think about potential sources of delay and risk.

Wishful thinking is your enemy. Look at everything from a skeptical perspective. For some people, this is natural, and for others, it requires practice.

You will often need information from others in order to update the project plan. I usually schedule a weekly meeting per team, or per project. At this meeting, I have the main leaders present, and walk them through the next couple of weeks in detail, showing them my current thinking with the project plan, and asking them to critique it. I then go through further out into the future, but with a little less fidelity the further out we get. Usually I invite someone representing product, technical leadership, and design.

Try to keep these meetings high level. You want them to not be a huge waste of time for the participants. I often find it best to join this activity into a meeting that has a larger purpose. For example, a local team's leadership meeting might cover this in 5-10 minutes every week. But the majority of our time we might discuss other matters, like the things we're concerned about or need to improve.

One anti-pattern for project managers is that you can get into a habit of pinging people for information all the time. This is annoying at best. Make it inexpensive for people to keep you informed. And also encourage people to surface what they're observing, so you don't have to update them. Switching communication to push, from pull, is preferable.

What should the project plan look like?

I make project plans that are week by week. Each week has a couple of bullet points. The bullet points are things we plan to demo that week. The demos should include technical demos, but the description of what we will deliver should be as customer focused as possible.

I also add time-based events that are relevant to the delivery of the project. For example, vacation time, on-call schedules, and offsites.

The goal for your project plan is to not be more specific that a couple of bullet points. You want a plan that is easy to change. The more specified your project is, the more expensive it is to alter it. A good project plan should be a tool for discussion and one that encourages change rather than discourages it. When I see lots of project artifacts, and things that require updating, I get skeptical. Gantt charts communicate effectively but are often a sign the underlying data is hard to change.

One thing to watch out for is many engineering teams work on a person per project. The more you can use the team as a unit of delivery for your projects, the better. This is a deeper topic, so I won't go into a lot of detail now, but in general, when I see project plans that specify what each person is doing, I view that as a sign that the project is unnecessarily brittle. And a team that is overly specializing, and not doing enough knowledge sharing. This can be appropriate for small companies, but otherwise, I tend to discourage it. If I see fractional allocation of "resources" in a project, I typically scream, take a lighter out of my bag, set my hair on fire, and run out of the room. The fewer projects you're managing, the better you'll do with them, so that's another argument for teams focusing on fewer projects.

People ask if you need a week by week project plan if you're on a team that has two week long sprints. Of course it is up to you, but I would do it week by week. Otherwise, you have very little visibility into whether things are on course or not. If I were doing it on a two-week long sprint cadence I would tell the team, this is a plan showing whether or not we could demo this by Friday every week.

You can read some more advice on week by week project plans.

What should your risk registry look like?

As you update your project plans, you should also ask yourself what could go wrong. Here are a couple of ways to ask yourself, or others, questions that will help you plan better:

• What is most likely to go wrong with this project?
• What projects have been similar to this in the past? What challenges did we have with them?
• What's the worst case thing that could happen on this project that seems like it could actually happen?
• If we had to bet this month's salary on this project going well, what are things we would be worrying about right now?

Risk management is a balancing act. You don't want to overinvest in many of your risks. But you do want to think through them. After you come up with new risks, list them as bullet points in your risk registry.

For each risk, you or your leadership group should determine if you want to take any action to mitigate that risk. Next to your risk registry, list your plan next to each. Or write down, "accept risk".

If your leadership group is functioning well, you can have rational discussions about the risks, and the cost to mitigate the risks. I suggest when you introduce a risk registry, you have a conversation about the costs of mitigation, and be sure to have conversations about not just how you'll mitigate risk, but whether you even want to.

Write a weekly project update

After updating my project plan and risk registry, I have a good idea of the current state of the project. So now I help the people around me by sharing that state, in as concise a format as possible.

The aim of a weekly project update is not to share everything about the project. It's also not to show how great your team is. When you approach project updates from either of those perspectives, your writing becomes excrutiating to read. Instead, you should focus on the needs of your reader, and give them just enough information that they can come to you if they need to, to have a longer conversation.

This tweet summarized a lot of the things I've learned over the years about writing concise, readable updates. It also features some wonderful examples, so I strongly recommend reading through it:

One thing I'd like to emphasize is that you should come across as a neutral party: objective, upfront about risk, exposing things without bias.

Your update should include a link to the source of the information: the project plan, risk registry, and a link to demo the current state of the project.

After you write your update, send it to your stakeholders. You can use a mailing list or a channel in Slack. You want to make it easy for new people to follow the information. You'll probably be surprised who finds it useful. You might find other parts of the company who find your updates useful.

These updates help the people around you to be effective. Your manager will be informed about the state of the project, so they can represent your work in meetings (you have no idea how helpful this is for them). Other teams that depend on you won't have to contact you to get updates. It provides a very nice communication scaffold. You may even get thank you notes!

Incidentally, these updates help demonstrate your ability to manage the project.

I find the project updates to be a good forcing function for doing the rest of the project management. By knowing that I have to send out my weekly update, it forces me to take the time to do everything else.

Ideally, the update should be just a tweet or two in length. You don't need a lot of detail -- people can talk with you if they have questions.

Thank you

I've had a long history with project management, even writing project management software in the past. But Bjorn Freeman-Benson really hammered into me the details of how to write a good update. He acted as an editor -- training me each week on how to do better.

Thank you to Gerd Altmann for the cover image.

When you're working in a group, you need to know how to coordinate. Coordination requires a shared understanding of how everyone will work together. Most human groups define Roles to clarify boundaries and expectations.

For example, in sports, these roles are called Positions. To play well, you need all the positions to understand how to work with each other. And you need everyone to focus on their position. It would be ridiculous in football (aka soccer) to have everyone try and be the goalkeeper.

People often fail to work well with each other, because they lack that shared understanding of the Roles each person plays.

Over the years, I've come up with a simple way to align people on roles. It's way simpler than alternatives like RACI charts (but we'll describe those too). You can use this exercise when it isn't clear who is responsible for what. You can also use it for teams to define boundaries of team ownership.

Getting set up for the ownership exercise

I'm going to assume we're using this exercise between an Engineering Manager and a Product Manager. But you can use it for any roles.

You'll first need a whiteboard. If you're not in person, you can do it with Google Docs, Miro, Notion, or Trello. Really most tools will work.

Start out by creating three columns on the whiteboard:

1. Write "Clearly Engineering Manager" as the first column, on the left.
2. Write "Clearly Product Manager" as the third column, on the right.
3. In between the two, write "Unclear"

Doing the ownership exercise

Next, you have the people involved write down stickies for all their responsibilities. One per sticky. Why stickies? You want to easily be able to move things between columns. If you're using a tool like Trello, you should be able to drag between columns, so that works as well.

You can do this two ways:

1. Have each person create a bunch of stickies, and then talk through each as you put them up.
2. Or, you can have people put them up as they create them. Then review to make sure everyone agrees they are where you think they should be.

I usually use the second approach, as it is a little more time-efficient. Basically anything you have ANY disagreement about or want to discuss should be moved to the middle column.

Here's a messy version of how it might look after adding some responsibilities. In this example, both the Engineering Manager and Product Manager think they should be doing project reporting. And they've both worked in places before where their role was responsible for defining user stories for the team.

Just doing this initial exercise has already been valuable. You've aligned on a lot about your roles, and you've made it explicit what conversations you still need to have.

If you did this on a whiteboard, take a picture. You'll need it later.

Discuss the unclear parts of your roles

Next, you want to go through each of the items in the unclear column.

You may not cover all of them within this first meeting. That's fine. Spend the rest of the time you have figuring out as many as you can. Here's what to do:

1. Figure out the tradeoffs. For each responsibility, ask yourselves: what's the best argument for the Engineering Manager being responsible for this? And what's the best argument for the Product Manager owning this?
2. Discuss the tradeoffs. Talk through any concerns or tradeoffs you see.
3. Make a decision, even a temporary one. If you can't come to agreement, flip a coin and agree to discuss how it's working after two weeks.

Get through as many Unclear items as you can, and schedule followup meetings to finish up anything remaining.

Document what you've learned

You've now done the work defining the roles. One of you now needs to write down what you've agreed to. Simply list the roles, and the areas of responsibility under each.

You can also categorize things, if you've had a discussion where you've come up with some larger themes. For example, you might say the Engineering Manager is responsible for People, Process, Projects. But it's always useful to include examples. The larger themes are optional.

This is also handy when teams go through mitosis

You can follow this exact same approach when a team gets large enough that it splits in half. (Splitting a team in half is a great way to form new teams).

The team will have a lot of existing things it owns: parts of the codebase, and interactions with other parts of the organization. Run these through the exercise, and you'll have a list of things that are clearly owned.

With teams, you'll often have some sticky items in the Unclear column. Some of them might require technical work or projects to untangle. For example, you might have a piece of code that both new teams will rely on.

Document what you agree upon, and agree on some short-term ways you'll move forward on the Unclear items. And make sure you have real plans in place to figure out what to do with the Unclear items.

Write roles and team descriptions down in a canonical location

Let's say you're a Product Manager, and you've just gone through this exercise with your Engineering Manager counterpart.

A question you should ask yourself is, "does anything like this exist for everyone else?". If you don't have role definition written up, then the artifact you came up with might be generally useful.

If that's the case, share it or put it in a place where it can act as the canonical definition for your roles.

For teams, you also want the responsibilities to be put in their canonical location. For code, you might put it in the CODEOWNERS file, so it's clearly marked in the codebase. For support responsibilities, you should outline how support should interact with each team. Make sure the clarity you've produced is put into a place where it can be used by others, if that makes sense.

People aren't fungible, so roles need some flexibility

Roles shouldn't be rigid. Every person is different, and every pair of people will have different strengths, backgrounds, and ways of working together.

It should be a valid option to identify that one role owns an area of responsibility, but delegates that responsibility to the other role. For example, Engineering can be responsible for projects, but be totally buried in other work. So you can agree to have the Product Manager run the projects for a period of time.

If you have standard role definitions, there may be times when it isn't a good fit for your situation. In those cases, you might flex the responsibilities between you and someone else if you think it will product better outcomes. I find having the roles written up allows you to have more explicit conversations about these things. "Hey, it looks like you're responsible for project reporting. How would you feel about me doing the reporting, or helping you with it?"

Some people like RACI charts

You will find a lot of managers that love RACI charts for this same purpose. I find them inscrutable -- they take a lot of attention (for me) to understand.

What is useful about RACI charts is that they define responsibilities in a more nuanced way. So, for example, they make it clear who needs to be informed about certain types of changes.

I generally find them to be more effective when you're dealing with a lot of parties. Stakeholders and multiple people, all doing similar work. The ownership exercise I share here is simpler, faster, and easier, for when you have less parties involved.

If you are in favor of RACI charts, please be aware that not everyone understands them intuitively. It can be useful to walk through them.

Let me know what you think!

I find this exercise to be a quick way to get clarity and alignment around roles. Try it and let me know what you think!

Thank you

Image courtesy of UK Black Tech. Licensed under the Creative Commons 2.0 license. Image was cropped.

Today we're going to do a whirlwind tour of compensation. Hopefully you'll learn a bit along the way. And I'll share an exercise for managers, called a compensation review.

What do you need to know about salary?

• The largest expense for engineering organizations is usually salary.
• Salary is mostly determined by supply and demand. Engineers are fortunate to be an industry with high demand. Companies have to compete based on salary and benefits in order to hire good engineers.

Structured pay and "pay equity"

Some companies use structured pay. Structured pay is when there is a "system" for determining pay:

• Pay can be based on the employee level, tenure, specialty, or other factors.
• Most companies have structured pay -- some do not.

Structured pay is a spectrum. Most companies end up having some sort of system for pay.

Companies that emphasize fairness and equity also will often implement a more extreme version of structured pay: "pay equity":

• Pay equity truly uses a formula to determine salary.
• The formula can be based on years experience, or engineering level, or other factors. But it's determined by a formula.
• Pay equity is less biased, but also less flexible.
• Companies that implement pay equity tend to attract underrepresented employees better. Why? It is a signal that the company tries to pay people in a less biased fashion.

As a manager, you need to know how pay works.

Common pay methods (when you don't have pay equity)

If the company hasn't implemented pay equity, then you have to figure out how it works. Here are a couple of variants I've seen:

• Manager discretion. Managers have discretion to set pay, based on a budget they can use.
• Manager proposed. Managers propose pay increases, which are approved by their managers or some group.
• Directors or VPs set pay, with input from managers. Using a budget.
• Cost of replacement. Compensation increases can be justified based on proving you could make more by going elsewhere. This perversely incentivizes people interviewing outside the company. Netflix is famous for this approach.

Your homework is to figure out how your company's system works. Is it structured? Is there a formula for how salaries are computed? Do you have salary ranges?

Geo-based compensation

Companies have to make a philosophical choice in how they pay. They can choose:

• Geo-blind. Pay the same amount without regard to location. Note this does NOT mean everyone gets the same salary, because taxes and expenses will vary based on location. Some people could end up getting twice the take-home pay!

• Geo-balanced. Pay so that everyone receives the same amount, regardless of where they live. This means the company will pay very different amounts for each employee. Gergely Orosz documents how the cost can vary by the employer by 13-42%!

• Geo-based. The pay is adjusted based on what compensation people are getting in that location. People in San Francisco get paid more than people in Kansas or Poland.

Each of these has tradeoffs:

• Geo-blind: you're paying high for regions where pay is, on average, lower. This makes you more competitive in those locations. But unless you "over-pay" (by a lot) in those regions, you're less able to hire people in expensive locations. In practice, this means you're not hiring out of tech hubs like San Francisco.
• Geo-balanced: you're more competitive in countries with high taxes. And less competitive in countries with low taxes. But also you pay more for the employees you're most competitive with. That can be a disadvantage. I believe geo-balanced is a rare choice for companies -- let me know if you are at a company that does this.
• Geo-based: you're able to hire in expensive markets, where there is a lot of talent and experience to draw from. But you're also able to save money by hiring outside of those locations at a lower rate. This can incentivize hiring in less expensive locations. But you face more competition in those location from companies that are Geo-blind. Often the best candidates in those regions will look for Geo-blind positions, because their pay can be so much higher. You also have to figure out your approach to having people move between locations. Do you change their pay?

People tend to be pretty ideological about which of these is best. But the important thing is you need to understand how it works at your company.

You may not be able to hire everywhere, because taxes

Most companies will have a list of locations you can hire from. The reason for this is that companies have to do paperwork and taxes for every location they have employees. And sometimes there can be additional legal ramifications for having a "presence" in a location.

There are companies that help minimize the overhead of this. But it's still an issue, so you need to be aware of the locations of your team members.

Because of these factors, an employee moving to a new location can be a "problem" you have to work through with your People Ops / HR team. And you may not be able to hire people everywhere you think you can. So consult with your People Ops team when hiring.

Generally, companies will be willing to incur the overhead for existing employees, but there are times companies will cut ties with an employee rather than deal with additional legal requirements.

Note that the differences between locales is very different between regions within a country, and between countries. The burden of dealing with international relocation, visas, and taxes can be so large that many companies will not bother to address them.

Budgets and pay

Finance usually provides a budget to each department. Depending on the size of your company, you may or may not have much interaction with the budget. But even if you don't know it's there, the budget is a hidden force that drives a lot of decisions.

Companies generally have a certain amount of money they can spend on hiring employees, promotions, and cost of living adjustments. Usually that pool of money is determined by Finance.

That pool of money for employee salaries can be divided into buckets ("salary", "promotions", "cost of living adjustments"). But it may not be.

Generally budgets are zero sum. If you give more raises, you hire less. If you give larger cost of living adjustments, you promote less.

You'll see a lot of people that talk about how things "should be". But the reality is that that is how it works in most companies. You can sometimes get exceptions, but the money has to come from somewhere.

One thing to be aware of is that saving money somewhere can often justify spending it elsewhere. As a frontline manager, you probably won't be in a position to be making a lot of these decisions, but understanding how it works can help you if you want to advocate for change.

How companies determine compensation

Most companies have engineering "ladders". They describe the difference between a Software Engineer and a Senior Software Engineer. And they outline the criteria for advancement between these levels.

Companies do this for a number of reasons. What you're probably familiar with is their use to help guide career discussions. And to attempt to create an objective standard you can make promotion decisions around.

What you may not realize is that companies use these ladders for another purpose. They use them to determine salary. You may think, "of course they do". But there are companies out there that provide salary information. By creating a ladder, and defining what is in that ladder, your HR people can match up your company's levels to what is done for the whole industry.

This is complicated, because engineering titles are not equivalent across companies. A senior engineer at Company X does not equal senior engineer at Company Y. So these companies have detailed ways you can map your titles to theirs.

Many companies also use salary tools, which attempt to compute a salary based on a job description and a bunch of keywords. These tools aren't perfect, but they are an attempt to use data to make sure compensation is fair.

Larger companies will have a dedicated compensation person. Their job is to determine compensation for all the roles in the company.

Pay percentiles

Companies generally target a "percentile" when choosing what salary to pay.

So you might hear a People Ops person say something like, "we target the eightieth percentile for salaries".

What this means is that in each market you're competing in, you'll use data to determine the salary ranges. You'll use the distribution of salaries to determine what to set the salaries you offer.

For companies that are not geo-based, this means you're competing in a global market. Many geo-blind and geo-balanced companies don't attempt to compete in expensive markets like San Francisco, because that means they're paying a very high rate for employees everywhere.

Pay tradeoffs

Compensation is a set of tradeoffs that companies make. They have to juggle business needs, market conditions, the employee experience, and growth and performance management.

For example, a company can offer a higher salary, but not promote its employees as quickly. Or it can choose to give more money for promotions, but not be able to hire as many people.

And, of course, each company chooses what budget to make available for hiring and promotions.

These factors are invisible to most people. But they explain why you see things like a low promotion budget, but rapid hiring. They're optimizing to bring more people in. But trading that off against future employee turnover.

Equity

The definitive guide to equity compensation is the Holloway Guide to Equity Compensation. I recommend you read it carefully.

You'll need to have conversations with your team about equity, and understanding it is beneficial in its own right.

A couple of things to remember:

• For startups, the value of a share of stock changes over time (hopefully it rises). When the company is younger, equity will be "inexpensive". As the company succeeds, the value of the company will increase, so each share will have a higher value.
• It's inexpensive early on, because it's risky.
• Because of this, earlier stage employees get more equity. This reflects the additional risk they take on joining a company. Many companies don't make it, so the earlier you join, the higher the risk. Also the higher the reward.
• The typical pattern for startups is for early employees to have lower salaries, but lots of equity. The discounted salary is somewhat offset by the large equity grant. As the companies grows, they usually correct these discounts as options vest.
• To make it more confusing, companies also issue different types of equity. ISOs, NSOs, RSAs, and RSUs are all different.
• So comparing equity becomes complicated quickly.

Why do a compensation review?

So, now let's talk about the compensation review exercise. Here are the benefits:

• It will help you understand and support your team better.
• You'll probably find some errors, and correcting those errors will earn you goodwill.
• It forces you to learn about compensation and equity, which will be good for your team, and for yourself.

Are you even able to do a review?

But before we start, let's make sure you can do such a review.

• Do you have access to your team's salary? Can you just look it up, or do you have to ask for it? In most companies, you'll have access to look it up. But sometimes you can get it by request. In some companies it's not available to you -- if so, you won't be able to do this exercise.
• Do you have access to your team's equity? In some companies, even if you have access to salary, you may not have access to equity. And if you do, you may not have access to full information about that equity. That's fine, but keep in mind equity is complicated.

Doing the compensation review

Doing the review is actually pretty simple. I have a template you can copy and adopt to your purposes. Fill it out for every member of your team. Highlight in red the things that stick out as most problematic.

Tips when doing a compensation review

1. Look for salaries that seem lower than they should be. Compare to people at the same level, one level below, and one level above.
2. If your company does geo-adjustment, you'll need to figure out how it works, and factor that in. Usually it is a multiplier on the salary. 100K * (geo-adjustment of 90%) = 90K. Companies will usually have a table of location, or use a tool that produces the geo-adjustment. I like to produce a "geo-reversed salary", but doing so will require you to know how the geo adjustment works.
3. If you're not able to find how geo-adjustment works, you can estimate using cost of living calculators to get a ROUGH idea. This is highly imperfect, because cost of living is not how these geo adjustments are determined. They are determined by what supply and demand.
4. In the notes section, fill out anything interesting. For example, I've worked at a couple of startups that gave early employees the option to tilt their compensation towards equity or salary.

Look for edge case scenarios

• If they were rehired into their position, would they get a raise?
• If they were rehired into their position today, would they have more unvested equity?
• Do a pass for fairness. Is everyone being paid fairly for their level?
• Are there any salaries that seem higher than they should be? If so, have you seen any patterns behind this, like the recruiters or hiring managers? Or when they were hired?

What do I do if I need to determine salaries?

If you're in a position where you need to determine salaries for engineering, here is my advice:

1. First, read my post on implementing pay equity. It lays out how to gradually role out a pay scale, along with promotion criteria. And it links to my favorite article on how to compute equity and equity refreshes.
2. To get an idea of salaries, talk with PeopleOps. They'll probably have access to a tool like Radford or OptionImpact/Pave.
3. You can sometimes get salary information from your investors, if you're a startup.
4. You can look up salary info in levels.fyi. Keep in mind they list total compensation, and break out salary.

What do you do now?

Once you've done the compensation review, you'll probably notice a few things that seem off. Your next job is to dig into that. Are there legitimate reasons for those anomolies? Or are they something that needs to be fixed? Make these changes as quickly as possible.

Thank you

I'd like to thank Zen Mak (founder and CEO of RallyWorks) for her feedback on this post. And I'd also like to thank Bailey Douglass (founder and CEO of Second Principles). Bailey is working with companies for free to come up with policies to support medical travel for abortion, transition, and other purposes. Bailey gave a ton of feedback on this post, and helped improve it significantly. Darin Swanson helped improve the edge cases to look out for. Gergely Orosz has tweeted and written a lot of useful content on this topic, and is a good person to follow for content on salaries and other topics.

Image by Nattanan Kanchanaprat from Pixabay

Today, we'll talk about managing change, and how to reduce obstacles as you do so. We'll also talk about how to ensure your changes are effective.

As managers, our job is to make our part of the organization better. This requires imposing change. Today we'll discuss how to make changes: how to overcome resistance, skepticism, change fatigue, and people who push back against proposals.

First, always be listening

First of all, you need to be steeped in the problems of your team. You need to understand them better than anybody. This requires a lot of listening and questioning. Use your 1-1s to ask people what's going on for them:

• What problems do they see?
• What's frustrating for them?
• What do they love about their team?
• How do they interact with other teammates, or other teams?

The more you understand the team's problems, the more the team will trust that you're there to improve things for them. Your proposals and changes will go over better if there is a track record of you listening to them.

Keep a management backlog

This isn't obligatory, but one thing I've found helpful is to maintain a management backlog. I usually keep a prioritized list of Problems I'm seeing. Each problem is something I think will require management work to improve. While I might not tackle everything in the backlog, I do shift the priorities as I hear things from people.

One of the reasons a backlog can be helpful is you can use it to keep notes of patterns you're seeing. That way, when you do implement a change, you have a lot more history and context into the problem. And it can give you some examples to use to make a case for why the change is necessary. I call these "observations".

I keep this backlog separate from my management tasks, and treat the problem I'm working on like my current project.

How to make a change: problem, diagnosis, remedy

Once you've selected something from the top of your backlog, here's how you might go about making the change.

1. You start with a problem. The problem is an observation of the issue that's happening. It may have a swirl of related problems associated with it. For example, the problem might be that "the team velocity is slowing down". You're getting complaints about it, and maybe the team is telling you about it. This is the issue you had in your backlog. I sometimes will write it down for clarity. What are people meaning by "velocity"? Is it that the team just seems to not be delivering as much as it used to?
2. Assess how large of an issue this is. If it's a problem that you understand reasonably well, and the solution won't face a lot of resistance, you can just do some small improvements as experiments. In that case, you can reduce the amount of time you spend on the rest of these steps. Including the socializing portion.
3. Next, your goal is to come up with a diagnosis. A diagnosis is the issue that is at the heart of the problem. This is the step most managers skip, and it's the most important step. You dig in and research it so you understand it better than anyone. You must do this as rapidly as possible. There may be 10 problems in the mix -- the diagnosis identifies where to focus. I usually do this by stacking up intensive conversations with the people involved: all within a couple of days. But I might also look at any tools, or any data that can help me understand the situation. If we're looking at the team velocity, I would stack up 1-1s with the team and ask people individually. It could be a million things that are slowing down the team, or it could be multiple things. When you understand it well, you have a diagnosis. A clear diagnosis is important, because usually the remedy is clear with a good diagnosis. Also note it is fine if the diagnosis is that there are lots of small problems.
4. Finally, you come up with a proposed remedy. This should be something that will make the situation better. It doesn't have to completely solve the problem, but it should make things a lot better.

I like to write it down to clarify my thinking. The sections are Problem, Diagnosis, and Proposed Remedy.

Socialize your plan

After you're done all this work, you may think you have the best plan in the world. You probably understand the tradeoffs of your solution better than anyone. But people resist smart plans all the time.

The best way to get people to get on board with a plan is to give them a chance to interact with it. Instead of laying it on them, give them a chance to provide feedback. This can often result in them improving it as well!

So the next step is to shop your plan around in a gradually widening set of people. It's important to really ask each person to try and improve the plan. Don't just share it with people -- ask people to critique it. Ask them to make it better.

You can do this in person, or via tools like Google Docs. My favorite approach is to schedule time where people can read and comment on the doc quietly together. You can have a discussion at the end of the meeting if you like. But you can do this in a discussion, or asynchronously.

As you get feedback from each person, incorporate what seems valuable, so the plan improves. Iterate quickly, and thank people who improve the plan.

Like magic, everyone who participates in the crafting of the plan will be more inclined to support it. And the plan will improve from all of the additional feedback. As it gets to a wider and wider audience, it will go from being a good plan, to a great plan, with widespread support.

The way you broaden the plan is important. If you take too long to broaden the circle, you'll get rumors and people will feel left out. If you don't include enough people, you won't get the support for the plan, and the feedback you need. Usually I like to show it to another person, then show it to a leadership group, and then the team.

Communicate the change

You've circulated the plan with leadership, and key people. You may have talked with a few team members about the changes. It's important to act quickly so that things don't start being spread by rumor, and the people affected have a chance to weigh in.

The next step is to communicate the plan to the whole team. At this point, you have already received a lot of feedback, so you're probably able to anticipate objections or concerns.

The most important thing when communicating the plan is to say it is the plan, and that you'd appreciate any feedback on it if people have ways to improve on the plan.

This may seem like a dangerous thing to do. But people don't like to have things imposed on them. And by following the previous steps, you will be better prepared to respond to any feedback. And they might improve the plan! Be open to changes.

When you tell people about a change, different people need to hear different things to understand a plan:

1. Goals. What are the goals? What do we want to happen?
2. Process. What is the process we'll follow?
3. Constraints. What are the constraints and requirements?
4. Roles. Who will be doing what? What will everyone be responsible for?

You need to communicate all of these things when you communicate a change.

Reducing the cost and fear around changes

I see managers get into trouble rolling out changes. Usually this happens because people are nervous about the changes. So here are a few things you can do to make the changes seem less threatening.

First of all, if you communicate the change as an experiment, that's often less threatening. Use language like, "I'm seeing this problem, and I'd like to propose we try this for three weeks. I suspect the first few weeks will feel awkward and it may even be worse for a while. But I'd like to hear your experience, and I'll be evaluating whether this is a good change according to these criteria. [insert criteria]." Then after a couple of weeks, you can share what you've observed, get feedback from the team, and learn from the experience. A lot of objections will fade when people realize that you're going to be looking for evidence it isn't going well, and that you will adapt.

A second element is to communicate whether the change is easily reversible. Some changes are one way doors. Others are two way doors, easy to reverse. Don't fret over two way doors. But be sure to communicate what type of change you're implementing.

Finally, you don't need consensus to move forward. Your role on the team (if you're responsible for team process) is to adapt and continually improve the way the team works together. You're a specialist focused on how people work together. Just like the team might have someone that is a specialist in reliability, or the frontend, or the backend, your role is team improvement. It's okay for you to say, "I've heard all your feedback, and we're going to move ahead with this plan. I've heard some concerns from a few of you, and I appreciate hearing them. But I believe we're making the right tradeoffs right now. I ask that you all try and give this a shot at being successful, and in return I'll be on the lookout for signs we should reverse course."

Of course, you have to be careful when you move ahead with substantial objections from the team. Ultimately, you have the power to do it, but you may not want to use it. What I try to look for is whether people are expressing concerns, or whether they are trying to veto the idea. If they're really trying to kill the idea, it's often a sign you have more work to do before rolling out the change. You either need to be listening more, or there is some relationship work to do.

How much change can a team absorb?

As a manager, you should assess your team's ability to absorb change.

Usually, product development occurs within a dynamic and shifting environment. This means that ideally, your team should be continually adapting and evolving.

Don't underestimate the power of lots of small, incremental improvements. I believe the best performing teams as the result of the compounding interest you accrue from these types of improvements.

However, change can cause stress on your team, and teams that are already under stress can reject change. Ironically, the more under stress a team is, the more change may be necessary.

When you're in such a quandry, look for disproportionately valuable changes. Things where a small change can result in big improvements. Ideally, the result of a change is more bandwidth on the team to absorb future changes.

Watch your iteration speed

I've seen many managers get into a trap where they implement changes too slowly. What matters is how long it takes from when you start working on a problem to when you're able to turn your attention to the next problem. (Also, it matters when the impact of the change occurs).

When you embark on a change, it's often better to solve a problem in an incomplete way if it takes less time. As long as the situation improves, that means you will have gone through a complete cycle of improvement. Iteration speed is more important than perfection.

There is a trap on the other side of this, however. Making changes without good information is destructive. The key is to accelerate how quickly you can gain context, and be confident you're making a good improvement.

You can use the people around you to help make sure you're making good decisions. Ask them what problems they see with your suggestions. Ask them to improve your plans. This only works with trust, so when trust is lacking, you may need to slow down.

Thank you

I've learned about this topic from a lot of people. Alex Kroman taught me the most about having other people improve your thinking. Jim Shore, more than anyone, showed me how participation in a plan gets people on board with it. Robert Goldmann, my executive coach for many years, taught me the many things you have to pay attention to when communicating changes to people. Marco Rogers showed me the value of deliberation, that going quickly isn't always the best course of action, and that teams have a capacity for absorbing change. Rebecca Vetter provided valuable feedback on early versions of this doc.

Image by 0fjd125gk87 from Pixabay

Some people naturally shine in meetings. Others have to work at it. I count myself in the latter category. I’d like to share some tips that have helped me to improve my executive presence.

Ask a question, and then wait

I once got feedback that I didn’t wait for people to answer my questions. Instead, I would ask a question, and then another question. For example, I might say, “what do you think about how that last meeting went? I noticed you were a little upset. Have you had issues with Pete’s communication style?”

Offering a barrage of questions and comments is more confusing than helpful. Instead, ask a question, and wait. If it’s excruciating, count to yourself.

One of the biggest enemies of executive presence is anxiety. Anxiety causes you to want to fill in gaps in the conversation. Practice asking questions and waiting.

What you will find is that this helps improve your executive presence. Asking questions is a very important leadership skill. Asking and waiting is your friend.

Use powerful language

Many leaders speak gobbledegook. You can do better. A few tips to improve your delivery:

Use active voice.

Instead of “it was decided”, say “I decided”.

Passive voice sounds like passing the buck. It is weak. When you use active voice, you sound like you’re taking charge. Use active voice!

Use simple words and short sentences

Try to say things in the simplest way possible. Ahem, let me try that again. Speak concisely!

This is typical management speak: “We’re implementing an operational improvement plan which should result in higher velocity”.

Instead, say, “Our plan should speed up engineering”

Avoid weak adverbs

Many people use adverbs to avoid responsibility. For example: “Likely this project will have problems”.

Remove the adverbs and add precision. For example: “This project will have problems, due to these reasons…”. Or, you might say, “This project resembles projects that have gone off the rails”.

Practice with your writing

The way to be a better writer isn’t just to write more frequently. It’s also to edit better. You can practice your executive presence by editing your writing. This forces you to think about how you communicate. As you tighten your written communication, it will help your spoken communication.

My favorite way to revise my writing is to copy and paste my writing into the free Hemingway Editor. It color codes your writing to show where your language is problematic.

Notice your intonation?

Some regional accents make statements sound like questions. I have such a dialect. It makes me sound uncertain, because I sound like I’m asking questions when I’m not.

I didn’t notice this until an executive coach pointed it out to me. I use a rising tone at the end of my sentences.

To combat this, I’ve practiced making my statements sound like statements.

Instead of “I think this is a good plan?”

I try and say, “I think this is a good plan.”

Be definitive, but make space for disagreement

I like to create space for people to bring up concerns. This makes me bias towards language that sounds hesitant. But you can do that without sounding weak.

For example, “I think this is a good plan” could be better phrased as “this is a good plan”. The “I think” is implied.

If you want to make space for disagreement, ask for it: “This plan looks great. But I’d like to consider all sides. What do you all think is the weakest part of this plan?”

Record and critique yourself

You can take advantage of virtual meetings by recording and watching them later.

I use Fathom to record my meetings. You can use a similar tool to record your meetings, and then watch and critique yourself. Look for times you weaken your own position.

Copy the people around you

I find it easier to copy other people than critique myself. I take an inventory of the skills of the people I work with. Who is good at what? Then you can observe how they do what they’re good at.

What has surprised me is that I could copy many of their skills. For example, I once worked with someone who was devastating in his strategic thinking. I watched him and learned from him. A few years later, I realized he wasn’t in a different world than me any more. He was still much better at it than me, but I had caught up.

You can use this same approach to improve your executive presence skills. Watch your peers, and copy from them.

Ask others to help you improve

You can ask a management peer, or your manager, to help you improve your executive presence skills. When I was working on this, I would ask my peers for feedback. I also asked my manager to observe me during meetings and give me feedback and suggestions.

They will notice things you won’t.

Don’t do all your thinking in meetings

One thing a coach pointed out for me is that I like to use meetings to get people to think through things with me. In executive settings, this can sometimes backfire. Executives look for certainty and don’t always want to dive into the details with you.

I now assess each meeting I’m in separately. They all have different expectations. A large executive meeting should bias you to concise and action-focused statements. An exploratory meeting might welcome investigation or debate. Your executive presence should vary depending on the circumstances.

Don’t worry about umms and ahhs.

Focusing on your failure modes while presenting in public may make it worse. A trick I learned in an executive presence class is to focus on connecting with your audience. They taught me to look around the audience, and look at a different person as I make each statement. To my surprise, that eliminated all my verbal tics!

Try it!

Schedule time to prep for meetings

If you are the expert of what you’re talking about, you’ll have more to say, and will say it better. Schedule time to anticipate how a meeting will go. And think through what you should do to prepare ahead of time. Do your homework.

You don’t always have to think on your feet

If you’re the type of person that takes time to integrate something, own it. Say “Hmm, that’s really interesting. I’m going to take a day to think that over, and I’ll get back to you Wednesday with next steps”.

I’m not the best at thinking on my feet, and I used to get stuck when someone was asking me something I wasn’t prepared for. Nowadays, I try to respond with a quick action item about getting back to them.

Gender and racial stuff

I wrote this advice from the point of view of a white male in engineering organizations.

Women and people of color will face double standards. For example, black men in America often will be criticized for being assertive, where a white man making exactly the same comments in the same way will be seen as strong. A black man being angry is “scary”, whereas a white man being angry is “passionate”. Women face similar double-standards.

This can often lead to “you’re damned if you do and damned if you don’t” situations. If you’re struggling with this, reach out to people who seem to be navigating it well and ask their advice. It sucks that you have to deal with this on top of the double-standard itself.

I’d love to hear how people have navigated this. If you have any resources you think will be helpful, please let me know so I can share them.

What else have you found?

Let me know your experiences with these tips. And if you have any other ideas to share, please do!

Thank you

Alex Kroman, Bjorn Freeman-Benson, and Robert Goldmann taught me several of these tips. I learned a lot from an executive presence / public speaking class by SNP Communications – it was one of the best management training courses I’ve taken, though it was a while ago.

Image by Shiv Mirthyu from Pixabay

Today I’m going to share my favorite way to organize specialist teams that want other teams to do work for them. I recommend this coordination model for security, reliability, and design teams. Non-specialist teams can also use it for some projects!

This approach gives your team leverage.

What is an objective expert team?

• Your team has some sort of expertise. For example, they might be site reliability engineers (SREs), security engineers, or designers.
• You want people in the organization to do work you identify as important.
• You produce reports that help other people decide to do that work.

Example: a security team that partnered with engineering

The best example I’ve seen was the security team at New Relic.

• The security team recorded vulnerabilities in Jira. They were assigned to teams. The security engineer categorized the ticket by severity. More severe vulnerabilities needed to be done faster than less severe vulnerabilities (according to a service level agreement).
• The security team emailed all managers, directors, and VPs a security report each week.
• The report included a list of all the teams in engineering. Next to each team was a count of overdue security vulnerabilities. You could click on the count to see the vulnerabilities.

Security team leaders told everyone they were responsible for acting on this information. They would escalate urgent issues. Everything else depended on engineering and product prioritization.

Their weekly report became an easy way for leaders to measure themselves. It also became an easy way to measure each other. Local teams had the autonomy to prioritize security concerns against their own roadmap. And it gave leaders a way to see how their part of the organization was doing.

This social engineering resulted in leaders taking ownership for security issues.

Handling security vulnerabilities as a front-line manager

When I was a front-line manager, I would get the report each week. I would make sure my team didn’t have any overdue vulnerabilities. If we did, that meant something had broken down in our team process. Usually it meant we hadn’t prioritized the work to complete it ahead of the security team SLA. I would dig into the issue and see what happened.

Notice that the security team was a passive participant at this point. They did all the analysis to determine the severity of incidents. Their SLAs communicated their expectations. It was clear what a reasonable time was to address vulnerabilities. But they mostly left managing prioritization to the teams.

Handling security vulnerabilities as a director or VP

When I became a manager of managers, I interacted with the report in a new way. I was now responsible for my organization's security vulnerabilities. The report listed the teams in my organization. And it showed how each of them were doing.

When one of my teams had a lot of overdue vulnerabilities, I would talk with the engineering manager. I wanted to make sure they had made a conscious decision about prioritization. If they hadn’t made a decision about it, it was often a sign that their team needed help. They might be under water, or they might have poor process. I would talk with the manager and make sure we put together a plan to improve things.

Thus the security team not only outsourced prioritization, they also outsourced accountability. They created some pressure and visibility around vulnerabilities. But they didn’t have to muck around in every team to get the work done. It was a very nice contract between teams, and worked quite well.

Site reliability engineering teams can use the objective expert model

New Relic went through tremendous growth during my time there. We confronted scaling and reliability every day. Monitoring companies receive insane levels of data from their customers. So the demands on our systems were high.

One of the best things we did was to create a repository for incident information. Nowadays, you can buy incident management tools from Blameless, Jeli, Fire Hydrant, or Incident.io. But at the time, we didn’t have such options, so we built a tool we called Upboard. (Blameless was built with the lessons from Upboard).

Initially, we used it to store all of the incidents we had, and the followup items. But over time, we developed a team by team metric for reliability.

Although there was a lot of discussion about the metrics we used, they helped people make the case for doing reliability work. And they helped the entire organization focus more effectively on pain points. It also helped us realize what parts of our efforts were making a difference.

Upboard also had a weekly report. It showed (on a team by team, and org by org basis), how teams were doing with reliability. The report transferred the responsibility for reliability to the teams that could do something about it. The reliability team then became the team that helped teams improve. They acted as Consultants.

This made for a healthy relationship. I always appreciated my interactions with the reliability part of the org. They were experts that had wonderful advice on how to make things better for my teams. Their reports showed me the experience I was offering customers.

Design teams can also use the objective expert model

Companies often neglect usability. It can be frustrating to see customers offer experiences that people can’t navigate. Design is often iterative, but companies fail to iterate.

Ideally, your designers are working with local teams to prioritize usability work. When that fails, consider the Objective Expert Model. It gives more leverage for design teams.

Here’s how it can work:

1. I assume your designers embed on product engineering teams. They identify issues that need attention.
2. The designers would grade the issues by some sort of severity. This would correspond to an SLA for addressing it.
3. You produce a weekly report: “Top product usability issues”.
4. Product managers then use the report as input into their prioritization decisions.

See later in this post for how to put this in place.

Non-specialist teams can use this model for projects

Platform teams can steal this approach for non-specialist teams. If you have something you need others to adopt, you can make it visible who has done the work to adopt it.

For example, let’s say your team is responsible for permissions, and you have created a way to do Role Based Access Controls (RBAC). Your work has to be integrated into the code of every product team in the organization.

There are many ways to solve that problem:

• You can do the work across all the engineering teams. (I’m a fan of this approach)
• You can create documentation on what needs to be done to integrate, and work through product management to get the work prioritized on each team.
• You can create documentation on what needs to be done to integrate, and create tickets for each team to request them to do the work (I don’t recommend this approach)
• If you have the right prioritization, you can have a Program Manager coordinate all the parties involved.

Another approach you can take is similar to the approach we described for specialist teams:

1. Make sure the prioritization is agreed upon. (Perhaps a deadline or stack ranking of priorities).
2. Tell every product manager it is up to them to decide, and ask them to surface risks of not achieving it on time.
3. Automate a report that shows progress by team.
4. Share that report.

What’s nice about this approach is you make it easy for the product managers to do their own prioritization. They might have other urgent work scheduled. You don’t have to know about their local details. You then focus your team on making it as easy as possible for every team to take part.

This again is a way to outsource the prioritization to the teams involved.

When to use the objective expert model

The Object Expert Model is one of the most scalable coordination models. One team can produce a report that affects every other team in engineering. The main consideration is the cost required to produce the report. Coordination costs are low once you have set it up. The process and SLA combine to make it lightweight.

If your team wants other teams to prioritize work from you, use this model.

You can take this too far. It isn’t a model that every team should be using. Think about what it would look like if every team tried this model. Your organization would generate too many prioritization artifacts. It would be unwieldy. So if you see more than a couple teams using this model, think twice before adding another one.

When using this model

The general approach is:

1. Make this part of a larger partnership effort, and consider the human incentives
2. Get buy-in and establish service level agreements or standards.
3. Send out a weekly report to managers, directors, and VPs. Ideally automated.
4. Be patient and look for opportunities for reinforcement.

Make this part of a larger partnership effort, and consider the human incentives

You won’t be effective if you do this as a mechanical change. Rebecca Campbell: “Security [at New Relic] wasn't successful because they emailed me a report. They were successful with me and my team because they met with me when I first started, explained how things worked, and checked in with me regularly. They were happy when I asked questions or reached out to them and delightful to work with. They saw themselves as enablers rather than blockers.”

She’s right. The security team used the Objective Expert Model. But they weren't just sending a report. They were doing many things to make their team approachable. And they aimed to make it easy to do the right thing with security.

Think about how your team interacts with the rest of the organization. How would other teams welcome your report? You are successful if people view your report as helpful. I looked forward to the reliability and security team reports. They helped me create a better product.

I have an example of a project that got it right. The Role Based Access Controls (RBAC) projects was initially a "disaster project". Everything seemed to be going wrong. But the team turned that around, mostly because of how they approached other teams. It became a known as a model project.

Rebecca Campbell said: “The thing that was most successful about RBAC was the roadshow we did prior to asking teams to do anything. We met with every team, had their PM lead a portion of the meeting (very important to demonstrate priority and buy-in), shared sample code, and told engineers what was needed and where to get help.” Pay attention to what will make people think your team is helpful.

You’ll likely need to do some work with your team to prepare them for how to interact with the rest of the organization. I interacted with several security team members at New Relic. They all told me the same thing. If I asked, "do I have to work on this vulnerability", they all said, "this is our assessment of the level of risk, and when it should be worked on. But it's up to you to prioritize it". They spoke with one voice. This requires a lot of work within your team.

Get buy-in and establish service level agreements or standards

After you've thought about the human side, you'll need to get agreement that the work is important. You then can codify this as a set of standards of a Service Level Agreement (SLA).

Start out by working with leadership. Socialize the idea so that the SLA seem like a natural next step. Talk through the problems you’re seeing, and share the SLAs for feedback.

It’s important to make your SLAs easy to consume. When you approach teams with work, you need to be using the SLAs to simplify the decision-space for them. They’re going to ignore your report unless it’s easy to consume.

A few considerations as you develop these SLAs:

• Some things are urgent. They may need a different process. For example, security teams may need to do this for remote code exploits that are live in the wild.

• Balance the timelines with the capabilities of the company. You can start with something achievable, ratchet up responsiveness over time.

• See below for an example to start from. You can vary the description and SLA turnaround, according to the needs of your product. This is a design example.

  Priority Description SLA (report to close)

  P1 <criteria for most urgent usability issues. Should be severe enough that a team should interrupt its work.> 1 day

  P2 <criteria for next class of usability issues. Should be severe enough that it should be addressed in the next body of work> 14 days

  P3 <criteria for next class of usability issues. Essential to user experience, but people can work around it> 45 days

  P4 <criteria for next class of usability issues. Things that are important for a good experience, but can be deferred.> 6 months

After you have priorities and SLAs, ask your team to start categorizing issues with them. You can use them to build your report.

Send out a weekly report to managers, directors, and VPs

As you craft the report, make sure teams are receiving a report that is actionable. Scope the issues to team's areas of responsibility.

Sending it to directors and VPs adds an accountability layer to it. So having a rollup by organization is helpful.

Send it out weekly, and monitor how it is being received.

Be patient and look for opportunities for reinforcement

It may take a while for the report to get traction. Many teams will ignore the report. And others will follow it. You should expect that to happen.

Focus on how to make your team helpful. And look for patterns in the data that you can use to drive further improvements. For example, if you are a design organization, you can look for patterns in usability issues. This can lead you to making more proactive improvements.

Also, be on the lookout for opportunities to reinforce the importance of the report. For a design example: when a VP complains about the usability of the product, show them the report is the solution. It might be a moment where they realize the report they are ignoring is the thing that solves their frustration. Resist the urge to point out they are responsible for the problem!

Objective expert versus other coordination models

This is one of many ways to handle team coordination. Other patterns to consider with specialist teams trying to outsource prioritization:

Controller or Work Demander: Traditional security teams use the Controller or Work Demander models. They dictate what needs to happen. Like a legal team, the product development group feels compelled to do what they say. This can be effective but has some downsides. The chief downsides are that security team member incentives aren’t always aligned with the company, and it can create an adversarial relationship. They can become the “team of no”.

Consultant: acting as an Objective Expert team almost always goes hand in hand with acting as a Consultant. Your team aims to provide a service to the organization: your expertise.

Embedded: specialist teams often act as embeds. The ideal is if you can have those embeds work with leadership in those local teams to make good prioritization decisions.

Coordination models

Objective Experts are just one of many coordination models. Coordination models give you a menu of choices to choose from when solving your leadership coordination issues.

Feedback

Any points I missed? Anything you wanted to hear more about? I love feedback.

If you liked this post, subscribe to hear about future posts!

Thank you

Rebecca Campbell, Kelsey Yocum, Stephen Weber, and Eric Dobbs reviewed drafts of this post. Rebecca, as always, helped point out some areas I missed, especially around the human side of using this model. Kelsey had helpful feedback on how this works with design teams. Stephen contributed some discussion about team level metrics and helped refresh my memory of how the reliability metrics were adopted at New Relic. Eric also brought up some very interesting points about how costly it was to implement aspects of the reliability program, and a lot more interesting context.

Image by qimono from Pixabay

Most engineering organizations focus on delivering projects. They should focus on milestones instead.

Managing projects is hard. Companies contort themselves to do it well. Instead of playing chess, switch to checkers. Milestones are an easier game, and you get better results.

More importantly, milestones unlock powerful behavior. One example is project shaping. Milestones make it easy to play with the contours of a project.

In this post, I describe:

1. A special flavor of milestones I recommend you adopt.
2. Why humans suck at projects.
3. Why we’re actually pretty good at milestones.
4. How milestones unlock project shaping, better handle exploration projects, encourage incremental design, improve teams, make engineering look good, and increase trust.
5. How to make the switch to milestone-based delivery.

How should milestones be defined?

I use a special version of milestones, and recommend you do too. These milestones have these properties:

1. Small. Milestones are one or three weeks of work, no more. One or more milestones form a project.
2. High-quality. Every time you complete a milestone, leave things better than when you started. The codebase and user experience shouldn't degrade. The one exception to this is if you’re doing a throwaway prototype. This preserves your ability to deliver value over time. When you’re done with a milestone, it should be releasable in some way. You may choose to not release it to all customers, but it should feel done in some defined way.
3. Understandable. Select milestone names which communicate the value you are delivering. Any non-engineer in the company should understand what a milestone does. I like to use "[value delivered] by [approach]". This communicates to both engineering and the rest of the business. For example: "speed up search results on the listing page by implementing ElasticSearch". Business people understand the value part. The engineering team will understand what we’re doing from the approach part.
4. Valuable. Every milestone delivers value. If you are working on a project with many milestones, you should be able to stop midway. Even if you don't finish all the milestones, you should have delivered value.

For the last piece, the value can be

1. Customer value
2. Business value, or
3. Information.

The acronym for these attributes of a milestone is SHUV.

Why use milestones? Well, let’s start out by talking about why not projects.

Humans are terrible at project management

Project management is beguiling. It offers the illusion of control over what is a very chaotic process.

Project management seems valuable. You do need some sort of structure. You do want to consider risks, review progress, and track dependencies.

A better alternative to project management is milestone management. All you do is “replace all talk of projects with milestones”

Let me explain why this simple change is desirable.

Engineers are generally poor at incremental design

Engineers are predisposed to design and sequence work in a monolithic fashion. Experienced engineers know how to work incrementally. But the gravity in engineering organizations is towards design in horizontal layers.

Here is how your engineering team will design a typical feature. This assumes there is a frontend and backend part of the work.

• Have a conversation or design doc you’re working from. A designer provides mockups.
• Agree on what the API will look like between the frontend and backend focused engineers.
• The backend engineers get to work on the API. They design the data model, create the CRUD operations, then add the interesting business logic in. Spin up a new service for it.
• The frontend engineers get to work building out the UI. They build it from mocks, creating components for each thing. Do it page by page. Mock out the backend until it is available, then hook it up.
• Work out the inevitable problems between frontend and backend.

Full stack engineers will often build things in the same horizontally designed way.

What’s wrong with this approach?

1. It is not incremental at all.
2. All the value is delivered at the end.

We will talk about a better way to do this later.

Designers are also poor at incremental design

Designers have similar challenges with incremental delivery. To design something well, you have to think about the end state. So designers will produce artifacts that represent the end state you are working towards.

This is necessary. Designers need to do the work to think about the end state. But this further biases your delivery towards the monolithic. It is a trap!

Product managers think about long-term end goals.

Similarly, product managers think of things in terms of the end state. They have a capability they want to deliver to customers, so they may not think about the path to get there. So they join the crowd of people all biasing towards monolithic delivery.

Effective project estimation is almost non-existent

Our field is notorious for unreliable estimates. It’s so bad that many people in the #NoEstimates camp argue you shouldn’t estimate at all.

This is contentious because it is important to understand the cost of building out a capability. If you can’t figure out the cost, your attempts to figure out the most valuable work (value / cost) are impossible.

The problem is that we aspire to unnecessary precision. We don’t need to know exact estimates to ensure engineering focuses on the most important work. The cost of producing exact estimates is wasteful. And it doesn’t even produce the outcome you’re looking for. You don’t need to pretend the feature will be done on June 20. This date is most certainly incorrect anyway.

Milestones reduce the complexity of putting together high level estimates. They give a shorthand that can be good enough for most decision-making, without the overhead of rigorous estimation.

Occasionally, I see people succeed at rigorous estimation. But it’s rarely systemic – it’s usually one individual that is good at it. And it relies on them. If they go on vacation for a week, nobody is able to feed their model, and it collapses. While this is a great skill, to me it is the exception that proves the rule. Think of it this way: if one in twenty people can estimate in a high complexity situation, how many could be more successful in a less complex situation?

Projects are often bad for people

Project delivery can feel like a slog. Often you work for months towards a goal that may not be realistic. Or the goals may shift all the time. You work for months to deliver something, only to find it misses the mark.

These are the symptoms of the disease: projects are the evil behind it.

Humans are actually pretty good at milestone management

The difference between projects and milestones is like the difference between juggling three balls and six balls. It takes orders of magnitude more skill to juggle six balls. Most people can learn to juggle three balls in twenty-four hours. It can take years or decades to learn to juggle six balls.

Program management is a much harder skill than project management. Project management is a much harder skill than milestone management. Why not play the easiest game, especially when the results can be so much better. Let's enumerate all those ways.

Estimating one to three weeks of work is easy

One to three weeks is short enough that people can reason about scope. They can make an estimate of how much work is involved and be reasonably accurate. One company found their team was about 97% able to hit their estimates for projects of that size:

He is anonymous by request.

This matches my experience. The sweet spot for setting goals is a week to three weeks. Humans just do better with work of that size. And they are much more accurate at estimates for work with that amount of complexity.

Breaking a project into milestones lets you shape projects

When people break down a project, they typically break it down into technical tasks. Some more experienced teams might break it down into user stories.

You will extract far greater value if you first break projects into a list of milestones. A project is a list of one or more milestones. Your high level plan for the project is an ordered list of milestones to deliver.

This is more valuable because humans can reason about a list of milestones way better than a list of technical tasks. And a list is more fluid. The opposite of that is a list of technical tasks. They will cement your approach and make it a lot of work to change the plan.

User stories, while much better than technical tasks, are also more difficult to reason about, simply because you might have a lot of them.

Milestones sit at just the right altitude that you can play with them.

There are many ways you can deliver a capability. One of the most unexploited aspects of product delivery is playing with the sequencing and manner in which you break up the work. One of the most valuable uses of time is to consider multiple ways to approach building a project.

I like to ask teams to come up with three different ways to sequence a project. I ask them to discuss the trade offs and choose the best one.

Trade offs to consider include:

• When will customers have contact with your work? Moving it forward provides information earlier. A general pattern you will see in successful projects is early customer feedback, followed by increasing customer contact over time.
• What information will you learn, and when? You might do riskier milestones early, to explore areas you’re less confident about. Or you might do a milestone that is explicitly about testing some assumptions early on.
• When will things be fully integrated? Experienced engineers will start out with a steel thread, and keep it working at all points. Be suspicious of orderings that rely on people merging their work later.
• How much optionality do you give your future selves? Sequencing that allows us to pivot to new priorities can be incredibly valuable. If you’re able to sequence a project so the value diminishes over time towards the end of the project, you can keep your projects lean by not implementing things that may be less valuable. This can also reduce technical debt over time. Just be sure that what you deliver early on is actually valuable when you deliver only part of it.

For example, let’s say your project is to create a Slack bot that displays charts from your application in a customer’s Slack channel. Here are two different ways you could break down this project:

Breakdown #1 Breakdown #2

Provide a user accessible API for accessing chart data, with several chart variants. Share with a few customers. Create a hardcoded Slack bot for our own team that displays a useful metric with a line chart.

Deliver a Slack bot internally and to a few customers which connects to API and displays a table of chart data within a configured channel and Slack organization. Make it possible to set the metric being charted, still with a line chart.

Add support for line charts in bot, and release to customers. Make it possible to set the channel and Slack organization for displaying metrics with line charts and share with a few customers with early docs.

Add support for additional chart types. Release to customers. Add support for more chart types. Release to beta

Incorporate feedback from customers, test scalability, and release to GA.

Tradeoffs:

• The second breakdown gives the team a more immediate understanding of the problem space. Dogfooding is strong with this one.
• The first breakdown has earlier customer contact, but not in a form customers probably would want.
• The second breakdown creates a steel thread at the beginning. Less integration risk.
• The second breakdown does narrower slices at the beginning. It preserves more optionality because it may turn out you don’t need other chart types, and you can move on to some other project. Also note how the second breakdown could result in changes more easily – incorporating feedback is built into the plan for the project.

Milestones encourage separation of exploratory and execution projects

Milestones help separate deliverables into two categories:

1. Execution. Work that can be broken down, and
2. Exploration. Work that has a lot of unknowns and can’t be broken down.

This allows you to separate buying capabilities from buying information.

One nice thing about the milestone definition we use is that each milestone delivers value. Some projects won’t be able to be broken down into a complete set of milestones. This happens when not enough is known about the project yet. That’s perfect! Because then the value the milestone is intended to deliver is information.

Projects that deliver information deliver value. For example, let’s say that you want to build something for customers. The new feature relies on a technology nobody in your company has familiarity with. The approach you take depends largely on the technical details, which are unknown.

In such a case, you can have the team do an investigation. The deliverable for the milestone could be a presentation, or it could be a prototype. The team should also aim to deliver three different ways they would sequence future milestones to deliver the capability.

When faced with prioritizing a milestone that delivers information, the product manager can make a rational choice: is this information worth a couple of weeks of the team’s time?

Milestones naturally encourage vertical, incremental design

Previously we gave an example of the way teams typically build software: horizontally. It looked like this:

Milestones tend to encourage more vertical slicing of delivery. That looks different, more like this:

When a team using milestones delivers a feature, the approach we will take is very different from what we described earlier.

• You’ll have a conversation or design doc you’re working from. Perhaps a designer provides mockups.
• You’ll be thinking both of the full feature, and the first milestone of the feature. If the feature is sufficiently complex, you have a technical design for the whole thing. If you can do it incrementally, you just design the next milestone.
• The designer has a design for the end state, but also has to do a design for just the milestone.
• You’ll focus on the first milestone, which has a narrowly defined subset of the feature that you can build that will provide value.
• The frontend and backend engineers work together to build out the milestone.
• The backend engineers may not build out the full API. They build out what is necessary for the milestone. They design the data model, create the CRUD operations, then add the interesting business logic in. Spin up a new service for it. But they leave out the parts that aren’t necessary yet.
• The frontend engineers get to work building out the UI. The mocks they are working from are slimmer than the end state, so they build out a slimmer feature. They create components for each thing. They may mock out calls to the backend APIs for a few days if they aren’t done, but when they are they should be able to rapidly provide feedback and iterate with the backend engineers.
• They demo their work along the way, and at the end of the milestone, have a completed set of work they can demo together.

As you notice, this isn’t all that different, except that the frontend and backend engineers work more closely together, and deliver the milestone together. They’re more synchronized.

Although the benefits of this may seem subtle, they matter. Incremental design results in work that delivers value after every milestone. Instead of every three to six months, delivering a huge chunk of unreliable value, you deliver value every one to three weeks, and learn and adjust if you miss the mark. You must invest in learning and adapting to take advantage of incremental design, but the payoff is huge.

How to break down project milestones

When you work on the milestones, break down the current milestone only. Use user stories instead of technical tasks. Vertical slicing can be fractal, and extend down within milestones as well. You may find user stories you can remove, defer, or alter.

For example, let's say we're starting with our first milestone: "Create a hardcoded Slack bot for our own team that displays a useful metric with a line chart."

This could be done is a lot of ways, but let's list a few potential user stories:

• "Team member can see a 'hello world' message displayed in #my-team room in Slack when they type /happybot test"
• "Team member can see the current happiness score displayed in #my-team room in Slack when they type /happybot test"
• "Team member can see a line chart displayed in #my-team room in Slack when they type /happybot line"

Each of these could be broken down into technical tasks if that's helpful or necessary.

Prioritization with milestones is good enough

When prioritizing, you will have a rough sense of the cost of features based on the number of milestones (for execution projects). You probably don’t need as much accuracy as you think you do to prioritize features.

But one thing that is nice about milestones, is you can get creative with your prioritization. For example, you might look at a project, and decide you can do the first half of the milestones, and deliver most of the value of that project. This type of fluidity in planning is one of the main things we miss in product development.

As Donald Reinertsen, author of the Principles of Product Development Flow (one of my favorite books on product development), puts it:

Teams thrive when they deliver value all the time

Teams that are delivering value to the business every couple of weeks feel great. They have a constant stream of achievements. Work feels meaningful when you’re delivering value to the world. People develop a sense of momentum and confidence, which helps bond a team together and develop an ability to critique and improve each other.

Milestones are agile, in the sense that change feels natural

The fourth property of milestones is probably the most important: that it is independently Valuable.

With milestones, you have natural stopping places every one to three weeks. This means teams can change their priorities and it won’t feel like a thrash to do so. And because some value was delivered, it doesn’t feel like your work is being wasted. With projects, priority changes feel like throwing work away, and can be dispiriting. With milestones, you can usually finish up what you’re doing.

While you're using Milestones, ideally each milestone feels like it's own independent project. And going on to the next milestone feels optional. This gives the product manager more optionality. Teams that use this approach are also more responsive to their environment, because their work isn't locked up for 3-6 months at a time. This allows for more follow-up work and iteration.

There are some situations where you’ll want to halt DURING a milestone: if something is truly urgent. But if you do that, you’re making a conscious choice to incur the cost of throwing away work. And it’s less likely to happen, because the end of the milestone is more likely to be near.

Milestones make engineering look good, and it builds trust

And a team that is constantly delivering small units of value earns the trust of the people around it. Instead of a huge thing every three to six months, you see a steady stream of value. It is reassuring to see a team constantly learning from customers, and adapting. When a VP sees a team assessing its own work, and doing follow up work to make their part of the product amazing, it inspires confidence, and often results in higher autonomy for the team.

I’ve seen sales teams that were skeptical of engineering be turned around when they see a steady stream of milestones being released to customers. This may not be rational, but a stream of work invariably feels like higher velocity, and leads to a perception of engineering doing a great job.

Milestones also serve as a good altitude to talk about matters between Product, Design, and Engineering. What are the next couple of milestones? How are we feeling about them? What are we concerned about? Because they inherently give more optionality to Product, it can improve the way Product and Engineering work together.

Using milestones also means a more steady stream of value is delivered to customers. If you’re concerned about the product changing too frequently, keep in mind that the milestones don’t necessarily have to be delivered to everyone. You can deliver them to a subset of your customers, and use feature flags to hold them back and batch the delivery. Even if you do this, the milestone approach is still superior.

Managing milestones is lighter weight

You generally need a lot less heavy-weight process to manage milestones than to manage projects. At the most, you may need a couple of bullet points per week as a project plan. You don’t need heavy artifacts and heavy tooling. You don’t need Gantt charts. You can track you dependencies and risks, and probably should. But you can scale the amount of investment in milestone management to the complexity and risk of the situation.

Project-based delivery vs milestone-based delivery

Project based Milestone based

Increments Usually three to six months, but unpredictable.Usually built horizontally and monolithically. One to three weeks.Biases more towards vertical, incremental design.

Value delivery At the end. Each milestone. Higher value per unit time.

Hitting the mark Less likely. Relies on a perfect plan, and you find out if the plan made sense after you complete the whole project. More wasteful. More likely. You design feedback into your milestones, and learn along the way, and can adapt or change investment.

Scope control Futile attempts to “cut scope”, balanced by “if it’s not in the project it won’t happen” Naturally segmented.

Agility and optionality You get 2-4 chances to learn and course correct per year.Lower optionality, because projects are monolithic. You get 17-52 chances to learn and course correct per year.Higher optionality, because you can change investment during the project, and achieve value throughout the project.

How it feels for the humans Big feeling of accomplishment or failure after each project.Feels thrashy if you change your mind during projects. And you do. Frequent sense of accomplishment, stronger team dynamics. More opportunities to recognize team.More natural to change priorities based on new information.

Milestones vs sprints vs kanban

Isn't this just sprints? Or, can't you accomplish the same thing by using sprints?

I don't recommend it. Milestones are mini-projects. I've seen organizations that have used sprints since the beginning be transformed by using milestones.

If you use sprints for mini-projects, you're saying every unit of delivery is exactly 2 weeks in length (or whatever you use). This incentivizes adding more scope to fill out a mini-project, or cutting out scope to make a mini-project fit.

What I've found is:

1. Milestones incentivize different things than sprints. Instead of "what can we accomplish in two weeks", you ask yourself, "what's the minimum amount of work we can do to provide value".
2. Milestones incentivize better shaping of projects, and better planning.
3. Sprints don't encourage vertical slicing. Milestones do.

So, I recommend you use milestones WITH sprints, or milestones WITH kanban.

When using sprints, I plan the current and next milestone. Each sprint, we break down the work for any milestones we're coming into contact with. And we look at whether we'll complete a milestone that sprint or not.

Another way of thinking of it is that milestones is like using kanban but preserving your weekly or biweekly meetings.

Using milestones with Kanban is simpler. Your unit of delivery is just a milestone, and you try to deliver one every one to three week cycle.

How to switch to milestone based delivery

Switching to milestone based delivery isn’t especially complicated. You just start reporting on milestones instead of projects.

I usually have a weekly reporting cadence where I ask each engineering manager to report on the state of their current milestone. The update is in the form of a tweet (to constrain the writing and keep it concise). I ask for an updated estimate for when the milestone will be completed, newly identified risks, and an objective assessment of the state of the project.

As each milestone is completed, treat it as an opportunity to learn and celebrate. That is a good time for retrospectives and team conversations.

You will often need project level plans as well as milestone plans. For example, complex project may require a project-level technical plan. It important to have a good way to decide whether to do more detailed long-term planning.

Sometimes, you will need project level planning. For example, you might have something that will be due at a conference. For a time like that, you may need to do estimation for several upcoming milestones, and manage the risks of these upcoming milestones. Even in this case, milestones can be useful, because they can give you an early warning if you’re off course.

Final thoughts

The other thing you can focus on instead of projects is outcomes. I’m in favor of this, but it is a more difficult change to put in place, so I consider it to be a more advanced technique. For companies that are used to projects, moving to milestones isn’t a huge change. Moving to outcomes may be ultimately better, but it’s harder to do.

Sometimes you are working within an organization where projects are the unit of work, and you won't be able to change that easily. In that case, I recommend still working with milestones, but to pretend each milestone is a separate project. Or alternatively, you can just use milestones within your team and communicate the larger project externally.

As always, I love to receive feedback on my writing. Let me know what you think – what resonates, what you have doubts about. And certainly let me know about your experiences implementing milestones.

Why listen to me on projects?

I have a lot of experience with project management:

• I’ve managed projects for a couple of decades.
• I’ve run programs crossing many teams and many projects.
• I’ve studied project management theory. And,
• I’ve designed and written project management software.

Decoding leadership podcast

I cover Minimal Marketable Features, and how we rolled them out at New Relic, in this podcast episode. It covers some of challenges we had getting people used to milestones.

Thank you

This whole article owes a huge debt to Jim Shore. He introduced the concept of a Minimum Marketable Feature to New Relic. It was the inspiration for this whole post. He has a lot of experience in this realm and the second edition of his book, The Art of Agile Development, has just come out. Donald Reinertsen’s tweet is included in this article, and his book on product development is a master course in the principles behind product development. I’d also like to thank [anonymous VPE] who shared his study on project estimation with me and let me include it here.

Image by derRenner from Pixabay

Why set up a DSLR for videoconferencing?

Almost all of my professional life is done via meetings online. So I thought I should invest in things that make it easier to interact with me -- easier to hear and see. I saw two people who had invested in DSLR setups: Jason Yee, and Derek Steer. They stood out in Zoom meetings, because they were very clear and the camera was tight in on their face.

How is it using a DSLR as a webcam?

I've gotten it to a pretty good state, but in the past it has been really fiddly. It stopped working occasionally, and I wasn't sure why until I spent a lot of time on it. I still have some issues with Google Meet.

The results are worth it to me, but the frustration is real. These are issues I have had:

• (Solved) For a number of months, my video feed would sometimes just cut out. It seemed to take a restart or two to get it working again. I believe this was due to the cheap video capture device I was using. After going back to the Camlink (as I describe below), things began being reliable again.
• (Solved) Sometimes the video feed looks distorted. Like I’m really skinny. This seemed to be another issue with the video capture device.
• (Solved) Up until recently I was having near daily kernel panics. They seem to have stopped when I started plugging in two USB-C connections to my laptop instead of one. Maybe having a large monitor, DSLR video, and a few other devices plugged in to the same connection was the cause? I have no idea. I do know it happened across three different computers, and with clean installs of Mac OS, and the kernel panics are all in Apple code. I'm not even sure this is related to the DSLR.
• When I use the camera with most web browser based videoconferencing solutions, I get an echo in the sound, because Zoom seems to be the only solution out there that can use a microphone as both a speaker and microphone.
• If I leave the camera on for a very long time, it needs to be cooled down or restarted.

So all of this is to say the cost of having a DSLR setup is that it is more complex. It can result in having to do things like restart you computer right before an important meeting. Diagnosing these problems can be hard, because there can be so many places the problem can be.

What does it look like?

These aren't the best examples, but give you an idea of what the difference is. This is the view from my Macbook, sitting in a corner:

For a while I used a Logitech webcam. It was ... fine.

And this is the view from the Sony a6000.

I do some comparisons below to the Lumina webcam as well, and added what it looks like now that I've added a third-party lens.

Choosing a camera

I initially purchased a Sony a5100. It’s smaller than many alternatives, and has a handy flip up screen, so you can see what you’re shooting from the front. However, the external battery seemed to not power it forever — it would need to be removed and reinserted every time I used the camera.

So I instead went with a Sony a6000. It seems like Sony cameras are some of the best supported for webcams, so if I were purchasing today, that’s probably what I would start with. But anything with clean video that can take a dummy battery and stay on for a long time is fine. Some DSLRs won't stay in video mode for long periods of time, so be sure to search for overheating and using it as a webcam. Here are a couple of sources that list cameras that should work (but you should search a bit before using it):

• List of camera you can use with Elgato Camlink
• This is another list of cameras with clean video from Atmos

I started out using the lens that came with my Sony, which has a 16-50mm, f/3.5-5.6 lens. It worked fine, but in retrospect I should have upgraded it much sooner. You can see what a difference it made if you look later in this article.

You’ll need a dummy battery

You’ll need your DSLR to stay on while you’re using it. So for whatever type of camera you get, you’ll also need a power adapter that allows you to plug a power cord into the battery compartment so it runs off A/C power instead of a battery.

These are called "dummy batteries". I don’t believe mine is still available. But search for one that corresponds to your DSLR.

Video capture

You’ll need something that can take the raw video and get it into your camera with low latency. To learn more about this, Jason Yee has a great explanation.

• Elgato Camlink 4k. I'm on my second or third Camlink. The first time I tried it, it stopped working after a short while. In December 2021, I purchased it again, and it seems to have addressed two of the main issues I was having (with the video feeds cutting out). The whole setup is more reliable again. Crossing fingers it won't stop working again like it did before.
• For two years, I used a cheap HDMI video capture device Jason Yee recommended. It was only $10 and worked fine. It did give a slight red tint to the image. However, it’s no longer available on Amazon, so you’ll need to find some other option.

Cables and such

It would be nice if things just plugged into USB-C on my laptop. Most cameras output Micro-HDMI, so you’ll need:

• Micro-HDMI to HDMI cable ($10)

So from the camera, I have a this cable plugged in to the video capture device. That in turn is plugged in to a CalDigit Thunderbolt dock. That is plugged in to the USB-C on my laptop. The CalDigit is not a necessary purchase -- I did this because I was trying to diagnose some VERY annoying USB problems and I read it might be a solution. Somewhere in this is a weakness in my setup, I suspect.

Setting the DSLR above your screen: get a mount

I use this mount to put the DSLR right above my screen. It’s easy to figure out when you receive it. It basically clamps onto the back of my desk.

• Elgato Multi-mount - this seems to be the best one, but when I was buying it, it was impossible to get in the US. I couldn’t find a better alternative. I ordered this from the UK, and it took a month to get here.

Bathe my face in light

Some people may not like the light in their face. I live in a climate where we yearn for sunshine and only see it for fleeting moments in the summer. The light does look good on camera. But I also like that it makes me feel like I’m in a bright space. I have two of these. You can set the color temperature and brightness. Weirdly they are controlled via an app or menu bar on the Mac. They work, but it’s weird that it’s always digitally controlled.

• Elgato Key Light Air

Here's the difference between light and no light. Before:

And after, with the lighting:

I want to talk without a headset: get a microphone

I got a Blue Yeti microphone. It’s been fine. One challenge nobody told me about (but I should have anticipated) is that if you have an external microphone, it will get feedback from any other speaker. However, some microphones (like the Blue Yeti) have an output for speakers. If you plug in your speaker to this output, apparently it helps reduce feedback. It seemed to for me. I purchased some Bose mini-speakers, and this setup ended up feeling great. I can talk without a headset on, and people hear me well. And I can hear them great, and control the volume with a volume nob. It's a real pleasure to not need a headset during meetings all day.

The challenge with this is that I’ve found some applications don’t handle the input and output device being the same device correctly. Zoom recognizes it fine on a Mac. Google Meet does not.

I use Krisp.ai to reduce echo, since my office has wood floors and echoes. Zoom now has some echo reduction, but I haven’t yet compared the two. Krisp isn't perfect -- I've had it just turn off (and require a restart). But it's been good enough I haven't ditched it.

Make it sound good with shock mounts and pop filters

A pop filter reduces extra noises on high quality microphones. They’re easy to install. A shock mount isolates the microphone from physical bumps and such. I never got one. But I did get a mount for my microphone.

• Aokeo pop filter.

Lumina webcam vs DSLR review

In December 2021, I purchased a Lumina webcam. It has a great form factor, and I love that it's USB-C. It also attaches nicely to the monitor. The dream is that it would offer the same quality as a DSLR -- then you wouldn't need the expensive DSLR, the mount, the capture device. And it would just plug in to your computer. The Lumina even features a nice little privacy cap.

The Lumina is a very fine device, but it's not for me. It doesn't have the same detail and quality of the DSLR. Note this part of the review was done before I purchased a higher speed lens, which lets in more light. So the photos are darker for the DSLR images.

I set it up in the evening. Sometimes I start work in when it's still dark out, so this is a realistic situation for me. Here's the Lumina device:

Here's my DSLR (note this is before I got a lens that lets in more light):

So comparing those two, the Lumina tries to balance the brightness more, but the image looks pretty poor. And it does a nice thing where it tracks your face, and keeps it in the picture, but the DSLR makes it easier to control what is in your background (I don't like to show my window as I think it's distracting). You can override these things with the Lumina. The DSLR image clarity is much better. Still, the Lumina isn't terrible, and it's possible as they improve their software, it will be a better option. I'm using beta software after all.

The next morning, I did another comparison. The Lumina actually offers two different camera inputs. The "raw" input, which shows what it does before all their software intervenes, and the "plus" input, which shows the input after the software has done its work. This is the raw input, with lighting:

Oh my god that is kind of terrible. Let's see if their "plus" mode, after software does it's magic, helps:

So that doesn't really help. Maybe it's being over-sensitive to my bright lighting? I tried turning off the front light, and the results look pretty good.

But I still prefer my Sony a6000 DSLR setup. Here's the Sony view, with light:

When there is better ambient light, it looks even better. It also looks way better today, now that I've invested in a high speed lens.

I like that companies like Lumina are starting to produce webcams that are intended to compete with DSLR setups. But they still have a lot of work ahead of them. Still, for people that don't want to hassle with a DSLR setup, the Lumina is a decent option. I will be returning the Lumina.

Note that after I wrote this review, I was contacted by Lumina, and they said their software had been updated and they offered to send me another one to review. So in October, 2022, I retested everything.

This comparison now includes the Sony with an upgraded camera lens vs the newer Lumina. As far as I can tell, the only difference in the Lumina is the software has changed.

Here's the Lumina. Note the image is less detailed. But it does look pretty good.

The Sony looks much sharper. The light balance looks better (I fiddled endlessly to try and get the Lumina to look similar).

The Lumina has a "cameraman" feature which will track your face if you move around, but it reduces image quality by about 10% and it's pretty unacceptable quality. I found manually zooming in to be best.

The Lumina also currently doesn't work in Google Meet on Safari. This is due to a framework Apple is updating to work with third-party cameras. Well, strictly speaking, it works, but it doesn't do any software manipulation of the image, so you get a poor quality video feed.

I like that the Lumina has its own microphone. I've had lots of sound issues with Google Meet, so tried out the Lumina in the hope it would improve things. I didn't test the sound much, because the image quality was a dealbreaker for me, and I use Safari.

The software still feels a little beta quality to me. Needs a little design refinement, but the capabilities of the webcam are quite nice, and you can get a pretty good quality webcam feed from it.

All in all, I think Lumina is filling a nice niche. It won't replace the DSLR for me, but it's a decent webcam.

Upgrading the camera lens

After evaluating the Lumina, I realized the reason my picture was so dark is that my camera wasn't letting in very much light. So I purchased a Meike 28mm f/2.8 camera lens.

Here's the original Sony 16-50mm f/3.5-5.6 lens that came with the camera:

Here's the view from Meike camera lens.

Some things to note:

• The Meike is a manual focus, fixed lenth lens. When using as a webcam, that means you set the focus and leave it there. If you move around a lot, that's not a good option for you.
• You can manually set the focal length on the Meike lens, to decide how much light and depth of field you want.
• One annoyance with a zoom lens is when you turn it on, you have to zoom in and frame it each time. The Meike is easier because it starts always at the same Zoom level. However, you better be happy with how it frames the scene -- it isn't something you can adjust.
• I'm amazed how much more light it brought in. It significantly improves the view.
• The Meike lens cap is also easier to put over the lens, as a privacy cap. The Sony cap is less satisfying to put over the lens.

What does your desk look like?

There are the aethetic aspects of the decision as well. It definitely adds a lot of cords to my desk. This is a minimally straightened out view of my desk (pretty close to how it looks most of the time, except I usually have a few more things on it):

Further reading

• Besides the articles by Jason Yee, I found one of the most helpful posts was this one by Matt Stauffer

What's next

I'll probably experiment with my video capture and see if that improves things. I've also heard of webcams that try to achieve DSLR quality recommended to me, so I plan to try one of them out and see if it's equivalent. If it is, I might use it to replace a lot of the things I've purchased. I'll report back here what I find.

Feedback

I'm far from an expert on all this, so if you have any ideas on things I'm doing wrong, let me know! Also happy to answer questions on it.

Also, be sure to subscribe if you’d like to be notified of future posts.

Thank you

Big thanks to Jason Yee for sharing his setup notes. Thank you to Drew Stokes for pointing out the challenges of overheating. Thank you to Adam Larson for getting me to post a photo of my desk.

Image by Farbsynthese from Pixabay

What should an Engineering Manager own?

Companies break up the roles and responsibilities of an engineering manager in many ways. This post describes the various ways to divide those responsibilities. I also provide the tradeoffs.

Tech lead responsibilities

Many startups start out with a tech lead model. It’s fine for the early stages of a company, but tends to be something you outgrow.

• A Tech Lead manages people, projects, and process. They also lead the technical decision-making.
• People management suffers, because the Tech Lead has so many responsibilities. And they’re often not an Engineering Manager by training.
• Process suffers, because the Tech Lead has so many responsibilities. And they’re often not an Engineering Manager by training.
• Project management suffers, because the Tech Lead has so many responsibilities. And they may not have experience with project management.
• The Technical Lead oversees the quality of the team’s technical work. They help their team get better at technical thinking. They ensure the team's technical plans are well reasoned and future-proof.
• The Product Manager talks with customers and integrates feedback from many sources. They prioritize the team's work. They also make sure the team has context so they can build high value software.

Engineering manager runs projects

This is the approach I gravitate towards. With this approach, you have an Engineering Manager, Product Manager, and Tech Lead.

• The Engineering Manager handles people management. They coach their team members to make them more impactful.
• The Engineering Manager runs projects: project breakdown, sequencing, risk management, and project communication. This gives them a day to day view of the team’s work, and helps them be effective coaches for their team.
• The Engineering Manager manages the team’s process. They adapt and improve the way the team operates. This helps the team always improve.
• A Technical Lead oversees the quality of the team’s technical work. They help their team get better at technical thinking. They ensure the team's technical plans are well reasoned and future-proof.
• The Product Manager talks with customers and integrates feedback from many sources. They prioritize the team's work. They also make sure the team has context so they can build high value software.

Product manager runs projects

The intention is to have the Product Manager heavily involved in the team’s work. And to have highly technical engineering managers, who review code and sometimes even write code. At least some parts of Google operate with this model.

• The Engineering Manager handles people management. They coach their team members to make them more impactful.
• The Engineering Manager manages the team’s process. They adapt and improve the way the team operates. This helps the team always improve.
• The Engineering Manager oversees the quality of the team’s technical work. They help their team get better at technical thinking. They ensure the team's technical plans are well reasoned and future-proof. Since the manager is in a position of power, this can cause problems. These problems can happen because people won't want to oppose their manager's views. Also, Engineering Managers can find it hard to focus on technical work. Alternatively, you can have a Tech Lead handle this area. That can work okay, but has a disadvantage. The Engineering Manager won't have a connection to the work. This will cause them to not be able to guide the team's process or coach their team.
• The Product Manager runs projects: project breakdown, sequencing, risk management, and project communication. This gives them a day to day view of the team’s work, and helps them give lots of context to team members.
• The Product Manager talks with customers and integrates feedback from many sources. They prioritize the team's work. They also make sure the team has context so they can build high value software.
• Because the Product Manager is so focused on the team, they spend less time with customers. It's difficult to balance both aspects of the job when you're responsible for projects. I view this as a major disadvantage.

Single threaded owner

The Single Threaded Owner owns everything. They can hire people to delegate parts of their job. I have a longer experience report on the Single Threaded Owner model. Amazon popularized this approach.

• The Single Threaded Owner (STO) owns everything. They either do the work themselves, or find someone to delegate to.
• The STO manages people. They coach their team members to improve their impact.
• The STO may run projects or have a project manager run the project. That person does project breakdown, sequencing, risk management, and project communication.
• The STO manages the team’s process. They alter the way the team operates to be more effective.
• The STO handles the quality of the team’s technical work. They can delegate that responsibility. They help their team get better at technical thinking. They ensure the team's technical plans are well reasoned and future-proof.
• The STO or a Product Manager talk with customers and integrate feedback from many sources. They prioritize the team's work. They also make sure the team has context so they can build high value software.

SCRUM model

The SCRUM approach is a classic approach to software development. It doesn’t explicitly call out the managerial responsibilities. Here’s an overview of SCRUM.

• It’s not specified in SCRUM how people management works. Usually that person takes on the Scrummaster or Product Owner role. Ignore that the Scrummaster should not have authority over the team. Coaching individual team members tends to suffer. The manager may not be close enough to the work to coach the team member.
• You don’t see a lot of project management with SCRUM. It’s all focused on points or burndown charts. SCRUM teams I've worked with have neglected project breakdown, sequencing, and risk management. SCRUM divides project responsibilities between the Product Owner and the Scrummaster.
• Incentives for the Scrummaster are to focus on process and meetings. They tend to go overboard with it. They tend to lean on process too much.
• The team owns the quality of their technical plans and work. It’s generally done in an egalitarian way. Which can be good if the team functions well.
• The Product Owner acts as a lightweight product manager. The Product Owner role is a subset of the Product Manager role. Generally, a Product Owner will not do the job as well.

Summary

Area of focus Model

Engineering Manager (EM) does projects Product Manager (PM) does projects Tech lead (TL) Single threaded owner (STO) SCRUM

People EM EM TL (poorly) STO Manager (poorly)

Process EM EM TL (poorly) STO Scrummaster (poorly)

Projects EM PM TL (poorly) STO or project manager Shared (poorly)

Technical leadership TL EM (poorly) TL STO or TL Team, undefined

Product PM PM PM STO or PM Product owner (poorly)

Talk with customers PM PM (poorly) PM STO or PM Product owner (poorly)

Reporting

Engineers report to EM. Engineers report to EM. Engineers report to TL. Everyone (engineers, PM, TL) reports to STO. Unspecified. Usually Engineers report to EM.

Transitions

Moving between these models is where things get hard. For example, let's say you decide to give your Engineering Manager responsibility for Projects. If you haven't been hiring people who have project management experience, that might be a stretch. You'll need a plan to uplevel their skills.

Feedback

I have experience with all of these models listed above. One model I didn't include is the Engineering Manager who runs People, Process, and Technical, but works with a Project Manager. I’m sure it’s possible to be successful with any of these models. I welcome your feedback and comments!

Also, be sure to subscribe if you’d like to be notified of future posts.

Image by Marcus Williams from Pixabay

It's inevitable that your team will need things from other teams. This is especially common in product engineering. You can use the "Independent Executor Model" to avoid making this a problem.

What pain do dependencies cause?

It's very common to get into a situation where you need something from another team. Typically you'll negoatiate with that team to get that work done for you. It can then be tempting to wait for them to complete it before you continue with the part of your plan that depends on them.

For example, you might need a team to add a field into their API. Or you might need them to build a new API for you. Sometimes without these changes, you can't deliver what you need to.

This is an organizational trap. It leads to pain and misery. Why? Your plans are brittle. Priorities frequently change, and the more you depend on other teams, the more your plans are unrealistic.

Any ambitious project will have dependencies. The more dependencies, the more you are in the danger zone. Multiply the odds of change by each dependency. Soon it is an inevitability. Your plans are being built in the air.

Rules for Independent Executor teams

To avoid this pit of despair ask, but never expect other teams to do work for you. Follow these rules:

• You can make plans for other teams to do work for you.
• Communicate what you want to other teams, and make sure they understand how important it is.
• But don’t ever expect any other team to do work for you.
• Always have backup plans when you depend on other teams. That way, you can proceed and provide value if their priorities don’t align with your own.
• Do whatever it takes to deliver your experience to the customer.

How should product engineering handle dependencies?

Some organizations try to solve dependencies with tracking. This doesn't work, because the problem is dependencies, not tracking dependencies. Adding a system to track needs in Jira isn't going to help. Underlying everything is the fact that teams each have competing priorities. And those priorities shift, so you can't rely on them.

Some will think they can avoid this trap by using experienced project managers. But the problem isn’t poor project management. Even if you manage dependencies and risks, you will have unacceptable risk. Plans with dependencies are risky. When you always have a fallback, you guarantee you can deliver a baseline of value.

Shift to a model where you insist on always having a plan that doesn't rely on others. When you do this, you always have a plan that will deliver value. Communicate your needs to the other teams, so they can maximize value for the whole company. But deliver the most value you can within your zone of control.

This is the only way to stay sane in product engineering.

What does do whatever it takes mean?

Consider using hacks and later migrations. Your backup plans can often include scrappy solutions. For example, you can hack around the lack of good APIs. You can recreate services that other teams own, with a plan to migrate to theirs once it has what you need. You can do the work in their codebase, even though it is less efficient because your team doesn't know that domain.

Often working around other teams' priorities means you have to be messy. When you don't do the work in the right place, it can make your solutions less elegant. So when you go this route, first, check if your team can do the work in the right place. You might use the Away Team model to do this, doing the work in the other team's codebase for them. See below for more on that.

If you must use a hack, you should consider the costs and benefits. Waiting is expensive. But with a hack, your costs are to create the hack, maintain it, and migrate to the new solution when it is ready. Don’t ignore the cost of these things. You may decide the cost isn’t worth it, and you should work on something else instead. Be sure to communicate these costs to the team you’re making the request to. But you can decide based on the value you deliver and the costs you incur whether it is worth it.

Don’t forget your power to influence. Although you can’t control what other teams can do, you can influence them. Be sure to communicate your needs, and explain why it’s important. Give them the context they need to assess global priorities.

When to act as an Independent Executor?

• This should be the default for product engineering teams. Not operating in this way incurs higher risk of project failure.
• This is more widely applicable than product engineering. The basic principle can be applied to all engineering teams.

When using this model

• Keep an eye on organizational structure. If the org structure is incorrect, you may have excessive dependencies. The most common example of this a Frontend and Backend engineering team. Every Frontend project relies on the Backend team. While teams can find ways to work around the dependencies, they aren't ideal. They might agree to an API contract, but miss things and have to iterate. It's usually better to work together. When you see these dependencies, usually your org design is incorrect. (I’ll write a longer piece on the Frontend and Backend team pattern later -- I have lots of thoughts about that).

• Good system-level prioritization is essential. You won't decouple your teams without good product management. You need a product management (or Integrator) listening and prioritizing for global needs. If teams prioritize based on local needs, you need to strengthen your Integrators. And you may need a organization-wide prioritization scheme, like a Product Council.

Independent executor versus other coordination models

Independent Executor is a model for reducing unnecessary coordination. You should look at other coordination models when you need increased coordination. When you need work from other teams, consider:

Program Managers coordinate large initiatives

• When you need to coordinate a large initiative, a Program Manager is ideal. But the large initiative needs to be a priority in the first place. This is usually done with clear centralized priorities for the most important initiatives, which help local teams figure out how to prioritize requests from other teams. One way of accomplishing this is via a Product Council. A Program Manager isn't a help for getting other teams to do work for you.

Integrators listen and prioritize

• Integrators listen to people's needs and prioritize based on what they hear. Product managers usually fill this function. You need Integrators when you use the Independent Executor model. If you move to an Independent Executor model, you should add Integrators. Otherwise, you’ll end up with a lot of messy product engineering, and a platform that isn't useful.

An Away Team can be a way to do it yourself

• Instead of having another team do the work, why not do it yourself? Amazon has an approach called the Away Team model. You can't rely on other teams to do work for you, but you can do work in their area of responsibility. The basic idea is you send one or more people to the other team. They do the work in their codebase. It isn't true embedding because they do the work there for a limited time. The way they interact with the other team is also flexible. They can be completely independent or have someone from that team that helps.

• Developing norms for this in your organization will help Away Teams be successful. Having standards for contribution will also help. I've found that it is useful to have someone from the other team work with the Away Team members and guide them. But if you have strong norms and standards, that might not be necessary.

• It is important that the work be compatible with the other teams' goals. So usually this requires conversation. Sometimes the other team may not even have the bandwidth for these discussions. In those cases, you may be out of luck.

• I will write up a longer post on this model later.

Assemble a task force for critical projects

• If the work is critical, a Task Force may be appropriate. With a Task Force, you assemble a temporary team to do the work across a few teams. Task forces have downsides and can be disruptive. So you should only use them in urgent situations.

• Task forces don't help you get your work done by other teams.

Convince a Tiger Team to do it for you

• If the other team is too busy, sometimes you can convince a Tiger Team to do the work instead. This is a messy way to do things, but it can work in some cases.

Enlist the Community of Practice if you can

• Some Communities of Practice develop ways to drive work. If the work you need aligns with the Community of Practice, you can take advantage of that.

Coordination models

The Independent Executor is just one of many coordination models. Coordination models give you many choices to solve your inter-team coordination issues.

Feedback

See anything I missed? Disagree with this? Please let me know your thoughts!

Thank yous

I learned of this pattern from Jim Shore.

Image by Лечение наркомании from Pixabay

What is embedding?

Companies need specialists. And specialists often do their work with people outside their department. For example, all these specialties work with software engineers:

• Designers
• Product managers
• Site reliability engineers (SREs)
• Quality engineers (QA)
• Application security engineers
• and Architects

Specialists often apply the most leverage when they work next to engineers. For example, an SRE might help a team improve their monitoring. They might help even more if they help the team to understand how to do monitoring themselves. A designer might work side by side with an engineering team. They design future features, and collaborate on current features.

One way to structure this relationship is by embedding these experts on teams. They still report back to a home team of specialists, but spend almost all their time on another team. This is the Embedded Model of coordination.

Sometimes you might embed an expert on several teams. Or you might embed them in an organization. For example, you can embed a QA engineer with a Director. This director will have several teams they support. The QA engineer can then help improve quality across this organization. When doing so, they might focus on the top priority project. Or they might rotate between teams.

When to embed

• Embedding can be one of the best coordination models to use. It is appropriate when you want specialists to collaborate within a team. I like embedding designers, product managers, sometimes SREs, sometimes architects, and sometimes QA.

• Embedding can be the easiest way to add specialist skills. In some organizations, you are not "allowed" to hire specialists into your organization. For example, an engineering leader can't hire a designer into their team. Embedding allows you to get around this. See later in this post for a discussion on this.

• The embedded person’s work should be mostly aligned with the existing team. It only makes sense to use embedding if you need collaboration. Embedding increases context, collaboration, and relationships within the team.

• You can only embed up until a maximum team size. If the team already has eight or nine people on it, you need to split the team, or not embed. This count includes other embedded people.

• The embedded model scales well with the growth in teams. Each new team gets a new member. It does reduce the maximum team size by one. A rule of thumb is that the specialist team member should add more value than a generalist would.

• You are tying company investment to be proportional to the growth of these teams. That can be good, or bad. Many supporting organizations should grow slower than the growth of teams. They need to become more efficient over time. Other specialists should grow at exactly the same rate as engineering. Embedding makes the most sense when the growth should be at the same rate. Embedding makes these growth plans part of the structure of the organization. If you know every product team needs a product designer, then that is a good thing. You know that the plan will account for those hires.

• Sometimes specialist teams should grow slower. For example, a security team should grow slower than a proportion to engineering. They won’t achieve this goal with embedding, so other approaches might make better sense. You can still use embedding, but it should be temporary.

How to embed well

• You may need a central team. It’s common to need a centralized team even while embedding. This team is usually small.

• For example, a design organization might embed, but have a central team or two. This central team can work on a design system, or design research.

• An SRE organization might have a central SRE team as well. This team can work on standardized tooling, and provide reliability information and reporting. This works together with embedding.

• This sort of pairing is natural. You may or may not need it to start. But you usually end up needing both.

• Support the home team too. Embeds maintain split allegiances. Embeds need support on both teams. Some people tend to make stronger attachments on their work team. Other people tend to make stronger attachments with their home team.

• Encourage embeds to view their work team as their primary team. But don't neglect the home team. Home teams can share practices and information among specialists. They act as a Community of Practice that can spread expertise throughout the company.

• Don’t shift people around too frequently. Part of the value of these long-term pairings is you build relationships and context. It is expensive to move people because it breaks these connections. Most embedded organizations tend to move people more than they should. It can be frustrating to have embeds on your team if you can't rely on them to stay. Yet it is tempting for the embed's manager to move people around. They want to react to changing needs, and sometimes there aren't enough people to go around. This can be a source of friction.

• Negotiate how it works and take care onboarding. A new embedded person has a more complex situation than new employees do. They have another manager, and things outside your team they may be paying attention to. Kick off the relationship with explicit conversations. Spend twice the care you would with onboarding a generalist employee.

• Keizan Shaffer offers this advice: “One thing that I found helpful when embedding staff engineers was to conduct a standard Kick-off process with a Statement of Work as output. Early on I learned the hard way that leaving success criteria loose leads to lingering engagements / disappointment / confusion.”

• Be clear who directs their work. When negotiating an embedded situation, decide who directs their work. It can cause problems if both the home and work team managers assign work. This makes their output less predictable. Even worse than that, it causes surprises because it seems predictable until it isn't. This can make it hard to depend on the embedded person.

• Clarify what type of work they’ll be doing. Another thing to be specific about is the type of work you’re expecting them to do. For example, is the SRE joining the team to do operational work? Or are they there to help the team get better at operational work?

• Many companies focus on cross training and T-shaped people. They do this to create more flexible teams. And they see value in spreading that skillset within the team. If you can, make it explicit that the specialist won't do all the work. The goal is for them to help the team get better at that type of work. This won't always make sense. Team members can't internalize every specialty.

• Good role definition can help clarify expectations. Develop role definitions. Review expectations when they join the team. This will help both them and the team know what to expect.

• Maintain good communication between managers. If you’re the manager of the embedded person, check in with the manager of the team your person works with. Set up a regular cadence for checking in. It’s good customer service, and it’s also vital so you can offer good feedback and coaching to your direct report.

• For the embedded person, they are living in a sort of matrix-managed world. Although they don’t have two managers, in practice it can feel that way. They attend two sets of meetings. And their manager isn’t seeing their work.

• As a manager of an embedded team, make sure you have a way to assess your team members’ work and coach them. They need career development and support like any other employee.

• Consider a "send back the embed" clause. If you're the recipient of an embed, one of the bigger challenges you might face is if there are performance problems. I've faced the challenge several times where the home manager was defensive of their embedded person, and prioritized that over listening to my feedback about performance challenges. They're often not as close to the work, and they may be inclined to let it be your problem. Yet you lack the ability to hire and fire the right person, so when you truly are facing skill gaps or performance issues, it's a tricky issue. From a systems perspective, I favor having a part of your embed agreement be that you reserve the right to "send back" anyone having performance challenges. The home team manager can decide whether to let them go or put them on other teams, or what to do about it. I've seen poor embeds cause immense damage on teams -- the diffusion of responsibilities can make it extend too long.

• Be careful of meeting load. Embeds will often have a higher meeting burden than other individual contributors. The easiest way to handle this is to keep their home commitments light.

• Watch out for embedding with multiple teams. Embeds with multiple teams have an even more complex situation. They have three or more teams they’re a part of, and they have a lot of relationships and complexity to navigate. It's usually best to pair them with the organization's leader. That way, the org leader can direct their work. This can help simplify some of the complexity they’re dealing with.

• When you don't align an embed with an org leader, it can get complicated. When a reorg occurs, you'll have to do your own reorganization. Even if less optimal, keep your organizational design aligned with the embed's organization. I've seen reorganizations with three separate hierarchies. It was a complete mess.

Embedding versus other coordination models

Single threaded owner: why embed when you could hire that skill directly on to the team?

One way to handle embedding is to take it even further and actually put the embedded person on the team. This has a lot of advantages, and some disadvantages as well. I wrote up an experience report on this model in The Single Threaded Owner model. See it for more details. It’s arguably a better approach. You should combine it with a strong Community of Practice (post forthcoming). You also need good role definition and career ladders.

Usually, you won’t have the latitude to do this. The designers will need to be on the design team. If they aren’t serving you well, you may consider hiring a designer onto your team. A sneaky way to handle this is to hire an engineer who has a design background or who is design-focused. Doing this as an organization can be difficult, so it's often a one-off approach.

Merged group: let’s do DevOps!

You can combine departments that historically passed work between each other. The classic version of this is to combine developers and operators. Eliminating the departments and having them on the same team improves flow. This approach is like the Single Threaded Owner model, but less extreme. You're not doing it for every specialty, but one.

This has similar tradeoffs, so read the STO model post for details.

If your eventual aim is to have a merged group, it can often make sense to start with embedding. For example, let's say you have a centralized operations team. You want to eventually have teams own their own operations. You could move to an embedded model to uplevel the development teams on operational practices. Then, you can move a merged group later.

Service provider: have them come to you instead?

I generally prefer embedding to a service provider approach. Most service providers lack the context and relationships to collaborate with teams.

Prefer the Service provider model when you want to scale your organization slower than the organization you’re serving. It’s also preferable if you offer a service where you don’t need much context from the team you’re serving.

Liaison: I don’t need to actually WORK with them, just want to spy on them

If you’re just gaining context from them, you don’t need a full blown embedding regime. Just use a liaison, and attend their meetings. Make your home team your main team.

Rotation: when context is less important, or you want to share the burden

If context is less important, a rotation may be preferable. This doesn’t build as deep of connections between people, or ease communication as much. But it does share the burden, so it can be useful if parts of where you serve are really hard work or are uninteresting. Some teams may also enjoy the variety of a rotation. Stay tuned for a post on this.

Objective expert: give the central team leverage!

When pairing the embedded model with a small centralized team, it can be useful to have the central team use the Objective expert model. A reliability organization might have a central SRE group that is in charge of reporting on reliability for the organization, broken down by team. This can drive a lot of work in other teams.

Coordination models

Embedding is just one of many coordination models. Coordination models give you a menu of choices to choose from when solving your inter-team coordination issues.

Feedback

See anything I missed? Disagree with this? Please let me know your thoughts!

Thank you

Keizan Shaffer provided a lot of feedback on an earlier version of this post, and had some good suggestions on how to embed effectively.

Image by PublicDomainPictures from Pixabay

Platform teams underestimate how much pain they cause when they deprecate their offerings. You should make deprecation rare and go through a process. Or the team should do the deprecation work itself.

Deprecation improves life for a team, but at an organization cost

A platform team’s natural incentives don’t match the needs of the wider organization.

The team will want to keep their code orderly and easy to work in. They will want to reduce complexity and improve things. Deprecation make life better for the platform team.

Yet, this is dangerous. It is easy for a platform team to create work for the rest of the organization. When they deprecate something, they are creating work on many teams. This type of “fan-out” work is dangerous, because it is expensive and hidden. I have seen product engineering focused 100% on platform projects. I've seen company-critical product work delayed due to lower priority platform work.

This is not because the platform teams are dumb or bad. Their incentives aren't set up right. Platform teams exist to speed up organizations, not to slow them down. So their incentives need to align. Externalizing costs is a moral hazard.

Deprecation is not customer centric

Deprecation is the illusion of communication. It appears to be a kind warning of upcoming changes. But it transfers responsibility for your internal problems to your customers (other teams).

Shifting problems to your customers is the opposite of customer centric. You are requiring them to do work for you.

When you deprecate, you prioritize your work over the whole organization's work. This is also not customer-centric, because the company's customers may lose out. You are making a choice with an unknown opportunity cost. For all you know, you could be making a decision that will kill the company.

Sometimes deprecation is necessary

So this may sound like a rant against deprecation, but it is true that deprecation is often necessary. Sometimes technical choices paint you into a corner, and deprecation is your only way out. The more a team has to maintain, the larger the surface area for vulnerabilities. And you do want to provide a consistent experience for a platform. And sometimes your team struggles to maintain legacy services that nobody understands. Having an unsustainable team load isn't realistic either.

Start with taking this as an opportunity to learn. What decision did we make that required deprecation later? Even if someone else made those decisions, you can learn from them.

And when you do deprecate, you should take great care to minimize the cost on others. Provide simple upgrade paths. Support legacy approaches for as long as you can.

How to handle deprecation

You want to have a system where you can be rational and discuss the impact of deprecation. You also want to minimize externalizing costs.

The two ways I see to solve this are:

Make it hard to deprecate things. Architects or engineering leaders can approve deprecation, for example.

Make platform teams responsible for doing their own deprecation work. This is my favorite approach.

With this approach, if a team deprecates an internal API, they do the work to eliminate all use of that API. If they upgrade a library in a breaking way, they do all the upgrades in their clients.

If you do this, platform teams can deprecate all they want. They're aware of the cost of deprecation, and they decide when it makes sense. They'll desire inexpensive and easy upgrade paths. This results in rational prioritization.

Doing this requires funding platform teams at higher levels. They will complain they lack the skills to work across codebases. When you hear that, ask how they can address that problem. Long term, that will be better for the organization.

If the platform team is unable to do the work, they should at least organize and work with the teams doing the work.

Feedback

See anything I missed? Disagree with this? Please let me know your thoughts!

Further reading

For an excellent approach to API deprecation that minimizes user pain while maintaining technical progress, see Will Larson's article on API Deprecation Strategy.

Thank you

Ben Bernard provided feedback on an earlier version of this article, and gave me insight into what he's seen at Amazon and Google and other places around deprecation. At Amazon they called this "owning your clients". Ralph Bodenner and I talked through the incentivizes and challenges of getting companies to work this way, and some of the failure modes we've seen in the past. He provided feedback on an earlier version of this post. Marcos Wright-Kuhns pointed out some other aspects of deprecation, like security and consistency, that helped me flesh out the "sometimes deprecation is necessary" section. Eric Dobbs shared a helpful talk from Laura Nolan on how to choose to kill or migrate something in place (registration required).

Image by wei zhu from Pixabay

Today, we'll talk about the use of Liaisons.

What is a liaison?

A liaison is a person who serves as a coordinator with another team or group of teams. They attend relevant meetings, and share context in both directions.

For example, product marketing has a dependency on engineering, product, and design. Product marketing needs to know many things to market to customers:

• When the team will deliver the feature.
• What benefits the feature provides.
• How the team talks about the feature.
• What terms the team is using.
• What the feature will look like.
• How the feature will fit into the existing offering.

A liaison can be very effective in this situation. You attach a product marketing person to a team or set of teams. They attend meetings and learn the context they need. And they also share that context back with their organization.

When to use liaisons

A liaison is one of the best coordination models to use when you meet these constraints:

• You are coordinating with another part of the organization.
• That part of the organization comprises a team or set of teams.
• You believe the liaison will be stable. They can be associated with a team or set of teams and that won't need to change for a long time.
• Your need for context is more than minimal. If you can replace a liaison with notes from a meeting or an email, you don't need a liaison.
• They can get that context from meetings. If the meetings aren't sharing a lot of context, it’s wasteful to have a liaison.
• You won't exceed the rule of eight. Attaching a liaison adds a person to the meetings they attend. You are adding a cost to the meeting and making it more complex. If the meeting gets too big, you have to restructure the meeting or choose another model.
• There is a natural way in which your people fit into other teams' meetings. You should have a direct need for context. You are misusing this model if you sprinkle liaisons in many places. If you feel a need for liaisons everywhere, you might have other problems. That can be a sign your company has bad information flow. Look for signs there is something structural at work.

One of the advantages of using a liaison is that it is inexpensive. Using a liaison only requires a few meetings a week for one person.

You can scale liaisons as the organization grows. For example, our product marketing department might assign a liaison per product line. These product lines map to engineering teams or organizations. When the company introduces a new product line, you hire a new product marketing person. This person is then attached to the new engineering group.

When using this model

• Make sure people understand what the liaison is doing. Using our example, many engineers don't understand product marketing. They will have no idea why a product marketing person is at the meeting. When you add a liaison to a meeting, tell everyone what their needs are. Explain why they are there and what information they need.
• Use the existing hierarchy if you can. When you map your liaisons to other departments, use the existing hierarchy. If you don't, you make the organization complex. I have seen situations where there are three or four mappings and hierarchies at the same time. Using the existing hierarchy reduces complexity during reorgs.

Liaison versus other coordination models

A liaison is a limited version of the embedded model. When you need deep context, embed someone instead. When it makes sense to work together, prefer the embedded model. If your need for context isn't significant, use a liaison instead.

When you want someone to be always available, a Rotation can be preferable. Rotations build less context. They also share the burden of being available better.

A collection of liaisons from many places forms a "Centralized Liaison". For example, you might have each team send their technical lead to form a technical leadership group. The technical leadership group then shares information and coordinates with each other.

My advice:

• Use liaisons with product marketing.
• Use embeds or liaisons with security.
• For support teams, use liaisons. Embedded support engineers would be a good experiment. For engineering teams, engineering Rotations with the support liaison create a natural pair. This pair can triage and sometimes solve customer problems together.

Next steps

Any important advice I missed? I love feedback.

If you liked this post, subscribe to hear about future posts!

Coordination models

Liaisons are just one of many coordination models. Coordination models give you a menu of choices to choose from when solving your leadership coordination issues.

Image by Couleur from Pixabay

Goal frameworks are a way to create a set of goals within the organization that attempt to be coherent across all the groups that need to contribute to that goal. The most common framework is OKRs, but you also see V2MOM and homegrown goal frameworks. You can even just track a list of goals across all departments. The military has a system called Commander’s Intent that you might be interested in, as well.

In this post, I talk about some general advice for frameworks, and in particular cover why not to use them for managing projects.

When to use goal frameworks

Goal frameworks are useful when you see a lot of confusion or a lack of coherence in what people are working towards. They allow local goals to be aligned more with the central goals of the organization, and can help rationalize conflicting objectives and align parts of the organization that need to coordinate their work together.

Plan for about half a quarter, not a full quarter

The biggest error people make with goals is they overcommit and underfocus. A good rule of thumb is to choose a few things that represent about half a quarter’s work, and make those your goals. You’ll have other stuff you have to focus on too, and you’ll have emergencies. You’re probably not accounting for that, so this helps you think critically about what you can accomplish.

Is it theater, or is it real?

When you’re using a goal framework, it’s usually a good idea to take it at face value. If you feel like you’re in an organization that is political, then you should bias towards goals that are more conservative -- after all, you’re making your own grading rubric.

In a higher trust organization, or when you have the privilege to not worry about your position, make goals that think backwards -- what should be true? I like to ask what the situation calls for long term, and work backwards to what needs to be true in the next quarter.

Reconcile priorities with sister organizations and teams

All of these systems can be useful because they force conversations about priorities and what you’re trying to accomplish. A lot of the value comes from improving coordination across separate parts of the organization. For example, you can make sure that the marketing organization and the engineering organization are preparing for the same big changes. So be sure that when you use goal frameworks, there is some sort of review of what other people in neighboring organizations and teams are doing. You’ll often find misalignment, and you can fix it early.

Don’t use goal frameworks to manage projects

There can be tremendous pressure on engineering organizations to use OKRs to run projects, and in general I think that’s a bad idea. My favorite way to use these systems is to drive improvements in the way engineering is operating, rather than to drive the work itself.

The way it typically works when you use OKRs to run projects is you ask each group what they commit to delivering in the quarter. Then these become OKRs for the quarter. What could be wrong with that?

• Listening to new information and changing your mind is penalized, while executing on the plan is rewarded.
• Projects that cross quarter boundaries are penalized, while projects that don’t aren’t.
• Projects that are new are penalized (even if they’re 100x more valuable), while projects that are the current plan are rewarded.
• Projects that are higher risk are penalized, even if they’re more valuable.
• Projects that learn from customers and incorporate feedback are penalized, while safe projects that are predictable are rewarded.
• Projects that do novel discovery and innovative work are penalized, while projects that lack innovation are rewarded.

Like a lot of management challenges, you’re creating incentives that can backfire. When you incentivize these things in a goal framework, you’re removing flexibility because you’re tilting the cost of change in one direction. Priorities change, and most product development work is variable enough that OKRs aren’t the most effective way to manage the work. You can manage projects better with other methods.

Instead, I like to use goal frameworks to improve the way engineering functions. Instead of an OKR on Project A, l might focus an OKR on reliability or testing or improving our hiring process.

You’ll often find this isn’t negotiable (the CEO might insist on it or there might be a need to coordinate across departments), so I just try to minimize it to the extent possible. One trick I have found useful here is when someone is pushing for a specific deliverable to be a part of the goals for the quarter, I ask what the outcome the project is intending to drive. That can provide some flexibility in how you reach that outcome.

I’ve found you don’t get out of projects in OKRs for free. Unless you provide your own system for tracking and communicating the status of projects, people's information needs won't be met. Then, the goal system will seem like a solution to fill the void.

Make goals fluid

If things change, should you update the goal, or leave it? I believe it’s a common mistake to focus on accountability more than having the goals make sense. If you don’t change the goals, then every meeting you’re reviewing something stale. My advice? Update your goals as you have new information.

The alternative to this results in a situation where people are reporting on the same dead goal every week. It becomes ridiculous -- but I see this all the time.

Consider trying “2-up” for organizational awareness

One goal framework you may not be familiar with is the military’s use of commander’s intents (abbreviated CSI for some reason). The military is a surprisingly good source of information on management, and this is no exception.

Commander’s Intents are expressions of the desired end state and key tasks that, along with the mission, are the basis of the subordinates work. “CSI acts as a basis for staff and subordinates to develop their own plans and orders..., while maintaining the overall intention of their commander. [CSI] is also a concise expression of the purpose of the operation. CSI may also include the commander's assessment of the adversary commander's intent and an assessment of where and how much risk is acceptable during the operation.”

One thing I particularly like about this approach is every subordinate is expected to understand the intent of the organization two levels up. I’ve never used this “2-up” system, but it seems like a potentially better system than OKRs or V2MOMs -- I’d be curious to hear from anyone that has experience with it. I like that people aren’t expected to internalize the goals from every level of the organization.

Reality is complex, goals are simplistic

One of the bigger challenges with goal frameworks is that they simplify the situation. This is both desirable (as they simplify what to focus on, and get people aligned on that), and a curse (if you look back with a critical eye, you’ll see most goals are trash).

Bjorn Freeman-Benson: “One of the problems of these goal frameworks is that they are used to be over-precise and that over-precision then limits the decisions that the lower levels of the organization can/should make. And thus you end up with a suboptimal overall result. [You want to] set goals without overly constraining how the lower levels of the organization should reach them.”

He gives this as a hypothetical example:

If team A's goal is 100 widgets then team A will work really hard to build 100 widgets. But if the real higher goal is 12 happy customers and the higher level just decided that 100 widgets delivered will create 12 happy customers, that constrains the lower level from doing something different to create the 12 happy customers. For example, maybe we can actually get happier customers with 50 widgets and then help the customers install the widgets? But if I was measured by an OKR, I couldn't do that because I'd miss my numbers.

Ideally, you want the local team using their local domain expertise to come up with an optimal solution. If you find your local teams aren’t able to do this, you may want to be more proscriptive. But you may also want to consider how to develop that capability on your local teams, as command and control isn’t scalable or effective.

Have a cadence for information sharing against the goals

All of these goal frameworks are intended to get people to understand what the organization is doing and why. And typically there is some cadence of sharing information on how everyone is doing against those goals, to get people to coordinate their activities better.

I’ve also found it useful to use goal systems to improve information flow within the company -- it can be really good to know what is going on in other departments, and be able to see if everyone is coordinating or not. Having a single place where you can see what everyone is up to is genuinely useful.

Typically the format for this is to report on how we’re doing against the goals, every week. This can be in a written location, and sometimes also done verbally. Sometimes leaders will only talk about things that aren’t green, if you’re using a “green/yellow/red” system.

Messy is fine

Goal systems tend to be frustrating and never feel perfect. Many organizations struggle to complete them on time, and they require a lot of planning and investment. My general advice is to expect it to be messy, but focus on the conversations that it will generate. You’re aiming to improve the conversations and decision-making, and that’s actually probably more important than the goal-setting framework itself.

Consider starting lightweight

It can be tempting to start right in with OKRs or another framework. But you might get a lot of the same value, without as much process involved, by just writing down what everyone’s goals are and talking through them each week in a leadership meeting.

Setting goals well

I really like this post by Molly Graham on using goals effectively. She recommends:

• Not using more than three company goals.
• Have one goal that is most important.
• Make the goals be dead simple.
• Strategy should hurt (I cover this in some shoulds for strategy
• Each goal should have an owner.
• Goals are just the start, you need structure to make them happen.

Feedback

Any important advice I missed? Let me know!

Coordination models

Goal frameworks are just one of many coordination models. Coordination models give you a menu of choices to choose from when solving your leadership coordination issues.

Thank you

Thank you to Bjorn Freeman-Benson for pointing out the problem of over-specificity as a challenge with goal frameworks. And to Keizan Shaffer for suggesting I add an example to the “don’t use goal frameworks to manage projects” section. And to Gustavo Aguiar for edits and feedback. Alex Kroman taught me some of the rules of thumb here. Merlyn Albery-Speyer gave me feedback, suggested a different (and better) title, and pointed out typos. Ian H Main pointed out some awkward language, and provided some helpful feedback.

Feedback

See anything I missed? Disagree with this? Please let me know your thoughts!

Image by Sasin Tipchai from Pixabay

Self-service is arguably the most important team coordination model you can use. If you are a leader in any moderately complex engineering organization, you should be actively pushing for self-service. Otherwise, you’re making a problem for someone else to clean up.

What is the self-service model

In a team that adopts self-service:

• The goal is for people to be able to use the team’s work products without talking with the team.
• The team treats their work like a product, and provides high-quality documentation, APIs, and tutorials.
• Every interaction with other teams is used as a learning opportunity. While they may not refuse to talk with other teams, the goal is to not need to do so.
• The team continually reduces the complexity of interacting with their work, with the aim to make it so easy to use it’s preferable to any alternative.

My favorite example of a team adopting this pattern was from New Relic. The database team started out as a service provider team, spinning up databases and setting up replication and backups for teams within New Relic. When they realized the value of self-service, they made a long-term commitment to move to a self-service model. They gradually automated and streamlined spinning up databases, and provided self-service documentation and tooling. Eventually, they were able to offer databases as a service within the company. They effectively removed themself as a bottleneck, which improved the efficiency of every team. This also reduced the communication needs for the team, so they were able to focus more and more on other high leverage work.

Self-service teams can provide a variety of offerings:

• Services, with APIs.
• Tooling, like deploy or testing tools.
• Configuration and defaults that make security, linting, monitoring or costs easier to manage.
• Libraries or components.
• Infrastructure.

When to use this model

I generally recommend to start using self-service as soon as you have platform teams. Product teams can have a self-serve approach too, but it’s something really to emphasize with platform teams.

Pros

• Highly scalable. There isn’t a better coordination model that I’m aware of for decoupling the teams within an organization than self-service. It dramatically reduces the coordination requirements between teams. It’s one of the few things a company can do to get a “compounded interest” type of productivity out of engineering.
• Customer-focused. Using self-service requires teams to think about and listen to their internal stakeholders. This can be rewarding for engineers who like to see the impact of their work. And this can often mean the overall Platform organization will be more focused and effective.

Cons

• It takes a lot of effort. Getting something to be truly self-service can often be a long journey. And the team’s existing fires and “service provider” work can compete with the efforts to build self-service tooling.
• It’s more expensive. The short-term costs of building self-service are greater than the alternatives. You’re trading off long-term scalability for a greater initial expense per capability. Switching to self-service can often require significant architectural changes, or require work from other teams to enable it.
• It requires long-term thinking. Because it can take a while for a team to make the journey to self-service, it can require long-term thinking. And in many environments, long-term thinking isn’t rewarded, or can be scarce.
• It can require wider skills from the team. As Merlyn Albery-Speyer put it: “Engineers more towards the ‘I built the thing I was told to build’ end of the spectrum will struggle on such a team or worse hold the team back.” This can require changing the way you hire and the skills you look for.

When using this model

Treat your offering as a product

“My advice is to treat the platform like a product. What you want is the platform to be so useful that people onboard without having to make it a mandate. Too many times, I see people building platforms and not doing the user interview work required to make sure it is actually solving a problem for the internal customer it serves.”
-- Aaron Erickson

When you treat your offering as a product, you engage in a different focus than when you focus on “building a platform”. You focus on solving your customer’s problems. This implies your team...

• Spends time with the teams using your offering, understanding their needs.
• Can accurately represent the relative value of requests they hear. Merlyn Albery-Speyer: “My first lesson was to not build anything for any single team’s needs. A test question to ask your customers is: if your team had to build/fix this yourselves, how much engineering effort would you be willing to invest in it? If the response is: ‘well, none of course!’ Then they’ve indirectly answered how valuable the need was to them.”
• Uses adoption and usage metrics to learn how people are using your work. Like a team launching a product, you carefully watch adoption, and think about the entire experience using your product.

I favor having a product manager on platform teams. However, you can also have a technical leader act in this capacity, or have the engineering manager do it. Generally which way you go on this depends on the complexity of the environment, and the time it takes to do this job well. However you structure this, you should endeavor to have the whole team be as customer focused as possible.

Don’t just create great documentation

“Too many folks think good self-service is all about extensive documentation. While it can be true that great self-service offerings have a lot of documentation, designing a thing intentionally to eliminate questions and doubt is the high art here.”
-- Merlyn Albery-Speyer

How do you do this? You need to radically reduce complexity in the user experience. When exploring a solution, John Hyland has a wonderful talk (So Simple A Well Trained Artichoke Could Do It) on an approach to building out a solution that minimizes the number of steps it takes to accomplish your goals -- I’ll post it here when it’s publicly available. In this talk, he advocates for an iterative approach where you create an implementation and then refine it multiple times, each time removing steps you have to go through to use your API or tooling, and reducing the amount of outside knowledge required. You need to actually go through the steps the user will go through. The user experience isn’t adornment -- it’s actually most of the value you’re providing, so spend a lot of time on that.

Create a place to find platform capabilities

How will people know about and take advantage of the capabilities of the platform?

If this doesn’t already exist, bootstrap a place where platform capabilities can be listed. You can start with a central place in the wiki, or in Github. Include things like how the build and deploy system works, create a place for APIs and libraries available to you. And include links to your design system and component library. These are things that should improve over time, so reward people who spend time improving them.

When that is in place, you should also find a way to broadcast new capabilities, and for customers of a capability to hear about updates. Mailing lists or Slack groups can work for this.

Move to self-service incrementally (consider a Wizard of Oz or Scripting approach)

Moving to self-service often takes a very long time. One thing to evaluate is how you’ll get there incrementally. Two approaches to consider:

• Wizard of Oz: if your team offers a service that can theoretically be automated someday, consider a Wizard of Oz approach. The database team at New Relic took this approach, I believe. Teams used to come to them and ask them to spin up a database for them. They knew that their end goal was to make that automatic. So they created a webform that people could use to request a new database, and gradually automated the steps behind it. To their customers, it took less and less time. After a year it still took manual work for the database team, but only a 5 minute process to create most new databases. The idea with this is to gradually fill it in, but start out with a service provider approach. The Wizard of Oz approach is similar to the Facade design pattern in object-oriented software design.
• Script it all: if you have a bunch of steps that people take, that you believe can eventually become automated, then you can create a script that basically walks people through the steps they need to do. For each step, they can do what they’re supposed to, even if it’s not automated. Over time, you can eliminate and automate steps, making it more efficient over time. Ada Cohen used this approach at Gremlin to improve a deploy process. She started with a list of steps in the wiki, and turned that into a script which told you step by step what to do. Then, she began automating and reducing steps in the script. This allowed her to roll out improvements incrementally and improve the deploy process significantly!

Minimize or even outlaw deprecation

You don't want platform teams to deprecate things. Teams shouldn't create massive work for other teams. This is a controversial topic, so I wrote a whole post on it.

Consider extending your offering to internal and external customers

One of the more interesting things about how Amazon operates is that they don’t distinguish much between internal and external offerings. For example, they offer the same trainings to internal employees and customers.

A useful exercise can be to imagine your team’s capabilities as a product that also goes to external customers. Would people pay for it? How would they interact differently with it? Conversely, imagine if you could replace your team’s offering with something that is available externally.

Thinking through these things can help you solidify your thinking of what you’re offering, and help you make sure that your team’s offering is essential and valuable to the company.

Mandate as little as possible

Self-service teams often focus on creating a “paved road” or “golden path” that makes doing the right thing the easy thing:

• When you mandate everyone use something, you distort your ability to measure if you’re being useful to your teams. And you can get poorer signals of where the problems are.
• Perhaps more importantly, mandates can be a form of coupling. They introduce harder dependencies, and force your team to be on the critical path for more.

Sometimes it is necessary to mandate the use of an offering. For example, you don’t want to have multiple SSO implementations, and it’s legitimate to require security in implementations.

One way to handle this is to have layers to your abstractions, and let your customers go into deeper layers if necessary. Aaron Erickson: “One painful lesson we learned was that if you are too opaque with your abstractions and can't break them when needed, it becomes very easy to box yourself into a corner.”

It shouldn’t feel like it’s imposed

If you’re seeing teams resisting your offerings, this is usually a sign something is wrong. The ideal reaction when you announce a newly available or upgraded offering is, “thank you, this is going to make my life so much better!”. When you’re not seeing that, dig into why.

Consider a plugin approach, and make it easy to contribute

One person I worked with at Gremlin, Ben Bernard, had a huge impact on the internal tooling at Amazon. What he did was he made a set of developer command line tools that were really handy. But more importantly, he made it easy to contribute to these tools and add new ones. When he came to Gremlin, he used this pattern to quickly spin up a set of incredibly useful tooling. But the developers found it helpful because they could contribute to and improve these tools themselves.

Plugins can be important with self-service because plugging something into an existing model requires less centralized work than doing the work in that centralized thing. For example, imagine a service that does A, B, and C versus a service that you can plug in the behavior for A, B, and C. The latter model will be easier to extend without involving the central team. So think carefully about anything you provide which will require you to make updates for other teams to use it.

Self-service versus other coordination models

The natural pair for self-service is the independent execution model, which I’ll write about in more detail later. In this model, teams don’t make plans for work that depends on non-existent capabilities. Or more precisely, they can make those plans, but they expect those plans to be high risk, and always plan for fallbacks where those things don’t occur.

Two other coordination models that are natural fits with self-service are embedded and consulting models. With self-service, you have a strong need to stay connected to the needs of your customers. So having team members embedded in other teams can be a useful way to see what their pain-points are. And if you provide a way for people to consult with your team when they have problems while using your work, you gain a valuable source of information on where your offering is falling short of the self-service ideal. So embedded and consulting models can be good complements to the self-service model.

Coordination models

Self-service is just one of many coordination models. Coordination models give you a menu of choices to choose from when solving your leadership coordination issues.

Feedback

See anything I missed? Disagree with this? Please let me know your thoughts!

Thank you

This post benefited heavily from the feedback and thinking of many people with deep experience building Platforms. Jim Shore introduced self-service to me. I’d like to thank Merlyn Albery-Speyer for offering a huge list of things he’s learned over the years building self-service teams, many of which became key points in this article. Aaron Erickson shared his experiences with platforms as well, which helped cover some points I wouldn’t have thought to include. Keizan Shaffer had a lot of wisdom to share on things to watch out for and in particular some of the challenges of moving to self-service. Marty Matheny double-checked my memory about the database team at New Relic and helped me see how they had gone about moving to self-service, which helped flesh out the Wizard of Oz pattern. John Hyland has taught me a lot about building platforms (as we built them together!), and provided valuable feedback on this article. I learned about the importance of the plug-in model from Bjorn Freeman-Benson and Kevin McGuire. Ben Bernard and Ada Cohen were some of the most effective people I’ve seen at being both customer focused and building developer experience tooling, so watching them in action helped me see some of the patterns I’ve mentioned in this post.

Image by Michel Bertolotti from Pixabay

The Service Provider is one of the most common and least effective coordination models in software engineering teams. There are legitimate reasons to use this model, but you should avoid it unless you’re aware of the tradeoffs.

What is a Service Provider?

When a team is a Service Provider, they...

• Have valuable skills they offer other teams.
• Are a dependency for those other teams. They do work for those teams.
• Do their work per ticket, per project, or per initiative. When their work is over, they either do work for someone else, or work on their own priorities.

Service Provider is a coordination model.

What types of teams commonly adopt this model?

• Design.
• IT.
• Infrastructure and platform teams.
• Functionally organized product teams. For example, a backend team might work on various APIs depending on the requests and needs from other teams. Or a frontend team might do work for various projects.
• Some cross-functional teams use this model, where they request things of each other in order to accomplish their own goals.

Pros

• High utilization and efficiency. Allows you to have a specialist that can apply their skills across a group of teams that may not need those skills continually.
• Cost-effective. You can often have less people serving a larger group of people than the alternatives.
• Maps nicely to org structure. It’s often what a manager will naturally think of, so it’s what you will end up with if you don’t design something different.

Cons

• High latency. This pattern always leads to poor flow through your company, because to avoid doing so would require you to have excess capacity, and there is almost no tolerance in companies for having people “without work to do”. But this results in problems. You’ll have projects that can’t be delivered because your infra team is busy on something more important. You’ll have projects without a design and it will either be built poorly or delayed. You’ll have people unable to work because their machine is having problems. Leaders tend to underestimate the cost of latency. A way to avoid some of this trap is to have something else valuable that the team focuses on, so being a service provider is only a part of what the team does. If it’s easy for them to swap out that other work, then they can have spare capacity and consume the queue of work.
• Underfunded. Funding for service providers tends to be treated as a cost, and often tends to get underfunded. So these teams will often struggle to keep ahead of a neverending queue of work. Scaling is typically done independently of other parts of the organization, which can lead to funding levels that are enough to avoid emergencies, but not enough to make your organization thrive.
• Long tail of request servicing can be quite high. Service Providers focus on prioritization. They often have ranking systems, and ways to pull out the most important work to be done. This results in a long tail of request servicing that can be unpredictable and cause large amounts of downstream thrash. For example, an infrastructure team might not get to a lower priority request for a month, because they have more important things to do. But the downstream impact of that might be a team working less efficiently than it could. Because the impact of these downstream prioritization decisions can also cluster, they can cause certain parts of the organization to become ineffective.
• Hard dependencies scale poorly. This model is a hard dependency, so it prevents teams from being able to deliver value independently. As the organization grows, complexity grows as well, and you’ll find you’re creating a lot of structure just to manage the dependencies. This can lead to excess process and people spending a lot of time just managing the unpredictability. You want to minimize complexity as your organization grows, and the Service Provider is a leaky abstraction, which allows complexity to dominate.
• Urgency can have unpredictable impact. If a new top priority comes in, it is usually a zero sum game, and some other priority will be cancelled. This can lead to unpredictable project performance across the organization, because the impact of these changes can ripple through. In some organizations I’ve been in, this results in “only the top priority project can ever be delivered on time”. If Kim the designer is working with a team, and something important comes along, then it will always be tempting to move Kim to a more important project. But this means a totally unrelated project is now in peril. So this structure can make product development more unpredictable.
• Experts can fail to develop context and depth in the areas they’re helping. This pattern also results in service providers that can lack context for the areas they’re helping, so this model is more effective in homogeneous or simple environments. For example, design teams who work across multiple products, or across broad parts of the product, may not develop the deep technical understanding of their area that will help them be more effective and innovative. They also fail to develop deep working relationships which can produce more effective results over time.
• A lack of slack can lower innovation. Since you’re moving from task to task or project to project, you often don’t get to see the larger patterns in an area, and have the time to explore larger impact changes to improve in those areas. Your work will tend to be more low level and tactical.
• Hard to manage incoming requests. It can be hard to understand the context of the requesting teams, and understand how the work maps to the value delivered for the business.

If you use this model

• Evaluate how realistic it is to service the needs of the organization. You can often use measurements to keep an eye on things -- turnaround time for requests, or 95 percentile for turnaround time. If your engagements are longer-lasting, interview people and make sure you’re aware of the impact of projects that don’t get prioritization. Try to be conscious of the impact of latency, and of shifting priorities. If the team isn’t set up in a realistic way, it’s time to have some serious discussions about its future.
• Use automation projects to give your team higher leverage. This is a balancing act, because you have to balance the need to help people with the need to get automation work completed. Sometimes the only way to dig your way out of the hole you’re in when using the service provider pattern is to make your customers less happy while you automate the improvements.
• Communicate prioritization and tradeoffs. You want people to understand what they can expect from you and what they can’t. A big source of frustration dealing with service provider teams is not being able to predict whether they'll be able to give you something and how long it will take.
• Switch to another model. There are often better coordination models you can use, that you can gradually transition to.

Service provider versus other coordination models

• I try to avoid this pattern whenever possible. I prefer embedded and consulting patterns with designers, self-service and consulting patterns with infrastructure, and a centralized liaison pattern with architecture. For software teams, I tend to prefer cross-functional independent executor teams, or self-service. For IT, I’m not an expert on what makes sense -- let me know if you’ve seen alternative patterns that are more effective, or if a service provider model really is the best fit.
• If you can move from being a hard dependency to a soft dependency, that will make the organization more effective. You can sometimes do this by switching to a consultant model. For example, having teams do their own infrastructure work, but having an infrastructure team there to help with hard problems, can be a better model than having one team do infrastructure work for everyone. Ideally, with the consultant model you’re not actually doing the work, just guiding people on what needs to be done and best practices. There is a lot of nuance to this, we’ll discuss more in the Consultant post (soon).
• One way to make other teams’ dependencies on you softer is to give them a default approach when you’re not able to help them. For example, designers might have a design library, and encourage people to “do their best” when design isn’t available to help. (A design library can be automation for designers). You can tell people not to plan on your team being able to do work for them unless they’re a “top 3 priority”. This is not at all ideal, but it’s better than the alternative of them planning for your help when it’s not realistic.
• It’s often best to have a long-term goal in mind which is a different coordination model, like self-service. You can often scare up investment by selling that vision and showing progress towards it. This can take a long time to transition to, but be a highly valuable improvement. Combining self-service and a consultant approach can actually work pretty effectively. You get customer awareness by being a consultant, but continually aim to make your work product self-service. This can also ease some of the transition pain to getting to a self-service model.
• Functionally designed teams tend to move towards the service provider pattern. Sometimes this makes sense: if you have a lot of code that isn’t worked on frequently, or something that requires deep expertise or has highly demanding reliability requirements, or a different form of engineering. But barring those exceptional cases, I tend to make teams cross-functional in nature, to eliminate the need for a service provider pattern, and improve the coordination within a team.

Coordination models

Service providers are just one of many coordination models. Coordination models give you a menu of choices to choose from when solving your leadership coordination issues.

Feedback

See anything I missed? Disagree with this? Please let me know your thoughts!

Thank you

Image by Gerd Altmann from Pixabay

Tenets are mental shortcuts that help an organization make decisions faster. They are a way to bias the decision-making of an organization in a particular direction.

An example tenet might be, “we build our software to scale by ten times our current baseline traffic”. This tenet helps reduce the number of decisions people have to make in the future, and aligns the organization around HOW we do things.

When to use tenets

Tenets are valuable when there is a particular way of thinking you want to encourage. If you see people over and over discussing the same tradeoffs of HOW to do something, you might have a good candidate for a tenet. For example, if you see engineering teams always talking about how much to invest in testing, you could create a tenet to give clarity about it (a no-bullshit version of it might be something like “we value tests so much we’ll push back project deadlines to make sure it happens”.

How to write tenets

A good tenet should have a perfectly valid opposite tenet that would make sense in a different context.

Example #1:

• “We build things in a cost-conscious way, even if it takes longer to build”. Vs.
• “We are willing to throw money at problems if it speeds us up”

Example #2:

• “We value clarity over moving quickly” Vs.
• “We value taking action over analysis”.

It’s often good to have a “We value this, over this” structure to the tenet. But you can also use a structure where you’re explicit instead. For example:

• We build our software to scale by ten times our current baseline traffic. Vs.
• We build our software to scale by twenty times our current baseline traffic.

Both of the variations are valid, so you know it’s a good tenet. It should feel opinionated! One tip: try to think of the things people are actually trading off and use that.

After you have the pithy sentence for a tenet, the next part is the commentary. You should have commentary after the tenet that gives more nuance, and helps people understand WHY we do things this way. Give a little context for why this makes sense for the company, and also outline when we might not follow the tenet. Be opinionated, but show the nuance. You can also give people some rules of thumb they can use to help them decide these things for themselves. For example, if the tenet is "we're willing to spend money on problems to speed things up", you might provide guidance that you can assume an engineer's time is worth a million dollars a year.

Note my examples above are how I do it. Feel free to use your own format -- just make sure it is clear and unambiguous.

How to implement tenets

1. Start with your team or organization. Discuss the pattern you see and the idea of using a tenet to help guide decision-making.
2. Write a draft and get feedback.
3. Introduce the tenet idea and the new tenet. Publish it in a company-accessible location.
4. Add tenets to onboarding.

If you’re in a company that encourages individual contributors to contribute these types of things, you can define a way for people to propose tenets. Designate a leader who manages the process.

How to structure tenets

Tenets can be centralized and global. They can also be federated throughout the organization, with each team creating tenets for itself. I usually start with a tenet or two, and see if it develops organically on local teams. If you do federate it, you should train your managers on how to create effective tenets, by doing a workshop or sharing this blog post with them.

How to not screw up tenets

• Don’t make too many tenets. You want people to be able to remember them. Just have a couple. You can retire tenets that don’t seem necessary.
• Don’t make obvious tenets. They should be things that the organization really needs to keep in mind and doesn’t have a habit of thinking that way.

Open questions

• I’ve never tried starting an organization or team with a set of tenets -- I’ve always viewed them as emergent. If you have experience with this, I’d love to hear the tradeoffs you found.

Coordination models

Tenets are just one of many coordination models. Coordination models give you a menu of choices to choose from when solving your leadership coordination issues.

Podcast on related topics

Decoding Leadership is my podcast on leadership. I talk about tenets in this episode on removing yourself from being a bottleneck.

Thank you

Shaun Yelle for introducing me to tenets. Seth Falcon for overall feedback, for pointing out the initial version had too many steps to implement tenets and basically rewriting it for me, for the interesting question about emergent tenets versus initializing a team with them, and for some excellent editing suggestions. Matthew Finlayson for feedback on this post and suggesting I go into more detail on how to implement tenets.

Image by chenspec from Pixabay

As an organizational leader, you’ll face situations where you need to improve the way people work together.

This is a list of coordination models -- approaches you can take to coordinate the way people work together. They are very much like tools in a toolbox. You want to have a lot of familiarity with the tools you can use, and which tools make sense in what situation. They thus are a pattern language for coordinating humans in large groups.

I am writing longer posts on each of these coordination models. The links below are to those posts, and to drafts for future posts.

How to use these coordination models

I’ve put these models into three categories:

1. Use centralized coordination models for problems that are global in scope, and have to do with alignment and the ability to coordinate the entire system.
2. Use role-based coordination models for the limited cases they apply.
3. Otherwise, use team coordination models.

Centralized coordination models

As we discussed in my last post, you should centralize coordination and goal-setting. These patterns are appropriate for achieving better global coordination and alignment.

1. Product strategy. A written strategy which outlines the environment and way forward.
2. Standards. Rules that define guardrails in team behavior and force conversations about non-standard choices.
3. Tenets. Mental shortcuts to help people globally make tradeoff decisions.
4. Goal frameworks. A way to set goals that attempt to be coherent across the organization, and translate across the functions to coordinate work.
5. Metrics -- early draft. Track numbers to drive particular outcomes, add observability, or force attention to an area.
6. Product council. Cut through cross-team projects gridlock by providing prioritization for critical projects and a way to break ties in prioritizations between decentralized teams.
7. Budgets. Use constraints, like money(!) to guide investment and focus, and to decentralize ownership.

Role based coordination models

These coordination models are limited in scope, so use them when appropriate but don’t overuse them. I’ll write more on applicability as I write deep dives.

1. Program manager -- draft. A role that runs programs (projects containing projects) and coordinates efforts across teams.
2. Integrator -- early draft. A role that solicits information from a broad variety of sources and synthesizes them into prioritization and plans.
3. Controller -- early draft. A role that has explicit authority to demand behavior in a particular arena.
4. Standard definer -- draft. A role that defines guidelines and guardrails that constrain behavior that can be done without discussion.

Team coordination models

Team coordination models are the models you turn to most regularly. Familiarize yourself with all the different models, as each is a tool you can use for particular situations.

1. Service provider. A team has a valuable skill or service they provide to other teams (and those other teams depend on them to succeed).
2. Consultant -- early draft. A consultant is available to help guide other teams to make better decisions or learn faster. They are never a hard dependency for other teams' work.
3. Self-service. A team offers its work product without requiring other teams to collaborate with them.
4. Independent executor. A team produces customer value without collaboration with other teams. They may make requests of other teams, but they don’t rely on those requests being completed.
5. Liaison. An individual serves as a communication channel with a team or group of teams.
6. Embedded. An individual has a source team, but spends the majority of their time working in another team and is treated like a part of that team. A variation of this is when the individual is associated with multiple teams or an organization, and they do work for those teams, or move between them.
7. Single-threaded-owner. A team where all the cross-functional contributors (most commonly engineering, product, and design) all report to the same manager.
8. Rotation -- draft. A variation on the Liasion model, where a person from a team (or set of teams) takes on a role for a defined period of time.
9. Centralized liaison -- draft. A variation of the Liaison model, where you have representatives from a number of teams form a working group.
10. Merged group -- early draft (aka DevOps pattern). Two groups that previously passed work between each other are merged.
11. Task force. Temporarily create a merged team that focuses on a particular outcome, maximizing short-term collaboration.
12. Away team -- draft. Part or all of a team does work in another team's area. Lasts for the duration of a project.
13. Tiger team -- early draft. A long-lived team that does work in other team's territories (and that's how they are defined, as a boundary-less team).
14. Objective expert. An expert or group of experts produce measurements or reports that help visualize problems, to drive behavioral changes.
15. Work demander -- early draft. A team demands work of other teams, and they have to do the work.
16. Cousin team -- early draft. Change the management hierarchy so teams that need to collaborate have the same Director.
17. Community of practice -- draft. Support specialists across cross-functional teams by creating a group that shares information and practices, and often defines standards.

More detail coming soon

The linked in posts include more detail on each of these patterns. The posts with "draft" or "early draft" next to them are links to the Google Docs where I'm working on my writeups for those patterns. Please comment on those drafts if you'd like to see more detail -- that helps me see the demand for it, and also see where things are missing. I define "draft" as not done but possibly useful. I defined "early draft" as likely not useful.

Thank you

Early versions of this blog post were becoming a full-on book. I’d like to thank all the people who contributed their thoughtful feedback: Brent Miller and Robert DiFalco had excellent critique of the structure of this post. Due to their help, I ended up writing a lot of content that will be broken out into a lot of different posts, so it might be difficult to thank everyone correctly for the parts they contributed to. I’d like to thank Matthew Finlayson for pointing out the need to clarify how to use tenets, Seth Falcon for pointing out areas where my language use wasn’t clear, and Keizan Shaffer for pointing out the costs and challenges of doing a Community of Practice well, and sharing helpful advice for doing self-service and embedded models well. Thank you Michael Stahnke for suggesting I include more detail on how to think about tradeoffs between the patterns. Finally, I’d like to thank Jim Shore for introducing me to many of these models, especially the Product Council and Self-Service models. Thank you to Gustavo Aguiar for pointing out some awkward language. Finally, I'd like to thank Charity Majors for amplifying this post. I get so much pleasure from people finding these posts useful, and she's brought a lot more attention to these topics.

Image by Pexels from Pixabay

“We need to break down silos between departments and get people to collaborate better” -- almost every leader everywhere.

Most leaders reflexively think of silos as BAD and collaboration as GOOD. This manifesto defends silos and challenges the value of collaboration.

Increasing collaboration can do harm

In general, you should aim to maximize collaboration within teams, and minimize collaboration between teams.

Why maximize collaboration within teams?

A collaborative team works together on one or two goals. Why?

• This maximizes shared state -- everyone has a common understanding of goals, progress, and who is doing what.
• This gives team members a better ability to focus and coordinate their work with each other.
• Team members have overlapping areas of knowledge, so they can critique each other’s work and help each other grow.
• They are more innovative, because the interplay between people as they work on the same goal helps generate more diverse thinking and improve decision-making.
• When someone leaves the team, the fact that others have a shared understanding of the work means the team can survive and continue to work effectively.
• People can go on vacation or leave without as much disruption.

Collaborative teams feel great to be a part of -- everyone shares the same victories and accomplishments together. A team that doesn’t collaborate often really isn’t a team at all.

Why minimize collaboration between teams?

To the maximum extent possible, teams should have what they need to succeed within the borders of their team. And where that is not true, you need some structure to ensure the team can get what it needs in a way that will scale with the organization’s growth.

As companies grow, communication and dependencies proliferate. Companies start out with many-to-many communication. As they grow, the communication patterns within the company must necessarily switch to being segmented and defined. Otherwise, the communication burden on teams will grow at an exponential rate, and the increasing complexity will degrade the effectiveness of the company.

I observed an example of this at New Relic:

1. As the engineering organization grew, we encouraged a collaborative culture and rewarded people for collaboration between teams (it was even part of our promotion criteria -- you can blame that on me!).
2. After a few years, the increasing number of teams made it more and more difficult to manage dependencies between teams, to the point that it eventually became impossible to accomplish any large project within the organization. I know, because I was one of the “best project managers”, so I got put on many of those large projects -- and they were systematically impossible to execute on. We all tried heroically to make it work, but the system was rigged -- there wasn’t a way to accomplish these larger projects.
3. The solution to this was to eventually define the interaction models between teams, reduce dependencies, and add some structure to prioritization and communication.

Looking back in retrospect, it was as obvious as math what happened, but I see organizations fall into this trap over and over. We’ll talk more about how to structure these solutions in the second blog post in this series.

Collaboration sounds great, but it’s something you want to actively be combating between teams. A little collaboration is fine, but excessive collaboration between teams is a sign your organization isn’t structured well. If you ever wonder why Bezos took such a hard line on his API mandate in 2002, this probably factored into it. Bezos structured Amazon so that teams were as independent as possible.

What are silos?

Silos are boundaries between groups of people, based on the organizational structure and teams they’re working on.

Why do silos exist?

Silos exist because humans have cognitive and communication limits:

• Humans can only handle a certain number of relationships
• Humans have a limit to how much communication they can handle.
• Humans are pre-wired to think in terms of teams and tribes. We work better in small groups.
• Humans have cognitive limits on the amount they can focus on.

It’s important to recognize that these limits are real limits, and not something you can wish away. Every company in the world (above a certain size) operates in teams that specialize. We organize into teams and have org structures generally because that is what human beings need to work in larger groups.

There are ways to flex certain aspects of this. For example, a company that is designed to be fully asynchronous can rely on process and tooling to change some of these constraints. But even then, you’re moving the constraints around, not eliminating them completely. You need to operate in a way that recognizes these limits.

Why do leaders want to break down silos?

Leaders start talking about breaking down silos when they see that individual parts of the company aren’t achieving larger business outcomes and are excessively focused on their own area. Alternatively, they talk about breaking down silos when parts of the company aren’t well coordinated with each other. Generally this happens once the organization has become complex enough that the structure is getting in the way.

Here are a couple of examples:

• Marketing is planning a major announcement of an upcoming launch, but can’t get a timing commitment from the product and engineering teams.
• Two teams in engineering are building the same service, but in slightly different ways.

Leaders see these things, and start blathering about “breaking down silos” and “increasing collaboration”. What’s wrong with that?

“Breaking down silos” represents incomplete thinking

“Breaking down silos” is an exhortation rather than a diagnosis or prescription of how to improve the situation. “Breaking down silos” blames individuals for not having a big enough vision and working across boundaries, instead of looking with curiosity at the system and asking why they are doing what they’re doing. It’s expecting people to have your level of perspective without figuring out why they don’t.

But the main problem is that it isn’t specific enough.

Communication != Collaboration != Coordination

When you hear someone say they want teams to collaborate more or break down silos, encourage them to look at the problem from three perspectives:

Coordination

Usually when people talk about collaboration, what they’re really looking for is better coordination.

Coordination is “the harmonious functioning of parts for effective results” (Merriam-Webster)

The US military found that the best way to coordinate groups of people quickly and effectively was to centralize coordination and decentralize decision-making and execution. This is still the state of the art for organizational design. You want local groups to be able to act independently and have what they need to be successful. You want centralized functions to set high level objectives and coordinate where necessary to produce the right outcomes.

We’ll talk in the second post in this series about a multitude of ways you can coordinate groups of people working together.

Communication

Communication is transferring information from one person or group to another.

When people talk about needing increased collaboration, you can often achieve this more effectively by looking at the flow of information between people, and redesigning it. Typically, you can do something like this:

1. Ask people how they are getting information today.
2. Find out what information people actually need.
3. Design the lightest weight version of this you can imagine.
4. Try it out
5. Get feedback and act on that feedback.

One thing I’ve done over and over in many startups is set up weekly communication on projects in engineering. This helps the marketing organization understand how to coordinate their work with engineering, among other benefits.

Collaboration

Collaboration is when people work together to produce an outcome. When teams are collaborating, it means they’re working with other teams to achieve outcomes together. That’s often a sign the team isn’t set up well. Ideally, it should have what it needs to do what it needs without dependencies on other teams.

A team that has to collaborate to achieve its objectives is going to be less reliably successful. In general, teams shouldn’t be collaborating with more than a couple of teams, unless they’re explicitly set up to be that way. For example, in some organizations you might set up the design team as a “service” organization which provides design for a larger organization. For teams that are set up that way, it can be fine, but it usually should be very carefully thought through. What is the interaction model for the team, and how will it deal with the inevitable fact that there will be excess demands on it? We’ll cover some of these patterns and their tradeoffs in the second post in this series.

Of course, the world is messy, and you shouldn’t expect a complete lack of collaboration between teams. But when you see teams collaborate, that’s something to pay attention to.

Technical analogy

You might think of silos as "encapsulation". Leaders want to dig into the internals of the classes holding the logic they need. But it's a bad solution to just bolt that on -- you really need to refactor things so that they are better structured.

How do you get teams to coordinate better?

In the next post in this series, I share concrete patterns that help teams and departments work across organizational boundaries.

What do you think?

I always appreciate hearing your thoughts and reactions to my posts.

Thank you

Thank you to the many people who helped improve this post. Rebecca Campbell always makes my work better. She helped me tighten up many of my arguments and helped me realize that I needed to make the section on coordination more explicit. Brent Miller, always the purveyer of astute observations, offered structural feedback that made the post much stronger. Chris Haupt, always thoughtful, pointed out a few areas that he didn't find convincing, and helped me see that I needed to go deeper on information flow. Aaron Erickson suggested the metaphor of encapsulation. Thanks to Neville Kuyt for suggesting I define silo. And additional thank you to Robert DiFalco and Darin Swanson for reviewing and commenting on the post. Always appreciate your insight! Nate Saul pointed out an embarrasing typo (thank you!).

Image by 1778011 from Pixabay

This is a part of a series of articles on hiring and recruiting. And a series of posts on what works to improve diversity, equity, and inclusion.

Imagine filling a position in a week (or a day!)

Several years ago, I was asked to build up a new team quickly. We seeded the team with a couple of people, and knew we would need to hire a few other positions rapidly.

Fortunately, we were using "bundled hiring" for these positions. For one of the first roles, I was able to extend an offer within a week! This wasn't unusual, in some cases we were able to hire in one day.

What is bundled hiring?

Bundled hiring is when you take multiple, similar positions, and use a unified process to respond to all of them.

The simplest form of this is when you merge job postings. For example:

• Senior Engineer, Services (Java)
• Senior Engineer, Authentication and Access (Java)

Becomes

• Senior Engineer (Java, several teams)

Bundled hiring eliminates one of your biggest bottlenecks in the hiring process

At New Relic, we ran an analysis of our hiring process, and found that the biggest driver of "time to hire" was spent in getting the job written, posted, and finding the initial candidates for that position.

When you bundle up similar positions, you have an "always on" queue of candidates. When a new position is opened, it already has a queue of candidates you can choose from. This is how I was able to fill that position so quickly. It wasn't always this quick, but it dramatically shortened the time it took to fill positions.

Bundled hiring can improve the candidate experience

When a candidate is looking through your jobs page for positions, they're trying to map their expertise to roles in the company. They want to know if there is an opening that maps to their skillset.

Most engineers think of themselves in categories. They are a "backend engineer", or "frontend engineer", or "full stack engineer". They may think of themselves as a "Java engineer" or "Python engineer". Although there are real downsides to these categories (because people can often pick these things up), most positions are listed that way, and most engineers think of themeselves in these categories.

However, most companies surface positions according to their own internal reporting structures, not according to what is most friendly to the candidate. A company might list eight backend engineer positions. The candidate then either applies for all of them, or has to read through each one and figure out which one they care about.

There are some things you have to be careful of with the candidate experience -- I discuss those in the implementation section below.

Bundled hiring reduces the workload for hiring managers

Let's go back to the example of a company that lists eight "backend engineer" positions. Each of the hiring managers typically writes their own job description (adding time to the process before the position is posted). The quality of these job descriptions will vary.

Because the process for the bundle is pre-defined, the hiring manager doesn't have to design the whole interview panel and interview process. They just plug into an existing system.

In addition, by joining forces, the hiring team for the "bundle" can split up the work. They can substitute for each other when someone is on vacation. They can take on roles they're more effective at.

Bundled hiring can drive better outcomes for underrepresented groups

I was stunned by how effective bundled hiring was in improving our ability to attract candidates from underrepresented backgrounds. At New Relic, our bundled hire numbers were much better than our normal hiring process. We saw a dramatic jump in the representation of people we hired -- about 50% of the candidates came from an underrepresented background, versus 25% from our previous approach.

Why was it so much better? I asked a lot of people, and here's what they said:

First, research shows if you have one woman in your candidate pool, there is 0% chance she’ll be hired, but if you have more than one woman in your candidate pool, you're much more likely to hire a woman. It may be nothing more than just having a couple of people to choose from. Bundled hiring increases the size of your candidate pool, helping to overcome a selection bias (for all types of underrepresented groups).

However, the thing people pointed to most was that our bundled hiring process was just better run and optimized for underrepresented folks. Because it wasn't reinvented by a hiring manager every time, it was something everyone made better and focused on making high quality. Instead of creating a throwaway process each time for each candidate, we made something better.

Bundled hiring can result in higher quality hires

Because you are blending the candidate pool between many roles, you get the absolute best candidates across those roles. Instead of the #2 person in a role getting rejected, they might be the next best candidate globally. So it can result in an overall increase in candidate quality.

Also, because the hiring process tends to be better refined, you may get better quality hires as a result of improved selection.

When does bundled hiring make sense?

Bundled hiring makes sense once you start seeing duplication in the roles you're posting. This means it's typically for mid-sized companies and above. The more uniform your positions, the easier it is to do it.

How to implement bundled hiring

• It's important to have a constant driver of the process. The biggest challenge we had implementing bundled hiring was that we let some things happen that should never have been acceptable. One issue we often saw was with candidates not getting replied to -- the hiring manager would find the candidates they were interested in, and respond to those people, but then leave everyone else back in the "pool" for everyone else. You need someone with good process chops who can manage the whole process.

• When I first joined a "bundle", it was a little disorienting. I was jumping into an existing process, and I had to get my bearings. Make sure you think through the onboarding experience for hiring managers, and have a very clear set of written documents on how to get started and plug in to the process.

• Whenever a position is filled, the hiring manager should do some sort of retrospective on the process, and incorporate what they've learned into the overall hiring process. This needs to be formally defined so people feel encouraged to treat the process as "open source". Usually it's good to involve the other hiring managers in those retrospectives. The overall process must be written down.

• You might consider having each position list the variety of roles within it. So for example, if you have a Frontend Engineer bundle, you can list the three teams that have openings, and describe the types of problems they might work on. This can make your postings be the best of both worlds: specific details to excite people and a less challenging application process (because candidates don't have to go through many roles).

• You have to think through who does the interviewing. One way to do it is to have people assigned to "tours" of a quarter or so. This is simpler than having a group of people swap in when their team is interviewing, and gives you a chance to place people who are good at interviewing (or who with experience can be good at interviewing).

• Once the position is filled, you have to decide if the hiring manager's involvement ends at that point. We had the hiring managers jump into and out of the process, but you could also do it for a period of time. One thing that surprised me is that hiring managers would sometimes volunteer to help out other hiring managers. For example, one engineering manager would be under the gun on a critical project, and someone else would volunteer to help them hire. Because the hiring process was well understood, it made it easier to help across teams.

• The part that hiring managers usually resist is having other people interviewing "their" candidates. You can overcome this once the process becomes refined enough that people are seeing good outcomes from it, but you have to be careful in the early days of bundled hiring to get over that gap.

• You can choose to bundle skill levels together, or separate them into separate bundles. We generally found it useful to bundle close skill levels together, as it gave us flexibility during the interview process. If we found someone was more skilled than we expected, we might focus them on a particular subset of teams.

• All the hiring managers need to be aware of the needs of all the other hiring managers. We usually did this with an orientation meeting, where each hiring manager went around and described their needs. They also wrote them down, so that the whole team was looking out for each other and looking for the right fit for each candidate. This took some maturity from the hiring managers -- if someone was only looking out for their own needs, it wouldn't work as well. So it's possible this may not work as well in some low-trust work cultures.

• We did sometimes experiment with showing multiple listings for the same bundle. That was a good way to test out different approaches to the job description. The most important thing is that the process behind all the postings is the same. But you don't want to confuse the candidates by having a lot of similar postings, so I would keep this to a minimum.

• We never found a way to do this with specialized positions. In general, think of this as supplementing your usual approach.

• One concern people had about this process was that the candidate wouldn’t have the experience of knowing which team they would be on. People care about the team they are on and want to learn about that team. This is especially true of underrepresented groups, because you want to know what you're signing up for -- who your coworkers and manager will be. The way we handled this was to narrow it down to the team or teams the candidate was eligible for, and then let the candidate choose. We would typically have the hiring manager for that team interview the person as the final interview. This helped ensure there was a good mutual match.

Pitching bundled hiring

It can be hard to sell people on doing bundled hiring. Even if the benefits are obvious, there are a lot of reasons people might not want to do it.

For example, hiring managers may oppose it because they want to control the hiring process. And they may not trust their counterparts in recruiting and talent to run the process well.

Probably the most important success factor is that you have a high level sponsor for the change. Someone that can operationalize the changes and make sure they're evolving and improving.

It's usually best to start with the most standard jobs possible. Whatever you're hiring for most, that's where to start.

What do you think?

Did I miss anything? Do you have experience with bundled hiring elesewhere? I'd love to hear your thoughts or your experiences as you implement bundled hiring.

Thank you

Belinda Runkle introduced bundled hiring at New Relic, and that's where I first learned about it. She was inspired by Iris Bohnet's book, What Works. It's thanks to her that I was able to write this blog post at all. Thank you to Darin Swanson for feedback and detail on how the bundled hires were run. Thank you to Amjith Ramanujam for the helpful suggestion to provide an early example of what bundled hiring is. Matt LeBeau shared his experience with me and pointed out some ways it hadn't worked well and the need for strong process management. Beth Klem had good questions about the candidate experience that helped me improve that section. Julie Pagano had great feedback on the importance of knowing your team. Thank you Bill Green for sharing your experience of being hired in one day! I'd also like to thank Allison Pollard, Andreas Kavountzis, and Jason Hobbs for sharing their experiences in implementing bundled hiring. This led to me fleshing out the "pitching bundled hiring" section, and the addition of the section that describes how the quality of candidates can improve.

Image by David Mark from Pixabay

This is a part of a series of articles on hiring and recruiting.

People vet a job based on the job description. If it’s well-written, they may be excited about the job, and give it precedence.

Content and format

• Your post should market the company and job. Don’t write the job posting as just a description of the job. A good job description markets the role. Tell them what will be interesting about the work and the team. What type of problems will they solve? Why is the company doing something valuable in the world? Characterize the role enough so they can get a concrete idea of what it’s like.

• Focus on them, not on you. The job description should make it easy for them to see what the job will be like. Describe things like: here's what you'll be doing, these are the problems you'll be focused on. This is what your team will look like, and this is how you'll know if this is a good role for you.

• Test what you write with existing team members. Ask them if they would be excited by the position. It's not finished until its exciting.

• Include a section for "Interview Process". Include a short section that describes the interview process. This is a great example from ReachSuite

Fight bias

• Use anti-bias tools like textio (paid), joblint (free), and gender decoder (also free) to make your job postings appeal to a wider audience. Even effective writers will find things to improve.

• Make sure you're following the law. Some locations require you to post salary ranges (I believe Colorado is one such state). Other states require you to tell the pay scale of the job if requested (California). Consider posting the salary in the job description, even if it's not required.

• Be explicit about what requirements require. Women tend to not apply for a role unless they meet 100% of the requirements, while men will apply even if they only meet 60% of the qualifications. Add language like this: If you feel like you don’t meet all of the requirements for this role, we encourage you to apply anyway. We know the confidence gap and imposter syndrome gets in the way of meeting incredible candidates, and don’t want it to get in the way of meeting you.

• Prune your requirements down to what is actually need for the role. Don't artificially constrain the position beyond what is essential for the role.

Did I miss anything?

I'd love to hear what else you like to see in job descriptions. And if you have good examples you'd like to share, send them my way!

Thank you

Alexa Stefanko took my stilted wording for the "don't meet these requirements?" and made it infinitely better, so I copied her wording in the post above. Colin Smith from ReachSuite inspired the section on including info on the interview format.

Image by congerdesign from Pixabay

This is a part of a series of articles on hiring and recruiting.

One practice that doesn't seem to be very widespread is the use of candidate packets. A candidate packet is a package you send to applicants that explains the interview process, tells them about the company and the role they're applying for, and helps them understand what to expect as a candidate.

Why bother with a candidate packet?

Interviewing is a stressful process. Candidates have no idea what to expect, because companies tend to jerk them around and expectations for the interview are unclear. By helping people understand what to expect, you're painting your company in an incredibly good light. It can help people feel more motivated to continue the interview process with your company, and it can help them perform better during the interview, giving you a better sense of who they are and how they'll contribute at the company.

Plus, since this isn't a widespread practice, it really stands out.

How to get started on a candidate packet

Create a Google Doc and gradually start adding useful information for candidates: answers to questions they ask, information about the company you think they might want to know. Treat it like an FAQ, and gradually fill it out until you think it reaches a critical mass of useful information.

Look at the suggested table of contents below, and adapt it to your purposes.

Then take an editing pass, make it look nice, create a PDF version of it you can send to candidates.

What to consider for your candidate packet

Candidate packets should outline the interview process, sell them on the company, outline benefits, and explain anything unusual about the company.

Here’s an outline for one candidate packet we used at Gremlin:

• Interview process. Walk candidates through what the interview process will be like. What are you looking for in each interview, what will you be covering. You can even explain why you're focusing on each interview. A great candidate packet will even give suggestions for how to prepare, or what questions they're going to be asked. In real life, people often can prepare for something, so why optimize your interviews for non prepared questions?
• Who we are. Explain why your company is important, what it's attempting to do, why customers care about the company, and go over things like core values. This is a good chance to introduce your company and explain what the company is like and anything that is unusual about it.
• Remote life. If you are a remote company, then talk about the practicalities of being in a fully distributed company. If you're a hybrid company, mention that. Explain any practices around this. For example, "we get together quarterly".
• Benefits. You don't have to include all the benefit information here, but give a high level summary, so they know what they're getting into, even before they get an offer.
• Our offers. Describe how your company approaches salary and equity. If you're a company that has implemented pay equity (see my post on how to implmement pay equity), then this is a good chance to describe that they won't have to negotiate salary.
• Reviews and opportunities for growth. People may be nervous about the company unless they see that they'll have opportunities to grow and be invested in. Talk about how your reviews work and how you evaluate people for promotion. A lot of startups don't have any system for this, and it's fine to be candid about that. Give them a sense of where the company is at and where you want it to be in the future.
• How we think about inclusion. Some of your candidates are going to come from underrepresented groups. They'll want to know how they'll be treated at this company. A frank, candid look at where you're at and where you aspire to be, in as concrete terms as possible, will help them understand what your company will be like for them to work at.

I update the candidate packet frequently, as I hear new information, and then generate a new PDF and send it to new applicants.

Which candidates should receive a candidate packet?

I decided to give it to all candidates who made it through the initial screening interview. I think it would also be fine to give it to candidates before the screen.

What do you think?

Did I miss anything? Please share your favorite ideas with me, I’d love to hear them!

Image by Harry Strauss from Pixabay

Thank you to Alexa Stefanko for driving forward the candidate packet at Gremlin.

This is a part of a series of articles on hiring and recruiting.

The easiest way to improve your hiring is to speed it up. Here is how to do that.

Hiring faster can drive immense improvements

Hiring for a position is a thoughtful and deliberate process. And it can take a lot of effort and time to get to the point where you're confident making a decision on a candidate.

But all that deliberation comes at a cost. Imagine your process took twice as long as it does now. Wouldn't that result in a lot of those candidates deciding to work elsewhere?

The same principle applies if you shorten the amount of time it takes to decide on a candidate. Imagine a theoretical situation where you could immediately decide on a candidate after they apply. How much more likely would it be that they would end up working for you? In my experience, this can result in dramatic improvements.

Time windows are a differentiator when hiring. Candidates have an amount of time within which they'll interview. And they have an amount of time within which they'll make a decision.

If you decrease your time window, you compete with fewer companies. You also speed up every aspect of your hiring.

Instead of filling a position in three months, you might fill it in one. The compounding effect of this on a growing company is immense.

Give your hiring team a service level agreement (SLA)

My advice: give everyone that touches candidates 24 hours to get back to them. Ideally you get back to candidates within a few hours.

Even if you do nothing else, this will give you many times better results than being average.

Most companies take a week or more to get back to candidates. This little change accelerates every step of your interview process. The end result in people get through the whole process faster.

This also results in a better experience. Candidates feel like they're valued if they get a prompt response.

Reduce the number of steps in your interview process

The second change to make is to remove steps in your interview process.

Most interview processes contain three to five rounds. For example, you might have a process like this:

Each of these steps adds a new cycle of turnaround time to your interview process. So removing steps is the best way to compete with other companies:

With these changes, your time window is a week or two, instead of a month or more.

You’ve just made your hiring pipeline AT LEAST 2-4 times more effective. You’re competing with fewer companies for the same person. This isn’t even accounting for the fact that the candidate experience is so much better.

Schedule all the steps ahead of time

The formula for how long your interview takes is:

(Number of steps) * (Turnaround time + Coordination time)

We've already optimized the number of steps, and turnaround time. The final optimization is to reduce coordination time.

One way to do this is to pre-schedule your team for interview times, so they're available for a certain number of interviews per week. Then you can offer times that are immediate, instead of far off in the future.

By creating set times each week that are reserved for interviews, you avoid overloading your team with interviews on any given day or week.

Now you're cooking with fire!

Interview until you find the right person, not in batches

You can use two different models for your interviews:

• Batch interviews. You interview a bunch of candidates, compare them to each other, and have a decision after you’ve interviewed a whole bunch of them.
• Rolling interviews. You interview until you find the right person. Whenever you find the right person for the role, you stop.

Batch interviews are slow, but more fair and thorough.

Rolling interviews are difficult at the beginning, because you haven't calibrated your candidates. They are also difficult at the end, because you may be reluctant to extend an offer when you you're still interviewing promising candidates.

I recommend using Rolling interviews. You should calibrate for a couple of weeks, until you have a good sense of how good each candidate is.

Best of both models for diversity

You can batch up candidates if you’re not getting enough diversity in your hiring pipeline.

Research shows if you have one woman in your candidate pool, there is 0% chance she’ll be hired, and I imagine that’s true of other underrepresented groups).

I like to not make a decision until I've interviewed people at least two people from underrepresented groups. To balance against the need to hire quickly, I give myself a time limit. If I don't meet that limit, I can proceed, but note down the failure as a signal for improvement.

This is how I try to balance my interest in hiring quickly, with my interest in hiring fairly and from a diverse pool of candidates.

Present an offer the day of the last interview

One way to speed things up is to monitor how the “onsite” interview goes, by talking with the interviewers right after the interview, or having them enter it into the applicant tracking system right afterwards (tip: have a meeting scheduled for them to reserve this time, right after their interview). If you see enough positive signs, prepare an offer to present them at the end of the day.

Improve the visibility of your hiring pipeline

One of the challenges I've had with a lot of applicant tracking systems is that it is hard to know which candidates are waiting on us. And it's hard to get a sense of how we're doing as a team in responding.

A tool I have not used, but would check out if I were using Greenhouse or Lever, is TalentWall. It visualizes your candidate pipeline as a kanban board, and shows how you're doing and what needs to be done for each candidate. Super valuable! Maksim Koutun describes his experiences with it here. Let me know your experiences with it -- I have not used it. Also let me know if you've used any other tools that help you visualize your hiring pipeline.

Did I miss anything? Please share your favorite ideas with me, I’d love to hear them!

Thank you

My appreciation to jtopper from Rands Leadership for his feedback on the post. Maksim Koutun shared the post on hiring improvements he made and his experience with TalentWall.

Image by Pexels from Pixabay

This is a part of a series of articles on hiring and recruiting.

Creating a hiring strategy

Why should someone work at your company? Every company answers this question differently. Here are some valid strategies:

• We’ll pay the median rate in your area, but we’re investing heavily in making this the place for you to do your best work -- the best work culture. (early days New Relic)
• We’re working in a compelling technical area, which attracts a lot of talented people, so you can work with a great team and solve interesting problems. Plus we’re making a great people-focused work environment. (Gremlin during my tenure there, also early days New Relic).
• We’re making Disneyland for engineers, lots of perks and top pay, in a university like environment (early Google strategy).
• We’ll pay really well for you to work your ass off, and in return you get a chance to work at a scale you won’t otherwise (Amazon today).
• We won’t pay you the most, but you’ll get to spend time each week working on open source software (a local software consultancy my friend ran for a while).

What your company looks like without a strategy

In practice, when you haven’t done the work to articulate why someone should work there, your hiring strategy looks like this:

• Work super hard at a stressful startup for low pay and equity that is probably worthless.

Developing your strategy

This step is to create your hiring strategy. Ask yourself, how are you going to stand out as a place people will want to work? Like any strategy, you should consider what your unique strengths and weaknesses are.

Areas to flex in a strategy

Some dimensions you can flex or emphasize to attract candidates:

• Remote versus in person. Having a strong stance on this will attract people who care about either of those options.

• Perks you are willing to offer. Consider offering allowances for teleconferencing (high quality cameras and microphones, for example). Flatfile offers an office makeover, with a designer helping you renovate your workspace. You can offer top of the line developer machines, or quality monitors. These are things that also help your team work more effectively together, so it’s offering a benefit that serves two purposes! If I were making my own startup nowadays, I’d focus on top-tier videoconferencing equipment (DSLRs and external microphones) as a main perk. When I was at New Relic, they offered free bicycle tuneups in the office to encourage people to commute by bike.

• Freedoms you’ll offer. You can offer a 4 day workweek. You can craft a narrative around offering no deadlines and no meetings. Or something else entirely.

• Transparency on salary and equity. Research shows diverse teams produce better business outcomes. There are a few things you can do to make your position stand out for candidates from diverse backgrounds: list the salaries (and equity!) for the positions you’re posting, and emphasize how you’ll make the workplace fair and equitable for all employees. Share your philosophy around how salary and equity are determined. Being more upfront about this than competitors will make you more desirable and competitive. Start by implementing pay equity, and then tell people you’re doing it. How bizarre is it that your hiring manager is expected to advocate for and win the trust of their team member once they’re hired, but they’re expected to negotiate against their best interests during the hiring process? Starting off the relationship without having to negotiate salary can be a huge relief for candidates.

• Haven for people from diverse backgrounds. The experience for many people in tech companies is far from ideal. Many people leave our field because they're treated so poorly. Most companies haven't invested the time and energy to build a work culture that retains and delights people who aren't overrepresented in tech.

• Family friendliness. You can offer generous paid parental leave. On-site daycare. Or flexibility to stay home with sick kids.

• Impact people can have on the world. Consider ways your company helps employees connect their work to something meaningful or interesting to them. Are there ways you can emphasize this more? People at Patagonia are given the option to volunteer once a week, paid by the company. Working at a nonprofit can have direct impact. For most companies, you can think of the value of what you produce in a way that people will find meaningful. Think about how that ties into your narrative for why to join your company.

• Attributes of the team. What is the team like, and how can you use that to attract the type of people you’d like to come to the company? For example, you might be good at hiring people that are collaborative, and not assholes. You might be in a place with a lot of creative thinkers.

What is the pitch?

As you look through these, come up with a one sentence description of why people should join your company.

You'll know you have a real strategy if it is rational for a certain type of person to select your company over all the other companies out there.

What's your pitch?

Let me know what you come up with!

Image by PublicDomainPictures from Pixabay

Tips for startup hiring and recruiting

I'm going to share some practices that I don't see used very often. These practices help ensure you'll find great candidates, offer a superior candidate experience, and be competitive against other companies that are looking.

I haven't seen many of these topics listed anywhere else. To be comprehensive, I'm listing a lot of topics, with breakouts for anything you want to go deep on.

If you do even a few of the things in this post, it should dramatically improve your hiring outcomes, and help you stand out.

Why listen to me?

I’ve interviewed hundreds of people over the years, and at times it was basically my full time job. As a consultant, I’ve also seen how a lot of organizations approach hiring, so I’ve seen both some depth, and some industry breadth in how people approach hiring and recruiting. And I’m very good at copying from the people around me!

Create a hiring strategy

You should have a pithy description of why someone should work at your company (and your position) as opposed to elsewhere. You shouldn't rely on hope as a strategy, but have a plan that identifies how you'll appeal to the best applicants you can attract.

There are a lot of ways you can attract candidates: pay, technical area, perks, and work culture. I've written up a whole post on creating a hiring strategy -- read it over and identify what your approach is.

Update your careers page to reflect why they should work there

Now you need to make sure you communicate this strategy to people who are evaluating your company.

The first way people evaluate your company is by your careers page, but most startups don’t spend much time on it. This is your “top of the funnel”, and even a modest investment in this area will make the difference.

Spend some time reviewing the careers part of your home page. How are you talking about the company? Is it compelling? Does it convey why someone should come work for you?

If you’re genuinely trying to make your company be a great place to work, this is a good place to highlight that. Think of it like a dating profile -- you want the people who read it to get a good sense of whether they’ll be a good match or not, and you want to portray yourself accurately and in a positive light.

Write compelling (and unbiased) job descriptions

People vet a job based on the job description. If it’s well-written, they may be excited about the job, and give it precedence.

Most job postings also turn away a large percentage of good candidates unnecessarily. I wrote up some tips for writing unbiased and compelling job descriptions. This includes what to include in your job posting, and some easy to use tools to improve your job descriptions and reduce bias.

Make your application process painless (for everyone)

• Make sure your application process allows people to specify the pronouns they use. Make it optional (as not everyone wants to be “outed”). Make sure your team uses those pronouns during the interview process.
• Don't require gender as part of the application process. It can be an awful experience for applicants. If you have to require it for some reason, explain what to expect, apologetically.
• Apply for the job and see what the process looks like. Test it out! How long until you get a reply? What flow do you go through? Are there unnecessary steps? Most people don’t know what their own interview process is like!

Create an interview plan

A good interview plan allows you to have confidence you’ll vet and find the right candidate. A poor interview plan results in a terrible candidate experience. I have a whole post and example on how to create an interview plan.

One tip: pair people up in the interviews. People learn from seeing others interview, so this will make your interviews automatically improve themselves.

Be faster than anyone else

One of the biggest improvements you can make in your hiring process is the speed at which you hire. This is so important I've made this its own blog post. Three things you can do are:

1. Implement an internal SLA for communication
2. Present offers the day of the last interview, and
3. Use a rolling approach (with some caveats) to hiring.

I go into a lot more detail in this post: How to speed up hiring.

Use a candidate packet

One practice that doesn't seem to be very widespread is the use of candidate packets for interviews. A candidate packet is a package you send to applicants that explains the interview process, tells them about the company and the role they're applying for, and helps them understand what to expect as a candidate.

You can think of it as an FAQ for candidates, but also as a way to sell the candidate on the company and position.

Give feedback to candidates

One way to stand out for candidates is to give them real, honest, but kindly worded feedback on how they did, even if they don’t get the job. Treating them like humans can pay off in the long term. I’ve had candidates refer their friends to me after they were turned down for a position, because they liked the hiring process so much. Think about what they’ll tell any friends about working at your company, if they feel like they missed out at working for a great place.

This is a good example of something that is both right for people, and can be in the self-interest of the company. You don’t have to take a lot of time with it -- just send some kindly worded summary of the feedback to them.

For especially promising candidates, you can even refer them to other companies, or make little suggestions on how they might improve their interviewing.

Ask for feedback from candidates

Set up your application tracking system to ask candidates about their experience. And ask them yourself when you give feedback. Periodically go over the feedback and continually improve your interview process.

Keep in touch with promising candidates

One of the bigger problems with the way companies hire is that they treat it like a transaction. But humans are relationship-focused creatures. Keep in touch with candidates you wish you had been able to hire. If someone turns down your offer, set a calendar reminder and check in on them, and ask them how the new position is treating them. Be nice about it -- hopefully they should be enjoying their new role and new position. But keeping in touch can bring people back in the door who had a good impression of your company and may have gone elsewhere for good reasons that could change.

Don’t rely on inbound candidates, use mining to find hidden candidates

One of the best ways I’ve found of bringing in candidates is using the social connections of the people already in the company. You have to be careful because this biases towards people that are similar to the people already in the company. But when you reach out and they already know someone there, it can be a lot easier to attract them, and they won’t feel like you’re a complete stranger. These two articles describe an excellent process for “mining your networks”:

• Mine your Network for Early Stage Hiring Gold
• Mining Your LinkedIn for Potential Hires

There is even a product that helps automate the process of finding candidates from the network of people your employees already know. (I came up with the idea for this product before it existed, so I’m so happy someone came up with the same idea and made it happen)

For this to work, you have to be extremely careful with consent and how you use people's network. When I've done it, I got consent for the phrasing of the message I would send people. And I took notes for each potential candidate on how to contact them.

Make a list of people who are impossible hires

Because companies approach hiring as a transaction, they neglect courting people who might be harder to convince. Sometimes these hires can be incredibly important.

Make a list of people who in a dream world would work at your company. Periodically review that list, and ask yourself, "is there a way I could make it a rational choice for them to work here?". Reach out to the them and extend an invitation. And importantly, treat it like a real relationship -- don't just send them a form letter.

The important thing with this approach is you need to invest time, and you need to handle it respectfully and strategically.

Work well with your recruiters

If you use recruiters, invest some time in making sure they understand what you’re looking for. Talk through how the whole process will work, and make sure you’re on the same page about expectations. You can make them much more effective by walking them through how to know if someone is a good fit or not. A strong partnership with your recruiters can streamline everything, get you higher quality candidates, and, if they’re organized, they might even be able to help manage parts of the process for you.

Have a review meeting when making a decision on a candidate

Candidate review meetings tend to be just reading the feedback everyone collected. Improve those meetings by following my guide to running a candidate review meeting.

Bundle up similar roles

One of the bigger inefficiencies in a hiring process is to treat every role as a separate queue of candidates. You can achieve way better outcomes by bundling similar roles together: no ramp up time for new positions, prefilled queues of candidates for new positions, process efficiencies, and even better success hiring candidates from diverse backgrounds.

Learn more about bundled hiring and how it can improve your hiring outcomes.

All of these things reinforce each other

The more you view your hiring process as a continually improving thing, the more these things will reinforce each other and combine to give you an advantage when looking for candidates. It can feel really good to see this in action. I often have received thank you emails from candidates (that I turned down), talking about what a great experience they had.

Did I miss anything? Please share your favorite ideas with me, I’d love to hear them!

Thank you

I learned the impossible hires approach from Alex Kroman.

Image by Katie White from Pixabay

Exploration and exploitation in technical standards

In engineering organizations, we live in constant tension between Exploration and Exploitation.

If you're not familiar with Explore/Exploit algorithms, let me give a restaurant analogy.

• Exploration is trying a new restaurant.
• Exploitation is going to your favorite restaurant.

In all aspects of our life, we're balancing two impulses: to Explore new things (and discover something wonderful), and Exploitation: to take advantage of the things we've discovered.

Successful engineering organizations:

• are disciplined about both Exploration and Exploitation.
• engage in a low level of constant Exploration, but are ruthless about cutting off experiments that aren't successful.
• bias towards standardization, because introducing something new immediately introduces debt into your whole tech stack. To be exploited appropriately, it has to be so valuable it's worth going through and updating everything to take advantage of what you've discovered.

Standards reduce organizational complexity

A standard means you require a conversation to do things in a non-standard way.

Many engineers resist standards. They view them as limits on their freedom. Standards make it harder to do things in non-standard ways.

Many startups put off standards because they are fighting to survive and these seem like costly distractions.

But there can be a huge payoff to establishing lightweight standards in your engineering organization:

• Fewer patterns in your code.
• Less tech debt to navigate.
• More deliberate conversations about tradeoffs with new approaches.
• A lower cognitive load required to understand your codebase and make changes to it.
• Lower onboarding requiremenst for new team members.
• Faster velocity in releasing new features, due to less time spent navigating complexity.
• Teams that can maintain and operate their code.
• More internal mobility within the company, as engineers aren't stuck with their proprietary systems that nobody else can understand.

The same people who resist standards are also the people who complain about the results when you lack standards: a messy codebase that is hard to reason about.

Startups should put in place minimal standards and roles around technical decision-making. Standards can ensure you have high-quality conversations about new technologies or patterns.

You avoid problems down the road by establishing patterns around which conversations to have when you want to do something new.

System-level benefits

From a systems perspective, standards help reduce overall organization complexity, along many different axes. Instead of N different ways of handling state within codebases, you'll have 1 or 2. Instead of 8 different programming languages in use, you might have 2 or 3.

As Coda Hale points out in his marvellous article on organizational complexity, organizations can achieve great benefits if they invest in internal tooling and "force multipliers". You make that type of work easier if there are less targets they have to work against.

Standards can also help distributed teams benefit from the overall learnings from local groups. If one team finds out that a particular coding pattern leads to disasterous results, standards (and linting against those standards, where possible) can help an entire organization not relearn everything over and over again.

Implementing standards

I generally like to see standards be a process that everyone can participate in. An RFC process, where you have a central place for standards, and people can propose new standards but there is a period of public comment, can be a good way to approach it. If you have a Chief Architect, or Technical Leadership Group, you can sometimes set up approval for standards through that group.

An anti-pattern for standards

Standards can dictate a lot of work, so beware of the danger of unfunded mandates. You'll see groups fall into this trap in two ways:

1. Not take on the burden of fixing things but ask teams to update to the new standard whenever they touch something. This just spreads the cost over time.
2. Require a new pattern that requires a lot of work.

If you have an architecture group or technical leadership group, it's good to have product management or engineering management involved, so they can remind the group that they can advocate for projects, but can't will them into being.

The intention is to reduce complexity over time, both cognitive complexity (choosing between options) and code complexity (a proliferation of approaches in the codebase). Look for opportunities to automatically enforce new standards: code linting, observability tooling, and tests.

Examples of standards

• New Relic's approach to standards

Credits to the book Algorithms to Live By for a inspiring some of thinking on exploration and exploitation.

Image by Julius Silver from Pixabay

The day I became brilliant

It was a proud day for me, but I remember it for the bizarre twist it took.

My boss had just given a heartfelt, remarkable speech to the whole product organization, praising my work and announcing my promotion to Senior Director. Throughout the day, I received congratulations from everyone around me. I was walking on a cloud, just feeling great.

It began in my first meeting after the promotion. Person after person would go out of their way to say why my suggestion was the “right approach”. All of a sudden, my ideas were BRILLIANT.

This persisted throughout the rest of my day. All of a sudden, I was a different person, a Very Important Person. By the end of the day, the initial excitement of the promotion had worn off, and I found myself wondering, “what the hell just happened?”

This day marked the beginning of the time I really started paying attention to how power shapes the relationships and effectiveness of leaders. People treat you differently, and you start to operate in a different bubble. These dynamics can warp the environment around you and stunt your effectiveness as a leader.

I start by sharing some rules of power, each of which makes you less effective. And then I share some habits and patterns you can apply to be a better leader:

Rules of power Rule 1: leaders don’t know what is going on even when they think they do

A lot of the struggles of leadership are around getting good information, and creating a management structure that gives you enough signals to make good decisions throughout the organization. Yet power directly counters your attempts to get good signals from your organization. Here’s the paradox:

The higher you go in an organization, the less you know what is going on. Yet, the higher you go in an organization, the more you’re responsible for fixing what’s going on.

Leaders often know enough about a situation to pass plausible judgement on the situation, but usually aren’t close enough to the details to know all the implications of their assessment, or to really get it right.

I’ve frequently seen leaders make a gut call on a situation based on the information they have and what feels right, and seen whole departments scurry to follow their direction. Sometimes this can destroy companies (I have seen that more than once). Leaders do this because they are encouraged to “make tough calls” in ambiguous situations all the time, and they are praised no matter what direction they choose. It is a trap!

Rule 2: people lie to leaders

The higher you go in an organization, the less likely people will give you the truth about what’s going on. Why? You have power over them.

Think about the implications of that. The higher you go in an organization, the less accurate the information you receive from people. Dangerous.

People watch leaders carefully. They emulate their behavior: they may dress or talk like their leaders. You’ll see this in meetings -- watch how people look around the room, and see how they treat the people who have the biggest titles in the company. If you don’t know who the leader of a group is, you can just watch people, and see where they look.

People don’t lie to leaders to be malicious, they hide things that are problems or downplay things because it’s often not safe for them to bring it up.

Even if the leader claims to want to hear it (they usually do), people know based on your actions if you really mean it. Many people fetishize straight communication and feedback but don’t understand what it takes to actually build a culture where it is safe (for everyone) to do that.

The more tenuous your role, the less you will feel comfortable bringing up problems. So many white male CEOs have no clue what is going on for their employees from underrepresented groups, because those employees are in a more tenuous situation and don’t feel comfortable bringing up hard problems. Why should they?

Rule 3: leaders are overconfident

Humans are hierarchical creatures. Like other apes, we naturally think of ourselves in hierarchy. Leaders are constantly deferred to. They have their ideas validated as brilliant. In meeting after meeting, they see signs that their ideas are taken seriously. This reinforces confidence in leaders.

I’ve seen this spiral out of control, where leaders become aggressively confident, and dismissive of other people’s ideas. Other leaders emulate the behavior of the person on top, and the whole culture tilts towards risky behavior, and aggression.

Think about the danger of being overconfident, with poor information on the state of the organization. Confidence is a valuable skill in a leader — overconfidence is a distorting trait in one.

Rule 4: leaders naturally get the credit

One of the weirder parts of our ape psychology is that the leader naturally gets the credit for the accomplishments of their organization.

The team that did all the hard work building something -- who coordinated, designed, problem-solved, and figured out how to make something hard work -- they don’t get the praise at the all company meeting. Instead, the leader of that organization does.

Getting the credit has a pernicious effect on a leader's psychology. They may have contributed to the direction or made suggestions that really did improve the trajectory of the project. But it reinforces the exceptionalist mentality that just reinforces the ego of the leader. This is definitely something to be wary of!

As an aside, one of the more unfair things I’ve noticed of leaders is that if you’re a white cisgender male, it’s easy to ride on the success of your organization and learn from the failures. If you’re not a white cisgender male, you may get credit for the success of your organization, but you also get blamed for the failures of your organization. There is a lot less latitude and the problems get more tied to you. Frankly I wouldn’t be where I am today if I had been held to the standards I’ve seen some of my colleagues be judged by.

Habits to be effective and humane as a leader

Over the years, I’ve seen a couple of things that seem to help counter some of the downsides of power.

Ask for the best argument against your idea

Leaders often expect the people around them to “push back” against their ideas if there are problems with the idea. But they don’t make explicit space for people to do so, so it doesn’t happen.

The easiest way to make space for other people’s thinking is to be very explicit in looking for it. Instead of saying, “I think this is what we should do. Anyone think different?” say it like this: “It is starting to look like this is what we should do, but I'm not sure it's really the best course. What are the best arguments against doing it this way.” With this move, you’re inviting everyone to critique the idea together. Instead of them feeling like they’re critiquing YOUR idea, they’re invited to provide tradeoffs. You’ll be surprised what you hear when you make space for it.

There are a lot of variants of this. You might ask, "what problems do you forsee with this approach we might need to watch out for?", for example. But the basic idea is to explicitly ask people to tell you why something might be a bad idea. Use the people around you to critique your thinking.

Some leaders might be insecure with this approach, because they feel like it’s opening themselves up to look “weaker”. I’ve found the contrary is true: if you’re able to show that you are bigger than your ideas, you earn a lot of trust and your team will see you as a stronger leader.

Propose things that are dumb

This might be a controversial approach, but it’s especially useful when combined with asking people to critique your ideas.

People can often get stuck analyzing a situation, and having an idea to start from can be an effective way of problem-solving. It also helps model the behavior you are looking for — if the team can critique your ideas, they will be more likely to work together to improve each other’s ideas as well.

I’ve found a very effective pattern is to put together my best thinking on a topic, even if it is incomplete. I write it down, and then share it with the people around me. I ask them to improve the plan, and invite them to critique it. “What’s missing in the plan?” “How would you do this differently?”

Proposing incomplete direction, and allowing people to shift it creates trust in the people around you. They’ll see you change your mind, making them more likely to offer their opinion in the future.

Proposing an obviously bad idea can be manipulative. One way to approach this was covered by one of the most interesting talks I’ve seen. Dave Rensin warned the people around he would mix good and bad proposals with people, to make sure they were correcting his thinking. He was explicitly testing how well the decision-making system around him was, and making it okay for people to disagree with him.

You can be explicit that an approach is a bad one. You can say for example, “unless we come up with a better idea, let’s do <this really bad idea>. What are some ideas for a better way to approach it?”. This makes the bad idea the default, and again aligns all the people in the room towards an impartial discussion of what would be better than the terrible idea you’ve suggested.

This is an approach that takes some guts. You have to be willing to be seen as foolish or be associated with bad ideas. Most likely, you’ve got some political capital you’ve built up, so you can use it to help strengthen your team around you.

Side note: make space for real analysis

Although acting on incomplete information can often be effective, like I outlined above, some situations call for real analysis and information gathering. This trick is to figure out which situations call for what.

Urgency in a situation often results in decisions being made from the gut rather than through analysis. This can result in decisions that miss the mark, or don’t solve things in a complete or lasting way. If you see a pattern of frantic decision-making, this is often a sign to slow down and solve things more thoroughly and completely.

Build relationships throughout your organization

One trap leaders get into is selecting a narrow subset of the organization, building close working relationships with them, and doing all their work through those people.

You can think of the network of people around you as a communication system, and the depth of your relationship with those people is how well information moves between you and them. If you invest in a skeletal structure of strong communication, you’ll have a group of people who are more likely to share problems with you.

I view this as a two way street:

• They help you understand what’s going on, and help you be less clueless and in touch with what’s happening in the organization. They provide a sounding board for you to float ideas and observations with. They help you see patterns you wouldn’t see ordinarily.
• You in turn, have to act on problems they share with you, and show that you’re making their work life better. You can provide mentorship, and be on the lookout for opportunities for them.

Doing this requires genuine vulnerability. People can see through insincerity, so you have to be willing to show your humanity, and approach them as one human being to another. I can’t claim to be the best at this, but what I do is show curiosity about their work experience. For example, I ask junior employees to tell me how well they’re being supported, and what it’s like to work with their coworkers. I’ll ask new employees to share what their initial weeks have been like, so we can improve it for other employees. I’ll also ask them what they see as surprising about the company, both good and bad. Anything they wish had been better? All of these things are useful information that will make you a better leader. But you’re also building relationships that will help you support your organization and the people in it more effectively.

The most common way to do this is with Skip Level 1-1s. Nowadays, I like to use Donut to set up a “skip level 1-1” room in Slack, and that automates setting up the invites and making sure it works for both people’s calendars.

As an Interim VP of Engineering, I once decided to skip building relationships within the company I was working at. I didn’t have a lot of time to invest in building relationships when I was going to be a temporary leader. I found to my chagrin that this ended up causing a lot of problems -- a manager quit and mentioned me not building a relationship with them, and I discovered that some of my thinking wasn’t as well developed as it should have been. After correcting for that, I was amazed how much more effective we were as a leadership team. I had wasted a few months, and contributed to a manager leaving the company. It was a hard lesson.

Consistently show appreciation for feedback and opposing points of view

To get people to share feedback and opposing points of view with you, you have to reduce the risk they perceive in offering you hard feedback or opposing your ideas. One way to show that things are less risky is by going out of your way to show appreciation for feedback people offer that makes you look bad. And to show in public that you are open to criticism and alternative ideas.

Market leaders as servants

One of my leadership mentors is Bjorn Freeman-Benson. When he built the engineering organization at New Relic, he had a number of ways he “marketed” leadership within his organization, which helped reinforce the type of organization he wanted to build:

Invisible leaders: Bjorn encouraged his whole management chain to think of themselves as invisible leaders: “The team should get credit for their work. Try to fade into the background whenever there is a success, and recognize the people who did the work.”

Inverted org charts: Bjorn created his own take on an org chart, with the leaders on the bottom, supporting the rest of the organization. It was a dramatic way to make a point that he expected the leaders to support the organization, and it made the managers within engineering reconsider the way they approached their role.

Use whatever opportunities you have to reinforce that leaders serve the organization.

Use a coach

Some of the best companies provide executive coaches for their leadership group. For myself, some of the most rapid growth of my professional career was driven by the advice, council, and mentorship of executive coach Robert Goldmann.

A good executive coach should help you see the world more accurately, and guide you in developing your skills. They can help you overcome some of the leadership traps here.

Many leaders stop developing their skills once they become leaders. An executive coach ideally has been in your shoes before, and can be a sounding board for you to better yourself.

Release your plans like you release software: incrementally and to get feedback

Since your plans may be based on poor information, and the people on the ground will have better information, it’s often good to roll out your plans incrementally. Involve people who have the most direct knowledge of the problem (or better yet, have the manager in that part of the organization solve it). But when you’re rolling out almost anything, roll it out like software: do previews to a wider and wider set of “customers”, and get feedback from them on the plan. See if they can improve on it, or if they can see problems with it. As you roll it out to each wider group of people, encourage them to offer feedback on the plan. You should have increased confidence in the plan as you proceed, because you’ll be hearing all the concerns and feedback as you expand its audience.

Another post I recommend

Nadya Duke Booke wrote a great post on how "Extremely Upper Management" may have different incentives, pressure, and information that you. I highly recommend it.

Let me know your feedback

So those are some of the things I’ve observed and patterns I’ve seen to help leaders serve their organizations. I’d love to hear your thoughts on this topic.

Thank you

Marco Rogers helped me understand the pattern of leaders not making space for deep analysis.

Image by OpenClipart-Vectors from Pixabay

One of the challenges of dealing with a diverse workplace is that all of your employees have different vacation needs:

• A Jewish employee may want Rosh Hashanah or Yom Kippur off.
• A Muslim employee might want to take Eid al-Fitr and Eid al-Adha off.
• American employees might want to honor Juneteenth
• And you may have different holidays between countries you’re operating in -- a company with employees in Britain and the US has different holidays off.

Part of the problem with accommodating all of these diverse needs is it’s expensive to add new holidays. CEOs and executives are beholden to the needs of the business, so they may hesitate to offer additional holidays.

One way out of this conundrum is to align the needs of your employees with the needs of the business. When I was at Gremlin, we came up with a nice solution to holidays that more companies should adopt:

Make an annual list of “holiday focus days” to make it easy for people to take time off during their important holidays.

Holiday Focus Days

With Holiday Focus Days, you have a set of days each year that are specially laid aside, not as time off, but as Holiday Focus Days.

Many companies have adopted “unlimited time off” as a way to give their employees flexibility with their vacation schedules (and to incur financial benefits). By forbidding collaboration and meetings during Holiday Focus Days, you make it easier to actually take off the days they care about.

For people that work during Focus Days, they aren’t able to be in meetings, so they can use the time for focused work. That’s beneficial to the company because it not only offers some time for deep work, but it also provides a contrast to the daily way of operating, helping team members notice differences in the way they work and driving innovation in your work practices.

For people that want to take a Focus Day off, this gives them an easy way to do so. It makes it less of a burden for people from backgrounds that might want to take these holidays to do so (always be on the lookout for situations that increase the burden for the non-”dominant” culture at work). I can’t emphasize enough how meaningful this can be for people from other backgrounds -- their entire work experience has been workplaces ignoring their needs. When you make it easy for them to take their important holidays, that is very significant!

How to implement Holiday Focus Days

Here’s how to implement this:

• Decide what the parameters of the Holiday Focus Days are. You can make them pure focus days, where people are encouraged to focus on deep work and minimize collaboration between each other except by asynchronous means. Or you can just make it a typical “no-meeting day”.
• Make a list of holidays you’d like to take off. You can be pretty liberal about it, since the days you’re adding are likely to be beneficial to the company. So make it a good list, and make sure there are at least a couple of days per month.
• Decide if you want the “dominant” holidays to remain on the calendar or not. Some people may not want to take Christmas off, so you could decide to make this a more expansive holiday program that is culturally neutral.
• Consider adding days like the anniversary of Stonewall or Trans visibility day.
• Ask your employees what days they care about to add to the list.
• Make sure you’re combining this with some way of encouraging or reinforcing people taking time off. Otherwise, people will see this as a cynical thing without any real teeth.
• Also be sure to communicate that you’re doing this to make it easy to take those days off. This helps people from the backgrounds that would want to take those days know that it’s being done to honor their backgrounds.
• You can also use the occasion to help educate people on the various holidays as they come up. Posting a link to an article a few weeks ahead of time is another way to honor the diversity of your team, and help people understand the significance of days like Juneteenth or Trans visibility day.

Some additional considerations

• You might not have control over the whole company, but you can also implement this within your team or organization.
• One thing to be sensitive to is that this can create tiers of cultural support if you're not careful. Being on the list of approved holidays becomes a sign of whether the company is supportive of a group or not. The most ideal situation I can imagine is doing this within a larger strategy of moving a company to being fully asynchronous. Then, the bar for adding new holidays would be very low. Otherwise, you might want to just be really clear on what your approach is for adding new holidays. If you have one employee, is that enough? Define these things ahead of time, and communicate them so people know what the criteria are.

Let me know your experiences

This hasn’t been very widely adopted, so if you adopt it at your company, you’re going to be doing something new and innovative. Share your experiences with me, so we can learn from each other about what works and what doesn’t. And if you have critique or feedback, would love to hear it.

Thank you to Bailey Douglass for her warning about the dangers of creating tiers of cultural support. And thank you to Jim Lindley for pointing out this can be done locally, not just at a company level.

Image by Michelle Raponi from Pixabay

Many people offer advice for getting the best out of Slack: use threads, post things in public channels, and use a naming convention for your channels. This post contains some controversial or unusual recommendations about managing Slack or other chat platforms.

Humans don’t naturally manage their limits

Human beings have natural limits. We can only focus on so much at a time, and thus can only manage a certain amount of communication in Slack.

Without real design and structure, communication in a growing company will naturally grow to each human’s limits. People will naturally have more and more communication thrown their way: more channels, more people messaging them, more posts from the increasing number of people in their orbit. They’ll do their best to adapt to this, and take on more and more communication until they reach a limit.

This isn’t desirable, because humans operating at the limit of their capabilities are less effective.

Thus organizations are systems where communication will naturally degrade. The result is noisy and disorganized Slack communication, and problems with alignment, communication, and focus. Countering that requires design, and an awareness of the underlying principles governing how humans work.

Below are a few rules I’ve found useful to keep in mind as you’re designing how people communicate over chat:

1. Answers to questions should take longer than it takes to look it up yourself.
2. Notifications should go to one or two people.
3. The more people in a room, the more structured the room should be

Rule: answers to questions should take longer than the time to look it up yourself

By default, it’s easier to ask a question in Slack and get a quick response than it is to look it up in the documentation. This is like designing a computer system where your cache is slower than the thing you’re trying to look up -- it’s just fundamentally broken to expect that to scale: the cost of looking it up yourself needs to be less than the cost of asking.

Introducing a delay to the time before you answer a question is actually a desirable way to make a more efficient, scalable communication system. Being overly helpful may feel good, but it’s not serving the long-term interests of the company.

The best way to have effective communication in a company is for the communication to be future-proof: reduce the need for future questions. The best way to do that is to write things down and organize them to make them easy to navigate. Every time you answer a question, you should know that someone six months from now should be able to find the answer to that question themselves.

A useful pattern to apply everywhere is: I'll document the answer to this question, in exchange for you providing the answer to this question.

Socialize this within your company. For the questioner, it can help reduce their hesitation in asking questions to things that are blocking them. To the person being asked, it shows that the questioner has already looked for the information, and that they’re willing to make it easier for future people to answer this question for themselves.

You could even go so far as to tell people they don’t have to answer questions unless people agree to document the answers! Bake this into the way people think about their role, and you’ll find that it drives a lot of automation and organization of information.

If you’re not in a position where you can demand this of the people asking the questions, then you can still do it unilaterally. Whenever someone asks you a question, document it somewhere, and point them to that document. This helps train people to look things up first, and know where the source of information is.

So how does this play out in practice?

Let’s say your support team is asking questions of engineering, because they need that information to serve customers well. They might, for example, ask whether a certain browser is supported or not. Here are ways to respond to that, in increasing order of effectiveness:

1. Answer their question.
2. Write up a document on the internal wiki or external documentation, document the answer, and give the link to that documentation back to that person.
3. Wait an hour, then do #2.
4. Have customer support and engineering meet. Write down an agreed upon way to get support the answers they need in a timely fashion that is future proof and “cached” in documentation:
   1. Customer support person asks in channel X, after looking at <internal docs> and <external docs>
   2. A designated person from engineering responds within a certain SLA of time.
   3. The Customer support person documents the response and provides the link back to the answerer to show it’s been done. Next time, they’ll use that to look it up.

How far down that list is your organization? Do you agree it would be better lower on that list? Some things to think about are: how easy is it to onboard a new customer support person? How many times will they ask questions that have been answered before?

Rule: Notifications should go to one or two people

If you have Slack messages like this, you have a problem:

@Engineering-team I just tried to access the site and got this error.

@here Wanted to celebrate a huge success in Engineering. We just shipped super widgets!

Notifications are designed to interrupt attention. We are giving a group of networked individuals tools that allow them to interrupt a large number of people at a time and get them to pay attention to something.

When you have a small number of people in a company, it’s not that big of a deal, but again growing complexity will kill you. Going from 10 people to 50 people results in a lot more notifications, and each of them triggers that horrible Slack notification sound, and a red notification to pay attention to (by the way you should turn those off). For many roles, a single interruption can prevent them from doing complex work for a couple of hours. You would never let someone in an office broadcast a question to 50 people, so why do we allow it in the digital domain?

So how do we improve on this? Let’s take the first example: you have a room where people ask the engineering team questions like this:

@Engineering-team I just tried to access the site and got this error.

The pattern to apply here is a rotating position to field these questions. Notifications should go to a rotation, not to an individual. You can use chat, tickets, triage bots, or something like Pagerduty for it, but the role should be a rotation.

The person in this role is doing interrupt driven work, so that should be their main focus. They can do any other interrupt driven or small work as well, and try to help keep their team focused. At New Relic, we called this role the Support Hero, and they were also the person that was on call for that team.

A second pattern is to discourage or disable @here and @channel usage. Turn off @here and @channel and @everyone usage in large rooms (here's how). If you don't have the ability to do this yourself, you can introduce use emoji reactions and cultural reinforcement to make using these more rare. Make it part of your onboarding!

Rule: the more people in a room, the more structured the room should be

One of the challenges with chat-based communication is that it is conversational. It’s very easy to post something out to a group of people. The cost of saying something is almost zero, so people say a lot. But the cost of reading what everyone else is saying is high, and gets worse the more people there are. So if you’re in a company with 50 people, the complexity and time to navigate Slack is worse than it is if you have 25 people. This is a problem -- you don’t want it to get harder and harder for people to do their jobs as the company grows.

The larger the chat room, the more you should optimize for things being read-only and scanable. Make it as concise and easy to consume for other people as possible.

Posts to #general should be pre-approved or restricted to predefined types of messages. Large rooms shouldn’t have much conversation and should be more notification based (because a large group of people usually doesn’t make decisions).

A couple of patterns that fall out of this:

• Write things down outside of Slack, post links in Slack - A good pattern to encourage is to write up documents in your standard tool (a wiki or Google Docs), and share the writeup with a short summary in Slack. This moves the communication around the document to the place it belongs (the document). The summary should give the person enough context so they can decide whether or not to engage with the document, what timeline they should do so, what actions are being requested of them, and the gist of what the document is saying.

• Use threaded replies. This makes it easy for someone to follow the conversation, and not be notified on threads they don’t care about. This keeps conversations distinct from each other, making it easier to scan a room and decide if you need to look deeper at a conversation or not.

• But beware long threads! Long threads are often a sign of poor communication. Why should someone be engaged in a 75 message thread on Slack? Usually there are better ways to decide or communicate on things. I think of Slack and other chat based tools as a good way to make shallow decisions, but not to engage in complex discussions or share context.

• Use emoji and formatting to ease scannability - Busy slack channels should be easy to read. Post your topic with a relevant emoji and the subject in bold. This helps people visually separate the various topics and scan them quicker.

Let me know if you find these tips useful!

Thanks to Chris McCraw for pointing out you can disable @here and @everyone usage.

Image by Stefan Keller from Pixabay

A rule of thumb is that a meeting with more than eight people isn’t a decision-making meeting.

When you have more than eight participants, you either need to change the format of the meeting, or you need to restructure the participants (and you usually want to do some deeper work on communication and organizational structure).

Meetings devolve towards dysfunction

A common example that illustrates this point is what happens with a company's main leadership group. I've seen this happen a few times now:

• You start with a small leadership team of four or five people.
• As the company grows, you add more leaders.
• Soon the group grows to twelve people.
• The amount of substantial, real decision-making goes down over time, and the meetings feel like a waste of time.

Splitting meetings in two may not work

The way most people handle meeting growth is to view it as a meeting problem.

There are too many people in the leadership meeting? Split the meeting into two groups, right?

• An executive meeting (for a smaller group of leaders)
• A leadership meeting (for all leaders in the company, including the executive team)

While straightforward, this doesn't usually produce the results you would expect.

Problems with just splitting meetings

Why doesn't this work? Let's use the rule of eight to analyze this situation. Remember, a meeting with more than eight people isn’t a decision-making meeting.

With that in mind, here's what we will observe:

• The executive team remains effective, because it is under eight people.
• The leadership team is still a large group, so it’s not a decision-making body. But often that isn’t explicitly decided upon, and the group continues to try and use the meeting as a decision-making body.
• Having two meetings is more complex than one, and there is less "shared state" between the two meetings, so everyone gets confused because information isn't being passed well between the two meetings.
• The result is usually a more effective core executive team, and a more confused larger leadership team.

It’s not that this structure is bad (it’s very common, and something that can work). It’s that the expectations for this structure are unrealistic. The leadership meeting isn’t going to be a decision-making group.

What are some better ways to design your meetings?

It's better to do the work to deeply restructure the communication paths and underlying teams.

In this example, you have two choices that would be better:

• Make the leadership team meeting a status or announcement type meeting.
• Structure the leadership teams into smaller decision-making bodies.

I tend to favor the latter approach. You want substantitive, impactful decision-making to be happening throughout the company. If you don't design it that way, people will add the extra meetings anyway to get things done. You'll end up with an ad hoc, informally-specified, parallel version of what you're designing. This results in twice as many meetings, making the organization much less effective.

You could divide the leadership team into several smaller leadership teams, each focused on a part of the business. For example, a GTM and Product leadership team. Each of these meetings could be under eight people, allowing them to be decision-making meetings.

Some leaders won’t completely fit into one or the other group. Do your best to align them with the parts of the business where they'll do the most good. For example, you might have a head of security. Security is tightly connected to the product, but also highly involved in the sales process. You might decide to put them in the GTM meeting, because that optimizes for the sales team being effective. Or you could decide to have them attend the Product leadership meeting. Choose based on which is more critical.

Design your communication pathways

The important thing is to then think about the communication needs of these groups. You might decide to share the notes from these meetings to other leadership teams, or publish them on the wiki. You could have some sort of weekly communication coming out of each meeting. Or have a short sync meeting between the two occasionally.

Other examples of redesigning meetings

Here are a couple of other examples of this rule in practice:

• You have a managers meeting, and you go from having four managers within your department to five, six, seven, and eight managers. As you approach eight managers, you should anticipate needing to split the group.

  -> I would usually divide this into two managers meetings, each led by a different leader. I might lead one of those meetings and have a director lead the other and report to me.

• You’re working on a project with a lot of stakeholders. They’d all like to attend your weekly meeting, where you’ve typically done a lot of your problem-solving. At first it’s easiest to just add a few people to your existing meeting. But when the group gets too large, it becomes unwieldy.

  -> I would typically make a list of who the problem-solvers are, and who the stakeholders are, and either move the stakeholders to receive status communication, or give them good notes from the meeting. Or have a separate stakeholders meeting, where we share status and solicit feedback (That can be larger than eight people). One thing to look at in this situation is why there are so many stakeholders, and if that truly makes sense. I’ve seen times when there were eight stakeholders and two engineers working on a project. That's something to squint at.

It's not impossible to make decisions in larger groups

Note that it’s not impossible to make decisions in larger groups, it’s just harder, and requires more structure. Some techniques you can use are breakout rooms, polling, simultaneously editing a shared document, or approaches that use a lot of parallelism like sticky notes and grouping.

Image by Gerd Altmann from Pixabay

Operating under a shared reality is a competitive advantage, but most teams share state poorly. Let's talk about gluing brains together!

Computers and people both have "state"

In computer systems, state is information that is remembered between interactions. A stateful service is one that keeps track of things that have happened before. Databases are where we typically store state, and we try to design our services so they don’t keep track of state themselves, but keep it in a central place.

People, as autonomous beings, require state. The way our thinking works literally requires it. So one of the great challenges of all organizations is distributing a shared understanding of reality: the organizations challenges, goals, and activities.

You don't copy state, you serialize it

With computer systems, and with people, you can’t just copy state around. I can’t clone my brain, or even a part of my brain, and give it to you for you to understand perfectly what I’m thinking. So in computer systems, we serialize information to put it in a format that can be transmitted between machines.

Human communication is also serialized. We have to put what is in our minds into words or writing to share it with others. This is necessarily imperfect. It is actually impossible for us to express our internal state to each other, so we have to choose what to serialize, and create a facsimile of what we have in our minds.

Human serialization is lossy, too

Our serialization is lossy -- people listen poorly, or use pre-existing state and biases as they listen. We are poor at serializing our thoughts to give to other people. The protocols we use for communication are ill-defined, and ambiguous. And just like computer systems, they require a lot of processing power to transfer even simple messages back and forth.

Errors compound the more people you're dealing with

This is hard enough with two humans. But as we increase the number of people involved in any group of humans that need a common understanding of something, you multiply the number of people involved. Each is deserializing that information in a different way.

What you’ll often find is that people use error-correction algorithms to verify the messages we hear. After a meeting, you’ll talk over what you heard and double check your understanding. And then the two of you will try and figure out what you each thought about what you heard. Consensus algorithms are hard!

One on one conversations can be inadequate for sharing state. Let's say you have ten people that you're trying to get on the same page about something. It takes an incredible number of conversations for that group of people to settle on the same shared understanding. This is especially true if the problem or solution is shifting. You'll often see a pattern where a couple of factions emerge after a meeting where people don't get on the same page. They each go through numerous smaller meetings afterwards, discussing possible approaches.

In-person teams share state more easily

It is easier for in-person teams to “share state”. When you sit next to someone, you are exposed to the same information, and it’s easier to build a shared understanding. This is a prime advantage of colocated teams, working in the same time zone, on the same project, in the same room.

Often, in-person teams do not share state well. They work on different projects, or don't sit together. Or don't communicate well. But what I'm pointing out is that sitting together does allow for an ambient flow of information that helps with shared state.

Distributed teams often struggle with sharing state

Distributed teams require explicit communication to share state. On distributed teams, it feels like things are the same. But it seems to me like there is more variance in how people receive information.

One particularly bad example of this was when I was working at a company preparing an important project for our user conference. We got two months from the release and found out the engineers thought we were delivering “something” at the conference, even though the leadership had been talking about a GA release for months!

When leaders don't share state, the results are devastating

One of the most common "state" problems on engineering teams is when the engineering manager and product manager aren't sharing state. They may each have their own version of the project plans. They might each have different ideas of what is being done or what's important.

Another variation of this problem is when senior leaders are in a different world from the people doing the work on the ground. This is a common issue, because it's difficult for senior leaders to understand what is happening locally.

How to merge squishy brains together!

There are a few solutions to sharing state.

Be explicit and build trust

One is to check in frequently and have a lot of explicit communication. The more trust you have with someone, the easier it is to share state, because your serialization protocols are better tuned for each other.

Probe for misalignment

Consensus algorithms require error correction. The human version of error correction is asking questions, and listening. That is our protocol for verification.

Use latency judiciously

Latency in communication is a prime factor in how quickly you can come to consensus. If everyone reads a document within a day or two, you have a cycle of 1-2 days. If you don't perfectly get everyone on the same page within that cycle, it may take you about a week to get everyone on the same page.

Well run meetings can be synchronous, and have a much lower cycle time. Questions can be done in real time. So when taken advantage of, a meeting can be effective to accelerate state building. However, meetings generally come with a catch. If you don't achieve your objective within that meeting, scheduling the next meeting has high latency. So you have to make sure your meetings truly accomplish their objective.

Use writing to share state

A solution to sharing state is to emphasize writing. When we write something down and collaborate around that shared document, we are all participating more closely in a shared version of reality. While we may deserialize differently, we’re all interacting with the same “database” of information, and the changes are something we all participate in.

Writing is less transitory than verbal communication, and it is more precise. I believe it’s one of the key skills for a successful remote company to master.

I would pay more to make Slack MORE transitory (all messages disappear after a day), because people could rely less on being able to look up previous conversations. Chat isn't a shared document, because usually the people involved aren't starting from a shared state. That's why chat is effective for solving simple problems together, but not complicated ones. I like the pattern of posting links to information in Slack, and having conversations and decisions made around documents. And to reply to things with links to where that information is being maintained.

One of the best patterns I’ve seen for accelerating organizational leadership work is to write down your best thinking of what the problem is and what to do about it, and share it with a group of people. Ask them to make it better. You can go through a couple of rounds of sharing it more broadly, and getting feedback, but you can go through those rounds pretty quickly -- a day or two per round. Then just get started! Your plan likely is better than it would have been because you spent the time to write it down. And it’s much better still since your colleagues have improved on it. And everyone’s basically on the same page about the problem and what is to be done about it.

In any case, the important thing is to be aware of state, and how you're ensuring your messages are being communicated. Think of it like a computer system, and it might help!

Thank you

Thank you to John Hyland for pointing out some oversimplifications in the post, that could be construed as "in person > remote".

Image by Willi Heidelbach from Pixabay

This is a part of a series of posts on improving equity at your company.

Change requires work

Improving your company to make things more fair for under-represented minorities (and thus for everyone) requires sustained effort. It’s not going to be solved overnight, and it takes real commitment. You’re fighting bias (which is built into how our brains work), and systemic issues that are pervasive and challenging.

OKRs are a good way to ensure you're making progress

Objectives and key results (OKRs), or other goal frameworks, can be a way to drive improvements in equity. Executives and leaders pay attention to OKRs every day (or they should). So it can be a way to ensure that the company is improving in this dimension. And this can result in higher retention, and a greater ability to attract candidates to your company.

Smaller things can be done without OKRs. But for work that requires real programs to make them happen, OKRs can make sure they get the attention they deserve.

Improvements stack up over time

If you make improvements in a lasting way, every improvement you make adds to the last one. Having four quarters of equity related accomplishments behind you will mean your company is noticeably better than it was a year ago. This will aid hiring, retention, and will result in human beings feeling better about the workplace they're involved in. It's one of those nice areas where it is good for the business, and good for the people within it.

Here are the steps to take

• Listen to people's feedback. Ask where the pain points are. In the past, I've seen people use employee surveys, ERG groups, and direct 1-1s and skip level 1-1s to uncover what changes were most important to prioritize. Unless you're sensing these things, you're making guesses about what to improve. That's better than nothing, but you might make your first OKR be to set these things up if you lack them.
• Choose one thing each quarter to improve.
• Craft an OKR out of that improvement.
• Assign the work to someone. Be sure you rotate who gets the "equity work". Take care not to make it seem like the least important goal, or the goal nobody wants. And you don't have to be an underrepresented person in tech to do this work -- it should be spread around evenly.
• Have that leader report on progress each week, just like every other OKR in the compmany.

Political considerations

One thing to be careful of politically: you are branded by the work you do, and people will often brand you as being whatever you’re spending time on:

• If you’re a white male, this is a good place to use your stored up privilege. I find it useful to think of it as a bank account. You can't use so much that you go into debt, because then you become less effective. But, for this bank account, you want to keep your balance low. You probably have a higher interest rate than your peers from less privileged backgrounds, so use that interest for their benefit!

• Whatever your background, it's best to balance your focus, so you’re seen as both delivery focused, and process or culture focused. This is a good way to keep yourself effective and maximize the improvements you can make at your company.

• At startups, the focus is usually on the company's survival. You don’t want your efforts to be seen as competing with the survival of the company (even if the things you're doing make the company better and more attractive to candidates). Delivery will always be the top priority of any engineering department. So having it be an OKR per quarter is a good way to make sure you’re committing to continual progress. I generally advise people to focus on improvements that don't require a lot of effort until the company has achieved product market fit.

General advice on OKRs

See advice on using goal frameworks to see more suggestions on how to use goals to drive the results you're looking for.

Image by Gerd Altmann from Pixabay

Over the years, I’ve developed a set of practices that I’ve started calling “demo-driven development”. I believe it incentivizes the right things to both improve your planning, and take into account the chaos and change in product development.

What is demo-driven development?

Demo-driven development is a practice where you use

1. regular demos,
2. a standard week-by-week project plan, and
3. value-based user stories.

You use this as a lightweight and flexible structure for planning the team's work. These then drive meetings that encourage more active tweaking and improvement of the project.

Why demo-driven development?

Some advantages of demo-driven development over other approaches are:

• You think backwards from the needs of customers. This connects the team with the business impact, yielding better results.

• Goals for the week are more clear, leading to better focus and collaboration within the team.

• Team members feel more satisfaction. Why? People have an innate need to feel progress, and to connect their work to the value they’re delivering. They love to solve problems and understand why it is important.

• Demo-driven development provides a structure to improve the quality of conversations around projects. Plans are simple and easy to play with. User stories are at a level of granularity that make it easy to control scope. This leads to more fluid and dynamic projects -- projects that are actually managed as opposed to projects that run on autopilot.

Step 1: Demos on Fridays

So how do you go about implementing demo-driven development?

The first thing to do, if you haven’t done it already, is to introduce weekly or biweekly demos (I'll use weekly during this post, but you can substitute biweekly if that makes more sense for you). You can structure them many different ways, but to start with:

• Have each team demo their work every week. If you're doing this within a team, have each team member demo their work.
• For each group of people working on something, rotate the person that demos that work. This helps ensure that everyone gets recognized for the team’s work, and gives engineers practice with the valuable skill of showing their work.
• The demos can be done in a meeting, or asynchronously recorded and posted in a room in Slack. If you do it asynchronously, copy what I learned from Bjorn Freeman-Benson: ask each team to do a two-minute video. Suggest to people that they ask questions in Slack threads, and use Slack react emoji to cheer on accomplishments. I like to tell people to spend 10 minutes on a 2 minute video, to prevent the recordings from being too large of a time suck to produce.

Be prepared to tweak the format until it feels good for the people involved. Here are a few things to be careful of, or that you might want to tweak after you've gotten demos set up:

Include all the work. One thing to be careful of is that the demos are inclusive of all the work required to build functional software. Prepare the team to demo all the parts of their work: the APIs, the infrastructure, the reliability work, and the testing. It's important for you to cheerlead the work that isn't customer facing.

Focus on customer. You can use demos to make teams more customer centric. One thing I like to do after a couple of weeks is to introduce a standard format for the demos:

1. Today, I'm going to demo XYZ.
2. [Thank anyone that contributed or helped you out]
3. The reason we're doing this work is... [explain the customer or business value in a couple of sentences]
4. [Show your work in 2-3 minutes]

Use demos to educate. If you're in an environment where there is less trust, or the leadership doesn't understand how software should be built, you may need to use the demos to educate and give context on the team's work. You absolutely don't want people to feel scared to demo, so don't make it a scary thing to do.

Critique during demos. When you do have a high trust environment, you can start to nudge the demos to involve a little critique from fellow team members. The ideal is if people are excited about each other's work, and looking for feedback from their teammates. And their teammates are thinking about the customer or business value of the work, and suggesting ways to do it better, or to make it simpler. You want a little of this, not a ton of it, so it can be a delicate balance, and is often something to introduce later.

Step 2: What are we going to demo this Friday?

Once you have demos in place, you can use the demos to focus the team's attention each week. Later, we'll show how this also helps with project planning.

Prep before the sprint kick-off meeting

• The engineering manager and product manager meet before the sprint kick-off meeting, and decide on goals for the next couple of weeks.

• You two come up with what a good demo for the next couple of weeks might look like. You may not be sure what is reasonable, but think about the goals you're trying to accomplish, and what might be a good thing to move towards. Keep in mind this is not set in stone.

• The EM and PM should prepare a list of bugs or usability issues that need to be addressed that week.

At the sprint kick-off meeting

• In the team's weekly kick-off meeting, the product manager gives some context on where we’re going over the next couple of weeks, and reminds the team of what the value and tradeoffs are in the upcoming work.

• You, the engineering manager say, “what should we demo on Friday?”. Depending on your team, you might be able to leave it at that, and let them come up with the items that meet the goals the PM talked about. Likely, they will need more direction than that. So you can share the things you're thinking the team might work on the next two weeks, and ask them what's reasonable to demo this week.

• Part of the discussion should be about other work that needs to happen. What bugs and usability issues should be addressed this week.

• The main focus of discussion, however, should be to talk through the demo plans -- what is realistic, and what the demo should look like.

• The team spends the rest of the time in the weekly kickoff meeting talking about how they’ll get this goal accomplished, and break down the high level demo goal into the technical work that needs to happen that week. They talk through how they’ll coordinate their work together, and any decisions or problems they might need to address. Collaboration!

Why "what should we demo on Friday"?

It is a clarifying question. It forces the team to think through the week, the goal they’re working towards, and the concrete outcome they are moving towards. Instead of a bunch of tasks, they’re working towards something, and something meaningful and valuable.

Anything to be careful about?

• It’s important for the demo goals to not to be a commitment. They should be a reasonable guess. Asking for certainty in a complex environment is asking for hedging, and will backfire. Product development is uncertain, and it has to be safe for teams to miss their goals, as long as they do the work to understand why they missed them. Otherwise, they estimate overly conservatively and act defensive, instead of learning to be more effective. This also makes the planning meetings lower stakes, which improves feedback and participation.

Step 3: Move the team to milestones instead of projects

I have a whole post on this topic. The basic idea is to replace projects with milestones. This improves project breakdown, sequencing, and biases towards incremental design.

You might do this later if you're a new manager, as it is something that some teams might resist or not understand.

Step 4: Move the team's work to using user stories

Your next step is to move to using user stories, instead of tasks.

What are user stories?

• User stories represent things you might demo on a particular week. For example, "a user can select a color for the chart in a dashboard". "A user can save the selected color for the chart".

• User stories should be something a product manager, or an engineer from another team could understand. (This gives the product manager a lot more power to control scope by moving around user stories). If you have a particular approach in mind, you can say so, but the user story should mostly communicate what capability you're offering the business or your customer.

• Ideally, their size should be a couple of days of work.

• User stories should be team-focused, not individual focused. A common mistake is to make user stories that can be accomplished by an individual. For example, a poor user story might divide the frontend work from the backend work. Write user stories so they are "cross-functional" in nature -- crossing skill boundaries. They should mention the user and what they are able to do: "User is able to see a list of items they have listed for sale in a table". Note this implies both frontend and backend work.

• User stories can be an increment towards something bigger, but if it is possible, they should try to be valuable by themselves.

How does the team break user stories down to technical work?

Teams tend to like to create todo items that correspond to technical work.

If your team feels a need to do so, Jim Shore taught me an approach that works quite well. Use subtasks underneath the user stories to represent the technical work.

When you do this:

• User stories represent something a product manager or technical person outside the team can reason about.
• Then under that user story, you create tasks that represent the technical work you need to do to accomplish that user story.

The task breakdown doesn’t need to happen until the week you get to it. That can be the activity in the kickoff meeting after you talk about what to demo. Or it can be done after that meeting by the people involved.

Ideally, the kickoff meeting then becomes the engineering manager saying: "these are the next couple of user stories. What do you think we can demo this week?"

The team decides what they think is reasonable to accomplish that week, and talks through the approach they might take. Then the rest of the meeting is the logistics and coordination around delivering that, including creating the task breakdown and figuring out who will do what.

How does this compare to other common ways of doing it?

Contrast the way this weekly kickoff feels to the task treadmill approach. In those meetings, the engineering manager might pull up a list of 20 or 30 tickets, and go over the many things that are still in flight. They’ll have a bunch of things typically that are moving between weeks. They team spends the meeting talking about these tasks rather than the goal they’re trying to accomplish. The tasks don’t feel meaningful, they just feel like a list of things to do.

In a million-meeting agile format, they’ll spend all their time talking about these tasks, and making sure everyone understands them, and who will do what. Instead of talking about the goal and how to work together to achieve that goal, the team focuses on the reviewing tasks in the ticketing system.

In a Gantt-aholic kickoff meeting, usually the focus is on the tasks and the points associated with everything. How many points did we accomplish, what are the estimates for the upcoming items, and how are we tracking. The focus is more on the timeline than the objective.

Instead, focus the meeting on the goal you want to accomplish, and use your tooling to record things at the level of impact to your customers: user stories.

Step 5: Create project plans or milestone plans

After you have user stories in place, the next step is to improve your planning. The way I like to do this is to have a week by week schedule which is just a list of a couple of user stories per week. You should also list risks or interesting things that might happen that week, like someone being out of the office, or someone needing to help onboard the new person on the team.

This becomes the project plan: a week by week plan that helps you plan, and think through the sequencing of the project. It should be updated frequently (at least once a week), and used during meetings.

The project plan should be a tool for conversation. It should help expose risks (will this dependency be done in time?), things you should plan for (Maria will be on call that week), and other issues.

It should prompt conversations like, “could we do this part earlier so we can validate things earlier?”, or “can we break this project in half and deliver this part to customers without this extra thingamajig?” “How are we feeling about the next few weeks, what could go wrong?”

Even though it’s okay to demo progress on incomplete work, teams should try to keep their work always shippable. The project plan should reflect this -- it should be possible to pull scope out of a project plan and deliver things earlier. Or pull in additional scope based on customer feedback. A good project plan preserves this type of optionality, and delivers incrementally towards the end goal. This is incredibly valuable to delivering things on time, and balancing the need for that with customer feedback. If you’re not managing your project with this type of optionality, most of your projects will fail.

I usually recommend you only do milestone planning, not project planning. If you're planning out an entire project, you're investing a lot of time to produce estimates on work you may never get to.

I've written more extensively about project plans and risk registries. And more specific advice on week by week project plans.

Step 6: Create technical plans

You should have a technical plan for your projects. It can be simple.

What should be in the technical plan?

A technical plan does NOT need to be a technical spec. The most important things to surface are things that might affect you in the future. I like to ask people to write briefly about these topics:

• Tradeoffs being made, and why.
• Anything new or nonstandard the team is doing. I call these technical bets. It's usually fine to have a bet or two in a project, but a red flag if there are lots of new approaches being done at once.
• For any new patterns being introduced, will it require the rest of the codebase to be migrated to that pattern, if we like it? (This requires strong justification)
• Consider adding a section for things that will be hard to change in the future, like APIs or data models.
• Any known shortcuts we're taking that might cause problems later.

A technical plan doesn't need to be long. It could even be a few sentences long, if there isn't anything the team is doing that is non-standard or surprising. The more complex the situation, the most the technical plan may need to explain decisions.

Why create a technical plan?

The technical plan should be a tool for conversation and coordination. It should help people understand and reason about the way technical decisions are made. And they should be shared for others to improve upon.

The main value of a technical plan is that it surfaces assumptions and decisions that are being made, so people can discuss them. The theme for both project plans and technical plans should be to “use the people around you to improve your thinking.”

The entire world shouldn't be able to weigh in on the technical plans but it should be a way to surface potentially risky decisions and discuss them.

Who should create the technical plan?

Two approaches I've seen work are:

1. Team members write the plan. The Tech Lead reviews and coaches the team to make sure the plan is good.
2. A rotating team member writes the plan. Team members review the plan together in some way.

The first is more explicit about technical leadership roles. The second is more egalitarian. I tend to prefer the first.

If you do the first approach, you do have to emphasize to the Tech Lead that part of their role is to improve the technical reasoning for all the team members -- they are there to coach team members. And also they are responsible for making sure the decisions aren't terrible. This requires good judgment.

How does a technical plan interact with milestones and projects?

If the project is really long, you can do technical plans a milestone at a time. Sometimes you may also need to do it at the project level.

How does the technical plan interact with the project plan?

There is an interplay between these plans. The technical plan surfaces technical tradeoffs and choices. The project plan sequences the work and breaks it down into increments.

Step 7: Project execution meetings

The project plans can then feed into project execution meetings. The aim of these meetings is for project managers (usually the engineering managers) to help each other problem-solve their projects together, and critique each other’s project plans. And to talk through how execution is going on projects and what can be improved in the organization to make projects work better.

It’s important for these meetings to be safe places to talk through problems, not a place for people to posture and show off. Alex Kroman taught me the value of kicking off these meetings with a frank acknowledgement that product development is hard, and unpredictable. We all know that we should expect challenges. We’ve all been part of horrible, messy projects. So let’s help each other out.

Step 8: Technical leadership meetings

You can use technical leadership meetings to offer a similar type of critique for technical plans. Your technical leaders can come together and share anything interesting about the technical tradeoffs they’re making.

These meetings can be extraordinarily impactful. One of the most common causes of project failure is projects that take too many ambitious bets at the same time. And many of the challenges of engineering organizations result from technical decisions made in previous years. Having a group that can set technical standards will prevent painful migration projects from blossoming. And creating a culture of discussing the long-term implications of choices can avoid lots of pain in future years.

Demo driven development

Demo-driven development organizes your practice at the right level of granularity to really manage your project. It provides goals and meaning in the work that help make the work more engaging, creative, and inspiring for the people involved. And it results in better innovation, more customer-focus, and better planned projects that alternative approaches.

By gradually introducing the set of practices behind demo-driven development, you should improve your engineering organization, and the success of your company. Let me know how it goes for you, and I welcome any feedback or questions you have about it.

Thank you

Many experienced engineering leaders give helpful feedback on drafts of this post. Thank you to Seth Falcon, Bjorn Freeman-Benson and Kenichi Nakamura for numerous structural and content suggestions that made this post stronger and more focused. And thank you to Brent Miller, Davy Stevenson, and Darin Swanson for their improvements! Thank you to davidkunz for pointing out potential failure modes. I learned a lot of these approaches from Alex Kroman. Daslan Govender suggested some improvements to how I introduced "cross-functional" terminology.

Image by Sasin Tipchai from Pixabay

Managers are in the best position to cause misery

As managers, we're in the best position to cause sadness on our teams. If you aspire to be truly great at inflicting pain on your team, pay attention! We'll cover three of the most popular ways.

Note that this is focused on project management "anti-patterns". There are many other ways you can make the people on your team miserable.

These techniques lead to slow delivery, and can make your company less effective. With enough participation, you can cause engineering to completely fail in its mission. So not only can these techniques make your team unhappy, they can also destroy your company!

The three so-called "anti-patterns"

Sadist managers have long copied these techniques, passing them from person to person. Today, I will share them with you so you can know when to use them.

The three secret anti-patterns are:

1. Task treadmill
2. Million-meeting agile
3. Gantt-aholic

I've done all three of these approaches, and they all have merits and risks. I can report that they all cause a reasonable amount of suffering.

Task treadmill

This is the most common pain-causing technique I see in startups today. Diabolical managers turn to it because it's so effective at numbing the mind.

Here's how it works:

1. Use Jira to store everything that people think about doing related to the team’s work.
2. Break projects down into a ton of technical tasks. Bonus if the task has no description and has a very technical title that only one engineer can understand.
3. The team works against those tasks.
4. Each week you carry over the stuff that doesn’t get done.
5. Ideally, at the end, you have the work completely finished.

Task treadmill advantages

1. Working in this way is assembly-line work, instead of problem-solving work. This results in engineers disengaging from the customer problems they should be solving. It results in shallow and incomplete solutions. You’re finishing tasks, not solving problems.
2. Working from tickets can discourage people from challenging the approach, or thinking about the work they’re doing. So this discourages innovation and fresh thinking.
3. If you have any discovery work as part of the project, or the project is subject to change (i.e., most product development work), your detailed breakdown has just become a liability -- you’ve accumulated a huge backlog of tickets that need to be reworked. A splendid waste of time!
4. Often, this approach is done without discipline behind the breakdown process, so you end up creating the tickets as you go. This results in having no idea how long everything will take, or how well you’re doing. You're damned if you do breakdown, and damned if you don't! Genius!
5. This approach also can tend to encourage the project manager to focus on “points completed” and mechanical estimates based on the stories in Jira. This works fine if the work is completely broken down, but there are often areas of risk or uncertainty that are not accounted for. Thus it often leads to overconfidence based on false measurements.
6. Another common failure is teams just don’t really plan out their work. They don’t think about what can go wrong. They don’t plan for learning. They don’t think through the contours of the project. They work against the grind of the tickets in Jira.

As you can see, this is a very effective way to demotivate your team.

Task treadmill disadvantages

You should be aware that this technique isn't fail-proof. Or rather, it can succeed.

1. If you use this approach with someone that is very detail oriented, and you break down your work completely, it can work. Typically I see it work only if a senior technical person breaks down all the stories completely, and the project is completely thought through.

Million-meeting agile

If you're a twisted manager, you'll want to know all the variants. This second way of managing projects is sure to infuriate your team! This approach is to follow the form of agile, but to make it heavyweight. I call this approach "million-meeting agile".

The approach is:

1. Have a meeting for backlog grooming (with the whole team). Go over each item with the whole group.
2. Hold daily standups. Go through each item in the sprint, and have each person talk through a script. Usually this is talking about the work you’ve done and plan to do, and obstacles. Ideally this will take an hour every day.
3. Have a sprint planning meeting (with the whole team). Plan out the work for the sprint. Go through each item and make sure everyone understands it. If you're doing this correctly it should take half a day once a week.
4. Have a project estimation meeting. Typically this involves using agile poker to estimate each story, and discuss any points of disagreement. Bonus if you estimate things you may not even work on!
5. Have a demo meeting. Each person demos their work. While these may be good demos, it is yet another meeting.
6. Have a sprint retrospective after each sprint. A great practice, but oh my god another meeting?!

Million-meeting agile advantages

1. One of the biggest advantages of this approach is that it seems egalitarian. Everyone knows about everything! Everyone agrees on everything! It's a wonderful cover for your perfidy!
2. Almost everything the team does is sequential. This ends up taking a LOT of time. The argument for doing it this way is that it builds up context as a group. That sort of shared context is valuable (and can help the team feel satisfaction from their work)! Fortunately, the meetings are so dull nobody is actually building context.
3. This works especially well if you use small stories. That way, the overhead for your process is multiplied manifold. Fortunately, most teams use fine granularity in their stories and tickets, making them a perfect fit for this approach. This means the team is essentially wading through a huge queue of junk to determine what to work on each week. And they’re building context on things that aren’t actually that important.
4. It is necessarily synchronous, which can be challenging in a remote or time-zone distributed organization.

The inefficiency of this way of working can lead to disengagement. Team members will feel like they go to a million meetings but don't get to solve any problems. This will also lead them to feel like meetings are useless, so even collaborating with their teammates on a problem will feel like torture!

Million-meeting agile disadvantages

These practices are not in themselves terrible. A lot of them have good rationales behind them. So you can attempt to cause great misery but actually do an okay job if you're not careful.

1. Teams that work with stories that are high-level will tend to avoid a lot of the pain of this approach. They will have stories that are large enough to have a couple of stories a week. This reduces the frustration of "million-meeting agile", so if you want be sure to cause grief, make sure you have a lot of stories, and make them small!
2. This approach does do a good job of building shared context.

Gantt-aholic project management

Last and certainly least is Gantt-aholic project management. This is an approach usually taken by someone who is new to project management. They read a lot about project management techniques and apply them to software development. That can cause a lot of pain!

If you're seeking to cause the world to be a worse place, I would start with the other two patterns, but this can do its part as well.

So how does Gantt-aholic project management work? It is an approach that uses fine-grained planning to determine a schedule.

1. The project manager builds Gantt charts or PERT charts, and is using project management software or complicated spreadsheets to manage things.
2. They spend a lot of time each week managing the project and maintaining their models.
3. They ask the team for estimates on everything, to build the information they need for their models. (This often goes hand in hand with heavy-weight estimation, like planning poker).
4. They often maintain a list of risks. And a decision-log.

Gantt-aholic project management advantages

Although this isn't as evil as some of the other approaches, it is effective as making people doubt why they entered this terrible world of tech:

1. Most projects in product development have a high amount of change or uncertainty. Gantt-aholic planning doesn’t map well to change. It is precise, but it isn’t accurate. Discovery or changes in the project are hard to account for, because they’re not estimated in the same way.
2. Because the plan is so complicated, it requires substantial effort to handle the inevitable changes. This can make the plan more rigid than it should be, because the cost of making changes is so high. This inflexibility can unnaturally contort the outcomes of your project, and also result in work that is hidden from the model or unaccounted for. Changes can take a while to estimate because the cycle time for building the new state can be expensive. It is usually difficult to experiment with variations to the plan, and since the project manager is a dependency for any alterations to the plan, and the plan requires a high cycle time to be updated, the agility of your planning is compromised.
3. Because it’s based on a model, estimates are usually computed. This results in estimates that swing all over the place.
4. The project manager has to spend a lot of time managing the model (sometimes even taking away from their management of the project itself).
5. Usually the project manager is the only person who can update the model, as it’s too complex for anyone else to understand. This means they can’t go on vacation without losing your ability to manage the project.
6. The level of detail required means the team has to spend a lot of time estimating stories.

Gantt-aholic project management disadvantages

1. Sometimes experienced project managers can use this approach successfully, because they're aware of the failure modes. Be careful if you're working with such a person. They might make things better than you want it to be.

What is "better"?

Occasionally, you may see fit to be merciful to the teams you serve. In that case, you might want to read about a painless alternative to these styles of project management, called demo-driven development.

Thank you

Many experienced engineering leaders give helpful feedback on early drafts of this post. Thank you to Seth Falcon, Bjorn Freeman-Benson and Kenichi Nakamura for numerous structural and content suggestions that made this post stronger and more focused. And thank you to Brent Miller, Davy Stevenson, and Darin Swanson for their improvements! Sarah Madden helped me with some awkward language, thank you!

Image by István Mihály from Pixabay

I’d like to share the results of one of the most interesting management experiments I’ve been a part of. At a company I worked at recently, we implemented the Single Threaded Owner (STO) model from Amazon, where you have everyone on the team report to a single leader, including product, design, and engineering. You can think of the STO model as a more extreme form of cross-functional teams, where the team has everything it needs to deliver on its mission.

What is a Single Threaded Owner?

The idea of the Single Threaded Owner comes from Amazon. The name itself is based on the idea of a team focusing on a single thing at a time (i.e., instead of multi-tasking, you are focusing on a single thread of work or focus at a time). At Amazon, each team is supposed to have a clear area of ownership (a product area, based on a customer need) that they obsess about all the time. A leader of a team is called a “Single Threaded Owner”, or STO.

The most common alternative to the STO model is what I call the triad, or “3 legged stool” model. In the 3 legged stool model, you have separate leadership hierarchies for Engineering, Product, and Design. Design sometimes reports to Product. But you then assemble a team of engineers, led by an Engineering Manager, with an embedded Product Manager and usually a Designer. The two or three of them become a sort of local leadership team.

In the STO model, they actually all report in and are part of the same team. Everyone reports to the STO.

Single Threaded owners drive alignment and ownership

We were seeing a lack of alignment across engineering and product, and we were about to do a reorganization that aligned the teams along clearer areas of ownership. We had a lot of ex-Amazon employees, including the CEO, and they thought combining the reorg with having a single leader responsible for that area would improve things.

At a previous company, I had noticed some pathologies from maintaining separate hierarchies between Product, Design, and Engineering. Reorgs were frequently done in incompatible ways, with Product organizing along one line, and Engineering on another. Or they would be done in one organization, but poorly coordinated, so they would result in a series of reorgs across the different organizations.

On the local teams, I often saw issues where the engineering manager (EM) and product manager (PM) didn’t work well together, and these teams tended to do quite poorly. Generally what would happen is their managers would attempt to “coach them through it”, but what would happen in practice is that those teams would just not work effectively for six months or more.

So even before I heard of the Single Threaded Owner model, it was something I had been advocating for.

Transitioning to a Single Threaded Owner model

Moving to the model required choosing how the roles would be defined, how the teams were organized, and what meetings and structure we would put around it.

One factor we had to decide is who would be the Single Threaded Owner. Would it be Product, or Engineering? This is a pretty tricky matter. This is how I saw the tradeoffs:

Product manager as STO Engineering manager as STO

Typically has better business and product context, which allows them to set direction more effectively. More experienced managing people and process. Essentially more suited to be an "organizational" or "organizing" leader.

Stereotypically: like a mini-CEO who is also a poor manager. Stereotypically: like a mini-COO who lacks business context.

I made the calculation that a good people manager can work around their lack of expertise by using the skills of the people on their team. And it is harder for a person with less people management skills to compensate for the lack of that skillset. A product manager is less likely to have the people management and organizational skills, so I decided to have the PMs report to engineering. A better heuristic is: whoever is the better organizational and people leader should be the STO. The ideal skillset is an experienced people manager who also can act as a product and process leader. I found out later from Ben Bernard that this is how it works at Amazon.

We had four teams: two product teams, an agent team, and an infra team. All four teams were 3-4 people, which concerned us as they seemed too small.

After the reorganization, we ended up with three larger teams. We changed the Engineering Managers to be Single Threaded Owners, and changed the reporting so that the Designer and Product Manager for each team reported in to the new Single Threaded Owner. I acted as the manager of managers, which made me a Single Threaded Owner for all product development, and responsible for Engineering, Product, and Design. We had a Senior Director of Product, who reported to me and focused on high-level product strategy.

Although this violated the Single Threaded Owner, we had a group of three principal engineers that reported outside of this hierarchy to the CTO. They were embedded on each team, and spent most of their time with the team. This served as an architectural group.

Some other changes we made at the same time were to introduce weekly metrics review meetings, where the STOs and PMs would present what they had learned and were monitoring in their areas, based on metrics they tracked. We set up a monthly business review meeting which served as an interface between executive and local teams.

The monthly business review meetings were a chance for the team to align on direction with executives and other stakeholders. The intention was for direction to be set by each team, within a larger strategy defined by the senior product manager.

How did it go switching to the STO model?

Our results were mixed.

One team flourished with the STO model, and became one of the best performing teams I’ve seen in my career. Although the team composition changed slightly, most of the team members came from a previous team, and that team had delivered a couple of lackluster projects. Spirits were low as we went into the reorg.

I believe there were two aspects to the change that really clicked: one is that the new leadership group for the team was a strong combination. But it seems like the Single Threaded Owner model made that combination even stronger:

• The Single Threaded Owner (STO), PM, principal engineer, and designer were very aligned. I would hear the same message from each of them, presenting fundamentally a united perspective. They were more strategically aligned than any team I’ve worked with.
• They took greater ownership of their domain and began challenging some of the product direction they were “given”. They successfully pivoted the team’s direction to something they thought more important, and got the leadership team on board with those changes.
• The team became incredibly productive and incremental, to the point that they started delivering things to customers every couple of weeks. They went from having a lot of monolithic releases that weren’t released on a schedule, to releasing increments of their larger projects in ways that were useful, all the way to the customer. They did this frequently, so frequently that people stopped caring about the timelines. But they also met their timelines consistently, to the point that they were usually within a few days of their objective, and were sometimes early.
• Their work took on a different nature than it had previously. Instead of delivering large projects every couple of months, they prioritized a lot of small followup projects. They would often learn new information from customers and make a small 1-2 week project that would start the next week. I believe this resulted in much better software for customers: they were more focused on the outcomes for customers than they were previously.

Here’s how the STO (Alexa Stefanko) described her experience:

I loved it. If I could be an STO forever, I would. It made me better, it made our team better. I felt inspired, excited, challenged, and invested in a way I had not felt before. I loved that my decisions had an obvious and tangible impact. My biggest takeaway was that it made my previous antipatterns unacceptable. By that I mean that my weaknesses (relationships with PM, technical oversight, lack of market awareness) were no longer tolerable in that role. I had to take all of those weaknesses and turn them into either non-hurtful things or strengths quickly. It taught me how to have excellent and productive relationships with my PM, for example. Which had been something I had struggled with.

It also forced all of my leadership team to be on the exact same page all the time. With everything going through me, we were able to challenge each other on assumptions that were not shared. Our forced alignment meant that our team received clear and consistent direction.

While I think our data was often insufficient (and I wish we acknowledged that more often), I loved having to be so in tune with the success or failure of our products. I felt very confident in the decisions that we were making, and felt even more confident in our ability to course correct if we had done something wrong.

The second team didn’t take quite as well. The Engineering Manager felt like the new role was a huge increase in responsibility, and felt unsupported by their Product Manager. Because one of the challenges their team was having was around product direction, this ended up being a big issue for the STO.

The best thing you could say about the STO model for this team is that it made it completely obvious that it wasn’t working. The PM ended up leaving, and the situation started to improve. The biggest benefit there was that the problems were more visible and obvious than they were previously, and the EM was able to take action to improve the situation. However, the EM did report that the whole experience was incredibly stressful.

What were the strengths and weaknesses of the Single Threaded Owner model?

One of the biggest challenges with the STO model is that it asks people to forgo their identity as engineers or product managers. There was constant friction from product and design about them feeling like the STO model gave them less authority. As primates, we’ve been part of a hierarchical system of power since before we were human beings, and it’s a very natural thing for people to regard reporting structure as a proxy for status and importance. The STO model runs counter to this, and that is something that I think will be a continual headwind when in the STO model. For example, let’s say you think the weakness in your company is that you need a better product direction. If you want to go hire an experienced Product person to address that problem, you may be more limited in who will be willing to accept that role in a STO model. A lot of experienced leaders want to have a group of people reporting to them, or they think the position isn’t “real”.

I saw some low-level resistance to the STO model throughout the experiment, mostly from Product and Design. One PM seemed to see any problem as a sign that the STO model wasn’t working. The most senior PM would evaluate the Single Threaded Owners according to his own standards and find them lacking because they weren’t as product-focused as he was.

The job of an STO is very different from an engineering manager (or product manager), and it’s a harder job. I think in a lot of ways it is similar to a Director level position, because you’re leading a more complex organization, and you’re leading Engineers, Product Managers, and Designers. The biggest skill gap you’ll see in typical engineering managers might be direction setting. You can’t just be a manager, you have to lead.

There was some initial confusion on the Product Management role, and who to involve in decisions. One question was: do you always invite the STO to meetings about direction? Ideally that would not be the case, but it was a question we heard often. My message there was that a product manager is like a tech lead: a specialist that you listen to and is knowledgeable about a particular domain. For the PM, they’re a specialist who is the expert in prioritizing problems, channeling the customer, and making sure we understand the context we’re working in. Just like a tech lead, their judgement should be the most deeply considered in their area of expertise. The STO and the PM should be on the same page about things, and the STO is ultimately the person that is responsible for direction. But the PM is the person who should be spending most of their time on it. They’re serving that function on the team.

Ben Bernard, Principal Engineer: "After being on 3 different teams at Amazon and trying this again with you, it's pretty clear to me that very strong product leadership at the team level is required. This can come from the STO (at Amazon it frequently did) or from the PM. Either way this person must both be the most knowledgeable about the customers and their use-cases and the most tuned-in with a product vision. They also have to be able to prioritize things (though this can be helped by a data-driven culture or other leaders on the team)."

One of the challenges we faced is that the company was small enough that dividing up responsibilities between teams still resulted in many responsibilities per team. The idea that you could focus on a single thing (the “single threaded” idea) wasn’t completely true, as the areas of ownership for each team was broad.

The people supporting the STO are incredibly important to the success of this model. If you have a Product Manager, Tech Lead, or Designer you’re not confident in, you may be setting up the STO for a losing situation, and you should adjust your expectations accordingly. According to Ben Bernard: “This definitely happened at Amazon -- Amazon just fired people.. Not without trying to fix things, but people changed roles or left the company fairly frequently. This had some bad knock on effects with old timers feeling like Amazon couldn't retain anyone, and not feeling very tied to the company. However, I thought they did a decent job of moving people out when needed. The exception to that was with higher level folks. If you had a team that did its job but had terrible people managers at several points in the chain, that org would be terrible to work in. Single Threaded Ownership contributes to these because each level is a STO at amazon, with a large amount of leniency in how things get done. This contributes to every organization being pretty different. AWS and Kindle were famous for having different engineering cultures, for instance. This also led to me giving this advice to fellow engineers: ‘never change teams unless you have friends on the new team and know it isn't messed up’”.

We were moving to be more metrics driven at the same time, and although we saw good behavioral adoption of metrics, and learned a lot from being “metrics-driven”, it sometimes felt contrived because the amount of data we had wasn’t actually enough to be scientific. The main benefit from having the “metrics review” meetings was to force the STOs to think about their domain and report on it. The metrics themselves seemed less important than the act of reviewing their area. Over time, however, I think we would have matured and become more and more metrics driven.

One of the challenges the STOs faced was in the interface between leadership and the STOs. Ideally, the way this works is that the leadership provides goals, and the STOs choose projects to meet those goals. In practice, this can be hard within a small company, when the founders and executives want to be highly involved in the team’s work. In our case, the local teams weren’t fully empowered to be autonomous, so they weren’t fully able to make decisions without them being overridden by executives. That kind of violated the expectations of the STO model, and resulted in back and forth and friction.

Another thing to note is that Single Threaded Owners can discourage standardation. Each area tends to solve things in different ways. So a side effect of this approach is that different teams and different organizations may feel very dissimilar to each other.

Do I recommend the Single Threaded Owner model?

I believe the STO model is more effective than the standard “3 stool” model of having separate leadership hierarchies. But it has downsides and requires more discipline to implement and maintain, so I plan to keep it as a tool I can employ, but not one that I will use in most situations.

Here is how I will reason about when to employ it:

• Ideally, you would start out with this model. It’s almost impossible to move to it once you have already set up the separate hierarchies, unless you have extremely opinionated and forceful leadership. Very few heads of product or heads of design are going to want to give up all their direct reports. They’ll see it as career limiting.
• I probably wouldn’t introduce this in a larger organization. There are just too many barriers to being successful with it that I wouldn’t use all my political capital in this dimension.
• A large decision factor for me would be the working culture of the company. A place where people are more flexible in their roles, or more obsessively customer focused will succeed better with the STO model. The flexibility of the culture is another factor. If I believed I could shape the organization fully over a long period of time, I’d be more inclined to go with the STO model.
• A work culture that accepts structure will be more successful as well. Being successful with the STO model requires good standards, and good process.

If I do employ it again, some things I will pay attention to:

• I will hire STOs differently than Engineering Managers and Product Managers. You need people who are really good at creating direction out of nothing, good people and process management, and ideally good product sense or even product management experience. A person who has both product management and engineering management skills is your perfect fit, but they are few and far between. Look for organizationally focused Product Managers, and product-focused Engineering Managers.
• I will spend a lot of time talking through people’s roles. They see themselves as “Product Managers” and “Engineering Managers”, so a role that crosses those two is going to be confusing for them.
• I will anticipate the challenge many managers will have to switching. In particular, I’ll evaluate the strength of their supporting roles (technical, product, and design) and better prepare them for the gaps. This is a very large role transition, and many may not be successful with it, especially if they don’t have the right people in place to support them. Some people may not make the transition and may need to be switched to another role or managed out.
• Switching to the STO model requires you to develop good standards for all the positions. You need role definitions and career ladders well defined, so that people can be promoted fairly even if their manager isn’t someone with their specialist skillset. This isn’t a prerequisite, but you are signing up for that work immediately if you implement the STO model.

I find it disappointing to conclude that “the STO model is better” but you might not want to implement it. I think what it has shown me is some of the benefits of truly decentralized teams, with strong ownership, that are radically aligned. It has shown me what is possible, and I have found myself pondering hybrid models, and wondering if they might achieve some of the same benefits. One thing I’ve wondered is if adding a few modifications to the conventional way of managing might provide some of the same benefits.

As a thought exercise, you might be able to achieve some of the same outcomes by a combination of these things:

Write down an organizational philosophy that works across design, product, and engineering. I would include team size, ratios between specialties, and what principal by which the teams are designed (for example, based on product area, or customer segment). Write down a short list of rules. One of those rules, for example, might be that all reorgs have to be coordinated (or perhaps one org can own reorgs), and done as a group. This is a pretty heavy lift, and something easy to ignore, so I don’t think most organizations would do this in advance. You get it for free with the Single Threaded Owner model.

Make Product and Design into service organizations. Product provides direction, context, and customer information. Design provides usability and usefulness. If either of them aren’t providing this to the satisfaction of the Engineering Manager, they can essentially “end the contract” with that service organization. As an example of how this would work, imagine Lisa is the Engineering Manager for a team. She doesn’t feel she’s getting good enough product direction for her team to effectively serve their team’s mission. She could tell the Product leadership that she’s not happy with the level of service she’s getting, and “send back” the Product Manager she’s working with. The Product organization then would need to provide another person as soon as they could, and they could either move that PM to another role or let them go.

I could see this working effectively in either direction, with either Product or Engineering having the ability to end the contract.

There might be ways to set up the responsibilities so that the Engineering Manager or Product Manager have an expansive view of their role, that is similar to the “you own everything” approach of the STO model. I once had my manager tell me, “everything that happens on this product is your fault. If customers are supported poorly, that’s your fault. If you have bad product direction, that’s your fault. You’re responsible for making this all work.” Some of this was actually a stretch -- I wasn’t technically responsible for the product direction. But that sort of expansive view of a role can be helpful to ensure people make sure things are working, even if it’s not in their lane. Giving people the message that the Team’s success is what counts, even if it’s out of your lane, is probably a message we need to emphasize more.

General Manager model

The Single Threaded Owner approach doesn't have to extend all the way down the organization. A variation is a General Manager model.

A General Manager is a person responsible for an organization. They might have separate hierarchies underneath them -- product, engineering, and design. But they're responsible for all three.

General Managers are used to separate an organization into divisions. The nice thing about this approach is that it is fractal -- you can choose to take it as far as you want to. But you can also just do it at the top level, and extend further downwards as needed.

Want to hear a deep dive?

I interviewed Alexa Stefanko on the Single Threaded Owner model for my podcast, Decoding Leadership, and we did a deep dive on what made this so successful for her team. We talked about the other changes we made at the same time, and a lot more nuance than I was able to go into in this article.

You can also hear it directly below. If you like it, please subscribe!

I also interviewed Jason Poole on some of the challenges of the Single Threaded Owner model. You can hear it below.

Want help?

If you haven't done these things before, it's often valuable to have an executive advisor. Contact me if you'd like to explore this!

Thank you

I’d like to thank Alexa Stefanko and Ben Bernard for contributing their thoughts to this blog post. And I’d like to thank Jim Shore for suggesting I write this article.

Image by Myriams-Fotos from Pixabay

I was listening to a podcast where the host was interviewing Dan Heath, the author of Upstream: the Quest to Solve Problems Before They Happen.

The author tells a story about Jeff and Marta, who are standing by the river. They see a child drowning, so they rescue him. As they're drying off, they see another child drowning, so they jump in and rescue her. As they get her to shore, they notice two more kids in the water, flailing around. After getting the two of them out of the water, Marta starts walking upstream. Jeff says, "where are you going, we've got to rescue these kids!" And Marta says, "I'm going to go find who's throwing them in the water."

Our daily work obstructs upstream thinking

In our daily jobs, there are a lot of factors that prevent this type of Upstream thinking. For example, when you're the person that is responsible for answering support issues from customers, or when you're handling lots of incidents, it is often rational to do what you need to to satisfy the current situation. You answer the ticket, you fix the bug, you tune the alert.

Three reasons transactional approaches don't solve things deeply

Far too often, this type of transactional approach doesn't really get at the heart of the problem, and prevent it from happening again. The author outlines a couple of things that prevent us from doing deeper work:

1. The first is "tunnel thinking". This is when you're focused on solving things in a transactional way.
2. The second is distributed responsibilities — when it's not clear who is responsible for something, or nobody really owns it.
3. And finally, an issue is when you don't have the time or space to stop and contemplate the situation and understand it deeply.

When should you engage "upstream thinking"?

We always have to use our best judgment on how to invest our time, but a good rule of thumb is to weigh the costs and benefits of addressing things more deeply. If you see something happen more than a time or two, solve it more thoroughly.

Some approaches to build the habit of upstream thinking

Here are some things to consider that can help you get better at upstream approaches:

1. Reply with documentation. When someone asks you something, document the response in a place that a future person can look it up, and send them the link.
2. Automated testing. Prove that the bug can't reoccur, which prevents it from happening again.
3. Identify "do not repeat incidents" work after an incident. Make sure you identify the minimum possible thing that would prevent or mitigate future examples of this problem.
4. Turn off an alert, or rethink how we monitor something. Rather than just bumping up the threshold, think about whether the alert is something we even act on. Move the channel of the alert to an information channel instead of alerting channel, or rework it to be actionable.
5. Automate the steps into a script. Instead of going through a list of things, put it in a script, which is a step towards automating it.

If nothing else, it can be valuable to take a few moments when you're working on something to look at the big picture, and not let the rush prevent you from looking upstream to where the source of the problem may lie.

Questions to ask yourself

To come up with your own upstream thinking, here are a couple of prompts to ask yourself:

1. What is the trend I'm seeing? If it continues, will it be unsustainable over time?
2. Is there a way I could make the impact of this decrease over time?
3. Are there any simple ways I could make this situation go away?
4. What would happen if we ignored this problem?

Beware solving everything deeply

Occasionally I see people so in love with upstream thinking that they only embark on huge efforts to solve every problem completely. This is misguided.

For many problems, solving things completely is too large of an investment. It may be fine to solve things incompletely if it requires significantly less effort.

Image by 德明 胡 from Pixabay

Hiring well is the most important thing for companies. Today I share an easy way to plan interviews that almost guarantees better results.

How to coordinate poorly

The most common (and terrible) way to organize interviews is to:

• Decide on a group of people who can interview the candidate.
• Come up with a list of areas you want them to dig in on.
• Have someone schedule the time slots
• Have each person give a recommendation to hire or reject the candidate.

This is problematic in a million ways:

• When I’ve been on panels like this, I’ve had candidates tell me every person asked them the same questions. So the candidate experience is pretty terrible.
• You’re not really guaranteeing you are going to learn what you need to with the candidate.
• You have no idea of what each interviewer is doing, so you have no way of knowing if their feedback is solid or not. You have no visibility into the quality of your interviews.

The main problem is that there is no coordination.

Coordinate your interviews with assertion-based interview plans

So let me share an interview format I use.

The basic process is:

1. Make a list of assertions for an ideal candidate.
2. Create questions to test your assertions.
3. Group and assign the questions.
4. Tweak the interview format continuously

Make a list of assertions for an ideal candidate

Make a list of assertions that should be true for the person to be right for this role. For example, for an engineering manager, you might come up with a list that starts like this:

1. Has improved the organization they were a part of
2. Able to attract and hire great talent.
3. Experienced with org design and scaling organizations.

Roughly order the list by what is most important for the role. You won't be able to test everything, so focus on what you care about.

Create questions to test your assertions

For each assertion, list an interview question (or action, like a programming assignment) that will uncover evidence of how strong the candidate is in this area. Indent the questions.

1. Has improved the organization they were a part of
   
   Tell me about the largest improvement you've made in your organization. I'm interested in process changes, or direction changes, or other things that had an impact on the company.

If you can, add what a good answer might look like. For example:

1. Has improved the organization they were a part of
   
   Tell me about the largest improvement you've made in your organization. I'm interested in process changes, or direction changes, or other things that had an impact on the company.
   
   A poor answer would be switching to Kanban, or introducing standups (yes these can improve things, but the impact is very local). A better answer would be something like changing the way leadership worked together, or how people communicated, if the impact were good. A perfect answer would relate the impact to the results on the company, and the results would have made a substantial difference for the company.

Do this for your top questions. If you'd like to see excellent examples of what good answers look like, this article from Medium engineering is quite good.

Group and assign the questions

Now we flesh out the interview plan and assign people to it.

First, group your assertions by category, like Managing Projects, Managing People, Managing process, and Technical decision-making. These are your interview sections.

Your list will now look something like:

Interview Section Category A (Interviewer i1, Interviewer i2)

Assertion 1

• Question to test assertion
  • What a good answer looks like.

Copy this this Google Doc and put your assertions and questions into the format, adding the people who will be responsible for each section.

(optional) Choose two people to do each interview. They'll observe the interview better, and learn from each other, which increases your organizations ability to select candidates over time. One is okay if you can't find a second person.

Work with each interviewer to improve the questions they will ask. Tell them what you came up with is a starting draft. Then, you act as editor, read through the interview and think about the interview experience, and whether questions duplicate each other. Make sure you have confidence asking these questions will generate the answers you're looking for.

Before you start the interview, copy and paste the questions you're asking to search for bias in the questions you're asking. I often find I'm gender coding in a way that might exclude women -- for example, in a recent interview I used two masculine-coded words: "active" and "force". Usually removing the gender coding also makes it more clear.

Tweak the interview format continuously

After every interview, encourage everyone to ask themself these questions:

1. Did I get the information I needed to make a confident assessment of the candidate? If not, I should change my "assertions".
2. Did I understand the results of the assertion by asking the questions I did? If not, I should tweak the questions I ask.
3. Did I hear a particularly good or poor answer to the questions? If so, I may want to tweak the "what does a good response look like" section.

Why does this format work?

1. It separates out what you want to learn from how you are going to achieve it. This makes it easier to iterate on the interview format after every interview.
2. It is explicit about what you’re looking for in a good answer. This reduces bias, because you will have thought of it beforehand. This helps with interviewees that may not be the best interviewee, but are giving answers that indicate they might actually be the right person for the job.

I hope you find this useful. Let me know what you think!

Image by Adabara Ibrahim from Pixabay