Note: I wrote this during my 18 months as CTO at MongoDB and titled it “A blog (novella?) I can’t figure out what to do with” — because I never quite figured out where to put it. It’s an open letter on culture to product and engineering organizations, including (at the time) MongoDB itself.
I think a lot about culture, and so should you
Great cultures and successful companies go together. If you have a bad culture, everything is hard - and even if you have one success, it’s not predictable that you’ll have another one. On the other hand, the business outcomes of a great culture are undisputed. As Jack Welch, CEO of GE, said, “An organization’s ability to learn, and translate that learning into action rapidly, is the ultimate competitive advantage.” Culture is important to foster innovation, to keep iteratively in touch with the market, to create and maintain employee engagement, to deal with setbacks and challenges, and to help retain top talent.
At MongoDB, we put culture front and center in our mission and values. It’s not enough just to have it up on a wiki page or a poster. We have to be intentional — seeking out what we’re doing well and where we can improve. We pay attention to culture just like we pay attention to product strategy or innovation. During my 18 months as CTO at MongoDB, I’ve spent a lot of time thinking about culture - and talking about it in All Hands meetings, written documents, and internal podcasts. Over time, my thinking continues to evolve, and I’m thankful for all the help I’ve gotten from my C-level peers in industry, my mentors, and many MongoDB employees. We’re all on a journey together.
My goal of putting all this thinking into an open letter to both MongoDB employees and to the industry at large is to enable us to keep having discussions about our culture, and for those discussions to facilitate changes where we need them. This letter isn’t perfect. It misses some things, I’m sure, and mis-states others - it represents my thoughts right now. We’re adapting in response to the needs of our customers, the wishes of our employees, and what’s happening in the world as a whole. I hope to look at this letter in the future and see it as a waypoint rather than a destination. I hope you’ll see it as one of yours, as well.
What is culture, for our product and engineering organization?
I’ve said how important culture is, but I’m surprised regularly by two things. First, how rarely people ask about culture, and second, how pleased they are when I talk about it. Of course, people often ask me about my predictions for the future, what’s important to me and the company, and where I spend my time and attention. People expect me to start my answers by describing the software we are building, the speeds and feeds it will handle, and the clever technical things it will do. In other words, they expect me to start with our technical model. After all, that’s the “T” in CTO. They also expect a similar speech about our product vision from my peer, our Chief Product Officer, Sahir Azam. He and I work on the technical vision and the product vision as “two in a box,” meaning that the two are intimately linked and interdependent.
Of course, these two things are very important. But they are just a part of what’s most important - our culture. Our culture comprises a lot more than our technical model or our product vision. We also need to be thoughtful about how we envision, prioritize, build, deploy, and operate our software — that’s our working model. And we need to be very intentional about how we work together as humans — what our rules of engagement are, what we can expect from each other, and what’s forbidden. That’s our social model. We have to have a vision for all of these things, or we function suboptimally. To sum up, our culture is our product vision, our technical model, our working model, and our social model. Our culture defines what we do every day, and how we act and feel while doing it. In it’s best state, it is a published and shared set of values that we’ve agreed upon and that we follow. Existing members of the company can measure their own and other’s behavior against those values, and new employees can use the public statements to ramp up on the culture quickly, or even to learn about it before joining the company, as I hope many of you reading this are.
Why is culture important? MongoDB has an awe-inspiring opportunity in front of us - and this is one of the key reasons I joined. As the data landscape changes over the last 10 and next 20 years, we have everything we need to fundamentally change how people write apps and manage their data. Succeeding at this will result in not only a wonderful company with delighted customers but also tens of thousands of successful careers, both inside and outside our walls. But, we can only do this amazing thing if we execute well at all levels, all the time. How we execute is what culture is - and we all have to agree on it to execute together. To that end, I’ve put below eight cultural guidelines that I’ve found help teams do excellent work, produce great products, and delight customers. I hope you find value in them, whether you’re practicing them here at MongoDB or in industry.
You have to have a framework for balancing non-functional requirements and features
A key element of our culture is satisfying customers. They have real and justified expectations of our software. After all, we produce, at our core, a database, one of the most complicated parts of any tech stack. As I’ve said elsewhere, databases make promises to customers that are among the most challenging of any in commercial computer science in terms of durability, availability, and correctness, among others. And, if you want, we wrap that database in a magical managed cloud service, a development platform for server and mobile apps, an analytics engine, and all sorts of other features. Our goal is to be the application data platform that our customers expect, wherever they need it, at the appropriate scale.
Let’s go into detail on 7 important expectations customers have.
At the most basic level, customers expect our products to be secure and not be breached. They expect that data they put in them will be durable — it’ll stay around through any challenges the world comes up with. They expect the data to always be correct, especially in an ACID-compliant database like MongoDB. They expect the database to be available (up all the time). They expect it to easily scale out when they have a burst of activity, and they expect it to be operable by mere mortals.
Boy, those are a lot of expectations. Go take a deep breath and come back. The six things above are all “Non-Functional Requirements”(NFRs). Across three decades of enterprise software development and deployment, I have come to believe that these six NFRs are just as important as features. After all, in order to delight our customers, the features they’ve asked for have to actually work. Otherwise, customers are faced with an unreliable product. Nobody cares about new features when the system is down. Nobody cares about availability if the data is incorrect.
Why on earth is this all related to culture? An element of any good culture is that it gives frameworks for choices to be made - i.e. it defines the “rules of the prioritization game.” Without a framework, it’s very hard for teams to make choices between these seven key areas to decide where to spend time and resources - and different smart people might make different choices. Any experienced tech leader knows just how much time we spend on making these choices.
The framework that I offer for consideration is this: In general, in the list above, the requirements I list first take priority over the requirements that come later - i.e. security, durability, correctness, availability, scalability, operability, and features.That said, I’m not advocating that NFRs are more strictly important than features — that doesn’t work either. You have to have all seven to be successful, in the right balance. And that balance is debatable and mutable across products, teams, and even time. That means that each team must own the responsibility for making the right choices, each planning period, for their unique requirements and goals. Our feature design process and our quarterly planning process all roll into our working model — which has to have this balance front-and-center in every discussion. If we get this choice framework part of our culture right, customers get the right quality at the right time - and our teams function at the highest possible efficiency and cohesion.
Technical debt is a choice
How and whether you incur and pay down technical debt is the single most important cultural element I’ll talk about in this blog.
Technical debt is like financial debt — it’s a tool that can allow us to do things that might otherwise be out of reach - by taking withdrawals, most often by releasing software that is incomplete on some dimension. But if we take on too much debt, we spend our lives chasing bugs and can’t build new features. If we take on too little, we aren’t moving as fast as we could in an ultra-competitive market. A really unsettling consequence of too much tech debt is that our best employees will get fed up and leave. They want to build innovative and exciting products, and spending their days working around the last 10 well-intentioned hacks makes that impossible.
It is so easy to take on tech debt. Cut the corner, add the manually set config parameter, or knowingly ignore an edge case — but ship it! That’s not necessarily evil — it’s a business decision. But be careful: We need to track how much debt we have, and we need a plan to pay it down. There is interest on the debt, in the form of software with rough edges, which is hard to understand and improve; and the fact that poorly written software becomes more difficult for developers to maintain and improve over time. The principal of the debt takes the form of the work needed to rip out the hack and build the capability correctly.
It’s important to separate technical debt from the need to improve our architecture. Tech debt is the consequence of doing things that we knew weren’t great at the time, most often to get something out the door fast. Architectural changes are different. We make those to improve something that was written as well as it could have been at the time, but thanks to new information, capabilities, or requirements, we now want to improve it. This is an important cultural distinction. You shouldn’t need to optically lower an architectural change to “paying down tech debt” to get it scheduled, nor should you have to lie and say that paying off a shortcut you took in a previous release is an architectural improvement. We’ll have different levels of technical debt in different parts of our organization, and that’s okay and intended. The appropriate level of tech debt depends on the criticality of a particular part of the software, or where the software is in its lifecycle. My advice is to be open and transparent about our tech debt choices — embrace incurring the debt and embrace paying it down, in a partnership between the product, engineering, support, and customer success teams. Having all those groups represented in our decisions is a necessary part of our working and social models.
I talk to a lot of other software folks, and I can often get a handle on the culture of a company by asking a handful of questions about the current state of technical debt and architecture. I start by asking “What are your customers happiest about?” and continue with “What are your developers struggling with?” And then I ask “What is your vision for the next 3-5 years and are you confident that you can get there with your current code, teams, and processes?” We need to constantly ask ourselves these questions to prevent regression to the mean of a mediocre product and engineering team. Sometimes, when we ask these difficult questions, we’ll find out that people aren’t happy, and the reasons they aren’t happy can be core to our ability to succeed as a company. Which brings us to taking pride in our work.
Listen carefully to the amount of pride your employees feel
Pride in our work is crucial on so many levels. As a developer, you know when your code is good, and you know when you’re taking ugly shortcuts in the name of expediency. Software quality, or the lack of it, is hard to define but easy to see. If you’re not proud of your product, your design, or your code, and you wouldn’t willingly put it up on GitHub with your name on it, something’s wrong. If you feel this way about what you’re producing, this is a serious problem, and you need to discuss it with your leadership. At MongoDB, “leadership” is up to and including me, Sahir, and the rest of the executive staff.
This feels obvious, but I’ve seen teams get it wrong for decades. Many teams overcomplicate this discussion. We don’t need or want fancy analysis metrics or tools to know whether we’re producing quality code — code has a smell and any experienced engineer can tell if it’s a good or bad one. Poorly designed products have that same smell, and a product manager or designer can sniff it out immediately. If a team asks to take some time to fix things so that they’re proud of their work, a typical Dilbert pointy-haired manager technique is to make the team stand up yet another project — this one to measure how bad things are—to justify it. This is a waste of time and a passive-aggressive way to tell your product managers, designers, and engineers that you don’t trust them. We hire excellent people at MongoDB and you probably do at your company too. Trusting them to know the quality of the work they produce is foundational. Give people the time they need to create work they are proud of and listen to them when they are concerned. They aren’t slackers or ivory tower idealists, the vast majority of the time.
Don’t get me wrong - pride is complicated. You might be justifiably proud of your cool workaround (a.k.a. tech debt) that gets something out the door this sprint - and you probably should be. Just make sure you’re also proud of how you remove that hacky coolness later and replace it with great production-quality code. Let’s revel in being told by our executives to be proud of what we do. As part of our culture, the social model needs to be structured so that we look for and reward pride in all the right ways. We’re a company building important products, and a company that should last for decades. Maintaining pride in all we do is one of the most important ways we can keep the right people here, building the right products.
Trust is an essential part of a productive work environment
Speaking of people, none of us work alone. In this complicated world, we constantly partner with people with skills both similar to ours and also with a complicated rainbow of people with skills that complement ours — that’s how we get amazingly complex products out the door and support them.
Of course, at some primitive neolithic level, there are days we’d all like to be able to do our work independently, without having to depend on anyone or anything. The cognitive load of working with other humans can be large. After all, there are plenty of dependencies to deal with in engineering: highly coupled code, scripts, branch strategies, etc. But on a project of any size, you have to depend on other people. So it’s just as important to know and trust your people dependencies as your technical ones.
Ask yourself - do you feel confident in your coworkers? Do they feel confident in you? When they do something wrong, do you think, “Wow, I’m not sure I understand that, but I’m sure they did something smart or at least well- intentioned.” Or do you think, “Wow, that person’s a jerk!”
If, as a group, we can have the first reaction, we have trust. And thus, we will have a chance to build great products. If we have the second reaction, it will take us longer to build products, and a ridiculous amount of time to debug and fix them. It will all be tortured, political, and unpleasant.
The action behind this guideline is simple: monitor the level of trust in your teams, whether you’re an individual contributor or leader. If there is a lack of trust, don’t put up with it. Raise it as an issue. You might think you’re alone, which is one of the key indicators of a trustless environment, but you’ll most often be surprised and find others are concerned as well.
Most people have been fortunate enough to work in a trust-filled environment at least once in their career, and unfortunate enough to have experienced the opposite. Let’s keep MongoDB the trust-filled environment that got us this far.
We ship our organization, whether we want to or not
Conway’s Law says that products made by any organization reflect the communication structure of the organization itself. So if two managers don’t get along, the product produced jointly by their two teams will have a terrible interface. Or if two parts of a team don’t communicate well, and each designs half of the customer-facing API, customers will find the method signatures to be bizarrely and frustratingly different. If you partition your code wrong across teams, and those teams have poor interactions, your code will reflect that poor team structure and dysfunctions.
Of course, it’s natural for our groups to grow and change over time. We add people and charters, and only after things settle do we realize that we need to reorganize. That’s normal. We want our teams to be highly aligned but loosely coupled, and we want to monitor this. In a functional organization, we spend time aligning on strategy and overall goals, but get out of each other’s way when it comes to the daily work. In a dysfunctional organization, there is little time for strategy. Instead lots of effort goes to boring tactical cross-team dependency management and coordination. Which kind of org are you in? What about your peer orgs? The key is to recognize the problem and then be open about restructuring.
MongoDB, like any large company, has organizational structures that have grown and morphed for years. Many of them continue to make as much sense as the day we put them in place. But not all. We need to always be alert for organizational over-coupling and be open to reorganizing in order to minimize it. Taking an output-first approach to our organization, and being willing to challenge the inertia of the current org chart, is the sign of a successful company—one that will stand the test of time. It’s ok to ship the organization if it’s the right organization.
Stay within your Dunbar number
Yet another complication is that not only is org structure crucial, but the size of the units in the organization is as well. Here I’ll give a guideline on how to think about how big your individual teams should be before you couple across an API or other trust-building mechanism. If building great companies was easy, everybody would do it, right?
The anthropologist Robin Dunbar discovered that mammals with larger brains tend to live in larger packs. For humans, Dunbar estimates the maximum comfortable social community size, or Dunbar Number, to be around 150. Simplifying Dunbar’s research (which you should read here and here) significantly, the optimal community size is used as a proxy for the number of people with whom each person can maintain stable and meaningful relationships – often seen as being the number of people they can trust. Of course, your number may be different; for example, mine is a lot smaller than 150. Maybe that’s why I like computers so much. If you have to work with more people than whatever your personal Dunbar number is, you won’t work as efficiently - your communications will be more formal, political, and slow. And if you have a whole team that needs to work together, and that team is bigger than the members’ Dunbar Numbers, then you need to think about reorganizing that team. Obviously, these are guidelines and not rules, and your personal and team experience may vary wildly from Dunbar’s research. The point is to watch out for size or coupling as an inhibitor of high functioning teams.
Being part of a pack or community is not, on its own, cognitively difficult, even at 150. We can pretty much all have shallow relationships at or near that level. However, with more research, Dunbar found cognitively richer relationships to have a lower Dunbar number. In his latest research, Dunbar figured out that there are natural Dunbar layers, with the inner layers being smaller and more intimate (~5-10) and each successive outer one (~20-30 and ~60-100) having less trust and intimacy with more people. Maybe that’s why the median guild size in World of Warcraft is 61, and the number of guilds above 180 is negligible. As techies, I surmise that our Dunbar numbers are relatively low — about 30 or 50. Why am I talking about this in an open letter about culture for a tech company? Because there are multiple levels of trust in a company, all the way from trusting each other to be good colleagues to trusting people with your own personal success. And, as I thought about what intimacy and trust meant for engineers who sit in front of a screen all day, I’ve come up with a corollary to Dunbar’s Number for developers. One of the most intimate things you can share with another developer is your stack or your address space. Anybody you share your address space with can (in most languages) ruin your day in a hard-to-detect and hard-to- fix way with a wild memory write. So, when you’re writing code, you should be concerned about how many people share your address space. If you are sharing that space with more people than can be contained in one of your inner levels of trust, you need to refactor your code or how you all work on it.
In summary, watch out for team size just as much as trust-creating ways of communication. Typically, by breaking into smaller teams and enabling people to trust each other, we can work more happily and for knowledge workers, that translates directly to more productivity.
Take risks and learn
No one plans to fail — at least I hope they don’t. But if we’re doing our jobs, we’re going to have failures. Some failures will be surprises, but others are literally designed into our systems - and aren’t really failures at all. For example, every time we run an A/B test, (at least) one of the two versions has poor results. It’s not a terrible failure. It’s just an experiment that didn’t work. And we need to keep experimenting if we want to make our code, our designs, and our products better. I’m allergic to the phrase “embrace failure”, and I’d advise you to be as well. It’s in style right now, but please don’t embrace failure. Instead, figure out where you need to experiment, and embrace experimentation instead. The very best decisions are often ones that we iterate towards over multiple experiments. You’ll know if your culture doesn’t encourage you to experiment - because you’ll instead feel pressured to be exactly right the first time on any new endeavor. In any knowledge-based innovation-driven organization, that’s impossible. So, propose experiments. Note that in the scientific method we all value, we’re encouraged to predict outcomes based on hypotheses before running the experiment. So, take it one step farther and include in the experiment proposal the things you expect to do or learn from the varied likely outcomes of the experiment. This will get everybody aligned on the value of the experiment and remove any possible stigma of just shooting in the dark.
When, as leaders, we see experiments that didn’t work, we need to hold them up as examples of wonderful things that people tried. Because of organizational power dynamics, people lower in the org will always be hesitant to admit to failures, no matter how much we preach their value. We have to offset that by always being positive and forward-thinking about experimentation. Otherwise, we can say good-bye to innovation, and good-bye to our goal of building world- changing software.
Give candor and context to get accountability
Accountability is defined as “an obligation or willingness to accept responsibility or to account for one’s actions.” An organization that has high accountability is a place where people say what they’re going to do and do what they say they will, knowing that there will be appropriate good and bad consequences, depending on the outcome.
Every manager wants accountability from their employees. Wouldn’t it be easy if it all just worked that way? The good news is that high-performing employees want accountability too! They want to do great things, and they want to feel the pride that comes from a job well done–one where they had impact and were recognized for it.
But if we’re not careful, we create environments where accountability is actually micromanagement on the way down the org chart and fear and politics on the way up. Poorly implemented OKRs are the current instantiation of this phenomenon plaguing the tech world, but that’s a story for another blog.
The best way I’ve found to create accountability is by communicating with candor and by giving a surprising amount of context. Let’s tease these apart.
Candor
When we communicate candidly, everything becomes low-stakes. Openness, honesty, and good intentions rule. There are no scary conversations; no “gotcha” moments. For example, we don’t use the power move of faking being surprised when somebody doesn’t know something. We don’t play power word games to win arguments. Another element of candor is that we endeavor to speak the truth — regardless of whether we are speaking good news or bad.
In a candid environment, we attempt to deliver all communications with both good intention and with the presumption of competence. Both are essential. After all, if you think somebody doesn’t have good intentions or isn’t competent, go talk to their manager or HR representative and stop wasting your time trying to get work done with them.
While speaking or writing correctly is important, listening and reading correctly is just as crucial. This means we have to receive communications where we presume good intent and competence from the giver, even if the person said something that seems boneheaded and clearly wrong. This is closely related to The Principle of Charity, which I encourage you to read more about. This means that we don’t just focus on the words that are used but also the intent. Obviously, some words are hurtful and should never be used, so there is a balance. The point is that in a fully-formed culture, people have agreed how to speak and listen constructively. In some cultures, this might involve avoiding certain conversation topics, while in other cultures, more things might be fair game. The point is to be aligned - so that people feel safe speaking within the social contract of the company, and people know how to listen, also within that same social contract. Notably, outside of work, I’ve personally found the principle of charity to have significantly improved my personal relationships.
Note that candor does not bring consensus; that’s not the point. Candor means contributing information so that all parties can work with the same facts - and that they were shared and received safely. Combined with a presumption of good intent, candor enables us to have the important constructive debates we need to have to keep the company moving forward quickly.
Frankly, I never believed a company that executed well on candor, charity, and intent was possible. No company I’ve been at prior to MongoDB exhibited these behaviors anywhere near as well as we do. I’m humbled to be here in this great environment and working daily to unlearn 30 years of other behaviors. For those of you starting your careers at MongoDB, don’t expect to find anything like this healthy a culture other places.
Context
Candor is a good starting point, but you can’t make decisions without information - and so many companies are reticent to share information throughout the company. Communicating context means that management and employees will share surprising amounts of information up and down the org – about profits, customers, tickets, teams that are doing well or struggling, and anything else that’s needed to make a thoughtful decision. If you need to make a choice, but we, as leaders, haven’t provided you with the proper context, we haven’t done our jobs. Alternatively, if you, as employees, expect leaders to give you resources or guidance, but haven’t shared what’s going well and poorly in your team and why, you should do so.
I, and all MongoDB leaders, want to be held accountable for treating each and every person we work with as a responsible professional, speaking with candor and providing all the information needed for us all to do our jobs. Obviously, there is lots of information that can’t be shared - from compensation to health to people issues. But that info is rarely needed to make work decisions. There is an escape valve here; if you don’t have the information to make a decision that’s part of your job role, you should ask to have your job role redefined - or get the information. In summary, a successful company needs employees who 1) feel safe having candid conversations 2) have all the information they need to make the decisions, 3) are empowered to make decisions, and 4) know they can take risks and not be punished if things go wrong. Once they have those four things (no easy task!), my personal experience is that employees are eager to take on projects and the accountability for them.
If you, as a leader, have a team that seems reluctant to volunteer for projects and own their outcomes, I’d advice to start your debugging with the four items listed above: Candor, Context, Empowerment, and Safety.
Avoid Unnecessary Approvals
There is a balance between independence and needing permission to do things. Too much independence brings chaos; over-reliance on permissions clogs processes and clouds accountability. To have a great culture, we need to strike the right balance.
One area that is particularly nettlesome is business approvals. Because they can be high-stakes, approvals are where the rubber hits the road on culture. None of us want a slow-moving, top-down, approval-driven company — even the people who are asking to be approvers. And none of us want a chaotic, wasteful workplace— even the people asking to skip approvals. So what’s the problem? There are at least two things that make this hard. First, there are rarely rules that determine whether something needs approval or not. The rules have often “just grown” over time and appear capricious, even when they aren’t. Second, an incorrect approval often has quite negative consequences, often irreversible, while an incorrect rejection merely slows the company down and is easily reversible.
Sorry, like many of the guidelines in this letter, there are no hard and fast rules for how to get this right. The goal of the letter is to raise awareness that this is an issue that you need to come to cultural alignment on at your company, in your market, at this time.
Instead of hard and fast rules, let’s look deeper. It turns out that there are ways to detect both appropriate and inappropriate levels of approval. Appropriate high levels of approval are for things that have legal or financial implications, ones that require very broad context (perhaps across the whole company), or decisions that have irreversible consequences.
Inappropriate high levels of approval can be identified just as easily. They involve decisions where the person making the decision knows less than the person asking for approval, decisions with tiny consequences, or decisions that are easily reversible. Another bad pattern is an approval where the requestor has far more responsibility in another area that the need for an approval would indicate — an empowerment inversion. Those kinds of requirements can be very demoralizing. Recognizing which approvals you need and which you don’t is a major step in building a great culture.
Of course, you’ll still need some approvals. But can you make them better? Yes, by figuring out ways to get opinions and input without making every single person feel like they are a blocking approver. At MongoDB, we try to accomplish this with a process we call Looks Good To Me, or LGTM. This is how a person confirms that within their sphere of knowledge, the proposal makes sense — without presuming that they are all-knowing — and leaving the decision ultimately up to the responsible individual. Sometimes, that responsible individual is indeed the executive in charge of the business unit - but very often it’s the individual running the project - at any level of the org. So we’ll have documents that have a list of requested LGTM’s, and people comment on them as they have time and energy. All with the safety of the final approval of the responsible individual. It’s complicated to build a culture with the right approval dynamics. And you’re never done. As your company changes, you’ll have to revisit this cultural guideline often.
Surprisingly, yes, this is my job - and it’s likely yours too
In the traditional CTO job description, you won’t find “joy” or “wellness”, and you probably won’t find “cultural alignment” featured prominently either. This is too bad, since fostering healthy teams is a huge part of being a good leader — and all of the items I discuss above are essential to that.
There are many other topics I could have talked about, and in other blogs, I’ll likely do so. The reason I chose these guidelines for this letter is that they are all woven together into a single fabric. The interdependent threads include context, candor, trust, charity, tech debt, pride, experimentation, accountability, and highly aligned but loosely coupled teams. This fabric, it turns out, is our culture, and it tears if we fail to deliver on them.
I hope this letter gives you things to think about as we build MongoDB together or you shape your own company. I hope it’s given you some kernels of value to either agree or disagree with. I look forward to hearing from you, either within MongoDB or on @MarkLovesTech.